Meet Apache Twill, the newest member of club Hadoop

Twill, formerly known as Weave, is now an Apache Incubator project, designed to make it easy to run apps on YARN/Hadoop

There's little question Hadoop's become an attention-getting technology. That goes double for in its 2.0 incarnation, which introduced YARN, a way for most any applications -- not just the usual MapReduce jobs commonly associated with Hadoop -- to be distributed across a Hadoop cluster.

But one of the big problems that's surfaced with YARN is that it's tough to leverage properly. Apps run through YARN have to roll their own ways to cross-communicate or deal with errors, which can be tiresome.

The folks at Continuuity, a major Hadoop shop, decided it would be worthwhile to roll their own libraries -- a collection called Weave -- to make it far easier to port apps to YARN and make them run well. Since then, they decided to take the next step and make Weave into a bigger, more community-driven effort, so they submitted it to the Apache Incubator for acceptance. It's since been approved and will now continue its life as an Apache project: Apache Twill.

When I talked with Continuuity CEO John Gray, I asked why they'd been motivated to do this, since it places stewardship of the project in Apache's hands. The answer, simply: Community.

"When we first released Weave," he told me, "originally it was just up on GitHub. With GitHub, there's a lot more consumption of open source and a lot less contribution. It isn't a community-building site; it's good for sharing, but not as good for community building." By contrast, the Apache way is a process, one where the software becomes the explicit center of a user community.

"There's a lot of avenues out there for open source, but we want Twill to be adopted, extended, and adapted to meet the community's needs."

For now, most of those needs revolve around applications written in Java, but one of the tentative plans for the future is to expand Twill to directly support other languages that run in the JVM. Python, for instance, is a likely candidate, although Gray feels it would best be done by someone in the community -- such as in an existing Python shop.

But the most general idea with Twill is to expand the application mix for YARN beyond its defaults. "The primary user of Hadoop and YARN is still MapReduce," Gray said, "but Twill can help a lot of open source apps that are not MapReduce be developed there." That, in turn, he says, will help drive requirements back into YARN and shape the future evolution of that framework.

This story, "Meet Apache Twill, the newest member of club Hadoop," was originally published at Get the first word on what the important tech news really means with the InfoWorld Tech Watch blog. For the latest developments in business technology news, follow on Twitter.