Hadoop's open community driving its evolution

A meritocratic, open community is unlocking opportunities for the ecosystem, which in turn is helping Hadoop evolve a lot faster than within any one vendor's corporate walls

News from today's Hadoop Summit '09 got me thinking about the importance of open community again. According to Joey Echeverria's tweet from Hadoop Summit '09, Yahoo representatives feel:

Yahoo!: Easier to take an open source project and add steam to it rather than write something from scratch. #hadoopsummit09

[ Find out what InfoWorld's Test Center thinks of Hadoop's contribution to Amazon's use of MapReduce for its Elastic Cloud | Keep up with the latest open source news with InfoWorld's open source newsletter. ]

What a difference a year and a half makes. Back in December 2007, there were roughly 10 Yahoo employees working on Hadoop, and only five or six outside contributors. Hadoop said at the time:

It's dominated by Yahoo, it would be great for the project to have a more balanced team.

Today, Hadoop's core has 185 contributors, only 30 percent of whom are Yahoo employees.

Also, Cloudera, a commercial company aiming to bring Hadoop to the enterprise, has just contributed a new database tool for Hadoop. The tool, SQOOP, enables users to directly import large database tables into Hadoop. According to Cloudera founder:

"SQOOP is a tool that enterprise customers were demanding," Bisciglia said. "Enterprises have lots of data in existing databases, and if you can't give them a way to interact with that data, Hadoop isn't as useful as it could be."

Much like Kernel.org, Apache HTTPD, and Eclipse before it, a meritocratic, open community is unlocking opportunities for the ecosystem, which in turn is helping Hadoop evolve a lot faster than within any one vendor's corporate walls.

Today, it appears that most of the contributing vendors are collaborators, with little, if any, head to head competition.  That will surely change over time.  But that's a good thing. More vendors, more developers, more ideas, more innovation.

One can't help but wonder what Google thinks of Hadoop's progress.

Follow me on Twitter at: SavioRodrigues.

p.s.: I should state: "The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions."

Join the discussion
Be the first to comment on this article. Our Commenting Policies