Just as important, YARN can become a unifying thread for diverse Apache open source initiatives around big data. As InfoWorld recently noted: "The biggest win of all here is how MapReduce itself becomes just one possible way of many to mine data through Hadoop."
That's the YARN promise, but seeing it realized requires that the industry retool their Hadoop stacks and tools to work with it. Per the article, "Apache claims that any distributed application can run on YARN, albeit with some porting. To that end, Apache's maintained a list of YARN-compatible applications, such as the social-graph analysis system Apache Giraph (which Facebook uses). More are on the way from other parties, too."
This is good, but notice that disclaimer: "albeit with some porting." The article says YARN's true test will be in the extent to which vendors port their analytic development tools to output jobs that are conformant with YARN. As the author states, porting development languages to YARN "isn't a trivial effort."
Will this take place consistently throughout the industry and diverse Apache and other open source communities? If so, to what extent? Those factors will determine the degree to which YARN, the defining feature of what some call "Hadoop 2.0," truly takes hold.
Considering that Hadoop 2.0 preserves backward compatibility with MapReduce and YARN requires some porting to bring MapReduce applications up to speed, that nontrivial effort may significantly slow developer adoption of the new framework.
Also, in light of the range of alternative languages (R) and alternative platforms (any NoSQL approach) with which big data applications are being developed, it's not even clear that Hadoop, 1.0 or 2.0, can maintain its current marketplace momentum indefinitely.
This story, "Devops can take data science to the next level," was originally published at InfoWorld.com. Read more of Extreme Analytics and follow the latest developments in big data at InfoWorld.com. For the latest developments in business technology news, follow InfoWorld.com on Twitter.