Hadoop makes all the big data noise. Too bad it's not also getting the big data deployments.
Indeed, though Hadoop has often served as shorthand for big data, this increasingly seems like a mistake. According to a new Gartner report, "despite continuing enthusiasm for the big data phenomenon, demand for Hadoop specifically is not accelerating."
According to the survey, most enterprises have "no plans at this time" to invest in Hadoop and a mere 26 percent have either deployed or are piloting Hadoop. They are, however, actively embracing other big data technologies.
'Fairly anemic' interest in Hadoop
For a variety of reasons, with a lack of Hadoop skills as the biggest challenge (57 percent), enterprises aren't falling in love with Hadoop.
Indeed, as Gartner analyst Merv Adrian suggests in a new Gartner report ("Survey Analysis: Hadoop Adoption Drivers and Challenges"):
With such large incidence of organizations with no plans or already on their Hadoop journey, future demand for Hadoop looks fairly anemic over at least the next 24 months. Moreover, the lack of near-term plans for Hadoop adoption suggest that, despite continuing enthusiasm for the big data phenomenon, demand for Hadoop specifically is not accelerating.
How anemic? Think 54 percent with zero plans to use Hadoop, plus another 20 percent that at best will get to experimenting with Hadoop in the next year:
Gartner This doesn't bode well for Hadoop's biggest vendors. After all, as Gartner analyst Nick Huedecker posits, "Hadoop [is] overkill for the problems the business[es surveyed] face, implying the opportunity costs of implementing Hadoop [are] too high relative to the expected benefit."
Selling the future of Hadoop
By some measures, this shortfall of interest hasn't yet caught up with the top two Hadoop vendors, Cloudera and Hortonworks.
Cloudera, after all, will reportedly clear nearly $200 million in revenue in 2015, with a valuation of $5 billion, according to Manhattan Venture Partners. While the company is nowhere near profitability, it's not struggling to grow and will roughly double revenue this year.
Hortonworks, for its part, just nailed a strong quarter. Annual billings grew 99 percent to $28.1 million, even as revenue exploded 167 percent to $22.8 million. To reach these numbers, Hortonworks added 105 new customers, up from 99 new customers in the previous quarter.
Still, there are signs that the hype is fading.
Hortonworks, despite beating analyst expectations handily last quarter, continues to fall short of the $1 billion-plus valuation it held at its last round of private funding. As I've argued, the company will struggle to justify a billion-dollar price tag due to its pure-play open source business model.
But according to the Gartner data, it may also struggle due to "fairly anemic" demand for Hadoop.
Moving beyond Hadoop
There's a big mitigating factor. Hadoop vendors will almost surely languish -- unless they're willing to embrace adjacent big data technologies that complement Hadoop. As it happens, both leaders already have.
For example, even as Apache Spark has eaten into MapReduce interest, both companies have climbed aboard the Spark train.
But more is needed. Because big data is much more than Hadoop and its ecosystem.
For example, though the media has equated big data with Hadoop for years, data scientists have not. As Silicon Angle uncovered back in 2012 from its analysis of Twitter conversations, when data professionals talk about big data, they actually talked about NoSQL technologies like MongoDB as much or more than Hadoop:
Silicon Angle Today those same data professionals are likely to be using MongoDB and Cassandra, both among the world's top 10 most popular databases, rather than Hbase, which is the database of choice for Cloudera and Hortonworks, but ranks a distant #15 in terms of overall popularity, according to DB-Engines.
Buying an ecosystem
Let's look at Gartner's data again, this time comparing big data adoption and Hadoop adoption:
Gartner A significant percentage of the delta between these two almost certainly derives from other, highly popular big data technologies such as MongoDB, Cassandra, Apache Storm, etc. They don't fit into the current Hadoop ecosystem, but Cloudera and Hortonworks need to find ways to embrace them, or risk running out of Hadoop runway.
Nor is that the only risk.
As Aerospike executive and former Wall Street analyst Peter Goldmacher told me, a major problem for Hortonworks and Cloudera is that both are spending too much money to court customers. (As strong as Hortonworks' billings growth was, it doubled its loss on the way to that growth as it spent heavily to grow sales.)
While these companies currently have a lead in terms of distribution Goldmacher warns that Oracle or another incumbent could acquire one of them and thereby largely lobotomize the other because of its superior claim on CIO wallets and broad-based suite offerings.
Neither Cloudera nor Hortonworks can offer that suite.
But what they can do, Goldmacher goes on, is expand their own big data footprint. For example, if Cloudera were to use its $4-to-5 billion valuation to acquire a NoSQL vendor, "All of a sudden other NoSQL vendors and Hortonworks are screwed because Cloudera would have the makings of a complete architecture."
In other words, to survive long term, Hadoop's dominant vendors need to move beyond Hadoop -- and fast.