Such variety came about to address differing needs among users, Baker said. MySQL, for instance, is really fast at reading data, but the Cassandra data store, on the other hand, can write data more quickly. The production company behind the U.K. television show "Britain's Got Talent," used a Cassandra database to log the votes of viewers choosing their favorite performer, because it could ingest a high number of writes simultaneously, Baker noted.
A number of companies have released commercial Hadoop distributions, such as Cloudera, Hortonworks, and MapR, in which all the software components are integrated. But even Hadoop itself is not suited for all jobs, Maitland argued. It processes data as batch jobs, meaning the full data set must be written to a file before it can be analyzed. Many jobs, however, involve the analysis of a continually updated data, such as click streams or Twitter messages.
Also, a stack would need to have support from more than one company to be an industry standard, Maitland said. "If there is going to be a stack, it needs to be [managed by] an open source organization and not necessarily managed by a specific company," Maitland said.
Another problem with not having a standardized stack is that it drives up the cost of hiring experts to manage and use such systems. Right now the competition for experts is fierce.
"Trying to build [a big data system] takes knowledge and skill. To plug those into your infrastructure can take time and money," Baker said. "There is no standard roadmap -- it is a feeling along process. Putting it all together is not a simple task."
"You can't have the explosive growth in an industry with so much specialized knowledge that is required as of right now," Maitland said.
"The average business analyst can't write queries against Hadoop," Staimer added.