Don't expect big-time results from big data

Big data is here, but barriers suggest success will be modest; for many companies, it may prove unsettling and disappointing

Big data is a big deal these days. It's all about storing and mining massive data sets that defy traditional storage models and technologies -- in many cases without the same level of design and planning that traditional data warehouses require.

Big data is the third hot industry trend to be analyzed in this space. As with the past two weeks, when we analayzed cloud services and BYOD, the question doesn't center on potential value. Instead, the focus is on its potential for industry acceptance, which is quite a different matter.

[ InfoWorld's Andrew Lampitt looks beyond the hype and examines big data at work in his new blog Think Big Data. | Download InfoWorld's Big Data Analytics Deep Dive for a comprehensive, practical overview. | For more of Bob Lewis' continuing IT management wisdom, check out his Advice Line newsletter. ]

Just to make sure we're talking about the same subject, while big data overlaps with NoSQL database systems -- another set of technologies that's been getting attention recently -- this analysis is limited to big data.

To recap: For any new technology to have a chance of success it must clear three hurdles.

  • The customer and consumer can't be different people. (Reminder: Customers make buying decisions about a product or service, in contrast to consumers, who are the people who use it.)
  • The "wallet" (the source of money) can't find the expenditure off-putting.
  • The technology can't be disruptive when mixed with the installed base.

Let's look at how big data fares.

Breaking down big data

Examining big data from along the three axes outlined above, we get the following:

Customer vs. consumer: Big data's consumers are business executives and managers, often the COO or CMO, who want to be smarter about their business than their competitors are. They're the ones who will dig into the data with picks and shovels to mine for nuggets of gold (more accurately, analysts, statisticians, or "data scientists" will dig into it for them).

Who makes the buying decision? This can go one of two ways. Either the executive who wants big data asks IT to take the technical lead, at which point IT becomes the actual customer (IT decides what to buy), or the exec engages an outside big data consulting firm, at which point the consulting firm becomes the customer.

Either way, while the customer and consumer aren't the same person, they are different in a relatively minor way. The consumer (the COO or CMO) still makes the decision to buy and so occupies the most important part of the overall customer role. IT or the consulting firm's role is limited to deciding what to buy.

Wallet impact: This is a bit more complicated.

Investing in big data takes more than downloading Hadoop from the Apache website and installing it on a server cluster. It's a major organizational undertaking that requires:

  • Staffing: Money, because data scientists don't come cheap.
  • Education: More money, because retraining your analysts so they know how to deal with, not just a new technology but a new philosophy of dealing with data, is a nontrivial and not-inexpensive undertaking.
  • Money: Even more of it, whether big data is implemented using on-premises equipment or the implementation goes into the cloud. On-premises implementations will need capex to cover storage and server capacity. Cloud implementations will spend opex on storage, server capacity, and either newer, faster network pipes or a boatload of flash drives transported via "FedEx Net" to the cloud provider of choice (some companies reportedly take this exact approach). None of this is free just because it's on the other side of the Internet.
1 2 Page 1
Page 1 of 2