The big data addiction

The data explosion isn't just about size -- it's about how our dependence on ever-expanding data feeds on itself

On the face of it, the data explosion seems like one of those out-of-control problems with no satisfactory solution -- another onerous fact of IT life. But that fatalistic view ignores our own culpability: Data is growing so quickly because we rely on more and more of it than ever before.

To wrap our minds around this, let's jump in the time machine and go back 10 years to the year 2000 (fans of "Late Night with Conan O'Brien" can sing along).  Back then, many enterprises had only recently embraced the Internet as a means to accomplish real work. The paperless office was a fanciful yet enticing dream. Your medical records were on a computer somewhere, but probably in order to bill you rather than treat you.

[ Also on See how metadata can help you manage ever-expanding data demands in our Log Analysis Deep Dive Report. | Looking to revise your storage strategy? See InfoWorld's iGuide on the Enterprise Data Explosion. ]

At that point, the enterprise datacenter was well into the phase of explosive growth we're still experiencing today. Even small businesses had embraced the use of technology to automate and streamline processes -- from e-mail communication to billing to CRM. Large businesses that had been using technology to provide this core functionality for many years were working to eliminate inefficient, non-electronic business processes. In that sense, things weren't terribly different than they are now.

Data Explosion iGuide

The subtle yet critically important difference between now and 10 years ago is that the paperless office has arrived. When was the last time you saw a rolodex on someone's desk? How about a filing cabinet used for anything other than signed documents? And when you see your doctor, chances are that the first thing she'll do is pull up your electronic medical record.

So here's the problem: 10 years ago, if there was a major datacenter failure, people would scramble to dig files out of cabinets to service customers. They would waste tons of time and money doing so, but in most cases, paper records and the phone would suffice, and life would go on. Today, most likely you're talking about stopping the business cold: Everyone, go home until we fix it.

In many enterprises, all you need to do is wander into the data center and flip the power switches on a few pieces of storage equipment; you've effectively paralyzed the entire operation, and not just big businesses. Even small businesses have heavy dependence on technology -- and are much less likely to have invested in hot standby resources in the event of disaster.

The good news is that the redundancy necessary to continue doing business has become relatively easy to deploy and manage. Server virtualization and nearly ubiquitous support for data replication on primary storage make it relatively simple and inexpensive to implement a completely mirrored application environment that can take over at a moment's notice. However, many businesses have yet to fully realize how dependent they are on the technology they use -- and have not yet invested in protecting themselves.

In a cruel twist of fate, our dependence on ever-expanding digital data has created a feedback loop that fuels its own growth. Within the past 10 years or so, we've grown more productive by using business technology. As a result, we've created even more massive mountains of data, and we rely upon those mountains to such a degree that we need to duplicate them -- multiplying the problem again.

As I look another 10 years into the future, it's clear that at some point, this cycle of data creation, dependence, and duplication will have to be broken before the burden of managing mountains starts to outweigh its usefulness. Part of me wants to delete my email, bury my cell phone, and go live in the woods. On the other hand, I can't wait to see how we solve these problems. All I know is that changes in behavior, not just in technology, will have to be a big part of the solution.

This story, "The big data addiction," was originally published at Read more of Matt Prigge's Information Overload blog and follow the latest developments in storage at