On the face of it, the data explosion seems like one of those out-of-control problems with no satisfactory solution -- another onerous fact of IT life. But that fatalistic view ignores our own culpability: Data is growing so quickly because we rely on more and more of it than ever before.
To wrap our minds around this, let's jump in the time machine and go back 10 years to the year 2000 (fans of "Late Night with Conan O'Brien" can sing along). Back then, many enterprises had only recently embraced the Internet as a means to accomplish real work. The paperless office was a fanciful yet enticing dream. Your medical records were on a computer somewhere, but probably in order to bill you rather than treat you.
[ Also on InfoWorld.com: See how metadata can help you manage ever-expanding data demands in our Log Analysis Deep Dive Report. | Looking to revise your storage strategy? See InfoWorld's iGuide on the Enterprise Data Explosion. ]
At that point, the enterprise datacenter was well into the phase of explosive growth we're still experiencing today. Even small businesses had embraced the use of technology to automate and streamline processes -- from e-mail communication to billing to CRM. Large businesses that had been using technology to provide this core functionality for many years were working to eliminate inefficient, non-electronic business processes. In that sense, things weren't terribly different than they are now.
The subtle yet critically important difference between now and 10 years ago is that the paperless office has arrived. When was the last time you saw a rolodex on someone's desk? How about a filing cabinet used for anything other than signed documents? And when you see your doctor, chances are that the first thing she'll do is pull up your electronic medical record.
So here's the problem: 10 years ago, if there was a major datacenter failure, people would scramble to dig files out of cabinets to service customers. They would waste tons of time and money doing so, but in most cases, paper records and the phone would suffice, and life would go on. Today, most likely you're talking about stopping the business cold: Everyone, go home until we fix it.