Welcome to the enterprise data explosion

The data avalanche is under way. Here's how to avoid being buried alive and left cold and penniless

As an IT consultant, I see it happening all around me: the heroic struggle to keep up with the exponential growth of data. IT finds itself confronted with larger and more diverse data sets than ever before, thanks to high-definition input devices, low-cost storage hardware, and the blazing processing power of today's servers and workstations. Even small enterprises can now afford to store mountains of data undreamed of only a few years ago.

At InfoWorld we've come to call this phenomenon the enterprise data explosion, and chances are, you're experiencing it right now.

[ For an entertaining look at a key storage management technology, fire up this InfoClip on storage virtualization. ]

Just five years ago, your average departmental file server had maybe 75GB to 100GB of primary storage, usually in the form of direct-attached disk. Today, that same server is much more likely to have -- and need -- five to ten times that amount. There's also a good chance this chunk of storage will come in the form of a NAS or SAN instead of a meat-and-potatoes server. Prices per gigabyte may have dropped precipitously, but the demand for capacity has escalated faster, raising storage hardware investments to new heights.

The less obvious cost of managing and protecting all that additional data is even higher. As more and more disk spreads throughout the enterprise, it becomes more and more difficult to monitor how that storage is being used and to ensure appropriate protection against data loss. Few IT shops I've seen these days are increasing their head counts significantly, so something has to give.

Yet there are no magic technology bullets when it comes to rampant data growth. Instead, you'll find a grab bag of useful of technologies for specific purposes -- plus a set of best practices from which to construct a storage strategy that anticipates constant expansion. Sure, hot technologies such as data de-duplication should play a role in your strategy, but they don't qualify as strategies in and of themselves. And failure to think through a plan at a high level is where many go wrong.

A good storage strategy almost always involves a combination of hardware, software, tools, and perhaps most importantly, policies. Together, these ensure that the correct data is being retained, stored on the most appropriate type of storage hardware, adequately backed up, and able to be restored in an acceptable amount of time. Assembling and fine-tuning that collection of resources is not always easy, but it's far better than being drowned alive in corporate data run amok.

This blog aspires to be a useful resource for those dealing with the enterprise data explosion. Based on my own experience and that of the IT community, I'll be offering the best of what I encounter: techniques, technologies, and tactics, from SAN architecture to cloud services that can help you create the infrastructure and strategy you need.

We'll start by digging into the ever more crowded landscape of primary storage options and try to put to rest some of the more persistent myths and misconceptions surrounding them. No primary storage architecture can exist without some means to protect it, so we'll take a close look at data protection options such as available backup software architectures, when to use Virtual Tape Libraries, and what role tape still plays in the enterprise today. We'll also help you find tools to help you manage, monitor, and control your data growth -- whether through tried-and-true methods such as disk quotas or more modern approaches such as automated data archiving.

Last, we'll look at what you can do with words. Can strong data creation and retention policies stem the growth of our data? Can you actually implement a storage chargeback model in real life and get away with it? Anything that involves how your data is stored, protected, governed, managed, monitored, or accessed will be fair game.

No single product will ever solve every storage-related problem you're likely to face. I can't pretend to do that, either. But I hope this blog can help you ease the pain and find solutions to some of the most vexing storage management problems that confront us today.

This story, "Welcome to the enterprise data explosion," was originally published at InfoWorld.com. Follow the latest developments in storage at InfoWorld.com.