Data explosion shakes up IT

In just three years, the bytes of data generated by digital cameras, mobile phones, businesses IT systems, and devices will equal the number of grains of sand on the world's beaches.

It's a mind-boggling estimation from market analysis company IDC. But it reflects the proliferation of devices and systems used by consumers and businesses, said Stephen Minton, IDC's vice president of worldwide IT markets and strategies, during a briefing on Thursday.

Over the next few years, corporations will face tough decisions on how to store data, find information and comply with regulations, said Minton, who laid out a forecast of what IT managers can expect from what's being labeled the "information explosion."

It won't be an easy task. While 85 percent of that data is predicted to come from consumers snapping photos, surfing Web pages, and sending e-mail, about 60 percent of that consumer data will still cross corporate networks, Minton said.

Much of the data is unstructured, meaning it's not clearly labeled as to its content, such as photos, video, and perhaps phone recordings, which makes it more difficult to use.

But technologies that enable deep analysis of the data are emerging, and could help business unlock what's important and improve their operations.

The mass of data companies are collecting -- the retailer Wal-Mart Stores, for example, generates 1TB of transactional data a day -- represents huge business opportunities "if companies can really get to a point where they can successfully analyze the data going back and forth on their network," Minton said.

Companies are "still a long way short of being able to analyze the unstructured data on their networks," he said. The data is coming from Web services such as the photo-sharing site Flickr, Skype, and even the virtual world Second Life.

The business opportunities are the positive side. But the security concerns still abound, as well as regulatory compliance and liability worries.

According to data from the U.S. Computer Emergency Response Team, the number of reported software vulnerabilties declined in 2003 and 2004 but surged in 2005 to around 6,000, an all-time high.

"This explosion in the number of applications and devices has created an explosion in the number of security vulnerabilities," Minton said.

The security and reliability of those IT systems has become so important that the IT department now has "a direct impact on the company's top and bottom line," Minton said. Customer experiences and the company's image are at stake when security fails, he said.

Accordingly, security has remained the top budget priority over the last five or six years, with 60 percent of companies placing it as the highest one in an IDC survey conducted earlier this year.

IDC is predicting that revenue from security and vulnerability management software products will grow by 20 percent this year over 2006, to $2.27 billion. That far outpaces the 6 percent to 7 percent growth expected for the software industry as a whole.

When a report of a data breach hits the front page that "has a very bad impact on a company's brand image," Minton said.