The big data journey has begun

Relentless data growth has conjured up new technologies for data analysis and unprecedented opportunities for insight

What we call "big data" today will one day simply be called "data." Big data is not a fad, but rather a new phase in the evolution of data management. It's a way to describe the exponential data growth we all struggle to accommodate -- and a way to underscore new opportunities to extract meaning from that data using emerging technologies.

This blog will be your front-row seat on big data as it evolves. I'll be highlighting case studies on a range of big data topics, including NoSQL databases, the Hadoop platform, predictive analytics, and more. In particular, data visualization -- which opens a new window on analytics -- will be a recurring theme.

[ Cut straight to the key news for technology development and IT management with our once-a-day summary of the top tech news. Subscribe to the InfoWorld Daily newsletter. ]

Big data has a host of different definitions. In this instance, I think Gartner got it right, describing big data as "high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization." Primarily through case studies, we'll explore how organizations are using new technology to handle tremendous data volumes, process huge data sets rapidly, and/or manage virtually any kind of data type.

Big data's big break
Why has big data "arrived" at this particular moment? The root cause is the Web itself. With the Web, we are no longer dealing with thousands of users of a client-server application powered by an RDBMS. Web applications are open to worldwide consumers. Millions of users instantly start generating data by interacting with those apps.

That level of activity requires highly scalable systems to keep users engaged -- and has helped spawn the NoSQL movement. And it requires analysis of huge, semi-structured data sets in order to figure out how to optimize the user experience and monetize behavior. That's given Hadoop its start and pushed the evolution of the MPP (massively parallel processing) analytic databases and new sets of business intelligence tools.

The innovators and early adopters of big data have found conventional technologies insufficient. They've embraced NoSQL and Hadoop in particular because they were not satisfied with traditional technologies in terms of cost and effort.

Those users have been willing to put up with a regression of features compared to traditional RDBMS, ETL, and business intelligence tools. For example, with NoSQL systems, they gave up ACID compliancy. With Hadoop, they gave up traditional ETL features, a nice GUI, and real-time querying. With business intelligence, they forgo real-time visualization and data interactivity. The list goes on. Whether this group perceives those features as real sacrifices or not depends on each use case.

Merging into the mainstream
The early majority of new adopters of big data technology will want those features back eventually. And when big data technology providers deliver them, big data as a distinct category will cease to exist.

At the same time, the rate of data growth will continue to accelerate. Today, Web clickstream data, systems events, and other sources close to the core technology we depend on supply much of the new, semi-structured data fueling big data processing. In the future, mobile devices and "the Internet of things" -- connected via RFIDs and other sensors -- will enable us to collect and analyze huge new waves of data from manufacturing systems, transportation infrastructure, medical instruments, and just about any vertical scenario you can imagine.

So it pays to look closely at the first big data applications as they arise. The emerging technologies we associate with big data today will be considered the standard data management fabric to handle these subsequent data explosions.

In this blog series, we invite you to join us on this evolutionary journey. We'll report on the latest technology developments and share thought-provoking case studies -- including big data efforts that are helping win the war on cancer or increasing the chances that money laundering can be detected. We welcome input and examples from readers and hope you enjoy the ride with us.

This article, "The big data journey," was originally published at Read more of Andrew Lampitt's Think Big Data blog, and keep up on the latest developments in big data at For the latest business technology news, follow on Twitter.

Copyright © 2012 IDG Communications, Inc.

How to choose a low-code development platform