Business intelligence is almost always among the top technology priorities of big companies. Yet by most accounts, satisfaction with business intelligence often falls short of expectations. Reasons abound: poor data quality, cost overruns, inaccessible data sources, and our old friend, miscommunication of requirements.
But businesses don't like to fly without instruments. So year after year, companies pony up for business intelligence initiatives despite lackluster results.
[ See InfoWorld's primer on Big Data and learn how a variety of companies are finding hidden business value in unstructured data. | Read Eric Knorr's interview with IBM vice president of emerging Internet technologies Rod Smith. ]
A recent technology surge promises break this unfortunate cycle. At InfoWorld we find these new developments among the most interesting we've encountered in a while, which is why we're launching an iGuide to the new business intelligence, a graphical portal that aggregates business intelligence articles and analysis.
Big Data dreams
The hottest new technology trend goes by the nickname Big Data. Conventional business intelligence software operates on SQL data; Big Data processing crunches on semi-structured information, from social networking clickstreams to security event logs. Hadoop, an open source software framework for distributed processing, is the best-known technology for processing unstructured data, but there are others, including GemFire, MarkLogic, and neo4j.
When I interviewed IBM's Rod Smith a few weeks ago, he characterized Big Data processing as "an exploratory tool that we haven't had before" to "help people sift through data where maybe 90 percent of it is not very useful." With the aid of data visualization and data mining, domain experts can then determine for themselves what's worth exploring using conventional business intelligence tools. (Big Data processing techniques, by the way, were instrumental in Watson's recent "Jeopardy" victory.)
