About seven years ago, I developed a 13-part functional model for what a big data and analytics (BDA) solution needed to contain. Key components included things like data governance/master data management, integration of data sources, integration of BDA with the underlying IT infrastructure, and collaboration capabilities.
Chief among them, however, was search. Think about it: We can do everything right in a big data implementation. Thanks to Google and Amazon, which created the world’s first NoSQL databases in 2004-05, and the continuing evolution of NoSQL solutions since then, we can now bring all data structures (structured, unstructured, and semi-structured) from all relevant sources into the mix. We can enable “both technical and business users” (to quote marketing messaging from seemingly every BDA provider’s website) to navigate the system.
Where it all comes together, though, is search: whether they’re running complex SQL queries or typing simple, natural language text into a search field, can users quickly get the answers they seek?
Google and Amazon have created an expectation of subsecond search results from every system
What else matters? Speed. Combining all of these capabilities with technologies and initiatives supporting real-time analytics—principally, stream processing, in-memory computing, and the 15-odd Apache projects sprouting various real-time-analytics-supporting solutions—has enabled providers to deliver comprehensive, subsecond search results.
So, what started with the pioneering work by Amazon and Google has created an expectation among all users of all systems for comprehensive, highly relevant, subsecond search results. By and large, those giants, the few other surviving search providers, and an exploding universe of e-commerce sites, are delivering.
In fact, they are overdelivering: Users who set out on online search and shopping expeditions are often presented with a bewildering array of search results, page after page into oblivion. The reality that most users are never going to look at more than the first few Google search results pages, for example, has germinated something of a cottage industry among digital agencies and others claiming to be able to land every new client on page 1 of Google search results. Search solutions are doing their job so well they are burying users under an avalanche of information.
AI is the new … search?
AI is forecast to drive global GDP gains in the trillions of dollars over the next ten to 15 years. AI fires the imagination with visions of driverless cars and pilotless planes, and intelligence that is not merely artificial but increasingly alternative: machines imbued with so much human capacity that they can think for themselves—and, while we’re not really looking—for us. Currently, and more realistically, chatbots are beginning to exhibit more of the widely hyped “human-like” qualities, fulfilling more complex tasks and roles, aided by chatbot analytics such as those emanating from Google’s Area 120 incubation creation, Chatbase.
Much of the value AI is delivering today, however, is far removed from the dare-I-say-sexy AI activities described above. No, most of the buzz is around AI (or machine learning, or deep learning, or cognitive) systems humming away behind the scenes crunching, say, every tweet on Twitter and triangulating that against weather a myriad of other data points to provide corporations with analytic insights to inform and power their product and services offerings. That’s one of the many things IBM Watson is busy doing. One Wall Street firm is saving millions of dollars annually by using AI to scan and interpret its massive internal knowledgebase to instantly select optimal investment options for clients, plus the documents and forms its team needs to put those investment choices into action. Sentieo’s AI-driven solutions analyze millions of SEC filings, transcripts, and presentations to find crucial information on any ticker and retrieve all mentions of key information in seconds. AI is hard at work at Airbnb, Amazon, Facebook, Google, Netflix, Pandora, Spotify, Tesla, and Uber; at Bristol Myers Squibb; in the large and growing stable of Slackbots; and in countless other places, accessing and triangulating breathtaking amounts of data and spitting out instant recommendations or reservations.
Those last points are crucial. A structural shift is under way. AI cuts through the clutter to provide not endless pages of results to wade through, but with specific recommendations tailored to you as the seeker of knowledge—or simply as the seeker of where to find the best Chicago-style pizza while away from home on a business trip. (Which is not to admit, certainly not in print, that I have not supplemented my normal whole-foods, plant-based, no-meat-or-dairy nutrition by indulging in such a cheesy, guilty pleasure. I present it merely for illustration.) The key construct: AI-driven systems present either the single best solution or a tight shortlist of best-fit solutions.
AI also means not having to search
Another way AI diverges from search is that part of the value AI is that it presents you with choices (or content, or opportunities) you didn’t ask for—that you didn’t have to ask for. That predictive capability is a clear differentiator. Search is reactive; the system reacts to your query.
AI, if properly trained and implemented, is proactive.
Does the rapid growth of AI mean that search has failed us? Not necessarily. AI is taking everything we love about search to the next level—and extending it into every touchpoint between the human and digital worlds.
Now let me say this about that
AI’s ability to cut through the clutter, to present not page after of page of search results for the seeker to ponder, but one best answer, or a shortlist of best solutions, is truly transcendent. A quiet voice in a corner of my consciousness says, however, that we should also proceed with caution. A world where the range of available choices in virtually every situation is defined by (which is to say, limited by) the AI algorithms presenting the choices—and thus defined by the knowledge, thoughts, proclivities, attitudes, opinions and perhaps even the politics of those developing those algorithms—could be a chilling new world.
Part of the magic of big data is, and hopefully always will be, the ability to analyze all of the raw data—not summarized data, or a curated subset—to obtain the analytic insights you need. To the extent that we collectively empower AI to decide what choices to present to us, we need to build in checks and balances that let us go back and get a taste of the raw data if we don’t like what’s on the AI menu.
What are your thoughts on the implications of AI on search and life? Let the games (er, discussion) begin.