Refining enterprise search

Endeca offers a technology that combines search with what it calls Guided Navigation. Here, a keyword search generates a search directory on the fly, which users can employ to drill down to progressively refined results.

Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld DOWNLOAD PDF Click here to download InfoWorld's special report Enterprise search

Customized tune-ups

According to Delphi Group’s Reynolds, creating an effective search interface for the enterprise user involves “knowledge-driven search applications” tailored to the business domain of the staffer.

“In order to achieve real accuracy, the search software has to be tuned to understand the context in which I’m working,” Reynolds says. “It’s a business-process-centered development strategy, so that you’re looking at a platform from the perspective of its ability to be tailored to specific users.”

Reynolds adds that Autonomy and FAST already prepackage offerings in the compliance, call center, market intelligence, and financial arenas. Verity offers multiple application templates as well. With this kind of tailored search interface, when financial brokers type “bonds” into a query, they never have to set eyes on a document related to glue.

FAST’s Marketrac layers an application on top of the ESP, which amounts to a search-powered interface that can access e-mail content, news feeds, competitor’s Web site content, and database content in a CRM. Moreover, the platform’s categorization facilities enable knowledge workers to explore content through patterns of meaning or subject matter.

Meanwhile, Google is taking a different approach with its enterprise offering, the Google Search Appliance (see our Test Center Review). It puts behind the firewall much of the successful technology that powers its public product, taking plug and play to new heights. In other words, the appliance is basically a search engine, not a comprehensive platform.

Dave Girouard, general manager for enterprise search at Google, cautions that ESPs “are putting a bigger burden on the user. As long as the results show up in the first page, [users] don’t care what’s behind it. … We have the right relevancy algorithms. So, in terms of [too much] content, we’re saying, ‘Bring it on.’ ”

The Google appliance may save the day for enterprises with broken search technology: Just open up the repositories and rev up the Google engine. But Delphi Group’s Reynolds thinks that “IT should stop investing in generic search tools and start concentrating on their professional domains. At the same time, the business side should be more involved, to ensure that IT commits the resources to develop business-oriented applications of search.”

Andrew McKay, vice president of direct sales at FAST, agrees but adds that vendors “aren’t necessarily fighting over a percentage of the pie. It’s about making the pie dramatically larger,” as information stores expand exponentially.

It’s all in the pipeline

For years, businesses have been fighting to get searches of unstructured data -- information that resides outside enterprise applications and databases -- to achieve the kind of accuracy and precision expected with structured data. According to Delphi Group’s Reynolds, with ESPs, the search-indexing process for unstructured information is evolving into a pipeline of different search algorithms and advanced technologies. These allow for dynamic categorizations or targeted text analytics to take place within the processes that parse documents when they come into the search platform, and within the processes that evaluate queries and return relevant information.

A relatively new addition to the pipeline is entity extraction, in which a search engine dynamically extracts terms from indexed content on the fly through grammatical analysis. The process includes identifying proper nouns and creating a list of people, places, and things from a document and then inserting a new level of metadata into that document.

Click for larger view.

Another is the use of NLP, which helps turn poor queries into good ones. The state of the art in search platforms involves a wide range of algorithms, rules, data enhancements, user- and context-profiling -- all of which work together to help zero in on what users need to answer their questions.

As for metadata, the old way of manually defining properties of a document is waning in favor of an intelligent search platform’s capability of autotagging based on users’ “custom logic,” according to FAST’s McKay.