Exclusive: Index Engines sparks search innovation

IT managers deal with a relentless sprawl of unstructured data. Research firm Meta Group estimates corporate storage for each employee was 3GB in 2003, and most analysts say this requirement is increasing by 50 percent to 70 percent each year. SANs make warehousing and archiving all these files more affordable, but for the past ten years, search agents have crawled LANs. That approach has now changed.

Put simply, Index Engines Appliance 1.0 leverages the efficiencies of SANs and backup processes you already have in place. In about 30 minutes, this elegant solution plugs into your SAN between your tape backup systems and your file and e-mail servers. That’s it. As a pass-through appliance, it does not interfere with existing daily backup operations. However, the system’s speedy indexing of content during backups removes the complexity, data duplication, and network overhead of traditional enterprise searches. Equally important, fast search results and easy operation are likely to cut the time employees spend looking for information.

For this exclusive test of Index Engines Appliance, I assembled a SAN with two Dell PowerEdge file servers, a Dell PowerVault LTO-2 tape drive, and 20 Dell OptiPlex workstations connected over a 100Mbps LAN. As you’d find in many datacenters, my primary file server ran Symantec’s Veritas NetBackup 5 to handle backup chores of all workstations and servers. Legato and Tivoli backup applications are also supported, plus any multitape robotics libraries or multiplexed backup servers your datacenter has running on your SAN. 

Plug and Search
With scheduled backups working, I swiftly installed and configured Index Engines’ solution. I merely connected the hardware to my SAN via an FC (Fibre Channel) cable, attached the SCSI tape drive, and registered the appliance on my Microsoft Active Directory domain using a simple Web UI. In addition to the easy setup, I also liked being able to connect the SCSI tape library directly, which eliminates having to retrofit robots with more expensive FC adapters. You can order the appliance with an FC card for tape backup units if you’re already using fibre end-to-end.

Index Engines Appliance proved virtually transparent and very fast when indexing. For example, a full backup of all systems that took one hour before I installed the appliance required about one hour and two minutes with the appliance running. As such, I have no reason to doubt the vendor’s claim that Index Engines Appliance indexes 3.5 million words per second.

Most search applications glut storage with large indexes of their own. With this solution, however, document catalogs were about 8 percent of the original file size. In other words, the entry-level appliance I tested indexes approximately 4 million typical Microsoft Office documents. The largest appliance handles 8TB of backup data (16 million files), and the servers can be clustered, so there is more than adequate headroom. The Index Engines Appliance hardware incorporates RAID hard disks, and the index is mirrored for added redundancy. As such, I believe the system incorporates the reliability measures datacenter operators expect of their systems.

Next, I plunged into the search experience. Index Engines catalogs the full content of typical office documents (.doc, .xls, .ppt, .pdf, text, and HTML). Additionally, it indexes archives (.tar, .zip), recognizes Acrobat .pdf, and scans Microsoft Exchange mailboxes and local .pst mail files. For an initial release, that’s a very good range of file types. The vendor indicated that with the core technology in place, upcoming software releases would expand the content that’s indexed. Databases are high on my wish list.

Another boon for efficiency is that catalogs point to the original document. The system doesn’t copy files or e-mail (unlike some other systems), so you needn’t worry about creating yet another data repository.

Searching is what I expect in an enterprise-class product. Users easily access the search engine with their Active Domain log-in information. After this initial authentication, the appliance uses security information embedded in the backup to restrict search results to those documents they’re authorized to view.

Searches are accurate, with response well under one second and with highly readable formatting of results. In addition to a basic search form, advanced search tools let me check for documents by attributes, such as size or creation time; use Boolean logic; and narrow my search to particular folders on a server.

Results show a document summary so users see search words in context. If Index Engines wants to go all-out, it should add clustering to results. Products such as Vivisimo Velocity, for instance, already do.

Strong Start
Index Engines Appliance breathes fresh life into enterprise search. By indexing critical business data as it’s backed up, search becomes far more efficient. Plus, it reduces administrative workload because IT staffs aren’t burdened with changes to existing infrastructures or backup processes.

Of course, established vendors aren’t sitting on the sidelines. For example, EMC’s Documentum embeds search into its content-management applications, and Google plans to expand its search appliance beyond Web searches of enterprise systems. Still, Index Engines’ fundamentally different way of searching will likely have the competition envying this startup company and wondering, “Why didn’t we think of that?”