Exclusive: Index Engines sparks search innovation

IT managers deal with a relentless sprawl of unstructured data. Research firm Meta Group estimates corporate storage for each employee was 3GB in 2003, and most analysts say this requirement is increasing by 50 percent to 70 percent each year. SANs make warehousing and archiving all these files more affordable, but for the past ten years, search agents have crawled LANs. That approach has now changed.

Put simply, Index Engines Appliance 1.0 leverages the efficiencies of SANs and backup processes you already have in place. In about 30 minutes, this elegant solution plugs into your SAN between your tape backup systems and your file and e-mail servers. That’s it. As a pass-through appliance, it does not interfere with existing daily backup operations. However, the system’s speedy indexing of content during backups removes the complexity, data duplication, and network overhead of traditional enterprise searches. Equally important, fast search results and easy operation are likely to cut the time employees spend looking for information.

For this exclusive test of Index Engines Appliance, I assembled a SAN with two Dell PowerEdge file servers, a Dell PowerVault LTO-2 tape drive, and 20 Dell OptiPlex workstations connected over a 100Mbps LAN. As you’d find in many datacenters, my primary file server ran Symantec’s Veritas NetBackup 5 to handle backup chores of all workstations and servers. Legato and Tivoli backup applications are also supported, plus any multitape robotics libraries or multiplexed backup servers your datacenter has running on your SAN. 

Plug and Search

With scheduled backups working, I swiftly installed and configured Index Engines’ solution. I merely connected the hardware to my SAN via an FC (Fibre Channel) cable, attached the SCSI tape drive, and registered the appliance on my Microsoft Active Directory domain using a simple Web UI. In addition to the easy setup, I also liked being able to connect the SCSI tape library directly, which eliminates having to retrofit robots with more expensive FC adapters. You can order the appliance with an FC card for tape backup units if you’re already using fibre end-to-end.

Index Engines Appliance proved virtually transparent and very fast when indexing. For example, a full backup of all systems that took one hour before I installed the appliance required about one hour and two minutes with the appliance running. As such, I have no reason to doubt the vendor’s claim that Index Engines Appliance indexes 3.5 million words per second.

Most search applications glut storage with large indexes of their own. With this solution, however, document catalogs were about 8 percent of the original file size. In other words, the entry-level appliance I tested indexes approximately 4 million typical Microsoft Office documents. The largest appliance handles 8TB of backup data (16 million files), and the servers can be clustered, so there is more than adequate headroom. The Index Engines Appliance hardware incorporates RAID hard disks, and the index is mirrored for added redundancy. As such, I believe the system incorporates the reliability measures datacenter operators expect of their systems.

Next, I plunged into the search experience. Index Engines catalogs the full content of typical office documents (.doc, .xls, .ppt, .pdf, text, and HTML). Additionally, it indexes archives (.tar, .zip), recognizes Acrobat .pdf, and scans Microsoft Exchange mailboxes and local .pst mail files. For an initial release, that’s a very good range of file types. The vendor indicated that with the core technology in place, upcoming software releases would expand the content that’s indexed. Databases are high on my wish list.