As for the big security players, a number are appropriating the traditional approach to viruses, using signature-based searches to track down known rootkits and applying related fixes. Two of the major vendors, Symantec and Trend Micro, however, are taking unique tacks in combating rootkits.

Symantec is leveraging mapping technology to discover rootkits on compromised systems. Oliver Friedrichs, director of emerging technologies for security response at Symantec, believes rootkit eradication requires a stable, reliable design that minimizes false positives and mitigates system instability during rootkit removal. To make good on this mission, Symantec has employed the expertise and technology brought on board during the Veritas acquisition. Using VxMS (Veritas Mapping Service), Symantec's Norton Internet Security 2007 maps data on the hard drive, compares it with the Windows file structure, and isolates any discovered mismatches in an effort to repair potential problems. In effect, VxMS enables Norton to compare file systems with the raw data on the disk. Differences are immediately suspect.

For example, say Windows Explorer shows five files in a directory, whereas VxMS shows 10. Clearly, the additional five files are cloaked. Norton sends the suspicious files to Symantec for analysis, eradication occurs during reboot, and the discovered rogue is removed from other systems worldwide as a result.

Trend Micro takes a different approach. Using experience gained in its security labs, the company developed a complete library -- the RCM (Rootkit Common Module) -- to replace the Windows APIs, says Geoff Grindrod, solution product manager at Trend Micro. According to Grindrod, the library includes double encryption to avoid spoofing, and its proxy for API calls is constructed as a special kernel module. With the RCM, the system sees hidden processes, hidden registry keys, and hidden files. As the RCM has matured, it has been integrated into more and more Trend products and is now a core component of anti-spyware and other Trend Micro products, Grindrod says.

Discovering rootkits, however, is only half the battle, as excising them can result in its own set of problems.

"Rootkits are so imbedded in the operating system," Mandiant's Butler says. "Plus, we're seeing firmware attacks and survivable rootkits installing themselves in the BIOS. Removing rootkits can also make the system unstable while it's running."

Admins should be aware of the implications of rootkit removal before lunging headlong into the endeavor, says Ron O'Brien, senior security analyst at Sophos, one of the first security vendors to offer a rootkit removal tool.

"Rootkits are not 'bad,' but they have developed a reputation for being bad," O'Brien says. "They are really just a form of hidden files" that may have legitimate uses. Ripping rootkits out before establishing their purpose can prove detrimental to overall system health, he adds.

Coping with an evolving threat
Despite advances in prevention and removal, Steve Manzuik, senior manager of security engineering and research at Juniper, sees no end in sight to the rootkit threat. In fact, Manzuik believes that rootkit.com, Joanna Rutkowska's work on the Windows kernel, and Microsoft's resource protections for 64-bit Windows Vista are "making it more difficult for both attackers and vendors."