Rootkits: The next big enterprise threat?

Capable of cloaking malware, rootkits are fast infiltrating the enterprise to expose sensitive data without detection

Late at night, a system administrator performed a routine check of a crashed server, one of 48 systems comprising a major online infrastructure that generated about $4 million per month in revenue. He was a bit surprised that the system had gone down, as it had been humming for months without any indication of being prone to crashing. The check uncovered three encrypted files. The administrator called on MANDIANT to analyze them.

What MANDIANT found was that an unauthorized kernel modification had caused the system to become unstable, and that the modification had compromised the system's security as well. To determine the extent of the breach, each of the 48 servers needed to be taken offline, booted in a controlled environment, and analyzed for three to five hours each. About half had the crack installed, forcing the company to assume that all credit card information had been compromised. What had first seemed routine resulted in a financial nightmare -- one that many companies are leaving themselves exposed to, unaware of the increasing pervasiveness of rootkits.

Every organization is aware of the importance of securing core systems, networks, and end-user equipment in an increasingly mobile and malware-saturated world. But what most may not realize is the growing threat of malicious software intended to keep its presence hidden from administrators and traditional anti-virus software. Termed after early Unix packages designed to replace commands that would otherwise alert admins to the presence of intruders who had "root" or admin access to systems, rootkits are on the rise among those seeking to steal corporate and personal information for financial gain.

Rootkits alone, of course, are not inherently malicious. But when packaged with malware, they can facilitate deeply compromising security breaches undetected, especially as they become increasingly popular for attacks on non-Unix systems, specifically Windows. And with Forrester Research recently estimating that security breaches cost companies between $90 and $305 for each record lost, who can afford to turn a blind eye to what may invisibly be leaching sensitive data from their network?

The rise of rootkits
Rootkits date back to the earliest years of the Internet, when crackers created cloaked variants of Unix commands to ensure their deeds on compromised systems would go undetected. A concern mainly of system administrators for Net-connected Unix systems, rootkits remained relatively low-profile for many years, until Sony BMG Music Entertainment's Windows rootkit DRM (digital rights management) boondoggle of 2005.

In an attempt to enforce copyright protection, Sony BMG developed a rootkit that surreptitiously installed XCP (Extended Copy Protection) or MediaMax CD-3 software when music CDs were played on a PC. Poorly designed, the software opened holes in the Windows OS, facilitating infection by viruses and causing other system problems. Mark Russinovich, now a technical fellow at Microsoft, discovered the rootkit's behavior, which he then announced on his blog. The resulting furor and further illustrations of the fallout of the rootkit led Sony BMG to recall the CDs and issue a removal program. Unfortunately, the removal program was equally poorly designed, leading to additional privacy and security concerns, as documented by Russinovich.

This incident awoke two groups to the potency of Windows rootkits: crackers and professional criminals who break into computers on the one side, and the companies who create software to protect systems on the other. Already entrenched in a high-stakes battle over malware, the two camps now had a new, potentially more damaging front on which to contend. The Computer Economics 2005 Malware Report, the organization's latest, put the cost of malware in 2005 at $14.2 billion. The ability of malware authors to hide their scripts from anti-virus software's capability of automatically detecting, protecting, and eradicating most malware would only serve to escalate the stakes, especially as malware authors' motivation "continued to shift from a general desire to inflict damage to an intent to gain financially, through theft of personal information such as credit card data or by gaining access to financial accounts," according to the survey.

The greater emphasis on mobility in the enterprise has certainly contributed to the increasing likelihood of infection with cloaked malware. So too are the various unpatched security holes in Microsoft Windows and related products, which provide access for automated rootkit installation. The proliferation of rootkits -- which are used to cloak files on disks, system hooks, and processes running on systems -- is alarming, as spyware developers and malware authors are creating bot networks that use rootkits to evade detection, hiding not only the malware but also what information is being obtained. Some of the more sophisticated rootkits even modify and corrupt Windows APIs. (For more detailed information on rootkits, visit rootkit.com or read Greg Hoglund and Jamie Butler's Rootkits: Subverting the Windows Kernel.)

Part of what's fueling the proliferation of rootkits is the ease with which they can be implemented.

"It has definitely ramped up over the last year and a half to two years," says Butler, principal software engineer at MANDIANT. "It has gotten very easy for malware authors to cut and paste these technologies into their code set to maintain a presence on the machine."

For the time being, malware rootkit use remains crude. "Many of the attacks are unsophisticated," Butler says. "We're not seeing leading-edge rootkit technologies." But the dynamics of intrusion and response that are the hallmarks of the security industry are fast pushing the use of rootkits in innovative directions.

The front lines of rootkit defense
Rootkits employ a variety of methodologies to conceal themselves. Some overwrite kernel structures to replace the hooks normally used by Windows commands. Others create files within the file system that are effectively invisible. Still others capture hooks in Windows commands to corrupt their outputs. Many hook into addresses used for kernel services, changing the address of the table entry so the rootkit gets called before the real Windows system call is performed. Extensive details on current approaches to concealment are available at rootkit.com and other Internet sites. One recent methodology posted on rootkit.com involves loading a drive in place of the Windows null.sys dummy driver. The same post outlines three other methods for hiding drivers and offers the code for null.sys replacement.

In terms of defending against infection, Microsoft Windows Vista 64-bit resource protection and Software Restriction Policies in Windows XP provide some assurance, but developers of rogue software have proven their ability to find new ways to hide code on compromised machines. In fact, the rootkit front is fast transforming into an arms race, with each side innovating in response to developments the other camp pushes forward. Keeping on top of the latest modes of prevention is essential, especially if you are responsible for a fleet of computers running any variations of Microsoft Windows.

As for the big security players, a number are appropriating the traditional approach to viruses, using signature-based searches to track down known rootkits and applying related fixes. Two of the major vendors, Symantec and Trend Micro, however, are taking unique tacks in combating rootkits.

Symantec is leveraging mapping technology to discover rootkits on compromised systems. Oliver Friedrichs, director of emerging technologies for security response at Symantec, believes rootkit eradication requires a stable, reliable design that minimizes false positives and mitigates system instability during rootkit removal. To make good on this mission, Symantec has employed the expertise and technology brought on board during the Veritas acquisition. Using VxMS (Veritas Mapping Service), Symantec's Norton Internet Security 2007 maps data on the hard drive, compares it with the Windows file structure, and isolates any discovered mismatches in an effort to repair potential problems. In effect, VxMS enables Norton to compare file systems with the raw data on the disk. Differences are immediately suspect.

For example, say Windows Explorer shows five files in a directory, whereas VxMS shows 10. Clearly, the additional five files are cloaked. Norton sends the suspicious files to Symantec for analysis, eradication occurs during reboot, and the discovered rogue is removed from other systems worldwide as a result.

Trend Micro takes a different approach. Using experience gained in its security labs, the company developed a complete library -- the RCM (Rootkit Common Module) -- to replace the Windows APIs, says Geoff Grindrod, solution product manager at Trend Micro. According to Grindrod, the library includes double encryption to avoid spoofing, and its proxy for API calls is constructed as a special kernel module. With the RCM, the system sees hidden processes, hidden registry keys, and hidden files. As the RCM has matured, it has been integrated into more and more Trend products and is now a core component of anti-spyware and other Trend Micro products, Grindrod says.

Discovering rootkits, however, is only half the battle, as excising them can result in its own set of problems.

"Rootkits are so imbedded in the operating system," Mandiant's Butler says. "Plus, we're seeing firmware attacks and survivable rootkits installing themselves in the BIOS. Removing rootkits can also make the system unstable while it's running."

Admins should be aware of the implications of rootkit removal before lunging headlong into the endeavor, says Ron O'Brien, senior security analyst at Sophos, one of the first security vendors to offer a rootkit removal tool.

"Rootkits are not 'bad,' but they have developed a reputation for being bad," O'Brien says. "They are really just a form of hidden files" that may have legitimate uses. Ripping rootkits out before establishing their purpose can prove detrimental to overall system health, he adds.

Coping with an evolving threat
Despite advances in prevention and removal, Steve Manzuik, senior manager of security engineering and research at Juniper, sees no end in sight to the rootkit threat. In fact, Manzuik believes that rootkit.com, Joanna Rutkowska's work on the Windows kernel, and Microsoft's resource protections for 64-bit Windows Vista are "making it more difficult for both attackers and vendors."

Manzuik sees that current approaches to rootkit discovery and removal are beginning to fail despite improvements in Windows security. Factor in the lag time before Vista protections are widely deployed, and you have a perfect breeding ground for rootkit innovation. For example, Manzuik points out that some rootkits can now bypass the security sandbox. They detect they are in the sandbox and lay low, effectively tricking the system into thinking they are legitimate apps.

MANDIANT's Butler, however, believes that Vista protections will have an impact. Not only will the protections make it more difficult for rootkit authors to break in, Butler says, but it will also require "another separate effort to conceal themselves and maintain their presence."

Manzuik and Butler do, however, agree on the importance of strict user access policies. Both view rootkits as further evidence against giving users admin-level access to systems -- especially at smaller organizations, where the practice is often promoted as a cost-cutting necessity.

"The culture in smaller companies is that they will only call the IT guys if they can't figure it out themselves, which leads to most users having admin rights on machines," Manzuik says. Any organization employing this policy -- regardless of its size -- will be compromised, Manzuik says.

Because of this, Manzuik believes policy should figure foremost as a means for protecting systems against rootkits: "Without buying special technology, [most organizations] can deal with the majority of the threats with proper security policy and management."

That said, recent attention paid to rootkits has resulted in a raft of discovery and removal tools, both free and host-based, including IceSword, RootkitRevealer, F-Secure's Blacklight, and Sophos Anti-Rootkit. Over time, these functions will be integrated into enterprise-grade anti-virus and host-based security solutions. In the meantime, however, most organizations remain unprepared -- all the more troubling, given that opportunism is pushing rootkit know-how deeper underground, out of the IT community spotlight.

In the past, innovations in the art of hiding rootkits was shared in newsgroups and posted to community Web sites. The financial upside of having rootkit knowledge, however, is changing that, MANDIANT's Butler says. Those who uncover new approaches may take their discovery to a security company as their calling card to obtain a job. More disturbing, however, is the amount of money malware authors are willing to pay for new techniques. And with both sides of the divide doling out cash for the latest innovations, rootkit development is clearly becoming a lucrative pursuit -- one that leaves most organizations in the lurch, unaware of what's coming.

To reduce the probability and impact of rootkit infection, organizations should take the following proactive steps:

1. Do not ignore the threat and do not rely entirely on deployed anti-virus or host security systems.
2. Develop and implement a plan to analyze the current state of all systems.
3. Establish proactive procedures for maintaining an expanding defense against rootkit installation attempts, including policies and end-user communication.
4. Create a plan to analyze any infections that occur.

Kevin Mandia, president and CEO of MANDIANT, notes two essential capabilities for discovering rootkits in the enterprise: "the ability -- tools and technology -- to detect the rootkit’s network traffic via network security monitoring; and the ability to perform a sophisticated host-based console review, [making sure you're] able to conclude that the host-based review did not identify the process that is generating the suspicious network traffic."

For organizations looking for added protection against rootkits, enlisting the assistance of security experts is a worthwhile idea. MANDIANT, for one, provides incident-response software and professional services, enabling organizations to tap experts when developing risk-mitigation strategies and when responding to incidents to determine what data was lost and how the attack entered and evolved.

Unfortunately, too many organizations will wait until they have lost data and have exposed themselves to great financial harm before taking steps. Don't be one of them.

Mobile Security Insider: iOS vs. Android vs. BlackBerry vs. Windows Phone
Join the discussion
Be the first to comment on this article. Our Commenting Policies