Unix: Rootkit detection and prevention

If a rootkit is hiding files, processes, log-ins, etc., how would you know? If you can't see it, how would you remove it?

Last week's posting was a reminder of how stealthy and dangerous rootkits can be. They hide in your operating system and compromise your systems and your data as quietly as possible. It often turns out to be some odd behavior or system peculiarity that clues admins into the presence of a rootkit -- if they're lucky.

If you've got a rootkit on your system, what it's going to do depends on the rootkit itself. It might be set up for keystroke logging or for giving a remote user control of the system any time they want. It might send spam or participate in a DDoS attacks. It might send your critical data to an external server, inject code into your web pages, or make your system unusable. And that's just a start. Whatever it does, it isn't good for you or your organization.

[ InfoWorld's Paul Venezia reveals the 9 traits of the veteran Unix admin. | Track the latest trends in open source with InfoWorld's Open Sources blog and Technology: Open Source newsletter. ]

Most rootkits can survive reboots. Many go to extreme measure to resist detection not just by Unix admins, but by the tools that admins use to locate and remove them. In fact, the efforts of security vendors to detect rootkits and rootkit architects to avoid detection is something of an ongoing "arms race". The threat landscape is continuously changing and what worked a few months ago might not work today.

Rootkits will often be specifically engineered to block security tools, especially those designed to detect and remove rootkits. Trying to remove a rootkit can be like trying to remove from your favorite shirt a stain that is designed to resist stain removers -- except, of course, that you can actually see the stain on your shirt. What do you do if netstat, du, find, ps, ifconfig, inetd, killall, and lsof all lie to you? How would you even know?

Replacement executables only represent the simplest form of rootkits. These would be easy to detect by comparing checksums with a known good system, though you would need to trust your checksum executable. One option I've seen people use for this is mounting a CD with an md5 binary and using it to compare their checksums.

More sophisticated rootkits might:

  •  
    • intercept and redirect system calls to a process that is running in memory
    • manipulate kernel objects -- hiding processes, changing privileges, etc.
    • modify kernel data structures
    • add drivers to your system
    • use covert channels to communicate with external systems

Fortunately, there are many commercial and free tools that are designed to both detect and remove rootkits. Examples

include RootkitReveaker, GMER, Sophos Rootkit Scanner and Rootkit Hunter.

Some of the techniques that rootkit detectors might use include:

  •  
    • using md5 checksums to compare with known good values
    • looking for file "signatures" based on byte patterns rather than checksums (these can lag)
    • looking for known files that might indicate a compromise
    • finding files that have been granted excessive permissions (such as setuid)
    • looking for hidden files in odd places
    • looking for pointers in memory in that refer to locations outside the expected ranges
    • comparing high level (user) views with hardware level views (files, registry keys)
    • for Windows systems, looking for registry discrepancies
    • looking at network traffic and comparing it what the system files acknowledge sending

The success of these techniques will depend on the sophistication of both the tool and the rootkit. A kernel mode rootkit can always alter the results of the commands you run trying to figure out what's going on.

1 2 Page
From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies