Don't think viruses, Trojans, and APTs are the only security problems that you need to worry about these days. Rootkits are still one of the most stealthy, potentially damaging and ultimately viable problems that can plague your systems. And they don't just infect Unix systems. Windows systems are also vulnerable to rootkits with the same concerns for detection that have made them such a problem on Unix systems. In fact, smart phones are just as vulnerable to rootkit attacks as the operating systems in your data center.
A rootkit is a piece of software (or a set of software components) that is able to hide within your operating system, often disguising itself as a kernel module or residing only in memory. Rootkits can also hide their presence by removing records from log files, failing to display when you type ps –ef.
[ Prove your expertise with the free OS in InfoWorld's Linux admin IQ test round 1 and round 2. | Track the latest trends in open source with InfoWorld's Open Sources blog and Technology: Open Source newsletter. ]
The name "rootkit" derives from the problem's historic roots on Unix systems where "root" is, of course, the power user and "kit" represented the suite of tools that co-opted root's authority. The first known rootkit was engineered in 1990. Since then, things have only gotten worse as now there are plenty of rootkits -- including many that one can download -- and, as with other forms of malware, detection tools have problems keeping up with the known rootkits, never mind the problem of recognizing new ones.
Since rootkits can sometimes thwart the activities of tools meant to identify them, detection is extra difficult and sometimes relies on the use of a trusted operating system to evaluate the potentially infected one. Other methods might involve evaluating the behavior or a system, looking for differences between systems that should be fairly identical, analyzing the content of memory and looking for signatures.
Not everything that might be labeled a "rootkit" is bad, though this term has pretty much come to equate to malicious in most peoples' eyes. However, some rootkits are not malware at all. Copyright protection systems may, for example, hide themselves for legitimate reasons. On the other hand, it's taken me until the 5th paragraph of this posting to even mention that rootkits can be good -- or, at least, benign. The term is rarely used in anything other than a negative context.
Rootkits don't generally install themselves. Instead, they are often part of a what has come to be called a "blended threat" -- an approach that uses several varieties of malware together to leverage the benefits of each. Often a "dropper" is used to install the rootkit. It carries the rootkit along as data and installs it when some action is taken to kick the infection into action. So, the infection often starts with clicking on a link or invoking a Trojan.