Microsoft on 'rootkits': Be afraid. Be very afraid.

Microsoft security researchers are warning about a new generation of powerful system monitoring programs, or "rootkits," that are almost impossible to detect using current security products and that could pose a serious risk to corporations and individuals.

The researchers discussed the growing threat posed by kernel root kits at a session at the RSA Security Conference in San Francisco on Tuesday. The malicious snooping programs are becoming more common and could soon be used to create a new generation of mass-distributed spyware and worms.

With names like "Hacker Defender," "FU" and "Vanquish," the programs are the latest generation of remote system monitoring software that has been around for years, according to Mike Danseglio and Kurt Dillard, both of Microsoft's Security Solutions Group.

The programs are used by malicious hackers to control, attack or ferret information from systems on which the software has been installed and are typically installed on a machine without the owner's knowledge, either by a virus or following a successful hack of the computer's defenses, they said.

Once installed, many rootkits simply run quietly in the background but can easily be spotted by looking for memory processes that are running on the infected system, monitoring outbound communications from the machine, or checking for newly installed programs.

However, kernel rootkits, which modify the kernel, or core request processing, component of an operating system, are becoming more common. Rootkit authors are also making huge strides in their ability to hide their creations, said Danseglio.

In particular, some newer rootkits are able to intercept queries or "system calls" that are passed to the kernel and filter out queries generated by the rootkit software. The result is that typical signs that a program is running, such as an executable file name, a named process that uses some of the computer's memory, or configuration settings in the operating system's registry, are invisible to administrators and to detection tools, said Danseglio.

The increasingly sophisticated rootkits and the speed with which techniques are migrating from rootkits to spyware and viruses may be the result of influence from organized online criminal groups that value stealthy, invasive software, said Dillard.

One rootkit, called Hacker Defender, which was released about one year ago, even uses encryption to protect outbound communications and can piggyback on commonly used ports such as TCP (Transmission Control Protocol) port 135 to communicate with the outside world without interrupting other applications that communicate on that port, he said.

The kernel rootkits are invisible to many detection tools, including antivirus, host and network intrusion detection sensors (IDS) and antispyware products, the researchers said.

In fact, some of the most powerful tools for detecting the rootkits are designed by rootkit authors, not security companies, they said.

There are few strategies for detecting kernel rootkits from an infected system, especially because each rootkit behaves differently and uses different strategies to hide itself.