Attackers launch malware that automatically alters itself to avoid detection, and they constantly create new domains where their command-and-control servers can hide, but researchers have come up with security software that detects the presence of attack code even if it has morphed and tracks down domains that infected client machines report to.
A platform called ExecScent can detect C&C traffic and ID the malware family involved or even the specific criminal group behind it, a team of researchers will tell Usenix Security 2013 this week.
[ Security expert Roger A. Grimes offers a guided tour of the latest threats and explains what you can do to stop them in "Fight Today's Malware," InfoWorld's Shop Talk video. | Keep up with key security issues with InfoWorld's Security Adviser blog and Security Central newsletter. ]
[MORE AT USENIX: New security scheme whacks text spammers in hours]
The software carries out a two-pronged attack on malware. First it defines protocol characteristics of HTTP/HTTPS requests between infected clients and C&C servers as a way to recognize malicious traffic, its source and its destination, say the researchers. They are led by Terry Nelms, a Ph.D student at Georgia Institute of Technology and also director of research at Damballa, Inc.
Damballa has incorporated ExecScent into its Failsafe network security appliance as a feature called Request Profiler. Failsafe is usually attached to a TAP port on an Internet router and sometimes to a switch port in order to monitor traffic.
The protocol characteristics ExecScent examines are rolled up into control protocol templates (CPT) for each known type of malware. When actual communications traffic matches a CPT, it is considered suspect. The templates include the family of malware involved and, if available, the criminal groups behind the malware.
Often when attackers craft morphing malware they alter the code to avoid being detected by signature based defenses. But the devices involved in the attacks still have to communicate, and the means they use can be used to identify them, the researchers say.
The second prong of their approach is designed to reduce false positives that may result from matching traffic to a CPT. ExecScent tunes itself to each network it runs on, figuring out what potentially suspect traffic components are actually legitimate. So a particular HTTP user agent header may be commonly used on a given network and so would carry less weight as a factor in determining whether the traffic it enables is malicious.
The value of these components as determining factors varies network to network. For example, a domestic bank network might never send legitimate requests to China, so request to URLs in China would carry a heavy weight in determining that such requests are to C&C servers. On the other hand, a manufacturer with suppliers in China would generate HTTP requests to China URLs frequently, lowering the significance of such requests as a determining factor, the researchers say.
This evaluation and weighting as well as blocking the illicit traffic is done automatically via TCP resets, Nelms says. The system analyzes enough traffic over the course of two or three business days to adjust to each network, he says.