Security vendors question accuracy of AV tests

Antivirus software is frequently tested for performance, so picking a top product should be straightforward: Select the No. 1 vendor whose software kills off all of the evil things circulating on the Internet. You're good to go then, right? Not necessarily.

The increasing complexity of security software is causing vendors to gripe that current evaluations do not adequately test other technologies in the products designed to protect machines.

Relations between vendors and testing organizations are generally cordial but occasionally tense when a product fails a test. Representatives in both camps agree that the testing regimes need to be overhauled to give consumers a more accurate view of how different products compare.

"I don't think anyone believes the tests as they are run now ... are an accurate reflection of how one product relates to the other," said Mark Kennedy, an antivirus engineer with Symantec.

Representatives of Symantec, F-Secure, and Panda Software agreed last month at the International Antivirus Testing Workshop in Reykjavik, Iceland, to design a new testing plan that would better reflect the capabilities of competing products. They hope all security vendors will agree on a new test that can be applied industrywide, Kennedy said.

A preliminary plan should be drawn up by September, Kennedy said.

One of the most common tests involves running a set of malicious software samples through a product's antivirus engine. The antivirus engine contains indicators, called signatures, that enable it to identify harmful software.

But antivirus products have changed over the last couple years, and "now many products have other ways of detecting and blocking malware," said Toralv Dirro , security lead system engineer for McAfee Inc.

Signature-based detection is important, but an explosion in the number of unique malicious software programs created by hackers is threatening its effectiveness. As a result, vendors have added overlapping defenses to catch malware.

Vendors are employing behavioral detection technology, which may identify a malicious program if it undertakes a suspicious action on a machine. A user may unwittingly download a malicious software program that is not detected through signatures. But if the program starts sending spam, the activity can be identified and halted.

Also, a program can be halted if it tries to exploit a buffer overflow vulnerability, where an error in memory can allow a bad program to run. Host-based, intrusion-prevention systems, which can employ firewalls and packet inspection techniques, can also stop attacks.

The ways in which a computer can be infected also make comprehensive testing complex. For example, users may infect their computers by opening malicious e-mail attachments or visiting harmful Web sites designed to exploit known vulnerabilities in a Web browser.

The different modes of attack also involve different defenses, all of which would need to be tested to arrive at an accurate ranking, analysts said.

By contrast, signature-based tests can take as little as five minutes. "This is a very basic test," said Andreas Marx of AV-Test.org, who wrote his master's degree thesis on antivirus testing. "It's easy, and it's cheap."