Security vendors question accuracy of AV tests

Other concerns remain, over sample sets of malicious software, the age of the samples and the relative threat those samples pose on the Internet as they become older. Security vendors also think tests should check how well security applications remove bad programs, a process that can affect a computer's performance.

For vendors, a failed test can be embarrassing, since the testing companies often issue news releases highlighting the latest results.

Testing companies make money in various ways. AV-Test.org is often commissioned by technology magazines such as PC World (a magazine owned by IDG). Virus Bulletin licenses its logo to companies for use in promotional material and publishes a monthly online magazine.

Earlier this month, Virus Bulletin announced that its latest round of testing produced some "big-name failures," including products from Kaspersky Lab and Grisoft SRO.

The company's VB100 tests antivirus engines against malware samples collected by the Wildlist Organization International, a group of security researchers who collect and study malware. To pass the VB100, products must detect all samples.

Kaspersky briefly removed a signature for a worm out of its product for "optimization" purposes on the day of the test, wrote Roel Schouwenberg, senior research engineer for Kaspersky, in an e-mail. The signature has since been put back in, he said.

"Obviously, we would have rather passed than failed," Schouwenberg wrote. "Had the test been conducted a day earlier or a day later, we would have passed."

Similarly, F-Secure initially failed its test also because of a technicality, but the failed rating was later reversed. All vendors are told after testing which samples they failed to detect, thus most end up adding signatures to their products.

So what should a user do? John Hawes, a technical consultant for Virus Bulletin, cautioned that the signature-based tests are "not enormously representative of the way things are in the real world."

But Hawes also noted that signature-based tests can indicate the reliability and consistency of a vendor's software. Virus Bulletin also writes reviews of AV suites, which take into account aspects such as usability, which may be just as important as detection for consumers. The company is developing more advanced tests that will test new security technologies.

AV-Test.org is already performing more comprehensive tests, although it uses between 30 to 50 malware samples, a much smaller sample set compared to its signature-based tests, Marx said. Those tests may give a better indication of how a security software suite performs.

At a bare minimum, through, users should install some security software, as computers without it can face high risks, Marx said. Several free suites are available that may be fine for light Internet use, he said.

Ironically, Marx doesn't use any antivirus software. That's because AV-Test.org collects malware for its testing, most of which comes through e-mail from other researchers. "I'm getting about 1,000 viruses a day," he said. "It [antivirus software] would be counterproductive."

Correction: Due to a reporting error, this story as originally posted incorrectly stated the number of malware samples. The article has been amended.