Nvidia, ATI said to massage benchmark tests

An organization that produces benchmark tests for measuring the performance of graphics chips has updated its test after it found code in Nvidia's graphics chip drivers that detected certain tests and altered the chip's performance to inflate results.

Futuremark, which developed the 3DMark03 benchmark, posted a statement on its Web site Friday detailing allegations that some Nvidia drivers have "detection mechanisms" that trigger a higher level of performance when certain tests within 3DMark03 are detected. ATI Technologies, Nvidia's rival, also appears to have altered its drivers in order to boost test results, Futuremark said.

Futuremark said it identified eight instances where Nvidia's Detonator FX 44.03 and 43.51 WHQL drivers detected specific 3DMark03 tests and inflated the benchmark results to the detriment of overall image quality. The allegations were first lodged by ExtremeTech, a hardware enthusiast Web site.

3DMark03 has been criticized by some as favoring products from ATI over those of Nvidia, said Peter Glaskowsky, editor in chief of the Microprocessor Report in San Jose, California. The benchmark contains certain DirectX calls that favor ATI, he said, and Nvidia appears to have been trying to rectify the situation by altering its drivers to convert that type of DirectX call into one that works better on Nvidia hardware, Glaskowsky said.

DirectX is an API (application program interface) developed by Microsoft to help games run on Windows operating systems, and 3DMark03 is designed to measure performance on hardware running DirectX 9.0, the latest version.

"It looks like what Nvidia is doing is trying to counteract their disadvantage, but some of the methods described (by Futuremark) aren't the correct methods for dealing with that problem," Glaskowsky said.

In some instances the 3DMark03 "shaders" -- pieces of code that render the appearance of a surface, such as a road or a tree --are discarded in favor of ones found in the Nvidia drivers that work more efficiently, according to Futuremark's statement. In other cases, code in the drivers artificially reduces the workload demanded by the benchmark test, thereby increasing performance, Futuremark said.

Nvidia, based in Santa Clara, California, did not respond to specific questions about Futuremark's allegations, choosing instead to issue a statement.

"Since Nvidia is not part of the Futuremark beta program -- a program which costs hundreds of thousands of dollars to participate in -- we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the statement said. "We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad."

A new version of 3DMark03 with slightly different code that blocks the Nvidia drivers from detecting specific tests reduced the benchmark score of a system with Nvidia's GeForceFX 5900 Ultra and the 44.03 driver by 24 percent, Futuremark said.