Futuremark says Nvidia didn't cheat, but broke rules

BOSTON - Futuremark, the developer of the benchmarking software 3DMark03, has retracted accusations of cheating it leveled against graphics chip vendor Nvidia over modified software drivers on Nvidia products. However, it remains opposed to Nvidia's techniques and is not reinstating the original benchmark results.

Futuremark has changed its description, but not its opinion, of Nvidia's decision to use application-specific optimizations in its drivers that detected the presence of specific 3DMark03 tests and altered the performance of its graphics chip to achieve a higher benchmark result. Benchmarks are software tests used to simulate application behavior on a processor or device. They are designed to be used by buyers of hardware to compare the performance of different products.

On May 23, Futuremark published a seven-page report detailing eight specific examples of where it believed Nvidia unfairly modified drivers after detecting certain types of shaders in 3DMark03 tests. Some shaders -- code that denotes how the surface of an object should appear -- in 3DMark03 were replaced in favor of ones in Nvidia's drivers that rendered the image more quickly, but at the expense of overall image quality, Futuremark said in that report. 

"Please note, that the cheating described here is totally different from optimization. Optimizing the driver code to increase efficiency is a technique often used to enhance game performance and carries greater legitimacy, since the rendered image is exactly what the (game) developer intended," Futuremark said in its report.

However, the rhetoric has cooled down, at least on Futuremark's part.

Futuremark, in Espoo, Finland, released a statement on its Web site Monday that read, "Futuremark now has a deeper understanding of the situation and Nvidia's optimization strategy. In the light of this, Futuremark now states that Nvidia's driver design is an application specific optimization and not a cheat."

However, application-specific optimizations are still forbidden under Futuremark's rules, said Tero Sarkkinen, executive vice president of sales and marketing for Futuremark. The benchmarking company put out its most recent statement because it believes the industry will not benefit from an extended "soap opera" between itself and Nvidia, he said.

Nvidia stands by its drivers, which have not been changed and are still available for download on its Web site, said Derek Perez, an Nvidia spokesman. The Santa Clara, California, company still believes it did nothing wrong and will continue to use optimizations, despite Futuremark's objections, Perez said.

"They made a bunch of rules without understanding the intricacies of the architecture," he said. "There are application-specific optimizations in every CPU and every GPU."

Rival ATI Technologies was quite willing to continue the mudslinging. "It's a shame that a large company with a lot of money and access to expensive lawyers can beat a small company into submission," said Chris Evendon, director, public relations for ATI.

Along with its report on May 23, Futuremark released a new build of 3DMark03 with slightly different code that did not affect the workload required by the tests. That release lowered the performance of Nvidia's GeForceFX 5900 Ultra processor by 24 percent and is the most accurate representation of processor performance, Sarkkinen said.