The developer of the most widely used test for ranking the performance of supercomputers has said his metric is out of date and proposed a new test that will be introduced starting in November.
Jack Dongarra, distinguished professor of computer science at the University of Tennessee, said the Linpack test he developed in the 1970s, which has been the basis for the Top500 list of the world's fastest computers for the past 20 years, is no longer the most useful benchmark for how well a system can perform.
The new metric, he said, could change the way vendors design their supercomputers and will provide customers with a better measure of the performance they can expect for the types of real-world applications they'll be running.
The Top500 list is published twice a year, in June and November, and is closely watched as vendors and nations seek bragging rights for who has the fastest system. The current leader is the Tianhe-2, developed by China's National University of Defense Technology.
Linpack has been used to rank the systems since the first Top500 list was published in 1993, but it's no longer an indicator of real application performance, Dongarra said.
"Linpack measures the speed and efficiency of linear equation calculations," according to a statement Wednesday announcing the new benchmark, called the High Performance Conjugate Gradient (HPCG). "Over time, applications requiring more complex computations have become more common. These calculations require high bandwidth and low latency, and access data using irregular patterns. Linpack is unable to measure these more complex calculations."
HPCG is needed, Dongarra said in a telephone interview, in part because computer vendors optimize their systems to rank highly on the Top500 list. If that list is based on an out of date test, it encourages vendors to architect their systems in a way that's not optimal for today's applications.
"We don't want to build a machine that does well on this 'fake' problem. We want to build a machine that does well for a larger set of applications," said Dongarra, who developed the new test with a colleague, Michael Heroux, of Sandia National Laboratories in Albuquerque.
Because of the way the new test is being introduced, however, it could potentially spark disagreements over who really has the world's fastest supercomputer. That's because HPCG will be introduced gradually over time, and it could be years before it becomes the primary method for ranking the Top500.
"One of the nice things about Linpack is that there's one number, so it's very clear what we mean by the fastest computer. This will in fact generate two numbers," Dongarra said.
He plans to maintain the Linpack test alongside HPCG in part for the valuable trending information that Linpack provides, he said. But it will also continue to be used because it could take years before a significant number of supercomputers are tested against the new benchmark.