Benchmarking Amazon EC2: The wacky world of cloud performance
The performance of Amazon machine instances is sometimes fast, sometimes slow, and sometimes absolutely abysmalFollow @peterwayner
Time of day? Yes, the engineer said. He really wanted to know whether we were paying attention to when we run the tests. Sure, operating systems are an obvious source of performance differences, and using the latest drivers and patches always makes sense. But he also wanted to know the time of day.
This was new. CPUs have clocks that tick extremely quickly, but they'll stamp out computation at the same rate morning, noon, and night. Was it really important to watch the time? Yes, he said. In other words, the cloud machines may be more like a '70s-era Detroit assembly line than a Swiss watch.
Cloud Camaros and Corvettes
To find out just how variable the cloud might be, I fired up instances on Amazon's EC2. I chose Amazon because it's one of the industry's leaders with one of the most sophisticated clouds, but the conclusions here are probably just as true for any of the other clouds -- they're all playing the same game. They're building a superfast rack of computers and finding a clever way to share it with many users. Even though the sales literature makes it seem as if you're buying another computer like the one sitting on your desk, it's usually closer to buying into a time-share for a condo at the beach.
At this point, I should pause and put aside these glass-half-empty allusions to Detroit and real estate time-shares. A gracious defender of the cloud would say that sometimes you get what you pay for, but often you get lucky. On a bad day, you could end up sharing a CPU with 10 crazy guys trying to get rich minting Bitcoins by melting the processor; on a good day, the CPU will give you many extra cycles for free. You could end up sharing a machine with 10 grandmothers who only update the Web pages with a new copy of the church bulletin each Sunday. In other words, the CPU glass is not only half-full, but the bartender will often top off your tumbler with the last remnants of the bottle too.
The results were surprising. I purchased two kinds of machines: the low-end T1 Micro instance and the medium version called appropriatedly an M1 Medium. The Micro is a test machine with 613MB of RAM and a promise of "up to two EC2 Compute Units (for short periodic bursts)." The Medium comes with 3.75GB of RAM and a promise of one "virtual core with two EC2 Compute Units." The Micro is listed at 2 cents per hour and the Medium costs 12 cents per hour. Both of these prices are the list for an "on demand" instance available to anyone walking in the door. Lower prices are available for those who make a longer-term commitment with a reservation.
In both cases, I started with the stock 64-bit Amazon Linux machine image that comes preloaded with a customized version of OpenJDK 1.6.0_24. After updating all of the packages with Yum, I downloaded the DaCapo benchmarks and started them running immediately.
Micros, Mediums, lemons
In all, I ran the benchmarks 36 times on 10 different machines (five Micro and five Medium). In general, the Micro instances were much slower than the Mediums, but the Micro actually ran faster more than a few times. This generally happened soon after the machine started, probably because EC2 was allowing the machine to "burst" faster and run free. This generosity would usually fade and the next run would often be dramatically slower, sometimes 10 times slower -- yes, 10 times slower.
The performance of the Micro machines varied dramatically. There was one instance that could index files with Lucene in three different runs of 3.4, 4.0, and 4.1 seconds; it was predictable like a watch. But another instance started at 3.4 seconds, then took 39 seconds for the second run and 34 seconds for the third. Another instance took 14, 47, and 18 seconds to build the same Lucene index with the same file. Micro's results were all over the map.
In the DaCapo tests, performance of Amazon's T1 Micro instances was everything but consistent. (Download a PDF of the T1 Micro results.)