Ultimate cloud speed tests: Amazon vs. Google vs. Windows Azure

If all of these factors leave you confused, you're not alone. I tested only a small fraction of the configurations available from each cloud and found that performance was only partially related to the amount of compute power I was renting. The big differences in performance on the different benchmarks means that the different platforms could run your code at radically different speeds. In the past, my tests have shown that cloud performance can vary at different times or days of the week.

This test matrix may be large, but it doesn't even come close to exploring the different variations that the different platforms can offer. All of the companies are offering multiple combinations of CPUs and RAM and storage. These can have subtle and not-so-subtle effects on performance. At best, these tests can only expose some of the ways that performance varies.

This means that if you're interested in getting the best performance for the lowest price, your only solution is to create your own benchmarks and test out the platforms. You'll need to decide which options are delivering the computation you need at the best price.

Calculating cloud costs
Working with the matrix of prices for the cloud machines is surprisingly complex given that one of the selling points of the clouds is the ease of purchase. You're not buying machines, real estate, air conditioners, and whatnot. You're just renting a machine by the hour. But even when you look at the price lists, you can't simply choose the cheapest machine and feel secure in your decision.

The tricky issue for the bean counters is that the performance observed in the benchmarks rarely increased with the price. If you're intent upon getting the most computation cycles for your dollar, you'll need to do the math yourself.

The simplest option is Windows Azure, which sells machines in sizes that range from extra small to extra large. The amount of CPU power and RAM generally increase in lockstep, roughly doubling at each step up the size chart. Microsoft also offers a few loaded machines with an extra large amount of RAM included. The smallest machines with 768MB of RAM start at 2 cents per hour, and the biggest machines with 56GB of RAM can top off at $1.60 per hour. The Windows Azure pricing calculator makes it straightforward.

One of the interesting details is that Microsoft charges more for a machine running Microsoft's operating system. While Windows Azure sometimes sold Linux instances for the same price, at this writing, it's charging exactly 50 percent more if the machine runs Windows. The marketing department probably went back and forth trying to decide whether to price Windows as if it's an equal or a premium product before deciding that, duh, of course Windows is a premium. 

Google also follows the same basic mechanism of doubling the size of the machine and then doubling the price. The standard machines start at 10.4 cents per hour for one CPU and 3.75GB of RAM and then double in capacity and price until they reach $1.66 per hour for 16 CPUs and 60GB of RAM. Google also offers options with higher and lower amounts of RAM per CPU, and the prices move along a different scale.