Hadoop and MapReduce: Breaking records in the cloud

Yahoo researchers create the longest pi yet -- and hint at the cloud's potential for highly intense and complex computation

Last week, a team of Yahoo researchers creating a long version of pi set a new record in the field of mathematics using the Yahoo cloud. According to Engadget's report, "The team, led by Nicholas Sze of Yahoo!, used the company's Hadoop cloud computing tech to break the previous record by more than double, creating the longest Pi yet."

The researchers leveraged Hadoop for this project. The widely distributed nature of Hadoop brought clear advantages by taking a divide-and-conquer approach. It cut up the problem into smaller pieces, then set different parts of the computer to work on different sections of the project.

[ Get the no-nonsense explanations and advice you need to take real advantage of cloud computing in the InfoWorld editors' 21-page Cloud Computing Deep Dive PDF special report. | Stay up on the cloud with InfoWorld's Cloud Computing Report newsletter. ]

As time passes, we'll see more of the massive amounts of resources now available in the cloud as-a-service being used to solve computationally intense and complex problems. Consider the options: You can purchase data centers full of servers, all working away on your problem for a very unreasonable price, or you can leverage the cloud to rent only the resources you need, just when you need them.

I've been pushing Hadoop as a killer system for the cloud, and this is just another instance of that value. While other databases are looking to scale through mechanisms such as hardware and software caching -- some with a million-dollar price tag -- Hadoop takes a more reasonable path. Hadoop uses MapReduce to spread out the data processing over a massive numbers of servers, then combines the results. It's an old approach in new open source wrapping, making use of the scalability and value of cloud computing.

This is just one example of new uses for the cloud over the next few years. As the growth of the cloud pushes down the cost of massive computing systems, both researchers and enterprises will find old problems with new solutions using cloud computing.

This article, "Hadoop and MapReduce: Breaking records in the cloud," originally appeared at InfoWorld.com. Read more of David Linthicum's Cloud Computing blog and follow the latest developments in cloud computing at InfoWorld.com.

From CIO: 8 Free Online Courses to Grow Your Tech Skills