Yahoo offers free supercomputing to Hadoop developers

Company hopes to get more developers creating applications that can scale around Hadoop

Yahoo has teamed with Computational Research Laboratories (CRL), a lab run by India's Tata Group, to offer supercomputing facilities free to academic institutions in India that are researching large-scale computing, particularly around Apache Hadoop, an open source distributed computing project of the Apache Software Foundation that Yahoo supports.

The deal is similar to one that the company completed last year with Carnegie Mellon University, providing Hadoop researchers at the university with a 4,000-processor supercomputer.

Yahoo aims to get more developers to research and develop applications that can scale around Hadoop, said Rajeev Rastogi, vice president and head of Yahoo Labs Bangalore, in an interview on Tuesday. To this end, the company plans to replicate the model it is trying in India in some other countries as well.

Educational institutions in India will have access to the supercomputer facilities of the CRL for their research for an unlimited period, according to Rastogi. "Our objective is to encourage long-term projects," he said. Academic institutions typically do not have access to hardware and software infrastructure such as a giant cluster of thousands of nodes, he added.

There are a number of educational institutions in India that are already doing research in the area of large-scale computing, and Yahoo will try to get some of them to work around Hadoop, according to Rastogi. The company may even sponsor research at some of these academic institutions, he added.

CRL's supercomputer is called the EKA, and is ranked the fourth fastest supercomputer in the world in the Top 500 Supercomputer list. The supercomputer is available for use on commercial terms.

EKA has 14,400 processors, 28TB of memory, 140TB of disk storage, a peak performance of 180T flops (floating point operations per second), and sustained computation capacity of 120T flops. EKA will run the latest version of Hadoop and other Yahoo-supported, open source distributed computing software such as the Pig parallel programming language developed by Yahoo Research.

Mobile Security Insider: iOS vs. Android vs. BlackBerry vs. Windows Phone
Recommended
Join the discussion
Be the first to comment on this article. Our Commenting Policies