The growing enterprise interest in Hadoop and related big data technologies is driving demand for professionals with big data skills.
Analysts and IT managers at the Hadoop World conference in New York this week repeatedly pointed to skills availability as one of the key challenges companies face in adopting Hadoop and said that those with the right skills could command healthy premiums.
[ Explore the current trends and solutions in BI with InfoWorld's interactive Business Intelligence iGuide. | Discover what's new in business applications with InfoWorld's Technology: Applications newsletter. ]
One indication of just how limited that skills supply is: IT executives from JP Morgan Chase and eBay who delivered keynote addresses at the conference used the opportunity to recruit from the audience.
Hugh Williams, vice president of experience, search and platforms at eBay, told audience members that the auction site is recruiting Hadoop professionals and he invited those interested in exploring opportunities to speak with him.
Larry Feinsmith, managing director at JP Morgan Chase, who followed Williams, only half-jokingly told the audience that Chase was also hiring and would be willing to pay 10 percent more than eBay.
"Hadoop is the new data warehouse. It is the new source of data" within the enterprise, said James Kobielus, an analyst with Forrester Research. "There is a premium on people who know enough about the guts of Hadoop" to help companies take advantage of it, he said.
Hadoop allows companies to store and manage far larger volumes of structured and unstructured data than can be managed affordably by today's relational database management systems.
A growing number of companies has begun tapping the technology to store and analyze petabytes of data such as blogs, clickstream data, and social media content to gain better insights about their customers and their business.
The increasing enterprise adoption is driving demand for people with advanced analytics skills, Kobielus said. That includes people with backgrounds in areas such as multivariate statistical analysis, data mining, predictive modeling, natural language processing, content analysis, text analysis, and social network analysis, he said.
"Big data in the broader sense -- and Hadoop in particular -- is driving demand for people who have experience doing advanced analytics using newer approaches such as MapReduce and R for predictive and statistical modeling," he said. These are the data analysts or data scientists who will work with structured and unstructured data in Hadoop environments to deliver new insights and intelligence to the business, he said.
Interest in Hadoop is also creating demand for Hadoop platform management professionals, Kobielus said. Their job will be to implement Hadoop clusters, secure, manage, and optimize them and to ensure that the cluster remains available for enterprise use. "These are the people who build out and optimize the platform" on which Hadoop applications run, he said.
"The database administrators who administer Teradata and [Oracle's] Exadata are the same people who are now beginning to redefine their roles as Hadoop cluster administrators," he said. "They realize this is a brand-new world." Also, expect to see demand for storage management professions and for those who can help integrate Hadoop environments with existing relational database technologies.