How to make big data beautiful -- and useful

For big data to be more than a buzzword, companies need to focus on design

Touch screen graph on tablet in hands 180442942
Thinkstock (Thinkstock)

Everyone’s talking about big data, but largely due to its lingering complexity, adoption remains relatively low. Indeed, a recent Capgemini survey of big data practitioners found that a mere 13 percent have reached full-scale production.

That’s pretty small for big data.

But it doesn’t have to be this way -- at least, if  Zoomdata CEO Justin Langseth is to be believed. As Langseth told me in an interview, design is as important as performance when it comes to big data. “Big,” it turns out, doesn’t matter much if it doesn’t translate into “useful.”

Small success with big data

One of the best features of the big data revolution is that it’s powered by zero-cost open source software. Whereas business intelligence has been plagued by complex, high-priced software, today’s most innovative big data technology is a download away.

At least, that’s the theory.

In practice, as anyone who has tried downloading Hadoop can tell you, it works somewhat differently. Cloudera co-founder Mike Olson is absolutely right when he declares, “No dominant platform-level software infrastructure has emerged in the last ten years in closed-source, proprietary form,” including the best data infrastructure like Hadoop, MongoDB, Spark, and Cassandra.

But dominant doesn’t necessarily mean easy.

As the Capgemini survey reveals, only 27 percent of those surveyed see their big data initiatives as “successful,” and a mere 8 percent described them as “very successful.” Even proof-of-concept projects have hit hard times, with a success rate of a mere 38 percent.

Some of the problems are difficult to separate from the people deploying the technology, including “dealing with scattered silos of data, ineffective coordination of analytics initiatives, the lack of a clear business case for big data funding, and the dependence on legacy systems to process and analyze big data.”

But all of these ultimately come down to the difficulty in translating big data’s promise to organizations’ ability to make use of the technology.

Designing big data success

Here's where design comes in. As Langseth tells me, one of his first hires at Zoomdata was an award-winning album cover designer from the famous New York-based jazz house, Blue Note Records (think John Coltrane, Thelonious Monk, Sonny Rollins, and many more).

Yes, a designer of album covers is now designing big data systems -- and this designer is more the rule than the exception at Zoomdata. He reports to a vice president who sits on the UX Curriculum Advisory Board at NYU.

Clearly, Zoomdata approaches big data differently. When I asked Langseth to distill this design-centric approach, he said it comes down to three main areas:

1. Top-down imperatives. First, there must be a mandate from the top that the company is building a design-driven app. This is very rare for enterprise technology (as anyone that has been forced to look at an SAP interface can attest).

Tony Fadell, one of the “fathers” of the iPod, showed how this works with consumer products like the iPod at Apple and later Nest, now part of Google. Steve Jobs is the iconic leadership example of that outlook with Apple. Tony worked for Steve and took that view to Nest, then Google.

Despite this success in consumer tech, there are virtually no examples of this leadership mandate in enterprise software. That’s why most enterprise software is terrible.

2. Staffed by design. Langseth also stresses that companies need to put their hiring where their mouths are. That is, organizations must have a ratio of UX designers to developers that reflects the design-centric mandate.

Zoomdata aims for a 5-to-1 developer-to-designer ratio. Most enterprise software companies have a ratio closer to 50-to-1, if that, which is a major reason why the user experience is so bad in enterprise software -- including in big data, which after all is a developer-driven trend.

3. Regular folks. The third suggestion from Langseth is that the whole UX team must be running constant usability tests with regular people, not only usability tests with analytics specialists. Geeks may be developing the software, but for it to be truly relevant to the enterprise, mainstream users must be able to grok it.

As Langseth notes:

This is what makes our app easy to use for the average person on the street who may be familiar with a program like Excel, but may not be a business-intelligence-analytics specialist. From a strategic point of view, this opens our app to a much broader audience than if we were focusing solely on the specialist.

This is critical. Ultimately, big data wins when it becomes available to all in some form. Data expert Peter Goldmacher made this point years ago when he said the biggest winners in big data are the companies building it into easy-to-use applications. Unless it translates into value for mere mortals, it’s not that interesting.

Designers are developers, too

While understanding that developers don’t make the best designers, Zoomdata wants its designers to know how to program. Langseth insists, “All good artists and designers have a mastery of their medium and know it intimately. For example, a good painter mixes her own pigments for paint and builds her own canvases.”

This means she knows the limits of what these things can do and how to work within them. When you don't know the medium, you can easily design something that will take 10 times longer to build. Zoomdata doesn’t want its designers imposing impossibly difficult engineering asks on its developers, so they have to know how to swim in code.

Designing for the rest of us

It’s also important to run a lot of research on the language used in an app. As Langseth suggests, it’s critical to “avoid industry-specific jargon like the plague.” Tempted to dive into the nuances of Hive or Pig? Don’t. The end goal for big data is to democratize an enterprise’s data assets, which means it has to be useful to those who aren’t hard-core data scientists or analysts.

Which, ultimately, is the main point. For big data to really take off, it needs to be more than a data scientist’s tool. As Gartner analyst Svetlana Sicular points out, “Learning Hadoop is easier than learning the company’s business.”

At least, it should be. But the point is you want those who have learned Hadoop to be able to translate it into the plainspeak of a company’s business. Microsoft is trying to do this for R, just as Zoomdata is trying to do for unstructured data spewing from NoSQL databases, Hadoop, and more.

While design isn’t the only factor involved in realizing this vision, it’s a critical component that most enterprises overlook. That myopia is a big reason big data projects keep failing. We can do better. Focusing on design can help.