The Target analysts got their breakthrough by looking at the buying histories of women who had signed up for new baby registries at Target. The analysts noticed that pregnant women often bought large amounts of unscented lotion around the start of their second trimester, and that sometime during the first 20 weeks of their pregnancies they bought lots of supplements like calcium, magnesium, and zinc.
The analysts then searched for these same "markers" in all females of childbearing age, found the likely moms-to-be, and sent them offers and coupons for baby products carefully timed to the various stages of pregnancy. Ka-ching.
This is a relatively simple example, and one that happened to be reported in the media. But, as the Duhigg article points out, most large companies in America now have "predictive analysis" departments and are learning to look for the kind of markers that Target discovered hidden in its data.
Big data puts privacy in a new light
In the Target case, future parents were served with highly relevant ads and offers, and the retailer found a new way to reach its customers and pump up sales. No problem, right?
Wrong, say privacy advocates. The warehousing and analysis of so much data, and so many types of data, might lead the curators of the databases to infer things about us that we never intended to share with anybody. The data might even predict our future behaviors -- what even we don't yet know that we're going to do.
The "predictive analysis" of big data is often called "inductive analysis" in academic and research circles because it induces large meanings from small sets of facts or markers.
"Inductive analysis concerns itself with singular things that can seem to be innocuous, but that when combined with other innocuous data points -- like your favorite soda -- can create meaningful predictors of behaviors," says Solon Barocas, a New York University graduate student who is working on a dissertation about inductive analysis.
Target, for instance, didn't even need to know the names of the women it ended up sending pregnancy ads to. It simply delivered a target ad to a group of addresses with the right demographics and a common pattern of past purchases. A process so totally cold and machinelike being used to predict something so human, so personal, like pregnancy, is creepy.
In the next 10 years, marketers and advertisers will spend more and more on big data science, focusing on finding analysts who can discern patterns in large pools of data. Big data analysis positions are the new hot jobs, and the people who will fill them are a new breed with new skills. "These people need traditional statistics and computer science backgrounds, but also some coding and basic hacking skills," Barocas says.
Big data analysts don't just help target ads for products. A political campaign might do a survey of 10,000 people to learn about their demographics and political choices. It might buy more data about those people from one of the large data sellers, like Acxiom or Experian, then search for unique markers in the data that would predict future political leanings.
But those predictors may bear no obvious relation to what they predict, Barocas says. "For instance, the analysts might find that something odd -- like what fashion-magazine subscription people hold -- is a strong predictor of the kind of candidate they're likely to vote for."
In future elections and ballot initiatives, billions will be spent on making inferences about voters, and about the issues, candidates, and political ad content that they might be sympathetic to. The campaign with the best personal data and the best analysts may win. That seems like a very undemocratic way to choose our policies and leaders.