Opinion

Stop future gazing and start taking data lessons from history

Richard Shotton argues that we shouldn't allow our fixation with the future crowd out an interest in the past, because there's a lot of lessons to be learned from history - even when it comes to data

The trade press overflows with interesting predictions about the future of advertising.

But there’s a problem.

Experts, marketing or otherwise, have an awful record of predictions. Philip Tetlock, a psychologist at the University of Pennsylvania, ran a 20-year study that analysed 82,361 forecasts from 284 experts. He found that their predictions were as likely to be wrong, as right. In his memorable phrase, the average pundit fared no better than ‘a dart-throwing monkey’.

It’s not just that these marketing predictions misguide us. There’s also an opportunity cost. Our fixation with the future crowds out an interest in the past. Yet there is value in looking backwards to people who grappled with similar problems to ours.

So, let’s consider a current problem, turning data into insights, and look back to what we can learn from the 1940s. In particular, the experience of one man: Abraham Wald.

Wald was the son of a kosher baker and grandson of a rabbi, born in the Austro-Hungarian Empire in 1902. He was a mathematical prodigy – at primary school he was correcting his teachers and at secondary school he was correcting his textbooks.

After a glittering early career at the University of Vienna, his success stalled in the 1930s with the rise of anti-Semitism. In 1938 he fled to America, a decision that probably saved his life. By the end of the war his whole family, bar one brother, had been killed.

Once in the US he worked for the government’s Applied Mathematics Panel, applying his talents to the war effort. Specifically, how to reduce the death rate among European bomber crews. Theirs was such a dangerous job that about half of bomber crews died during the war.

Wald’s team needed to determine which parts of the planes should be armoured. Too much and the bombers would be slow and cumbersome, easy targets for German anti-aircraft defences. Too little and the crew were exposed – a handful of hits might bring them down.

The researchers set to work collecting data. As planes returned from a sortie his team recorded which parts had been punctured by bullets. When hundreds of bombers had been logged, a pattern emerged. As the diagram clearly shows, the areas most regularly hit were the wings, fuselage and tail.

The top brass was ecstatic; Wald’s methodology had convincingly identified the critical parts to armour. Case closed.

But was it?

Wald’s interpretation was radically different. His bosses had fallen victim to “survivorship bias”: assuming returning planes were representative of all aircraft. That was a mistake. The data collected omitted the crucial planes: downed ones. They held the clue to solving the problem, not the returning planes.

The parts peppered with bullet holes could be ignored on returning planes, after all they had limped home. It was the unscathed spots that needed armouring. Any plane hit in these areas was at the bottom of the channel.

Wald’s counter-intuitive thinking was quickly tested and shown to improve survival rates. His approach became standard US policy until the Vietnam War.

This is more than a historical anecdote. It’s a guide to how best to use data, whether in the military or marketing. There are four key principles that can be drawn from Wald’s experience.

Don’t rely on intuition

Wald was a genius (a genuine one, not in the way we now bandy about the term) yet even he didn’t rely on introspection alone. He collected data, and only then, developed his hypothesis.

Far too many plans and approaches are based on gut feeling alone. This is an issue because agency staff are not representative. A study by Newsworks investigated this problem by recruiting 30 young media planners to complete the IPA Touchpoints diary for a week. This meant that their media habits could be compared to the broader population.

The differences were stark. Take TV. Young planners watched half the volume of TV compared to the population as a whole. Other media showed a similar variation.

This discrepancy is a concern because of what psychologists term the false consensus effect. It’s the idea that we over-estimate how much others share our beliefs and behaviours.

Again, this bias is prevalent in agencies. Jenny Ridell and I tested this by asking media agency planners to estimate what proportion of the population had an iPhone.

We then crossed that data with whether they had an iPhone or not. Those who owned one thought half the population did, while those who didn’t put the figure at just a third.

Relying on intuition is dangerous – it leads to plans that would influence us but not the consumer. Instead, every plan needs data at its heart. And Wald gives us guidance on how to achieve that.

Collect data in the simplest manner that answers the question

Wald’s methodology required pen, paper and a numerate assistant. Basic even for the 1940s.

This is important for marketing as too often we obsess over the complexity of our methodologies: machine learning, artificial intelligence and so on. It’s as if we believe we can replace the hard work of thinking with the high costs of measurement.

But insights don’t need high budgets. For a brief into a male incontinence drug we wanted to help the planners understand the target audience. We had no budget, so we used a technique we call ‘method planning’.

Over a weekend we texted the planners at random times. Each time they received a text they had to stop what they were doing and get to a toilet within two minutes.

This helped the planners understand the experience of the target audience. From this experiment, we uncovered two useful insights. First, incontinence is not a great concern when people are at home. After all, a toilet is a few seconds away. It’s when they’re out of home that it’s a worry. This led us to recommend media which reached people at the maximum moment of concern, such as tube car panels.

Second, our participants mentioned that while the experiment was inconvenient for them, it was also a burden on their families. This led us to the insight that maybe it was better to encourage older, male sufferers to rectify the issue not for their benefit, but for their family’s.

The cost of all this? About $1 on my phone bill.

Good insights can come from the simplest of places. Only if we recognise this will we undertake insight generation as frequently as we should.

Focus on the representativeness of the data

A seemingly simplistic point but the majority of online measurement collects short term effects only: sales, views or visits. This is a problem as what we measure shapes what we do. If we just measure the short-term, then that is what we optimise to: more budgets go to the best performers, while the worst are struck off the plan.

And if we optimise just to short-term metrics then we under-perform in the long term. Binet and Field have showed in their analysis of the IPA Effectiveness database, that what works best in the short term isn’t ideal in the long term.

Data alone is worthless, analysis is key

Data are nothing but raw material. They’re a necessary condition for insight, but not a condition that necessarily creates insight.

Henri Poincaré, the 19th century French mathematician, pointed out the difference between science and facts. What he said is as relevant for insight and data: “Science is built of facts the way a house is built of bricks: but an accumulation of facts is no more science than a pile of bricks is a house.”

And just as facts require theory to become science, so does data require theory and analysis to become insight.

If agencies want a competitive advantage they need a combination of best in class data collection and interpretative skills. The best source of those interpretative skills is the study of human decisions making: behavioural science and social psychology. These tell us about insights like survivorship bias but also hundreds of others that affect consumer decision makings.

Stop worrying about the changing man

The experience of Wald is useful. But there are plenty of other historical figures you can learn from. They shouldn’t be ignored because technology has changed – fundamental human motivations are the same as they ever were.

As Bill Bernbach said: “Human nature hasn’t changed for a million years. It won’t even change in the next million years. Only the superficial things have changed. It is fashionable to talk about the changing man. A communicator must be concerned with the unchanging man with his obsessive drive to survive, to be admired, to succeed, to love, to take care of his own.”

The agencies that focus most on the unchanging man will win the insight war.

Richard Shotton is deputy head of evidence at MG OMD and the author of The Choice Factory. This post was first published on WARC.

ADVERTISEMENT

Get the latest media and marketing industry news (and views) direct to your inbox.

Sign up to the free Mumbrella Asia newsletter now.

 

SUBSCRIBE

Sign up to our free daily update to get the latest in media and marketing