Jon Puleston Vice-president of innovation, Lightspeed

1. Distilling the Crowd
Jon Puleston
VP Innovation
28th Nov 2016
The next evolutionary step in crowd
prediction techniques

2. This is the story of the development of a research methodology

3. Market research is not so good at predicting certain
things…
Will you buy
this product?
Does this
ad work?
Who will win
the election?

4. Political
Polling

5. • The US election poll error = 2%
• EU referendum error = 4%
• UK election error = 5%
Show me a brand that would not be happy with a 2-5% error margin on
volumising their market?
BEFORE WE BEAT OURSELVES UP TOO MUCH ABOUT THIS LET’S
CONSIDER…
5
No other method not reliant traditional market research polling
information gets anywhere near doing any better:
• Social media analytics directionally helpful but proven completely unreliable at predicting
• Betting markets have not only miss-called all 3 votes, they have magnified the error

6. • What respondents say and what they do is so often different – we are often not
very good at observing our own behaviour
• Often asking people questions they don’t know the answer themselves to, or are
too difficult to work out
• Not always asking the right question in the right way to get the right answer
• Sometimes compounded by not asking the right people and the difficulties of
accessing representative samples
DO I NEED TO EXPLAIN TO A ROOM FULL OF RESEARCHERS WHY
PREDICTION IS SO DIFFICULT?
6

7. • Well, for the last few years we have been very interested in prediction protocols
THE WISDOM OF THE CROWDS
• Instead of asking people what they will do, will they buy a product, you turn then
tables and ask them to predict the outcome – who will win, will the product be
successful
• We have conducted about 3 years’ worth of experiments looking at the differences
between traditional and predictive research approaches
SO WHAT ARE THE ALTERNATIVES?
7

8. • 100 year old evidence to show that when used in the right way, it can be
uncannily accurate
• Accesses a different part of the brain “we think”
• You can sculpt a different type of survey using these techniques, that is more fun
• With a prediction you can feed back the answer and turn it into a game
WHAT WE LIKE ABOUT PREDICTION PROTOCOLS
8

9. • We have had some success at using this approach
• It can be really effective for ad testing…
WHAT WE LIKE ABOUT PREDICTION PROTOCOLS
9
0.89 correlation 5x differentiation

10. • They can miss fire ~ 15% of the time
• Highly sensitive to a wide array of cognitive biases
WHAT WE DON’T LIKE ABOUT PREDICTION PROTOCOLS
10

12. 35% LIKE BEST
9% PREDICT THIS IS
MOST POPULAR
25% PREDICT THIS MOST POPULAR
Prediction error:
FAMILIARITY DOES NOT EQUAL LIKING

13. PREDICT IF THIS A GOOD AD?
“Its an Apple ad and
everything they do is great,
right!”
some people find it easier to put themselves in someone else's shoes, some
people find it easier to step back and look at things more objectively.

14. • Crowds are particularly rubbish at predicting things like elections!
• Our predictions for many people are really proxies for what they want to happen…
WHAT WE HAVE LEARNT…
Predict who will win the election

15. • But its more complicated than that
…we now all live in social bubbles, so even if we are able to step back from what we
want to happen and look out and all we see we see is that nearly everyone has the
same opinions as us
WHAT HAVE WE LEARNT?
15

16. ME!
The minority illusion

17. More social people have more connections but not necessarily the same
points of view as the crowd

18. This person is a
socialite has lots of
dinner parties. She
also goes to London
a lot where blue is
the most fashionable
colour
Thesr 2 people are
also more social and
aware that blue is the
trend colour
The rest of this crowd are
less social don’t have many
dinner parties so have not
seen many other peoples
bathrooms
The minority illusion: predict what coloured bathrooms people have?
The crowd predicts blue!

19. SOLUTIONS?

20. Isolating the good
predictors

21. Philip Tetlock
THE GOOD JUDGEMENT PROJECT

22. PHILIP TETLOCK’S GOOD JUDGEMENT PROJECT HAS DEMONSTRATED THAT
SOME PEOPLE ARE A LOT BETTER AT PREDICTING THAN OTHERS
By asking people to make ongoing predictions you can begin to calibrate them and cream off the good
predictors into panels of super predictors

24. LEARNING FROM THE GOOD JUDGEMENT PROJECT
• Those with an actively open mind-set are better at making predictions
• Knowledge and understanding of the topic is important
• For more complex predictions, numeracy and intelligence are important
• Each participant needed to make 20 predictions before you could tell if they
were any good or not

25. Are researchers good predictors?

26. Kantar prediction experiment….examples of what we asked them to predict:
Competitive Brand performance
Global economic growth figures
TV audience figures
Global warming estimates
Assessment of fashion trends
Performance of teams in the Olympics
Outcome of a variety of political events

27. 48%
25%
Kantar staff Consumer
Close Predictions

28. Who will win the US election: Kantar Staff prediciton
80%
20%
Clinton Trump

29. How many superforecasters predicted the US election?
Even if they are brilliant at predicting, if the raw data they are working from is flawed
they will simply amplify the error (the minority illusion)
Some poll agitators e.g. Polypoll were predicting Clinton victory with 99% certainty

30. Solution: Find the people with the best
vantage point on what is going on

31. C
C
A
B
Predicting what coloured bathrooms people have…

32. C
C
A
B
Person A, has a blue bathroom and so do all their friends so predicts
blue

33. C
C
A
B
Person B, has an orange bathroom and so do most of their friends
so predicts orange

34. C
C
A
B
Solution = Ask the plumbers

35. ME!
Research people
in this territory

36. • Find a group of people with good judgement but also with a good vantage point!
Then combine these 2 bits of thinking…
Good
Judgement
Plumber
Plumbers with good judgement

37. Predict the polls experiment
• Regular monthly survey in the 5 months leading up to the UK election
• Each month a group of 1,500 randomly recruited panellists were asked to predict the opinion polls

38. DISTILLING THE EXPERTS
Scored each respondent each month for the prediction accuracy
Isolated the top 100 predictors and aggregated their predictions
Compared their predictions to the average from the 4 leading polls

39. It worked…sort of

40. EU Referendum forecasting leave…

41. Issue
F#@k me it is a lot of work
to find, calibrate and keep these people!
Issue 2
We had our suspicions that we were just lucky.
The only way to validate the method would be to test out on 20
or so similar 50/50 elections which come round once in a blue
moon (we thought!)

42. Could we apply this technique to something that
does not occur once in a blue moon?

43. Distilling a crowd who could predict the success of movies

44. Predicting box office takings…

45. Results of experiments
• Recruited 1,500 participants in the UK & USA
• Survey experiment ran for 5 months with adapted versions in each
market
• 70% of participants completed all 5 waves
• Collected a total of 38 film predictions
• The average participant correctly predicted the box office
performance (+/-20%) with 40% accuracy
• Isolated a group of the top 100 participants who predicted box
office takings (+/-20%) with an average accuracy of 60%
45

47. Could we switch to something that
was easier to calibrate?

48. Price testing experiment
48
A simple test taking 6 minutes

50. Ad testing experiment
50

51. 51
Source: Based on an average of 4 large scale experiments
= 90% better

52. Crowd accuracy (Assuming no network errors)
Accuracy of individual predictors
100
people
50
people
20
people
10
people
5
people
50% 50% 50% 50% 50% 50%
55% 86% 80% 74% 74% 60%
60% 98% 94% 85% 82% 68%
65% 100% 99% 95% 91% 79%
70% 100% 100% 98% 95% 84%
75% 100% 100% 99% 99% 91%
80% 100% 100% 100% 99% 94%
85% 100% 100% 100% 100% 97%
Sample Prediction maths

54. • Crowd distillation techniques I believe is the next evolutionary step in crowd prediction methodology
• The maths of expert crowd prediction work on smaller numbers - whether or not the market is ready for a
expert micro-sampling techniques is another matter
• It takes work to distil a crowd – finding and maintaining relationships with good predictor is not easy
• Creativity has been require to think how to apply this methodology within the Market research arena
Summary & learnings
54