Presentation from recent KR event held in Cape Town. The presentation is shareable and downloadable.

2. From Hunches to Hypotheses
The business case for building a culture of experimentation within your organisation

3. • Taking Organisational Decision Making Seriously
• 4 Hurdles to Good Decision Making in our Current Environment
• A Formula for Building a Culture of Experimentation
• Some Key Take Outs
Presentation Contents:

4. • Taking Organisational Decision Making Seriously
• 4 Hurdles to Good Decision Making in our Current Environment
• A Formula for Building a Culture of Experimentation
• Some Key Take Outs
Presentation Contents:

5. “Whatever else it produces, an organisation is a
factory that manufactures judgments and decisions.”
- Daniel Kahneman

6. ‘When you get down to it, more successful
businesses are just making better decisions’
Decisions concerning how best to create value for the clients
Decisions around designing products and services that will engage clients
Decisions about how best to help clients avoid making damaging mistakes
Decisions concerning the best ways to communicate with clients and what to say
Decisions relating to getting clients to behaviour in ways that are in their best interest
Decisions about…...

7. ‘Organisations have a good understanding of what they are
trying to achieve as a bottom line. The tricky thing is
deciding on the best ways to get there.’
Improve
profitability
Increase the
number of
users
Lower client
churn
Increase
in-app
conversions

8. And it’s only going to get trickier. Why?
There are 4 hurdles that need to be overcome to improve
business decision-making within an organisation
1. Status Quo Bias
2. Cognitive Overload
3. The Illusion of Validity
4. Authority Bias

9. Using story to help explain these hurdles

10. 4 user characters to help you understand these hurdles..
MarleyAva Barry Mike

11. Marketing Manager at a
large insurance company
Ava (35)
1
Status Quo Bias

12. Over the past 20 years, grown in a successful
market leader. However, the industry that Ava
finds herself in is changing. Financial
technological solutions and the scalability of
their services and communications are on the
rise, which threaten the comfort zone that
Ava’s company operates in.

13. Ava's knee jerk reaction is to stick to what she
knows, use the same channels that have
worked in the past, and ultimately rely on the
same systems she advocated so hard for and
feels she can control.

14. Ava is not alone…
Although it may feel comfortable, sticking to the status quo can often be
a dangerous approach to take in a rapidly changing environment
Continuous, successful innovation at all levels of business is not just nice to
have anymore. It is crucial to a company’s success going forward.

15. 2
Digital Strategist at
successful young financial
services company
Barry (29)
Cognitive Overload

16. Barry is preparing for a large internal strategy session by mapping out
the channels and tactics for the year ahead.
He's been excited about the opportunities that technological solutions
have created, but upon further exploration, Barry's mood has changed
from one of excitement to anxiety and angst.
He has realised the shear amount of options available and it has
become impossible to choose.
Barry decides to shift gears and focus on something else, promising
himself he'll get back to it.

17. Marketing technology landscape grew
by 40% in the first quarter of 2017….
Barry is not alone…

18. ‘Innovation Overload!’ – Louis Gudema
An organisational variation of ‘choice overload’
Implications:
A. Put our heads in the sand and just keep following the
status quo
B. Use a cognitive heuristic that may ‘feel right’ but lead
to sub-optimal outcomes (simplicity, salience,
familiarity)
Study by Sheena Iyengar and Mark Leppe (Jam purchases in supermarkets)

19. 3
Arthur (54) Mike (32)
VS
The Illusion of Validity

20. Arthur (54)
Business Owner of a
retail store in a small
community

21. Arthur lives 2km away and walks through town every
morning to open his store at 7am sharp, greeting
friends and engaging with many of the locals as he
makes his way. His shop is the only one of its kind in
the area, and, as a result, Arthur is on a first name
basis with most of his customers from over the years.

22. Arthur has built a successful business by relying
on his intuition and using his gut feeling to make
almost all of his strategic and tactical decisions.
From the products he stocks to the in-store
architecture and interactions he has with his
customers.

23. Mike (32)
Digital Growth Strategist for an
international fintech company
based in Johannesburg

24. Although Mike has worked in his current role for the past three
years, unlike Arthur, he has met only 1% of his customers, and
most of these interactions have been remote and skewed towards
those who tend to voice their opinions. As a result, Mike has a
biased representation of the entire base, who remain very much
silent.

25. The majority of Mike’s understanding of his customers from
online research projects and analytics, which are often content-
dense and difficult to internalise. In addition to this, Mike's
customers are constantly exposed to other services and, at the
same time, he is desperately trying to keep up with a rapidly
changing, technological environment.

26. Arthur (54) Mike (32)
Small Business Owner Digital Growth Strategist
 Stable environment
 Not much change overtime
 Continuous, accurate feedback
 Highly unstable environment
 Rapid change over time
 Infrequent, selective, remote feedback

27. Arthur and Mike work in completely different environments
Yet their approach to decision-making is very similar.
Even if all the information is available to him, Mike often simply goes with his gut, and
using research as a tool for seeking justification rather than using it to update his
beliefs.
Outcome: Reliance on invalid cues and so poor decision making outcomes
Skilled intuition arises where:
• There are valid cues
• The environment is stable
• The decision maker has the opportunity to learn

28. Take out:
Relying on intuition in an inappropriate
environment leads to poor decisions

29. 4
Innovation and
design expert
Marley (30)
Authority Bias

30. Marley is working on the next iteration of an
important digital interface for her expanding user
base. In order to guide her approach she has
explored many credible case studies and global best
practice playbooks.
Innovation and design expert

31. However, when Marley applies the recommended
solutions she finds them largely ineffective and in one
particular case, extremely damaging.

32. Contextual sensitivity calls into question the
generalisability of credible externally sourced solutions
• Scientific replication failures
• Faulty measuring approaches
• Impact decay
• Cultural nuances
• Problem misdiagnosis
Marley is not alone…

33. Contextual sensitivity calls into question the
generalisability of externally sourced solutions
• Scientific replication failures
• Faulty measuring approaches
• Impact decay
• Cultural nuances
• Problem misdiagnosis
Marley is not alone…

34. Problem Misdiagnosis Example: Defaults
Save More Tomorrow
Richard Thaler & Shlomo Benartzi
A programme whereby employees opt-in to a retirement savings escalation
rate that is triggered every time the employee gets a raise.
3.5%
6.5%
9.3%
11.6%
13.5%
Every time the employee gets a
raise, the savings rate is escalated.
Estimated increase in retirement
savings of 29.6 billion dollars.

35. Clearly defaults work!

36. • Researchers tested the impact of defaults on saving behaviour
• Context: Earned Income Tax Credit of low-income population
• Intervention: Default placing some of their tax credit in a savings bonds
• Control: No default, free to put money in a savings bond if they wanted to
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Opt-in Opt-out
Applying Defaults
What about Opt-out group?
(Using the defaults like Save More Tomorrow)

37. Diagnosing the behaviour
Out of every 100:
68 reported they were
saving too little
28 plan to raise savings
rate in the next two
morning
Only 3 people followed
through
Clear intention-action gap
Had specific plans for
what they were going to
spend the money on.
It was mentally
accounted for
Money already had a
clear intended purpose
Save More Tomorrow EITC

38. Clearly defaults work. But..
Only when certain working conditions are present!
How much money has been wasted on launching loyalty programs,
applications, popular products or novel technological solutions because
the situation looked the same as the successful case at face value?

39. Summary: The Four Hurdles
If you stick to the status quo, you will get left behind. Be proactive.
But there are so many options, which makes decisions is overwhelming
In an unstable environment with poor feedback, avoid relying heavily on intuition
Best practice is not always generalisable due to decay, cultural nuances and misdiagnosis

40. Challenge
With the four decision-making hurdles in mind, how can we consistently
develop innovation that is impactful at scale, in order for the business to
reach its objectives more effectively and at a faster rate than the market?
Solution:
The Planned Integration of a Systematic Framework for
Business Experimentation

41. Solution
The Planned Integration of a Systematic
Framework for Business Experimentation
What is Business Experimentation?
What is Systematic Framework?
What about Planned Integration?

42. Solution
The Planned Integration of a Systematic
Framework for Business Experimentation
What is Business Experimentation??
And a Systematic Framework??
What about Planned Integration??

43. What is Business Experimentation?
Typical case:

45. Solution
The Planned Integration of a Systematic
Framework for Business Experimentation
What is Business Experimentation?
What is a Systematic Framework?
What about Planned Integration?

46. What do you mean by systematic framework?
• What is a systematic framework – Structured approach to continuously
design, setup and run experiments
• Learnings are stored in a centralised space so that all can access them
• Ad hoc is fine when you’re doing one or two experiments

47. The Gravity Experimentation Framework:

48. Solution
The Planned Integration of a Systematic
Framework for Business Experimentation
What is Business Experimentation?
What is a Systematic Framework?
What about Planned Integration?

49. Planned Integration
Andrea Burbank
Head of Experimentation at Pinterest

50. 1. Get Started – low hanging fruit, low development costs, build cases
Planned Integration
Pinterest’s Experiment Maturity Model - A five step process:
2. Get Big – internal roadshow with case studies, education, lobbying
3. Get Better – but-in exists, guidance and facilitation, documentation
4. Get Out – single point of failure, templates, video tutorials, checklists
5. Get Tools – automate certain tasks to lower error and improve efficiencies

51. Which company’s have a mature culture of experimentation?
(At step five of Burbank’s maturity model)
• Netflix – 1000 experiments/year
• Amazon – 2000 experiments/year
• Google – 10 000 experiments/year
• Facebook – 10 000 experiments/year
• Capital One – 80 000 experiments/year

52. Judging by their leaders’ attitudes towards
experiments, these numbers are likely to increase
exponentially
“One of the things I’m most proud of, that is really key to
our success, is this testing framework. At any given
point, there isn’t just one version of Facebook running.
There are probably 10 000.” - Mark Zuckerberg, Facebook
“Our success at Amazon is a function of how many
experiments we do per a year, per month, per week, per
day.” - Jeff Bezos, Amazon

53. Why are big business leaders so happy with
experimentation?
Three Reasons:
1. The technological and data collection capabilities are finally here! This allows for
accurate, frequent experimentation on a large audience but at a low cost
2. Experiments successfully facilitate the interaction between our intuition and the irregular,
continually changing business environment by providing valid, frequent feedback.
3. Experiments are a proven approach for surfacing and scaling more successful ideas
through increasing the amount and quality of ideas.

54. • We are just as human as the users we design for
• Behavioural and psychological insights are not the only dimension of the academic value
proposition. There are scientific techniques, procedures and ethical standards that are well
worth considering too.
• One of them being randomised controls trials (experiments), which is currently one of the
best investments you can make in growing your company
Key take outs

55. www.gravityideas.com
Gravity Ideas
@gravityideas
david@gravityideas.com
Thank you