1.
1
Three Stages to
Experimentation
Excellence
Elizabeth Gabster
Kenneth Kutyn

2.
2
Elizabeth Gabster
Lead Value Consultant
Housekeeping: Ask questions in the Q&A widget
We will share the recording and slides
Kenneth Kutyn
Lead Solutions Engineer

3.
3
Getting experimentation started
Experimenting with ease.
Experimenting continuously.
Experimenting across departments.
Experienced but not at the head of the pack (yet).

4.
4
Customers move fast, you need to move faster
Customers today have sky-high expectations.
Digital transformation is ongoing and organizations need to repeatedly optimize
their customer digital experience using experimentation.
Established brands are under threat from digital dominators:
Spotify, Netflix, Amazon, Booking.com, Revolut.

5.
5
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

6.
6
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

7.
7
Scaling - Number, depth, quality

8.
8
Step 1
“Our success at Amazon is a function
of how many experiments we do per
year, per month, per week, per day.”
- Jeff Bezos
Scaling - Number, depth, quality

9.
9
Where to start?
Where do you want to be? Where do you need to be?
What more can you do? Where can you go?
Scaling - Number, depth, quality

10.
10
Upping the quality
Look deeper into where the best ideas for experiments come from.
Stay close to your customers and leverage your hard-won
relationships.
A small change can have a massive impact.
Scaling - Number, depth, quality

11.
11
Upping the quantity and breadth
More experiments means more insights and a greater impact on
the business. More is more.
However, closing the experimentation gap is not easy.
Scaling - Number, depth, quality

12.
12
Operationalize your experimentation.
Build a team with various skills sets including data
analysis and developers.
Establish a rock solid experimentation pipeline.
Upping the quantity and breadth
Scaling - Number, depth, quality

13.
13
Develop your hypotheses as fully as possible. Complex
hypotheses means complex, rich data.
Going deeper
Go beyond A/B testing. Save time and create more data.
Scaling - Number, depth, quality

14.
14
When you’re scaling and speeding up, don’t forget where
you’re heading.
Speed, scale and direction
Moving in the wrong direction is just as bad as not moving at all.
Manage your experimentation team, to extract value from experiments
and direct it.
Scaling - Number, depth, quality

15.
15
Maximise the output from experiments by connecting data from any
and every source.
Get all the data you can
Measure the impact of experiments across almost any metric.
“Understand the entire customer journey
by measuring the impact of our
experiments across all digital touchpoints ”
- Just Eat
Scaling - Number, depth, quality

16.
16
Encouragement from management raises the perceived value
of experimentation.
Digital experimentation as business experimentation
Connect every experiment to business strategy, strategic
goals and the bottom line.
Prove experimentation is a cash-positive part of the business.
Scaling - Number, depth, quality

17.
17
Keep your business competitive with digital experimentation.
Survival of the fittest
Competitors are still only one click away and always improving too.
If you’re not moving forwards, you’re moving backwards. Don’t
get squeezed out of the market.
Scaling - Number, depth, quality

18.
18
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

19.
19
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

20.
20
Shedding legacy technology

21.
21
Great digital experimentation is not just down to great people.
Great technology is needed too.
Don’t get held back
Legacy technology suites can be an obstacle.
Don’t let anything keep you from going from good to great.
Shedding legacy technology

22.
22
The keys to greatness
55% of experimenters going from good to great want to
automate more of the experimentation process.
This is key to scaling and speeding up.
You need an all-in-one technology stack. But, also one with
sophisticated solutions to complex needs.
Shedding legacy technology

23.
23
The good to great wishlist
Export in-depth statistics for rich data.
Machine learning tools for real-time analysis.
Automation for speed.
Shedding legacy technology

24.
24
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
Shedding legacy technology

25.
25
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
TEALIUM + CONTENTFUL
Shedding legacy technology

26.
26
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
TEALIUM + CONTENTFUL
AMPLITUDE + FULLSTORY
Shedding legacy technology

27.
27
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
TEALIUM + CONTENTFUL
AMPLITUDE + FULLSTORY
ATLASSIAN
Shedding legacy technology

28.
28
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
TEALIUM + CONTENTFUL
AMPLITUDE + FULLSTORY
ATLASSIAN
NEW RELIC
Shedding legacy technology

29.
29
Technology stack dream team
Marketing
Product
Collaboration
Monitoring
Cloud infrastructure
TEALIUM + CONTENTFUL
AMPLITUDE + FULLSTORY
ATLASSIAN
NEW RELIC
AWS
Shedding legacy technology

30.
30
Technology which allows you to remove the guesswork can
amplify your human talent and expertise.
Shedding legacy technology
The right technology stack can level-up the quality, quantity of
experiments and deliver rich data at lightning speeds.
Faster action keeps you competitive and gives the best return
on investment.
Shedding legacy technology

31.
31
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

32.
32
Your new ‘business as usual’: The three steps to
get started
Scaling - Number, depth, quality.
Challenging culture.
Shedding legacy technology.

33.
33
Challenging culture

34.
34
You’re not getting started anymore, it’s time to step up from
being a cultural upstart.
Now is the time
Snowball, pick up momentum and win over the majority of
your business.
Keep transitioning from the traditional top-down to the more
innovative bottom-up culture.
Challenging culture

35.
35
Management can fan the flames of digital experimentation.
Management styles for experimentation
More opportunities means more valuable data and insights
to improve the business.
Two management styles:
• Peer review
• Top metrics
Challenging culture

36.
36
It won’t discourage experimentation.
Management style 1: Peer review
More hoops means more thought. More thought means
greater complexity and rationale.
Experiments become more valuable. Netflix and AirBnb’s
adoption shows it ca be successful.
Challenging culture

37.
37
Monitor business performance from revenue and user acquisition to
catch checkout rates and more.
Management style 2: Top metrics
Communicate which metrics are most important.
Focus experimentation where it is needed most.
Challenging culture

38.
38
Siloed business functions miss out on:
• The news of success.
• Experimentation terminology.
• Valuable insights being discovered which they could implement.
Breaking down silo walls
Communicate effectively. Try distributing a monthly experimentation
newsletter.
Challenging culture

39.
39
Good to great summary
Scale the number, depth and quality of
digital experimentation.
Challenge your organization's culture.
Invest in shedding legacy technology.

40.
40
Download the report.
Stay for Q&A.

41.
41
In the next webinar:
Revitalising hypotheses and the digital experimentation engine.
How to embed digital experimentation across
an entire business.
• Scope
• Function

42.
42
You’ve got questions,
we’ve got answers

43.
43
Thank you!