How do you know where to start with experimentation? What if you don’t have enough information, or simply too much to decide where to begin and where to invest your time/effort/money?
In this breakout session we will cover how to cut the BS by treating experimentation as an “internal services startup”, where the customers are the teams in your business: commercial, trading, marketing, product, SEO etc.
You wouldn’t start a startup by hiring a bunch of people without a tool or an idea to work on, or buy an office or expensive work management solution for a startup of 3 people without developing a product and taking it to market first. So why treat experimentation that way?
2. PROBLEM: You’re a business who want to start experimenting, but don’t know where to start:
- Where should you invest your time, money or resources first?
• In these cases, panic often sets in - this leads to a scatter gun approach to trying to get
experimentation going:
- Let’s buy a tool
- Let’s pay an agency
- Let’s hire a specialist
- Let’s add more tools or get rid of an existing tool
- Let’s get everyone in the business to come up with ideas
PANIC!!
3. SOLUTION: This talk! Grab your towel and listen to a guide
who has done the hitch-hiking so you don’t have to:
• Advice on approach
• A strategy to get started
- Mindset
- Enablers
• Finally, how to assess how things are going
DON’T PANIC!!
4. 1. Think of optimisation like a mini-business or services startup within your business:
whose customers are the product and tech teams, commercial, marketing, the leadership
team…… anyone who could be more successful if they can run experiments.
2. Develop your optimisation startup like a real business:
- Hire a great CEO: get a CRO owner with vision and expertise
- They develop & rollout a strategy: give them ownership of the programme and strategy
- Invest: get the tools, people and skills they need to make the strategy successful
- Run and assess: Let the programme run and see what is working
- Double-down or pivot: keep doing what makes your customers happy, and fix the things that
don’t work
Advice
5. • Build a Mindset for experimentation: getting everyone to think about experiments
constructively
- The How, Why and What of Experimentation
• Utilise a Model to enable experimentation:
- Across the experiment lifecycle try different approaches and ways to enable experiments
- Measure the results
• Quick aside: The structure of your team doesn’t matter…
What is my strategy for my experimentation startup?
6. Mindset
Experiments consist of 3 key variables or decisions:
‣ What you are testing: Changes you think will impact your KPI
‣ Why you are testing: The impact you want to make on your KPI/metric or goal
‣ How you will test that effectively and rigorously:
- Process/Planning
- Ideation
- Execution
- Analytics
- Value
- … many more
The How, Why and What of Experimentation
7. Mindset
In many teams, the decision about what to test often comes first
They are employing the following strategy:
The How, Why and What of Experimentation: As-is
8. Mindset
In many teams, the decision about what to test often comes first
They are employing the following strategy:
The How, Why and What of Experimentation: As-is
• What phase: The team are doing ‘feature-led’ testing,
where they decide what to test and then justify why they
are testing it.
• Why phase: the team look around for KPIs to support
their pre-selected idea: And so increase conversion…
• How: Finally, they work out how to test their feature, or
whether it is even possible/feasible to run the
experiment.
This happens well after choosing the experiment.
9. Mindset
How:
Do we find out customer behaviour and problems
from our metrics?
Do we ideate experiments?
Do we plan experiments?
How do we do experimental design?
Do we build experiments?
Do we test statistically and analytically?
Do we design the feature?
Do we know an experiment has won?
In actual fact, this hierarchy needs to be inverted:
- Enable the team by teaching the principles of How to Test, which leads naturally into the
Why and What to Test.
The How, Why and What of Experimentation: To-be
10. Mindset
• After the How, it becomes clear which KPIs can be
influenced and whether these are actually important:
The Why.
• This leads to a growing number of much better
ideas: The What.
• The correct training and enablement free the team
to be more autonomous and agile with their
experiments.
In actual fact, this hierarchy needs to be inverted:
- Enable the team by teaching the principles of How to Test, which leads naturally into the
Why and What to Test.
The How, Why and What of Experimentation: To-be
11. Enablement
This is designed to show how we can inject the How Why and What into our ways of working,
and help us diagnose where this is going wrong.
The enablement model
Having got the right mindset, we then need to do the second half of our strategy:
Utilise an enablement model across the experiment lifecycle
14. Team Goal
Impacted
Team is successful
and Healthy
EXPERIMENT
DESIGN
SKILLS/
TRAINING
ANALYSIS
TECHNOLOGYEXECUTION
IDEATION
@ellieexperiment
Business is
Successful
17. Enablement
• Lack of Impact on goal over time or over many experiments
• Poor team health
• Using experiment Retros
How do you diagnose a failure of How, Why, What?
18. Enablement
• An experiment retrospective looks back at
what happened during the experiment, and
what the team did and how well they did it.
• Make it quantitative!
What is an experiment Retro?
• Ask a lot of questions:
- How did the experiment go?
- Was it a winner?
- Did it impact their goal/KPI? If not did it
impact there metrics instead?
- Where did they feel it went wrong?
- What knowledge and skill gaps were
revealed?
- What processes need to change for next
time?
- ….
19. Enablement
• Using a simple spider scorecard, I work with
our teams to reflect where they need
additional help
• For all the areas where they score 1 out of
5, we focus to bump that up to a 2 or 3.
Diagnose a failure of How, Why, What
20. Enablement Diagnose a failure of How, Why, What
• Low Maturity: The team biases towards
their comfort zones: technical skills and
execution/delivery.
@ellieexperiment
21. Enablement Diagnose a failure of How, Why, What
• Low Maturity: The team biases towards
their comfort zones: technical skills and
execution/delivery.
• Medium Maturity: The team starts to focus
more on analytics learnings and quality of
ideation, leading to more impact on their
goal.
@ellieexperiment
22. Enablement Diagnose a failure of How, Why, What
• Low Maturity: The team biases towards
their comfort zones: technical skills and
execution/delivery.
• Medium Maturity: The team starts to focus
more on analytics learnings and quality of
ideation, leading to more impact on their
goal.
• High maturity: The team invests heavily in
training (whole team), analysis & product
insights, and experiments with
planning/process models and ideation
techniques
@ellieexperiment
23. Enablement Diagnose a failure of How, Why, What
So what? The retro shows:
- Specifically how the team are struggling
- Where the “How, Why, What” has failed
- How they feel about it
- The impact on their goal
Now what?
- Use the enablement model, and invest in
the lagging areas.
24. Hopefully this has made it easier not to panic!
In Summary:
- Treat optimisation like a start-up inside your business
- Get the experimentation mindset right
- Enable experimentation by enabling the teams running
experiments
- Measure and fix!
Need more ideas and tools you can use to improve
experimentation, or want to try them out?
- come talk to me!
- checkout my blog post on medium
Twitter: @EllieExperiment
Medium.com/@EllieExperiment
DON’T PANIC!!
27. Session Exit Slide
(Do Not Remove)
Up next…
▪ Announcement 1
▪ Announcement 2
▪ Announcement 3
Notas del editor
A common problem I find when I join a business who want to start experimenting, is that they do not know where to start:
Where should they invest their time, money or resources first?
This is an experience which can be incredibly costly: I once started at a B2B travel provider who spent >£200k/yr on their analytics suite, but only had 5000 active users.
In these cases, panic often sets in - this leads to a scatter gun approach to trying to get experimentation going:
Let’s buy a tool
Let’s pay an agency
Let’s hire a specialist
Let’s add more tools or get rid of an existing tool
Let’s get everyone in the business to come up with ideas
The typical comments in the business as a result are:
We Built lots of tests but don’t seem to be getting anywhere
We aren’t building any tests or not fast enough
We aren’t able to analyse our tests
We don’t have enough ideas or we have too many!
The literature provided by most AB testing vendors, agencies and consulting firms doubles the confusion:
An inspirational aspirational pitch deck about your future optimisation maturity and growth
A survey about your business or optimisation maturity
A plethora of optimisation maturity models (saying almost but not quite the same thing?)
This talk is designed to give you some help and advice to avoid the mistakes many businesses make.
So, grab your towel and listen up to a guide who has done the experimentation hitch-hiking so you don’t have to!
Don’t undermine your “CEO” by buying a tool or bringing in an agency/consultants before they join: this gives them less ownership of the strategy and they may have recommended a different/better approach.
The type of team you run is actually irrelevant, so long as you get the MINDSET and ENABLEMENT right.
Go for squads, feature teams, product teams, marketing/commercial teams with a specialist, 1 internal person running an outsourced agency team…
I am not going to sell you some magic beans and make you build a bean stalk too
So first up: How, Why and What
In many teams, the decision about What to test often comes first, especially as it is the variable over which the team has the most control.
In many teams, the decision about What to test often comes first, especially as it is the variable over which the team has the most control.
By Understanding the methods and constraints of experimentation, and so success and failure, this gives product teams the right mindset.
Laying that ground work gives teams the ability to “play” in the space, getting an intuitive idea of the KPIs to test (Why) and therefore What to Test (impacting their KPI).
After the How, it becomes clear which KPIs can be influenced and whether these are actually important: do they indicate a customer behaviour we want to change or problem to fix.
This leads to a growing number of much better ideas: The What.
The correct training and enablement free the team to be more autonomous and agile with their experiments.
By Understanding the methods and constraints of experimentation, and so success and failure, this gives product teams the right mindset.
Laying that ground work gives teams the ability to “play” in the space, getting an intuitive idea of the KPIs to test (Why) and therefore What to Test (impacting their KPI).
After the How, it becomes clear which KPIs can be influenced and whether these are actually important: do they indicate a customer behaviour we want to change or problem to fix.
This leads to a growing number of much better ideas: The What.
The correct training and enablement free the team to be more autonomous and agile with their experiments.
Business is Successful
Team Goal Impacted
Team Successful and Healthy
Ideation
Experimental design
Execution
Technology
Training
Analysis
Business is Successful
Team Goal Impacted
Team Successful and Healthy
Ideation
Experimental design
Execution
Technology
Training
Analysis
Business is Successful
Team Goal Impacted
Team Successful and Healthy
Ideation
Experimental design
Execution
Technology
Training
Analysis
Running a retro is common practice for a lot of teams
Used in agile/scrum to check team health and improve the way the team works
An experiment retrospective looks back at what happened during the experiment, and what the team did and how well they did it.
Make it quantitative!
This talk is designed to give you some help and advice to avoid the mistakes many businesses make.
So, grab your towel and listen up to a guide who has done the experimentation hitch-hiking so you don’t have to!