What does it take to run a best-in-class experimentation program? What sets these programs apart isn’t some kind of magic formula; instead, they invested in making experimentation a core practice across the business with a dedicated team in place, best-in-class technology and a practice of evangelizing their impact on the business.
Learn from Optimizely’s own Experimentation Program Manager, Becca Bruggman, and the Product Manager of Optimizely Program Management, Kris McKee, on exactly how they took our own internal Product experimentation program from ad-hoc to a robust Center of Excellence. They will take you through the lifecycle of an idea: from concept, to experiment, results, and then ultimately iterating to the next experiment — all in just 30 minutes!
(Best) ENJOY Call Girls in Faridabad Ex | 8377087607
Program Management 101: Best Practices from Optimizely-on-Optimizely
1. Congratulations to our
Outperform Award Winners!
Most Dramatic
Business Impact
Most Customer-
Obsessed Company
Culture
Most Transformative
Innovation
Most Inspiring Social
Impact
2. 2
Welcome to Day 2 of Opticon!
Sessions today will be recorded and will be available after
Opticon
Share your feedback and rate sessions on the mobile app
3. Kris McKee
Senior Product Manager
Rebecca Bruggman
Staff Technical Program Manager
Program Management 101:
Best Practices from
Optimizely-on-Optimizely
4. 4
Experimentation is the
next great business
transformation
The Surprising
Power of Online
Experiments
“Our success is a function of
how many experiments we
do per year, per month, per
week, per day.”
Jeff Bezos
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“One of the things I’m most
proud of, and I think what is
the key to our success, is
the testing framework we’ve
built.”
Mark Zuckerberg
“Instead of saying ‘I have an
idea,’ what if you said ‘I have
a new hypothesis, let’s go
test it.’ ”
Satya Nadella
5. 5
Experimentation velocity is an
indicator of unlocked potential
What HBR says:
“Growth takes off once the experimentation platform
allows a user to take part in multiple experiments at
the same time, supporting virtually unlimited
concurrent tests.”
What Forrester says:
“Insights-driven businesses are growing by more than
30% annually and are on track to earn $1.8 trillion by
2021.”
What MIT Sloan says:
“Digital experimentation should not be a
‘once and done’ effort. Rather, it is about developing
a culture of experimentation in which new business
ideas are continually tried and tested. Build and iterate
on the successes, and learn from the failures.”
7. 7
… but, challenges are manageable with right strategy & tools
Coordination
As you scale, coordinating across
additional teams and stakeholders
becomes more and more difficult
Resources
Time, people, and effort is wasted
duplicating and reinventing different
processes
Prioritization
Ideation can be siloed leading to
inefficient prioritization and execution
Oversight & knowledge sharing
Lack of executive oversight can lead to serious
problems in a global organization
8. 8
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Combine a foundation of mindset and goal setting for
experimentation program success
9. 9
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
10. 10
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
11. 11
● Implement ‘test and learn’ decision
making
● Base all decisions on data
● Understand the “why” behind your
data by asking the right questions
● Avoid complacency through
continuous iteration
Build a foundation of
culture and mindset
12. 12
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
13. 13
Velocity & Win Rate
Testing velocity
lets you measure
your team’s
operational
performance
Measure Program
Win rate helps
teams measure
the quality of their
hypothesis and
impact to the
business
14. 14
Using Program
Reporting Right
● Set monthly, quarterly, and annual
goals for performance
● Measure and report on a monthly
basis
● Re-assess on an annual basis
● Integrate the learnings into daily
operations of your work
15. 15
Some things you may learn:
● Some teams are more effective than
others in moving an idea to an
experiment
● Win rates may be significantly lower on
certain pages of your site
● Testing velocity may slow around
holidays
● Certain experiment strategies work
more effectively
Insights on your
program
16. 16
As you begin to understand your
program:
● Track how win rate and velocity
impact your overall program value
● Think about ROI and where you
can maximize return
● Take a portfolio approach to your
experimentation program
Assessing
Opportunity
17. 17
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
18. 18
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
19. 19
● Train teams on how to write a
hypothesis
● The best hypothesis are rooted in
data and deep insights into your
business
● By systematic: create a process
for submission, scoring, and
acceptance
● Give constructive feedback
Democratize (good)
ideation
20. 20
Capture Ideas
Give structure to testing
backlog by categorizing
ideas by site / page
Collaborate widely
Always start with a
hypothesis - give teams
examples of what a good
one looks like
Naming conventions can
help keep things organized
and easy to report
21. 21
Capture Ideas with Slack
Integrate your Slack
instance to make idea
capturing easy
Collaborate Widely
Allow your collaborators to
work where they spend the
most time - Slack
22. 22
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
23. 23
● Set expectations by answering
why, when, where, who and how
of each idea
● Define primary, secondary, and
monitoring goals
● Calculate your min detectable
effect (MDE) to determine if results
will be valuable enough to justify
cost
Define your
experiment
24. 24
Commenting
Give structure to testing
backlog by categorizing
ideas by site / page
Prioritize backlog
Always start with a
hypothesis - give teams
examples of what a good
one looks like
Naming conventions can
help keep things organized
and easy to report
25. 25
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
26. 26
–
Selecting & Accepting Ideas
Backlog status lets
your team know it’s
still under review by
the central group
Prioritize backlog
Active status lets
everyone know
you’ve selected this
for experimentation
and are beginning
work
27. 27
Scoring
Potential
of the experiment
to positively impact
your goal
Prioritize backlog
Impact
to the business if
the experiment
wins
Level of Effort
to implement the
experiment in
Optimizely
LOVE
strategic
importance,
executive support,
biased opinions
28. 28
● Create a prioritization framework,
starting with product and business
goals
● Evaluate testing ideas based on
potential, impact, effort, and LOVE
(personal preference)
● Consider doing an objective
comparison based on a key metric
● Prioritization is critical to the quality
and velocity of experimentation
Prioritize with
scale in mind
29. 29
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Framework for experimentation program success
31. 3131
Data-based
Idea Generation
A strong understanding of
business goals and key
metrics should guide idea
generation and experiment
prioritization
Democratize
Ideation
Cast a wide net and
encourage ideation
throughout the firm to drive
innovative ideas and a
culture of experimentation
Celebrate Widely
Share wins across entire
firm and celebrate
collaboration and encourage
continued experimentation
mindset
Ongoing collaboration and communication is core to
building a strong experimentation program
Set Goals and
Publish results
Set win rate and
experimentation velocity
goals and publish results
widely
33. 33
ADEPT Experimentation Program
Center of Excellence
Center of Excellence
Enablement
Best Practices
Sharing Learning
Program Measurement
Ideation
Prioritization
Design
Execution
Results Analysis
Team 1 Team 2 Team 3 Team 4
37. 37
Promote an
experimentation
mindset
Set program goals
and measure
performance
Collaborate to
generate great
ideas
Refine
experimentation
ideas
Build an
experimentation
roadmap
Share and
evangelize
learnings
Focus on Ideation Loop