1.
FORGETCULTURE!
STARTBUILDINGAN
EXPERIMENTATION
ANEXPERIMENTATIONECOSYSTEM
www.effectiveexperiments.com
Manuel da Costa
Webinar 25th Jan 2023

2.
Manuel da Costa
Founder of Effective Experiments - a platform and service that
helps companies centralise their experimentation workflow and
insights with the aim of scaling and improving visibility and
engagement.
In the industry since 2012 and passionate advocate of
experimentation as a tool to drive innovation.
I help organizations build & scale their
experimentation programs with foundational change
management and process governance.
Developed the framework and strategies that we call
Experimentation Ops.
Follow me on Linkedin
www.effectiveexperiments.com
https://www.linkedin.com/in/manueldacosta/
Learn more about us

3.
WHO IS THIS WEBINAR FOR?

4.
WE NEED TO BUILD
A CULTURE OF EXPERIMENTATION

5.
WE NEED TO BUILD
A CULTURE OF EXPERIMENTATION
Blog posts
Linkedin Posts
Videos
Courses
Conferences

6.
WHAT EXACTLY IS -
A CULTURE OF EXPERIMENTATION?

7.
A LOT OF TBU INFORMATION
TRUE BUT USELESS

8.
EVERYONE SHOULD TEST!!
GET THEM EXCITED!!
SHARE INFORMATION!!
SHOW THEM HOW MANY WINS WE HAVE!!

9.
Sisyphus

10.
THE REALITY OF CRO PROGRAMS
Lack of awareness
Lack of buy-in
Lack of engagement
Low adoption

11.
THE PRESSURE ON THE CRO TEAM TO BECOME
CULTURAL CHANGE MAKERS
IS LEAVING A LOT OF CROs DISILLUSIONED.
THERE IS A NEVER ENDING LIST OF TASKS TO ACCOMPLISH

12.
SOME TRUTHS
CRO sits in a silo within the organization
with no authority or influence to make change beyond
the silo it operates in.
Cultural change happens because of top down
initiatives
Companies claiming to have Culture of Experimentation
are mostly lying and presenting a false reality.
Organizations treat CRO/Experimentation as an optional
nice-to-have/ an add-on and its value is not clear
beyond conversion increases and revenue.

13.
What's different about an
ECOSYSTEM?

14.
A complex, interconnected
system where one impacts off
the other

15.
How do you build an ecosystem?
You need to review, engineer the conditions,
monitor and iterate.
But you cannot do this alone.
This has to be a top down led approach...

16.
PRIORITY
PEOPLE
PROCESS

17.
PRIORITY
PEOPLE
PROCESS

18.
What is the current perception &
awareness of CRO?
Go find out!

19.
IT'S TIME TO CHANGE THE
NARRATIVE ABOUT
EXPERIMENTATION

20.
IF YOU'RE ONLY TALKING ABOUT WINS...WHAT
HAPPENS WHEN THE WINS DRY UP?
WE'RE FEEDING THE C-LEVEL WITH THE IDEA
THAT EXPERIMENTATION IS JUST ABOUT
MAKING MONEY AND NOTHING ELSE

21.
PEOPLE
PRIORITY
PROCESS

22.
EXPERIMENTATION TEAMS
MAINLY COMPRISE OF
TECHNICIANS & PRACTITIONERS
Analysts
UX designers
CRO specialists
Researchers
Development support

23.
DRIVING CHANGE REQUIRES
A different skillset
Patience to do "boring work" that doesn't yield
tangible results ASAP.
Focussed on people instead of technology

24.
THIS CAUSES THE LANGUAGE SPOKEN ABOUT
EXPERIMENTATION INSIGHTS TO BE JARGON
HEAVY AND TECHNICIAN FRIENDLY BUT NOT
STAKEHOLDER FRIENDLY

25.
THE INFORMATION GOING UP STREAM IS FULL OF
STATS AND FIGURES BUT NEVER ENABLES THE
STAKEHOLDERS TO UNDERSTAND THE MEANING
OF IT AND THE IMPACT TO THEM SPECIFICALLY.

26.
PRACTITIONERS - Responsible for the technical side, research, analysis etc. This is where most
experimentation programs are at.
ORCHESTRATORS - Focussed on managing the teams with the view of improving quality,
onboarding new members, monitoring and coaching.
AMBASSADORS - People oriented and extroverted. Able to build relationships with the C-suite
and transfer the insights and knowledge.
EXECUTIVE SPONSOR - Supports the orchestrator and ambassador in growing
experimentation. Is able to set the remit and requirements for the various teams involved.

27.
PROCESS
PRIORITY
PEOPLE

28.
IT'S ALMOST TOO EASY TO LAUNCH A TEST.
WE HAVEN'T PAUSED.
WE HAVEN'T CREATED
GOVERNANCE & GUARDRAILS TO ENSURE QUALITY

29.
THERE IS A HIDDEN EPIDEMIC THATS GROWING IN
EXPERIMENTATION.
AS EXPERIMENTATION GROWS TO TEAMS THAT
HAVE NEVER TESTED BEFORE, THE QUALITY OF
THEIR WORK AND PROCESSES BECOMES
QUESTIONABLE.

30.
THE NEED FOR SPEED IS PRIORITISED OVER A SLOW,
STRATEGIC APPROACH.
BASIC WORKSHOPS AND ACCESS TO TOOLS WILL
NOT INCREASE THE EXPERIMENTATION
CAPABILITIES IN THE ORGANIZATION.

31.
PROBLEMS OCCUR SUCH AS
PROCESS ADHERANCE ISSUES
ONLY REPORTING WINNING TESTS (And hiding others)
RANDOM IDEAS BEING TESTED WITHOUT PRIORITIZATION
HARK-ING (changing the hypothesis)
FAULTY ANALYSIS & REPORTING
NO INVOLVEMENT FROM STAKEHOLDERS
WITHOUT PROPER OVERSIGHT
IT'S IMPOSSIBLE TO SPOT THE PROBLEM

32.
WITH THE CURRENT EXPERIMENTATION SETUP -
NO ONE HAS THE TIME OR BANDWIDTH TO PROPERLY MANAGE
THEY LACK THE AUTHORITY TO ENFORCE ANY CHANGE.
THEY LACK THE EXPERTISE IN CHANGE MANAGEMENT
LACK OF INVOLVEMENT OF STAKEHOLDERS HINDERS CHANGE

33.
GROWING THE
EXPERIMENTATION CAPABILITY
IS NOT LIKE LAUNCHING A
ROCKET.
IT'S MORE LIKE TEACHING
PEOPLE HOW TO RIDE A BIKE.
FIRST WITH STABILIZERS UNTIL
THEY'RE CONFIDENT.

34.
TOOLS?

35.
PUTTING IT TOGETHER
ONE PIECE AT A TIME

36.
THIS ONLY WORKS IF THERE IS
TOP DOWN INITIATIVE
FACILITATED BY SUPPORT FROM THE TEAMS

37.
FIRST
FOCUS ON THE SENIOR MANAGEMENT
THEIR PERCEPTION OF EXPERIMENTATION
& ITS VALUE.
THIS WILL ESTABLISH THE NEED TO BUILD AND
ORCHESTRATE A SUSTAINABLE
EXPERIMENTATION PROGRAM.
THIS WILL GET YOU THE KEY EXECUTIVE SPONSORS
& THE ABILITY TO SET THE REMIT & GUARDRAILS

38.
SECOND
LOOK AT THE EXPERIMENTATION SETUP &
HIRE/UPSKILL ORCHESTRATORS TO FOCUS ON
QUALITY OF EXPERIMENTATION.
REFINE THE PROCESSES & CREATE GUARDRAILS
WITHIN IT TO ENSURE EVERYONE FOLLOWS IT.
STANDARDISATION IS KEY.
THIS WILL IMPROVE THE TRUSTWORTHINESS OF
DATA & INSIGHTS IN THE PROGRAM AND THE NEW
TEAMS WILL RUN HIGH QUALITY TESTS.

39.
STAKEHOLDER AWARENESS & REMIT
EXECUTIVE
SPONSORS CENTRE OF EXCELLENCE /
ENABLEMENT
(ORCHESTRATORS &
AMBASSADORS)
EXECUTIVE
SPONSORS
Team A
Team B
Team D
Team C

40.
THIS IS SMALL SAMPLE OF OUR
EXPERIMENTATION OPs FRAMEWORK &
STRATEGIES TO IMPROVE YOUR
EXPERIMENTATION PROGRAM

41.
Thank
Thank
Thank
you!
you!
you!