Mailchimp, an all-in-one marketing platform for small businesses, has built innovative, beautiful products that empower those businesses to scale. No longer just an email marketing tool--the shiny, new marketing platform’s mission is to empower the underdog to connect with their audiences and create the customer loyalty that’s needed to grow. In this webinar, Anne Reints, Senior Manager, Growth Analytics & Experimentation at Mailchimp, will discuss how Mailchimp is continuing to build a culture of experimentation and has leveraged testing across both their product and marketing teams. Join this webinar and learn: The history of growth and experimentation at Mailchimp How Mailchimp grew their experimentation program by 6,000%+ in 12 months, going from just 1 experiment to over 60 About the launch of their internal Experimentation Resource Group How they created a data-driven decision-making framework across the company

1. 1
How Experimentation has
Moved from Marketing to
Product
Anne Reints & Alex Lee
+

2. 2
Alex Lee
Customer Success Manager
Anne Reints
Senior Manager, Growth Analytics &
Experimentation
Introductions

3. 3
Housekeeping
● We are recording this webinar
● You’ll receive the recording and slides
● We’ll answer all questions at the end

4. optimizely.com/digital-change-agents/

5. 5
Agenda
● Introduction
● About Optimizely
● Mailchimp’s Journey
● Q&A

6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for
enterprise-wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield

7. “Our success is a function of
how many experiments we do
per year, per month, per week,
per day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale

8. What
We Enable
1500+
+51%
+$21M
+19%
+28%
Improved subscription
acquisition
Users Scale and agility
of experiments; users
incremental
in revenue
Improved Driver
Acquisition
increase in cross-sell
performance

9. Spent with Digital Media
19hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage

10. Experimentation: Testing variations of digital experiences
with real users and deploying what wins
+21%
-5%

11. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal

12. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience

13. 2019
Growing Growth
at Mailchimp
Digital Change Agents Webinar Series
Anne Reints
Senior Manager, Growth Analytics and Experimentation

14. 2019
The History
of Growth at
Mailchimp

15. 2019
The History of Growth at Mailchimp
20182017 2019
July
Marketing
Optimization
is born
Nov
It’s not just
a team of one
anymore
July
Experimentation
Resource Group
is Born
Feb
Product
Analytics
begins
Jan
Marketing
Optimization
CFX is born
April
The Growth
Product team is
Born
June
We launch our
1st experiment!
Nov
We move to
Domains/Squads
x 30

16. 2019
Getting your
team on board

17. 2019
Start with
a story

18. 2019
Invite everyone
to help solve
the problem

19. 2019
Find blockers and
optimize together

20. 2019
What this looks
like in action

21. 2019
Create a shared understanding of the
problem or opportunity
Problem Statement: Once a user uploads
2,000 or more contacts, they are no longer eligible
for the free Mailchimp plan. How can we help make
this transition easier?

22. Users begin on the
Dashboard
90%
Navigate to Lists
(Audiences) page
80%
Import Match
25%
Import Review
20%
Many steps later, users
finally land on the Sent
Campaign screen
19%
Import Setup
50%

23. 2019
Sneak peek into our
“Innovation Lab”

24. 2019
Sneak peek into our
“Innovation Lab”

25. 2019
Our Experiment

26. 2019
Control
Variant 1
Variant 2
Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more
positive language to make this a celebratory moment, we will increase paid user conversion rate.
Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not
have billing information stored in Mailchimp

27. 2019
Control
Variant 1
Variant 2
Results: Variant 2, which contained copy and button updates, was this winner with a 20.4% lift in new paid
conversions compared to the control variant.
Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more
positive language to make this a celebratory moment, we will increase paid user conversion rate.
Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not
have billing information stored in Mailchimp

28. 2019
Analyze results
as a team

29. 2019
Create a framework
for data driven decisions
across the company

30. 2019
Defining KPIs
and your North
Star Metric

31. Proprietary & Confidential 2019
Strategic Business
KPIs
Business Health
KPIs
Total Revenue, Profit Margin,
Operating Cost
Business
Level KPIS
Domain
Level KPIs
Clarifies how we prioritize
and drive business value
Articulates the customer
problems we are trying to
solve w/ our North Star
Key data points that inform
the customer problem
North Star
Problem Statement
KPIs
Input metrics
KPIs that measure
your company’s growth

32. Proprietary & Confidential 2019
Strategic Business
KPIs
Business Health
KPIs
Total Revenue, Profit Margin,
Operating Cost
Business
Level KPIS
Domain
Level KPIs
Clarifies how we prioritize
and drive business value
Articulates the customer
problems we are trying to
solve w/ our North Star
Key data points that inform
the customer problem
North Star
Problem Statement
KPIs
Input metrics
KPIs that measure
your company’s growth

33. Proprietary & Confidential 2018 33
An Example: Email
Sign Up Onboarding EngagementCreate a HabitAha Moment
Uploads a List Sends first
campaign
Sends 3
campaigns in
first 90 days
Sends monthly
campaigns +
uses other
features
Sign up

34. Proprietary & Confidential 2018 34
An Example: Email
Uploads a List Sends first
campaign
Sends 3
campaigns in
first 90 days
Sends monthly
campaigns +
uses other
features
Validate
Sign up
What % of sign ups
upload a list?
Are users who upload
a list more likely to
reach the Aha
Moment?
What % of users
complete this action?
How does this map to
our business KPIs?
Are users who
complete this action
more likely to be
retained?
How does this map to
our business KPIs?
What other factors
deepen engagement
and help retain users?
Total # of sign ups
(Top of funnel)
Sign Up Onboarding EngagementCreate a HabitAha Moment

35. Strategy
Lift
Required
Impact
on Paid
Users
Drive
Engagement
500% 30%
Improve
Activation
25%
Less
Effort
30%
More
Impact
Look for smaller efforts and bigger impact

36. 2019
The customer problem you
are working to solve
North Star
Metric
How your company
makes money
Business level KPIs

37. 2019
It measures the moment a customer
finds value in your product
It represents the core of your
domain’s strategy
It can be measured immediately
after launching a feature or creating
a new experience
It correlates with business KPIs that
measure Mailchimp’s success
Qualities of a
North Star
Metric

38. 2019
It measures the moment a customer
finds value in your product
It represents the core of your
domain’s strategy
It can be measured immediately
after launching a feature or creating
a new experience
It correlates with business KPIs that
measure Mailchimp’s success
Qualities of a
North Star
Metric

39. 2019
It measures the moment a customer
finds value in your product
It represents the core of your
domain’s strategy
It can be measured immediately
after launching a feature or creating
a new experience
It correlates with business KPIs that
measure Mailchimp’s success
Qualities of a
North Star
Metric

40. 2019
It measures the moment a customer
finds value in your product
It represents the core of your
domain’s strategy
It can be measured immediately
after launching a feature or creating
a new experience
It correlates with business KPIs that
measure your company’s success
Qualities of a
North Star
Metric

41. 2019
Use data and insights to
evaluate where you have
the most opportunity

42. 2019
Dig into your
input metrics with
both qualitative and
quantitative research

43. 2019
Make a statement
about how you
think you’ll impact
your metrics, and be
able to measure that
once you release

44. 2019
When you see the
results, be humble—
we’re often wrong
about the solution the
first time.

45. 2019
An Example: Themed Templates
Hypothesis: By surfacing themed templates in the default template selection window, we can give users a head start
and content, helping them to activate more quickly.

46. 2019
An Example: Themed Templates
Hypothesis: By surfacing themed templates in the default template selection window, we can give users a head start
and content, helping them to activate more quickly.

47. 2019
Once we know we’re
wrong, dig in with
more research

48. 2019
Understand why
and let these new
insights drive your
future roadmap

49. 2019
Creating a Culture of
Experimentation and
Inviting Others to
Join You

50. 2019
Engage others in the company and
make them a part of the process.
Share your learnings. Tell a story with
your experiments and help your
company learn.
Share your impact. As your experiments
lead to impact on the business, show
what impact you’ve had and how
experimentation helped you get there.
STEP 1:
Tell everyone
what you’re
doing

51. 2019
Control
Variant 1
Variant 2
Results: Variant 2, which contained copy and button updates, was this winner with a 20.4% lift in new paid
conversions compared to the control variant.
Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more
positive language to make this a celebratory moment, we will increase paid user conversion rate.
Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not
have billing information stored in Mailchimp

52. 2019
We created Experimentation, Ideation,
Analytics, and Developer trainings and
shared them with other product teams.
Provided consultations to teams
getting started.
These were great opportunities for
everyone on our cross functional teams
to get involved and grow the number of
subject matter experts in the
organization.
STEP 2:
Teach them
how to do
it too

53. 2019
Invite all teams to share the experiments
they’re launching and the results (we do
this using our company messaging app).
When a team is getting started - celebrate
with them. Even when they lose, help
them celebrate the learning.
At some point, with team’s that are new to
experimentation (and not so new) things
will go terribly wrong - make sure you
celebrate every attempt as a success in
terms of learning, even when it’s painful.
STEP 3:
Celebrate
their success

54. 2019
Changing your culture will never happen
as quickly as you want it to, and not
everything you do will work.
Be open to changing your process, how
you’re communicating, and empower
teams to take ownership of what they’re
learning and change the process with you.
Free food is always a great motivator for
getting others involved.
STEP 4:
Keep iterating
and applying
your learnings

55. 2019
Q&A

56. 2019
Thank you.