Integrating Testing & Optimization into your Digital Marketing Efforts

TESTING, TESTING:
IS THIS THING ON?
Integrating Testing & Optimization into your Digital Marketing Efforts

Kate McRoberts, Evantage Consulting

© 2012 EVANTAGE CONSULTING
© 2011 EVANTAGE CONSULTING 1

I AM A HERO!

http://s183.photobucket.com/profile/gruaig_rua/index © 2012 EVANTAGE CONSULTING

@
@kmcroberts
www.evantageconsulting.com

Let’s start with a case study….
y

How one organization made $60 Million
by running a simple experiment.
y g p p


The Question.

How can we convert site visitors to give us their
email address?

The Key Metric.

Email signup rate.
g p


Experiment tested
two parts of the
splash page:
splash page:


Elements Tested

Media area
Media area
• Images and Videos

Buttons

6 media x 4 buttons = 24 combinations


The Winner

Family image with
y g
“Learn More” button.
• Sign up rate: 11.6% (vs.
g )
8.26% original)
• 40.6% sign up rate
improvement


Results
Res lts

• “Roughly 10 million people signed up on the splash page during the
campaign. If we hadn't run this experiment and just stuck with the
original page that number would be closer to 7,120,000 signups. That's a
difference of 2,880,000 email addresses”

• “Each email address that was submitted through our splash page ended
up donating an average of $21 during the length of the campaign. The
additional 2,880,000 email addresses on our email list translated into an
additional $60 million in donations.”

http://blog.optimizely.com/how-obama-raised-60-million-by-running-an-exp
http://blog optimizely com/how-obama-raised-60-million-by-running-an-exp


The Lessons.

• Simple tests can make a BIG impact
• What YOU think is going to win, may not
ACTUALLY win


How do I get started optimization?
H d I t t t d ti i ti ?


Who took Psych 101?

MASLOW


morality, creativity,
spontaneity, problem
spontaneity problem
solving, lack of prejudice
self ‐
actualization

esteem
(confidence, respect)

love/belonging
(friendship, family)

Safety
(security of body, resources, employment)

Physiological
(air, water, food)

HEIRARCHY OF NEEDS © 2011 EVANTAGE CONSULTING 15

Testing &
Optimization!
self ‐
actualization

esteem
(confidence, respect)

love/belonging
(friendship, family)

Safety
(security of body, resources, employment)

Physiological
(air, water, food)

HEIRARCHY OF NEEDS © 2011 EVANTAGE CONSULTING 16

Hierarchy of Needs for the Modern Marketer
testing & optimization style


But first, a few words about data…


Consider this staggering statistic…
gg g

“ We now create as much information
W t hi f ti
every two days as we did from the dawn
of civilization until 2003.

- Eric Schmidt, CEO, Google

http://techcrunch.com/2010/08/04/schmidt-data/ © 2012 EVANTAGE CONSULTING

http://www.mamapop.com/2011/07/facebook-skype-chat-feature-debuts-to-thunderous-shrugs.html/sad-trombone

Marketing organizations everywhere are struggling
with what to do with all this data.
ith h t t d ith ll thi d t

1,700 CMO’s consistently
ranked “Data Explosion” as
the #1 “pervasive, universal,
game changer” challenge to
be addressed.
be addressed

IBM Global CMO Study 2011

http://www-935.ibm.com/services/us/cmo/cmostudy2011/cmo-registration.html

“ Data was once the domain of tech geeks and direct‐marketing
gurus, while chief marketing officers focused on loftier things
like shaping brand perception. But those days are over.

Thanks to an explosion of data …CMOs who want a seat at
the table will have to harness customer data and leverage it ‐‐
or risk being relegated to chief promotions officer.

‐ AdAge 2/13/2012
“When CMO’s Love to Learn Data, They
will be VIP’s in the C‐Suite”

http://adage.com/article/cmo-strategy/cmos-learn-love-data-c- © 2012 EVANTAGE CONSULTING
suite-vips/232699/

i h f d f h d k


at

I have a website. I send emails.



I have web analytics. I see standard reports.




We have defined KPIs. We track them
regularly.  We can spot issues.





Analysis.  Insights.
We take action on the data.






Optimization.

Analysis.  Insights.
We take action on the data.






A/B and Multivariate Testing
/ g


/ g

How it works.
How it works.

1

Add a line
of code to
your site.


/ g

How it works.
How it works.

1 2

Add a line Set up
of code to rules you
your site. want your
test to
follow.


/ g

How it works.
How it works.

1 2 3

Add a line Set up Set up
of code to rules you challengers.
test to
follow.


/ g

How it works.
How it works.

1 2 3 4

Add a line Set up Set up Launch test!
of code to rules you challengers.
test to
follow.


/ g

How it works.
How it works.

1 2 3 4 5

Add a line Set up Set up Launch test! Run until
of code to rules you challengers. statistical
your site. want your significance
test to achieved.
follow.


Bonus question!
q

Who knows the formula for statistical significance?
Who knows the formula for statistical significance?

*
* May or may not be the actual formula. But
it sure looks scary, right? Good thing the tool
scary
calculates this for us.


/ g

How it works.
How it works.

1 2 3 4 5 6

Add a line Set up Set up Launch test! Run until Full scale
of code to rules you challengers. statistical winners
your site. want your significance
test to achieved.
follow.


A/B vs. MVT ‐‐ When to use which kind.
/

Use A/B testing when you want to…
Use A/B testing when you want to…
• Get started with testing
• Get results quickly ‐ smaller sample size to achieve statistical significance
• Test dramatic changes/radical redesigns – e.g. does design A work better than design B?

Use Multivariate testing when you want to…
• Increase your level of sophistication
Increase your level of sophistication
• Test multiple elements on the same page
• To understand how different page elements interact with each other
• To isolate impact of different elements
p
• Drive incremental improvements, “fine tune“


Examples.
p


Examples.
p

What if…
Instead of this style of navigation
y g


A/B
/

Let’s try a fly out menu


A/B
/

What if….
Instead of manually moving your
cursor over to find new topics


A/B
/

Let’s try automatically rotating
the banners
th b


A/B
/

Instead of a hero image
I t d f h i
that is this size….


A/B
/

We tried a smaller one that introduces
We tried a smaller one that introduces
more content above the page fold


Or what if we tested them all at the same time?

Fly‐out menu
Using a Multivariate Testing
tool allows us to see how
Smaller hero image each of these elements
contributes to the most
Rotating banner
Rotating banner effective combination.
effective combination


HealthSavingsCheckup.com
g p


How does it work?
Step 1 – Enter basic information

Step 2 – Health history
St 2 H lth hi t
Step 3 – Financial information
Step 4 View results
Step 4 – View results

Step 5 – Take action


What’s driving abandonment?
How can we deepen engagement and action?
H d t d ti ?

Analytics review and usability
testing to diagnose issues and
understand reasons why. y

Findings showed opportunity
for A/B testing:
• Call to action on myUHC.com
• Field labels
• Field‐level validation
• Results page redesign
Results page redesign


Tools – Entry Level
y

These tools can be used for free!


Tools – Mid Tier


Tools – Enterprise
p

http://www.whichmvt.com/

Want to get to the top of the pyramid?
W tt t t th t f th id?

We have seen people get stuck at the various
levels. Why?
y

CHANGE


Change.
g

This is a mindset shift. And a commitment to a new
This is a mindset shift. And a commitment to a new
process. Individual advocates alone won’t succeed.

Change tips:
• Create compelling vision for the future
• Have clear and visible sponsorship to champion it
• Create small wins and communicate the heck out of them
• Find your change advocates and embrace them
Find your change advocates and embrace them
• Demonstrate tangible benefits to the business


Thank you!

Want a copy of the deck?

@
@kmcroberts
kmcroberts@evantageconsulting.com

Integrating Testing & Optimization into your Digital Marketing Efforts

Recomendados

Recomendados

Más contenido relacionado

Similar a Integrating Testing & Optimization into your Digital Marketing Efforts

Similar a Integrating Testing & Optimization into your Digital Marketing Efforts (20)

Último

Último (20)

Integrating Testing & Optimization into your Digital Marketing Efforts