This document discusses multivariate testing (MVT) across multiple brands. It provides an overview of why companies should optimize websites through testing, including to improve customer experience, increase conversion and revenue, and deepen visitor engagement. The document then discusses where to start with optimization, identifying opportunities through analytics and testing to improve performance. Several case studies are presented of MVT performed across the Phones4U and Dialaphone brands, showing testing of things like search results, upgrades, and multi-device experiences that led to increased conversion lifts. Key lessons are provided, such as only testing what is willing to be implemented, occasional re-testing, and that simple tests can provide big results.

1. MVT Across Multiple Brands:
One approach to successful
testing
Martin Sheerin
Head of Online Product Development, Phones4U
Michael Schirrmacher
UK Sales Director, Webtrends
Webtrends |
© 2011 Webtrends, All Rights Reserved.

2. Why should we Optimize?
#1. Improve Customer Experience
#2. Improve Conversion & Revenue
#3. Deepen Visitor Engagement
© 2011 Webtrends, All Rights Reserved.
|2

3. © 2011 Webtrends, All Rights Reserved.
|3

4. Where do I start?
Analytics
Optimization
Identify and quantify
opportunities through
insight
Test, segment and target
to improve performance
© 2011 Webtrends, All Rights Reserved.
|4

5. What Optimisation Involves?
© 2011 Webtrends, All Rights Reserved.
|5

6. Full Factorial….Fractional
factorial….?
“ok this looks complex”
© 2011 Webtrends, All Rights Reserved.
|6

7. So, for now…a managed service is still more
popular
© 2011 Webtrends, All Rights Reserved.
|7

8. Optimization Maturity Model
Operational
Reporting
“Crawl ” Phase
“Walk” Phase
“Run” Phase
• Personalization
Conversions
• Web Analytics
• A/B/n Testing
• More Advanced MVT
• Measurement
• Basic MVT -1-3
areas
• In-session
Segmentation
• 1 Element Targeting
• Multi Element Targeting
• 1-Project at a Time
• Frequent
• No Testing
• Ad Network
Targeting
• Integrated
Optimization
• External Segment
Use
• Social & MultiChannel
• On-going
Optimization Maturity Phases
© 2011 Webtrends, All Rights Reserved.
|8

9. Multivariate Test
Control
£
© 2011 Webtrends, All Rights Reserved.
|9

10. Multivariate Test
Control
Challenger
£
© 2011 Webtrends, All Rights Reserved.
| 10

11. Multivariate Test
+£3.2m
Lift in special offers sales in UK
© 2011 Webtrends, All Rights Reserved.
| 11

12. Martin Sheerin
Head of Online Product
Development
Phones 4U
@martinsheerin1
linkedin.com/in/martinsheerin1

13. Phones 4u
•
•
•
•
•
Multi-channel retailer
Over 600 stores
Contact centre for sales and service
Website: www.phones4u.co.uk
Mobile site: m.phones4u.co.uk
•
Online only brand:
– www.dialaphone.co.uk
– m.dialaphone.co.uk

14. Why should you MVT?
New idea
of creative
Build on
gut instinct
with
science
Campaign
Support
Increase
Conversion
De-risk
Projects
Avoid
mistakes

15. Why not MVT yourself instead of using a
partner?
•
•
•
•
Time
Skills
Ensuring that a test is statistically significant
Ensures the result is true and not biased to an internal individuals
viewpoint
•
You can do this yourself:

16. What to look for in an MVT provider?
Business
understanding
Business
alignment
eCommerce
insight
Ease of
reporting

17. Our Approach
•
Test simultaneously on both sites
•
Use results of test on one site and apply to the other (if its possible)
•
Be happy to repeat tests annually (or more regularly if you deem
necessary)

18. Our Test Successes

19. Free Gifts
6.27%
Optimal
Control Page
lift

20. Customer Behaviour
Phones 4u
Research
Purchase
Choose
Handset
Dialaphone
Select
Package
Allowance
(Lifestyle)

21. Phones 4u Review Tab
Control
Winner
+2.99%

22. Search Results
+16.56%

23. Simple and Usable
Source: Simple and Usable Web, Mobile, and Interaction Design (Voices That
Matter) - Giles Colborne (CX Partners) (2010)

24. Search Results
Optimal
Control Page
+16.56%
Experiment Page
Lift
Control
Optimal
--+16.56%

25. Upgrades - Variants

26. +81.64%
Optimal
Control Page
Lift at next step
Experiment Page
Lift
Control
Optimal
--+81.64%

27. white background
Level 5 –
grey background
Level 4 –
white background
and upgrade promise
Level 3 –
control
grey background
and upgrade promise
Level 2-
Taking this finding we applied it to Phones
4u
Optimal

28. Phones 4u Winning Variant
+70.64 %
Optimal
Control Page
Lift at next step
Experiment Page
Control
Optimal
Lift
--+70.64%

29. Not satisfied we went with a hunch…
Optimal
Control Page
+0.89%
Experiment Page
Lift
Control
Optimal
--+0.89%

30. Moving on with the upgrades test

31. Control Page
Multi-device testing

32. Applying this idea to Phones 4u

33. All Traffic (non-iPad)
Optimal
Control Page
Optimal
Experiment Page
Control
Optimal
Lift
--+4.11%

34. iPad Version
Control Page
Optimal
Optimal
Experiment Page
Control
Optimal
Lift
--+4.27%

35. Summary
Apply common
sense
Only test what you
are willing to put
live
Don’t be satisfied
that the result of
one test cannot be
improved upon
Re-test if you think
its necessary
Sometimes the
simplest things
give the biggest
results
Geo targeting
Persona driven
segmentation
Touch commerce
(Multi Device)
Define what
success looks like

36. Martin Sheerin
Head of Online Product
Development
Phones 4U
@martinsheerin1
linkedin.com/in/martinsheerin1