1. How Do You Know…
Designing Multi-Channel
Campaign Tests That
Optimize Your Budget
Jim DeLash
November 18, 2014

2. Page layout
One version had a Conversion Rate 24.6% higher
than the other one

3. Page layout
Moving registration to the right side form
increased the site’s Conversion Rate by 24.6%

5. How do you know?

6. The right frequency

7. The best subject line

8. The optimal channel mix

10. Designing successful multi-channel
tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

11. Establishing hypotheses
Statements to be proven
All tests must support a hypothesis
Extrapolating the results to other brands

12. Establishing hypotheses
Low unaided recall scores for
advertising creative will require
additional frequency….

13. Establishing hypotheses
Low unaided recall scores for
advertising creative will require
additional frequency….

14. Establishing hypotheses
Low unaided recall scores for
advertising creative will require
additional frequency….
BUT HOW MUCH MORE?

16. Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency

17. Establishing hypotheses
TEST…
More direct mail or e-mail
More frequency in multiple channels
Different levels of frequency

18. Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
Different levels of frequency

19. Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
10 15 20

20. Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency

24. The best design is not always the best approach
Do not let design conflate the objective of an email or message
Every unnecessary piece of content is waste and reduces your
chances of getting a click

25. Designing successful multi-channel
tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

26. Developing Test Strategies by
Segment
Specialist or Generalist
Current User or Prospect
Decile

27. Developing Test Strategies by
Segment
Multiple
Brands Cadence
Awareness & Trial Offer

31. What will we learn from this test….
1. Best frequency by Specialty and Quintile
2. One channel vs. Two channels (Segments 5,7,9)
3. Total frequency: 10 vs. 12 vs. 14

33. Version B, the one with the ‘Save Now’ button, increased opt-ins
for logged-in users by 333% at a 99.9% confidence level.

34. Designing successful multi-channel
tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

35. Creating a Measurement Plan
Control Groups
Measurement Milestones
Test Analysis

36. There are only two ways to get
fired from Harrah’s:
stealing from the company,
or
failing to include a proper
control group in your business
experiment

37. Creating a Measurement Plan
Control Groups…
Not a ‘holdout’ group
Must include all relevant variables
Large enough to be statistically valid

38. Creating a Measurement Plan
Measurement Milestones…
Track progress against goal
Forecast test impact early
Connect KPIs with Rx impact

39. Creating a Measurement Plan
Test Analysis…
Evaluate overall campaign impact
Both TRx and NRx

40. How can an e-mail test be
successful with a low open rate?

43. MICRO TESTS

44. MICRO TESTS

46. Version B, the subject line broken into three consumable
sections, increased opens by 73.7%.

48. Version B, the ‘Get My Free Paper’ Call to Action,
increased downloads by 25% at a 95% confidence level.

50. Version B, the ambiguous subject line, increased
opens by 112% at a 95% confidence rate.

51. Designing successful multi-channel
tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

52. What is the optimal number of
samples to offer in a multi-channel
campaign?

53. So, how will you know?

54. This is the last slide

55. The Influence of Color

56. The Influence of Color
21.4% Conversion
Rate improvement
with the red button

57. Headline Copy

58. Headline Copy
32.3%
Conversion Rate
improvement

59. A Picture is Worth 1000 Words…
But Which Picture? (Images)

60. A Picture is Worth 1000 Words…
But Which Picture? (Images)
The female model in the header convinced 108%
more visitors to fill out and submit the lead gen
form