Experimental statistics is only one of the many powerful analytical techniques companies are using to supercharge their experiment ideation, segmentation, and analysis. Check out this content for a refresher of key stats issues and a discussion on how to use data for better test and bigger wins.

1. Losing is the New
Winning
Presenters
Justin Christianson, Co-Founder & President, Conversion Fanatics
Sean Condren, Strategic Partnerships, Optimizely

2. Today’s Speakers
Justin Christianson
Co-Founder & President
Conversion Fanatics
Sean Condren
Strategic Partnerships
Optimizely

3. Housekeeping
• We are recording
• Slides will be available after the webinar is complete
• There will be time to submit questions at the end of
the presentation

4. Agenda
• Overview
• How to develop fool proof test hypotheses to
ensure more winners
• How to leverage your test data to set yourself
up for bigger wins
• What to do when only 1 in 8 tests are a winner
• Q&A

5. Write This Down
What are your 3 biggest problem areas right
now in terms of conversions and optimization?

6. Write This Down
What will it cost you if you do nothing to fix it?
What is your desired outcome?
What does success look like in 12 months?
What is holding you back from getting there?

7. Assumptions Kill
You can’t assume anything is going to work
better and for that you must test.
Voiceover Example

8. Magic Of Compounding
Conversions
10,000 Unique Visitors
3% Conversion Rate on a $99 Sale = $29,700
15% Upsell Take Rate @ $149 = $6705
Total = $36,405

9. Magic Of Compounding
Conversions
10,000 Unique Visitors
3.5% Conversion Rate on a $99 Sale = $34,650
18% Upsell Take Rate @ $149 = $9387
Total = $44,037
$7632 Increase

10. The Little “S” That Could

11. Even the SMALLEST changes can have
dramatic impact on your conversions
and visitor engagement.

12. +31%

13. +158%

15. -89%

18. “Big doors swing on small hinges.”
~ W. Clement Stone

19. Sometimes the only way to get a bigger win
is to make big changes, but only after
exhausting incremental changes/tests.

21. It’s All A Numbers Game
The GOAL is to reach the tipping
point as quickly as possible then optimize
so you can scale and profit more.

22. How to develop fool proof
test hypotheses to ensure
more winners

23. Don’t let a single
day go by without running a test.
Remember it is the only way to get
and keep a competitive advantage.

24. Don’t just blindly test things. Find
out problem areas and test them to fix
and confirm assumptions. Then learn
from those tests and test again.

25. Why Use A Process For
Testing?
Random testing can work. It is a process.
Can yield big gains to just go through a
checklist of tests, but you won’t always know
WHY the changes are happening.

26. With A Process
• You waste less time on tests that are of low
value.
• You learn something from each test, so no test
is ever a “loser” (extremely important for
morale).

27. To double your business how much would you
have to improve each conversion rate?
Or…if you improved each conversion rate
100%, how much difference would it make?

28. Prioritize your results so you
know which order to test.

29. Every metric has it’s limit. Use common
sense. If your model tells you to raise
your upsell conversion rate from
50% - 100%, it’s not going to happen.

30. The results are VERY different for each e-
commerce store even if they are selling the
same types of products. Don’t just change your
store because someone said a certain element
improved his checkout rate . You have different
inputs, hard good costs, fulfillment process,
continuity, stick, etc.

31. Making It Effective
No campaign should be deployed without first
understanding and creating a test hypothesis.

32. What’s A Hypothesis?
A hypothesis answers the question…Why?

33. A hypothesis is a bold, specific prediction
to be proven or disproven by
your experiment.
It includes:
- What you’re testing
- What you predict the outcome will be
- Why you think that’s the case

34. If we change X, then Y, because Z.
Or…
If (variable), then (result), due to (rationale).

35. From your data analysis you discover
your front end offer bounce rate is high.
94% are leaving within 15 seconds.

36. Why?
• Is the price hard to find or missing?
• Is the navigation confusing?
• Is there lack of a clear value proposition?
• Is the landing page a missing information
promised in your ad?

37. Wouldn’t it be nice to know WHY they are
leaving? Rather than just this button colors
raises or lowers conversions?

38. So craft a hypothesis to test every reason why
you can imagine across your offer page to
figure out what really drives your audience.

39. If we prominently reveal (instead of hide) rebill
price, then fewer people will bounce, because
they want to know pricing before putting in the
effort to read and consider the free trial.

40. That was a real case study that resulted
in a 40% lower bounce rate and 31% increase
in trial signups.

41. This seems simple but it quickly
iterates and compounds in the
right direction when done correctly.

42. Now What?
1. Document your findings.
2. Share them = makes your entire team smarter.
3. Push winners. Profit!
4. Do it again! This is the beginning of a data-
driven transformation of your entire organization.
You’re building a knowledge base that will take
you to the top of your industry = happier and
more productive organization with more winning
products, more customers, higher average order
values, and higher lifetime value = winning.

43. How to leverage your test data to
set yourself up for bigger wins

44. With testing we either win or we learn.

45. The main objective is to learn how visitors
interact with your site. What are they paying
attention to, what are they ignoring or where
are they falling off that is causing them to not
take the desired action you want them to take?

46. You can’t just make a list?
It would be great if we could just make a big list
of elements to test and go down that list. But
effective testing follows the data and the plan
can quickly change.

47. Watch Your Data
1. Watch for trends in the data over time
2. Pay attention to the interactions your visitors
have with the variations
3. Test bigger changes to sometimes yield
bigger wins

49. More Examples
1. Video placement
2. Benefit bullet statements
3. Adding more testimonials
4. Price testing and how you could lose money

50. When To Call A Test
• If not showing promise early with plenty of
visits or conversions
• Statistical significance
• 1000 visits/variation
• At least 25 conversions per variation

51. What To Do When 1 in 8
Tests Is A Winner?

52. What To Do?
• Learn as much as you can. Follow heatmaps
• Develop hypotheses for your results and data
• Don’t just follow a step by step list
• Don’t get discouraged

53. Testing Mistakes
• Don’t let tests run long enough
• Run tests too long
• Don’t implement fast enough
• Test too many small elements
• False positives or inaccurate reporting

54. If You Are Testing Correctly
• You can afford more traffic
• You get faster and more dramatic split-tests
results
• You can outbid your competition
• You’ll attract more and better affiliates
• You can afford to invest more in growth
• You can add more back-end products, build new
funnels, enter new markets

55. Key Takeaways
• Always leverage a test hypothesis
• Follow your data and let your visitors be the
guide
• Don’t assume anything and test everything

56. Questions?
Please submit your questions via the text box on
your screen.

57. 1200+Attendees
20+Sessions
100+Giveaways
30+Speakers
Wynn Las Vegas, October 17-19
Registration
Early Bird
$750Expires on 09/01/2017
Early Early Bird
$550Expires on 07/01/2017
Regular
$850Expires on 10/16/2017

58. Thank you