This document outlines an approach for continuous experimentation and improvement. It recommends starting with data to identify high traffic, low conversion areas to improve. Form hypotheses about potential solutions and test them through small experiments. Document learnings to establish best practices and avoid regressions. The goal is to continuously run small tests to drive learning and improvements.
What is A/B testing? A SCIENTIFIC METHOD OF PROOF
WHAT IS A/B TESTING FOR MARKETING
Where do you start testing? THE DON’TS
WHERE DO YOU START TESTING?
DON’T JUST GUESS
SUB OPTIMIZE • Button colors • Small copy changes • Small layout changes • Fonts, font size
WATCH OUT FOR BUZZ • Don’t just test what the gurus say • What works for others may not work for you
PREMATURE TESTING • Testing without traffic • Before you know what and why you are testing
Where do you start testing? THE DOS
DO YOUR HOMEWORK You have to put in the work to get great results • Learn as much as you can about A/B testing • Study the great people out there • What’s their process and insights?
A framework for success The basis of a successful testing program
Research Know what to test and see the greatest improvements
DIVE INTO YOUR DATA
FIND YOUR BIGGEST OPPORTUNITIES
INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT
FORM A HYPOTHESIS By doing A, B will happen, Because of C
BUILD THE TEST
LAUNCH THE TEST VS. Control Variant
THE TEST VS. +24.01% increase in registrations!! 100% Significance
DOCUMENT AND SHARE YOUR FINDINGS Documenting all your tests and results • Increases company transparency • Helps others know where tests are going on • Makes it easier to call tests • Makes learnings available for everyone in company
REPEAT THE PROCESS AS OFTEN AS POSSIBLE
Control vs. Variant
“Only 1 out of 8 A/B tests have driven significant change.” Noah Kagan AppSumo*
“The goal of a test is to get a learning, not a lift. With enough learnings, you can get the real lift,” Dr. Flint McGlaughlin MECLABS
RULES OF THUMB FOR TESTING Things to remember when we are testing • Min. 2000 visitors • Min. a few hundred conversions • At least one full cycle > 2 week • >99% statistical significance • Keep a backlog of test hypothesis • Build next tests as the current one is running *More info at http://kiss.ly/1vF4wUc and http://kiss.ly/1Q4vTm0
Tooltips AND BURGERS
How I like my testing burger Tooltips for your testing framework
Tool Company Why do I like it? Analytics Kissmetrics Shows me my actual customers and their behavior Surveys Qualaroo and/or SurveyMonkey Easy to use/Deep qualitative insights Project Management Trello Manage my time + communicate with collaborators
Tool Company Why do I like it? Company Wiki Confluence Document initial thoughts for everyone Testing platform* Optimizely, Pardot, Adwords Easily build tests and get data Analytics Kissmetrics Let’s me confirm and verify down my funnel Company Wiki Confluence Documentation and learnings for everyone *Includes actual testing platform and tools that have testing capabilities
Not everyone likes their burger the same way How do you take yours?
Trust the process Share your leanings Stack your results A/B TESTING FOR SUCCESS
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The real problem
One month Winner! Two months Three months Four months Five months Six months Seven months Eight months Nine months
1 winner per year
QUOTE FROM ELON MUSK BIOGRAPHY “ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out on that money.
We’re missing out on this
But…why?
TOO MANY IDEAS!
170 ideas in 3 minutes
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
Er.
We need a process
Find out what’s stopping folks from converting…
…fix it.
Thanks
I’ll be here all night
Step 1: Take these ideas…
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
…and throw them away.
Step 2: Listen to your customers and prospects
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price These ideas are good. They just might not be for you.
Gut Your gut is good. It just knows too much.
Curse of knowledge
USER TESTING
Send 3 users through your funnel
Send 3 users to your competition
Send 3 users to a HUGE company in your space
PEEK
QUALAROO
CHAT TRANSCRIPTS
SALES PEOPLE AND SUPPORT
Step 3: Use a framework to make decisions
Test summary Test objective Hypothesis Opportunity size Time to test Likely scenarios Next steps What worked? What didn’t work? HTTP://BIT.LY/ABTEST-FRAMEWORK
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The problem opportunity
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
Not so much Winner! Not so much Not so much Not so much Winner! Not so much Not so much Not so much Not so much
Not so much Not so much Not so much Winner! Not so much Not so much Winner! Not so much Not so much Winner!
13. Case: Ervaringen met conversie optimalisatie bij Ahold (Roel Willems)OrangeValley
Ahold is in Nederland één van de koplopers als het gaat om data driven marketing. Aan de hand van een case legt Roel uit welke drie principes er zijn om aan het verbeteren van de conversie te werken. Roel gaat er op in hoe traditionele research technieken helpen om als data driven marketeer succesvol te zijn.
A/B Testing You Might Be Driving in the Wrong Direction TOMASZ BORYS
@Kissmetrics #KissWebinar @thuelmadsen
Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a Kissmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras. THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Tomasz loves dipping his feet in the river while fishing, injuring his thumb while gaming, hacking away at a golf club, and driving demand at Kissmetrics. He’s also the biggest fan of gummy bears. TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali
@Kissmetrics #KissWebinar @tbcali
1 The optimum strategies for A/B testing 2 How we A/B Test at KISSmetrics 3 A/B Testing beyond click conversions - having a pulse on the entire funnel and why it’s important TABLE OF CONTENTS 4 The influence social traffic has in the funnel
• Set a Goal • Baby Steps • Aim for Statistical Significance • Never Lose Sight THE OPTIMUM STRATEGIES FOR A/B TESTING
Baby Steps People want to move the needle and see results fast, by implementing multiple changes at once. But how can you be sure what element had an impact?
Aim for Statistical Significance SET A GOAL Aim for Statistical Significance getdatadriven.com
Never Lose Sight We can caught up on what we think is best to optimize conversions or traffic…but data doesn’t lie
• 4,000 and above sample size • 99% Statistical Confidence • Blind Eye for 1 Week HOW WE A/B TEST AT KISSMETRICS
4,000 and Above Sample Size Significant or decent amount of data is needed when you’re testing beyond click conversions
4,000 and Above Sample Size significant or decent amount of data is needed when you’re testing beyond click conversions 99% Statistical Confidence Police
Blind Eye for 1 Week Data can be very erratic the first several days, so it’s easy to hit the panic button
A/B Testing Beyond Click Conversions
ORIGINAL
VARIANT
SIGNUPS
Signups Install JS
Custom Data
Custom Data Opportunities Created
SOCIAL AD FUNNEL
TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali tborys@kissmetrics.com THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen tmadsen@kissmetrics.com Questions?
Website Redesigns: Why they Fail and How to Ensure SuccessOptimizely
Learn how to avoid website redesigns that don’t deliver results.
Website redesigns are a tremendous effort: they take a lot of planning, involve many team members and are very costly. The worst about them is that they often fail, meaning that the “new” website will perform worse than the old.
- How common redesign fails really are, and how costly they can be
- The key reasons most redesigns fail to increase sales/conversions
- A better approach to increase chances of redesign success
To learn more about Optimizely, find more info here: https://www.optimizely.com/
To get more inspiration for better website redesigns, visit our blog:
https://blog.optimizely.com/2014/08/14/2-alexa-500-site-redesigns-that-should-inspire-you-to-ab-test/
Optimizely Workshop: Mobile Walkthrough Optimizely
Testing and optimizing your mobile apps can help with shorter development cycles, data-driven decision-making, and higher user conversion rates. In this highly interactive session, we encourage you to bring your app (or a sample app), and we’ll walk through the top-to-tail process for using Optimizely on your mobile app. This training is designed for iOS and Android developers who are looking to use Optimizely on their mobile apps.
Optimizely Workshop 1: Prioritize your roadmapOptimizely
When your testing roadmap includes dozens of ideas (each with unique requirements) and each team member is vying for her idea to be run first, effective prioritization becomes paramount. This session will focus on the considerations, tools and frameworks you can use to make sure your roadmap is appropriately prioritized to meet your goals.
What is A/B testing? A SCIENTIFIC METHOD OF PROOF
WHAT IS A/B TESTING FOR MARKETING
Where do you start testing? THE DON’TS
WHERE DO YOU START TESTING?
DON’T JUST GUESS
SUB OPTIMIZE • Button colors • Small copy changes • Small layout changes • Fonts, font size
WATCH OUT FOR BUZZ • Don’t just test what the gurus say • What works for others may not work for you
PREMATURE TESTING • Testing without traffic • Before you know what and why you are testing
Where do you start testing? THE DOS
DO YOUR HOMEWORK You have to put in the work to get great results • Learn as much as you can about A/B testing • Study the great people out there • What’s their process and insights?
A framework for success The basis of a successful testing program
Research Know what to test and see the greatest improvements
DIVE INTO YOUR DATA
FIND YOUR BIGGEST OPPORTUNITIES
INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT
FORM A HYPOTHESIS By doing A, B will happen, Because of C
BUILD THE TEST
LAUNCH THE TEST VS. Control Variant
THE TEST VS. +24.01% increase in registrations!! 100% Significance
DOCUMENT AND SHARE YOUR FINDINGS Documenting all your tests and results • Increases company transparency • Helps others know where tests are going on • Makes it easier to call tests • Makes learnings available for everyone in company
REPEAT THE PROCESS AS OFTEN AS POSSIBLE
Control vs. Variant
“Only 1 out of 8 A/B tests have driven significant change.” Noah Kagan AppSumo*
“The goal of a test is to get a learning, not a lift. With enough learnings, you can get the real lift,” Dr. Flint McGlaughlin MECLABS
RULES OF THUMB FOR TESTING Things to remember when we are testing • Min. 2000 visitors • Min. a few hundred conversions • At least one full cycle > 2 week • >99% statistical significance • Keep a backlog of test hypothesis • Build next tests as the current one is running *More info at http://kiss.ly/1vF4wUc and http://kiss.ly/1Q4vTm0
Tooltips AND BURGERS
How I like my testing burger Tooltips for your testing framework
Tool Company Why do I like it? Analytics Kissmetrics Shows me my actual customers and their behavior Surveys Qualaroo and/or SurveyMonkey Easy to use/Deep qualitative insights Project Management Trello Manage my time + communicate with collaborators
Tool Company Why do I like it? Company Wiki Confluence Document initial thoughts for everyone Testing platform* Optimizely, Pardot, Adwords Easily build tests and get data Analytics Kissmetrics Let’s me confirm and verify down my funnel Company Wiki Confluence Documentation and learnings for everyone *Includes actual testing platform and tools that have testing capabilities
Not everyone likes their burger the same way How do you take yours?
Trust the process Share your leanings Stack your results A/B TESTING FOR SUCCESS
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The real problem
One month Winner! Two months Three months Four months Five months Six months Seven months Eight months Nine months
1 winner per year
QUOTE FROM ELON MUSK BIOGRAPHY “ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out on that money.
We’re missing out on this
But…why?
TOO MANY IDEAS!
170 ideas in 3 minutes
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
Er.
We need a process
Find out what’s stopping folks from converting…
…fix it.
Thanks
I’ll be here all night
Step 1: Take these ideas…
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
…and throw them away.
Step 2: Listen to your customers and prospects
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price These ideas are good. They just might not be for you.
Gut Your gut is good. It just knows too much.
Curse of knowledge
USER TESTING
Send 3 users through your funnel
Send 3 users to your competition
Send 3 users to a HUGE company in your space
PEEK
QUALAROO
CHAT TRANSCRIPTS
SALES PEOPLE AND SUPPORT
Step 3: Use a framework to make decisions
Test summary Test objective Hypothesis Opportunity size Time to test Likely scenarios Next steps What worked? What didn’t work? HTTP://BIT.LY/ABTEST-FRAMEWORK
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The problem opportunity
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
Not so much Winner! Not so much Not so much Not so much Winner! Not so much Not so much Not so much Not so much
Not so much Not so much Not so much Winner! Not so much Not so much Winner! Not so much Not so much Winner!
13. Case: Ervaringen met conversie optimalisatie bij Ahold (Roel Willems)OrangeValley
Ahold is in Nederland één van de koplopers als het gaat om data driven marketing. Aan de hand van een case legt Roel uit welke drie principes er zijn om aan het verbeteren van de conversie te werken. Roel gaat er op in hoe traditionele research technieken helpen om als data driven marketeer succesvol te zijn.
A/B Testing You Might Be Driving in the Wrong Direction TOMASZ BORYS
@Kissmetrics #KissWebinar @thuelmadsen
Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a Kissmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras. THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Tomasz loves dipping his feet in the river while fishing, injuring his thumb while gaming, hacking away at a golf club, and driving demand at Kissmetrics. He’s also the biggest fan of gummy bears. TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali
@Kissmetrics #KissWebinar @tbcali
1 The optimum strategies for A/B testing 2 How we A/B Test at KISSmetrics 3 A/B Testing beyond click conversions - having a pulse on the entire funnel and why it’s important TABLE OF CONTENTS 4 The influence social traffic has in the funnel
• Set a Goal • Baby Steps • Aim for Statistical Significance • Never Lose Sight THE OPTIMUM STRATEGIES FOR A/B TESTING
Baby Steps People want to move the needle and see results fast, by implementing multiple changes at once. But how can you be sure what element had an impact?
Aim for Statistical Significance SET A GOAL Aim for Statistical Significance getdatadriven.com
Never Lose Sight We can caught up on what we think is best to optimize conversions or traffic…but data doesn’t lie
• 4,000 and above sample size • 99% Statistical Confidence • Blind Eye for 1 Week HOW WE A/B TEST AT KISSMETRICS
4,000 and Above Sample Size Significant or decent amount of data is needed when you’re testing beyond click conversions
4,000 and Above Sample Size significant or decent amount of data is needed when you’re testing beyond click conversions 99% Statistical Confidence Police
Blind Eye for 1 Week Data can be very erratic the first several days, so it’s easy to hit the panic button
A/B Testing Beyond Click Conversions
ORIGINAL
VARIANT
SIGNUPS
Signups Install JS
Custom Data
Custom Data Opportunities Created
SOCIAL AD FUNNEL
TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali tborys@kissmetrics.com THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen tmadsen@kissmetrics.com Questions?
Website Redesigns: Why they Fail and How to Ensure SuccessOptimizely
Learn how to avoid website redesigns that don’t deliver results.
Website redesigns are a tremendous effort: they take a lot of planning, involve many team members and are very costly. The worst about them is that they often fail, meaning that the “new” website will perform worse than the old.
- How common redesign fails really are, and how costly they can be
- The key reasons most redesigns fail to increase sales/conversions
- A better approach to increase chances of redesign success
To learn more about Optimizely, find more info here: https://www.optimizely.com/
To get more inspiration for better website redesigns, visit our blog:
https://blog.optimizely.com/2014/08/14/2-alexa-500-site-redesigns-that-should-inspire-you-to-ab-test/
Optimizely Workshop: Mobile Walkthrough Optimizely
Testing and optimizing your mobile apps can help with shorter development cycles, data-driven decision-making, and higher user conversion rates. In this highly interactive session, we encourage you to bring your app (or a sample app), and we’ll walk through the top-to-tail process for using Optimizely on your mobile app. This training is designed for iOS and Android developers who are looking to use Optimizely on their mobile apps.
Optimizely Workshop 1: Prioritize your roadmapOptimizely
When your testing roadmap includes dozens of ideas (each with unique requirements) and each team member is vying for her idea to be run first, effective prioritization becomes paramount. This session will focus on the considerations, tools and frameworks you can use to make sure your roadmap is appropriately prioritized to meet your goals.
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and TacticsCXL
Everyone knows video can be an incredibly powerful conversion tool, but few know the secrets of running lean and mean CRO campaign with video. Armed with data from studying the performance of Wistia's 10 million videos, Phil will explain how to get the most of the videos across your channels and will explain how video analytics data can be used to automate and improve your creative processes.
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
Peep Laja, founder of ConversionXL, talks about some of the original UX research studies ConversionXL Institute has conducted over the last months. Inspiration for effective tests and website changes.
During this webinar conversion expert Peep Laja shares his 6 step framework for continuous optimization. Learn what simple steps you can take to maximize online conversions and help turn clicks into customers.
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
The Million Dollar Optimization Strategy - Andre Morys - ConversionXL Live 2015CXL
"Conversion Optimization" and "Testing" became the hottest topics on the strategic roadmaps of online marketing managers. But still, a lot of companies suffer from easy to solve problems that probably cost a couple of million dollars.
In this talk, André Morys shows some unkown conversion barriers that are not visible on the website. He shows some practical examples and cases that show how effective optimization programs gain a lot more momentum and drive more optimization ROI than usual testing efforts do.
[CXL Live 16] You Can’t Make This Stuff Up by Alex HarrisCXL
See how moderated user testing is a proven tactic to gain valuable insights that you could never think of yourself. Learn how these discoveries turned into dramatic results for website growth.
SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...Distilled
Discover how you can increase the performance and velocity of your testing program. Stephen will share his experience at Conversion.com, the UK’s biggest conversion optimisation agency. You’ll learn how to develop a process for personalisation (and avoid the mistakes most people make), how to accelerate your testing (with an advanced framework for prioritisation), and how to run advanced tests on product and price.
Where do You Start?
Where to begin? How do I track the data? How do I setup an A/B test? When do I know if the test is conclusive?
1.The CRO Mindset 1
Not hitting goals?
Do more with less.
Failing is Part of the Game
Follow the Data
2.Start with a Plan 2
Don’t just test to test
Create a backlog
3.Collecting Data 3
Lots of Options
The standard
My preference
They are directly integrated into most AB testing tools
First we need to build our funnel
KISSmetrics makes this dead simple
Experiments show in funnel reports automagically
LOG IN WITH GOOGLE Start Your Free KISSmetrics Trial
4.Collecting Insights 4
Qualitative is important
On-page survey tools
Ask questions at funnel drop off points How can we help? Are you looking for something we don't have? Do you need assistance? How can we improve? What is preventing you from purchasing?
UX and Usability testing
Heatmapping
Look at your analytics :)
Use these insights to improve backlog
5.Start Testing 5
What is an A/B Test?
A/B Testing Tools
Add Javascript to site
“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.
The Top 5 Ecommerce Tests You Should Run Right NowOptimizely
In this webinar, WiderFunnel CEO and conversion rate optimization expert, Chris Goward, and Optimizely's testing specialist, Ryan Lillis talk about the tests you should be running on your e-commerce site today to maximize revenue NOW and when online shopping peaks – Christmas time.
It’s 2015, and the customer experience is increasingly being driven by techniques like A/B testing and optimization.
Optimizely recently surveyed digital experience owners to learn just how they think about and allocate resources towards their testing and optimization programs.
These slides answer the questions:
- How often do teams optimizing run A/B and multivariate tests?
- What are the top benefits that optimization programs are seeing?
- How do optimization teams manage their experiment process?
[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...CXL
Most optimizers aren’t big fans of the traditional redesign cycle. They promote a continuous optimizing process. And that’s fine. But sometimes websites just suck donkey balls. And a new website is a must. Also: new companies need new websites too. So how can we stop those clients from making shitty websites? Which research methods and tools can you use in which stage of the development cycle? And how do you this?
A/B Mythbusters: Common Optimization Objections DebunkedOptimizely
For every $92 marketers spend to drive traffic to their website, only $1 is spent on optimizing the experiences visitors encounter when they get there. But with the proven benefits of testing, why aren’t more companies spending on optimization? We’ve compiled a list of common objections to testing and asked digital marketers what they have to say about them.
Improve your content: The What, Why, Where and How about A/B Testingintrotodigital
A/B testing, also known as split testing, is a user experience research methodology where users are randomly split into two or more groups to see different versions of the same element. This presentation explains what is A/B testing, why you need it, where you can apply it and how to conduct an A/B test.
Unlocking the data possibilities of Big Data presentation shared at the Big Data / Internet of Things Conference Board Conference June 25-26, 2015
http://www.pwc.com/us/en/analytics/big-data.jhtml
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and TacticsCXL
Everyone knows video can be an incredibly powerful conversion tool, but few know the secrets of running lean and mean CRO campaign with video. Armed with data from studying the performance of Wistia's 10 million videos, Phil will explain how to get the most of the videos across your channels and will explain how video analytics data can be used to automate and improve your creative processes.
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
Peep Laja, founder of ConversionXL, talks about some of the original UX research studies ConversionXL Institute has conducted over the last months. Inspiration for effective tests and website changes.
During this webinar conversion expert Peep Laja shares his 6 step framework for continuous optimization. Learn what simple steps you can take to maximize online conversions and help turn clicks into customers.
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
The Million Dollar Optimization Strategy - Andre Morys - ConversionXL Live 2015CXL
"Conversion Optimization" and "Testing" became the hottest topics on the strategic roadmaps of online marketing managers. But still, a lot of companies suffer from easy to solve problems that probably cost a couple of million dollars.
In this talk, André Morys shows some unkown conversion barriers that are not visible on the website. He shows some practical examples and cases that show how effective optimization programs gain a lot more momentum and drive more optimization ROI than usual testing efforts do.
[CXL Live 16] You Can’t Make This Stuff Up by Alex HarrisCXL
See how moderated user testing is a proven tactic to gain valuable insights that you could never think of yourself. Learn how these discoveries turned into dramatic results for website growth.
SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...Distilled
Discover how you can increase the performance and velocity of your testing program. Stephen will share his experience at Conversion.com, the UK’s biggest conversion optimisation agency. You’ll learn how to develop a process for personalisation (and avoid the mistakes most people make), how to accelerate your testing (with an advanced framework for prioritisation), and how to run advanced tests on product and price.
Where do You Start?
Where to begin? How do I track the data? How do I setup an A/B test? When do I know if the test is conclusive?
1.The CRO Mindset 1
Not hitting goals?
Do more with less.
Failing is Part of the Game
Follow the Data
2.Start with a Plan 2
Don’t just test to test
Create a backlog
3.Collecting Data 3
Lots of Options
The standard
My preference
They are directly integrated into most AB testing tools
First we need to build our funnel
KISSmetrics makes this dead simple
Experiments show in funnel reports automagically
LOG IN WITH GOOGLE Start Your Free KISSmetrics Trial
4.Collecting Insights 4
Qualitative is important
On-page survey tools
Ask questions at funnel drop off points How can we help? Are you looking for something we don't have? Do you need assistance? How can we improve? What is preventing you from purchasing?
UX and Usability testing
Heatmapping
Look at your analytics :)
Use these insights to improve backlog
5.Start Testing 5
What is an A/B Test?
A/B Testing Tools
Add Javascript to site
“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.
The Top 5 Ecommerce Tests You Should Run Right NowOptimizely
In this webinar, WiderFunnel CEO and conversion rate optimization expert, Chris Goward, and Optimizely's testing specialist, Ryan Lillis talk about the tests you should be running on your e-commerce site today to maximize revenue NOW and when online shopping peaks – Christmas time.
It’s 2015, and the customer experience is increasingly being driven by techniques like A/B testing and optimization.
Optimizely recently surveyed digital experience owners to learn just how they think about and allocate resources towards their testing and optimization programs.
These slides answer the questions:
- How often do teams optimizing run A/B and multivariate tests?
- What are the top benefits that optimization programs are seeing?
- How do optimization teams manage their experiment process?
[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...CXL
Most optimizers aren’t big fans of the traditional redesign cycle. They promote a continuous optimizing process. And that’s fine. But sometimes websites just suck donkey balls. And a new website is a must. Also: new companies need new websites too. So how can we stop those clients from making shitty websites? Which research methods and tools can you use in which stage of the development cycle? And how do you this?
A/B Mythbusters: Common Optimization Objections DebunkedOptimizely
For every $92 marketers spend to drive traffic to their website, only $1 is spent on optimizing the experiences visitors encounter when they get there. But with the proven benefits of testing, why aren’t more companies spending on optimization? We’ve compiled a list of common objections to testing and asked digital marketers what they have to say about them.
Improve your content: The What, Why, Where and How about A/B Testingintrotodigital
A/B testing, also known as split testing, is a user experience research methodology where users are randomly split into two or more groups to see different versions of the same element. This presentation explains what is A/B testing, why you need it, where you can apply it and how to conduct an A/B test.
Unlocking the data possibilities of Big Data presentation shared at the Big Data / Internet of Things Conference Board Conference June 25-26, 2015
http://www.pwc.com/us/en/analytics/big-data.jhtml
University of Nebraska Think Tank: Google Analytics as your Digital Marketing...Caleb Whitmore
A presentation covering the what, why, and how for using Google Analytics as the central hub for gaining marketing insights and acting on them to drive business growth.
Google Partner Agency Digital Breakfast with Geary LSFKatie Fellenz
Geary LSF and Google partnered up to present the latest innovation at Google and in digital marketing at an exclusive Google Partner Agency event in San Francisco.
Evolution of TV Measuring Across All Screens | Think With GoogleArgent media agency
No matter how measurement is structured in the future, or what changes are on the horizon in terms of technology and viewer behavior, digital will continue to lead the way to greater clarity and precision. Advertisers will have better efficiency, programmers can maximize the value of their inventory, and consumers will have a better viewing experience. And that’s a win-win for everyone.
Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing ...Online Dialogue
Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.
Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.
Keynote Wheel of Persuasion - Bart Schutz of Online Dialogue on how to persua...Online Dialogue
Chief Persuasion Officer Bart Schutz will be keynoting about his Wheel of Persuasion on marketing & optimization conferences throughout Europe in october & november 2014:
Conversion Jam - Stockhom, Sweden
Conversions@Google - Dublin, Ireland
iLive - Riga, Latvia
Conversion Conference - London, England
Conversion Conference - Berlin, Germany
Conversion Summit - Istanbul, Turkey
Conversion Hotel - Texel, the Netherlands
At these conferences he will present his Maxima (System 1) and Willem (System 2) story and inspire you how to persuade these two parts of your customers brains. This slideset is the main slideset for each conference and is extended with extra A/B-testcases.
More info:
http://wheel-of-persuasion.com (the knowledge source)
http://online-dialogue.com (hire our team)
http://testing.agency (fast and quality A/B-test development)
http://abtestguide.com (you A/B-test workflow tool)
and http://Persuasion.tips for an overview of Bart his recent work.
Auto aftermarket digital driver research studiesHouseLead Italia
The needs of aftermarket shoppers haven’t changed. There are two types of behaviors when it comes to vehicle parts and accessories purchases: Proactive drivers buy for the season, enhancement and regular maintenance, while reactive ones purchase for repairs. But what has changed?
A/B Testing at Pinterest: Building a Culture of Experimentation WrangleConf
Presenter: Andrea Burbank, Pinterest
A successful experimentation program consists of much more than mere randomization and measurement. How do you help stakeholders understand the right things to measure, avoid common pitfalls, and learn to rely on A/B tests as the best way to measure a new system or feature? In this talk, Andrea will explain how building a culture of experimentation and the right tools to support it is just as important as the statistics behind the comparisons themselves - and potentially much trickier to get right.
Your competition now matters: Building a SaaS Company Isn't What it Used to B...Price Intelligently
Growth and software luminaries have preached for decades that "your competition doesn't matter...focus on your customers." While true in theory, Hiten Shah in his presentation at Price Intelligently's SaaSFest 2016, shows us that in the second wave of SaaS, your competition now matters. The shift took place because software is relatively easy to build now with infrastructure and marketing advances. You need to think about your customer first, but if you're not aware of your competition or doing things to circumvent them, you'll get left behind.
Millward brown vermeer organizational design think brand 2014 snfVermeer
MB Vermeer partnered with Google for Organizing for Marketing in a Digital Age – a thought-leadership initiative that codified best practices and practical insights into how winning teams are organizing for the evolving digital landscape. CMO Marc de Swaan Arons presented the findings to +500 senior level marketers at the Google ThinkBrand conference this October.
A brief, high-level overview and introduction on A/B testing
- What can you test?
- Famous A/B Testing Case Studies from the 2008 Presidential Election
- The technology behind A/B testing including Javascript, jQuery
- Tools you can use including: Optimizely, Visual Website Optimizer, Unbounce, Google Analytics
- Pluses and minues to A/B testing
Auto shoppers are open to influences. Today’s auto intenders are open-minded with 72% of search sessions involving cross-shopping. In-market shoppers are researching more than ever. While time in-market is roughly the same, these shoppers are turning to 24 research touch points on average. Connected devices are driving greater research activity. Mobile usage has increased 35% year over year, with a focus on research and comparison activities. Video influences auto brand discovery and consideration. Video research is on the rise and online video was the #1 format for encouraging brand consideration. Dealer interaction and post-purchase experiences matter. 62% of the vehicle owners said customer service could influence future purchases.
How to Create Winning A/B Tests through Stronger ResearchVWO
This is the accompanying presentation to a webinar on doing research for A/B testing. It describes how online marketers can create better A/B tests by investing in rigorous, useful research to create great testing hypotheses.
The topics covered are
- Analytics (digging into analytics with an eye on useful insights)
- Usability Testing
- On-site surveys
- Visual Analysis
- And finally, pulling all of these together
Creating a culture that provokes failure and boosts improvementBen Dressler
Everyone fails - but not everyone uses failed attempts as a source of learning and improvement. This talk outlines a framework to turn failure into gaining knowledge by understanding IF, HOW and WHY something fails.
We’ll cover the ins and outs of running a high velocity A/B testing and experimentation program so that you can run the most meaningful tests as quickly as possible without sacrificing the quality of insights.
Let's make testing your new text and design ideas for your landing page as straightforward as possible. Our landing page and tracking template add the last piece to the growth marketing puzzle: A/B Testing with Google Optimize. Follow our step-by-step guide and upgrade your setup. You can show 50% of your visitors your new and hopefully better version of your landing page from here on out. Build-in reports will show you which version has a statistically significant higher conversion rate. Back up your intuition and creativity with data and build up your confidence to make the right choices. You'll be rewarded with lower cost-per-acquisitions and higher customer-lifetime-values, aka happier customers.
Radically Improve Conversion Rates - eMSF 2009Kayden Kelly
See how to methodically and radically improve your conversion rate through quantitative and qualitative tools like Google Analytics, Website Optimizer, Crazy Egg, 4Q Survey and more.
Should UI/UX be gut-feeling or data-driven? How to stand out from the tough competition by perfecting your owned asset?
A/B testing a long-grind road but it does not have to be tough! Demyth the 4 steps approach to optimization and what it can bring to you
Experimental statistics is only one of the many powerful analytical techniques companies are using to supercharge their experiment ideation, segmentation, and analysis. Check out this content for a refresher of key stats issues and a discussion on how to use data for better test and bigger wins.
Intuit - How to Scale Your Experimentation ProgramOptimizely
Here’s the playbook Intuit uses to increase its experimentation velocity — even when they face traffic limitations.
Mike Loveridge is not new to running experimentation teams. Before Intuit, he built out programs at Ancestry.com, GE, Humana, and CheapOair. He's an expert at making experimentation work at high velocity, even in traffic-challenged situations.
In this webinar, Mike Loveridge shared his best practices for making CRO work at high velocity, key lessons from scaling multiple teams, and why he's bullish on the future of "test and learn".
Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdfVWO
You can utilize various forms of Generative Research to deepen your understanding of how people interact with your product or service.
Craig has amassed a vast toolkit of research methods, which he has employed to optimize websites and apps for over 500 companies. He'll share which methods yielded the highest return on investment, identified key customer pain points, and generated the best experiment ideas.
By sharing the top inspection methods essential for our work, Craig will provide advice for each technique. Anticipate insights on driving experiment hypotheses from research, a list of essential toolkit components for tomorrow, and additional resources for further reading.
Conversion rate optimization is a powerful tactic to get more value from your visitors, but there's a lot of mythology out there about what really works and doesn't. CRO is about more than testing button colors, headlines, photos, or videos.
Learn more about the big picture, and what truly moves the needle, with Rand Fishkin himself.
Check out this presentation to learn:
- Conversion Rate Optimization basics
- What's in a conversion decision
- How to find out what needs optimizing
- 10+ tactical tips for CRO
A Universal Theory of Everything, Christopher MurphyFuture Insights
Taken from the Future of Web Design, New York 2015 Conference. https://futureofwebdesign.com/nyc-2015/
Drawing on over two decades of experience designing and developing digital products, Christopher will walk you through everything he's learned along the way. He'll break apart the creative process, exploring how an understanding of that process, leads you to become a better designer. In this session, he'll explore how the best designers: firstly 'prime the brain' by ensuring it is constantly nourished with new material; then explore that material from multiple perspectives to gain a deep understanding of it; before, finally, putting those pieces back together again to create exciting new ideas that stand the test of time. In short, he'll ensure you leave the session fully creativity-hardened and never short of ideas again.
Horizon Interactive Awards, Mike Sauce & Jeff JahnFuture Insights
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
Mike Sauce, Founder and President of the Horizon Interactive Awards will present an award to the Most Awarded Developer in the 13th annual competition to DynamiX Web Design. Jeff Jahn, owner and founder of DynamiX, will discuss design trends, processes and technologies that led his company to achieve such a high honor in the Horizon Interactive Awards competition.
Reading Your Users’ Minds: Empiricism, Design, and Human Behavior, Shane F. B...Future Insights
Taken from the Future of Web Design, New York 2015 Conference. https://futureofwebdesign.com/nyc-2015/
How do you decide what your users really need? The difficult truth is that the best web design comes from finding out for yourself. Luckily for anyone passionate about improving web-based human interaction, the field of psychology can shed light on common motivations, needs, and biases that are powerful influences on human behavior. In this session, you’ll learn about how these psychological forces—such as prospect theory, metacognitive fluency, and the introspection illusion—can shed light on UX, design, and conversion.
Structuring Data from Unstructured Things. Sean LorenzFuture Insights
From FOWA Boston 2015
Structuring Data from Unstructured Things. Sean Lorenz
Data coming from Internet of Things (IoT) product sensors can be hard to manage or know what to do with. In this talk Sean will discuss ways to tame IoT data sources by organizing and pruning that information effectively. He will also discuss the importance of time series when culminating sensor, metadata and other data sources together, making it vastly easier to query or perform analytics on your newly structured data.
Taken from the Future of Web Design, New York 2015 Conference. https://futureofwebdesign.com/nyc-2015/
The process behind making a blockbuster film is similar to creating a meaningful website or app. Through the lens of cinema, we’ll walk through practical ways that UX design teams can work together to deliver an award-winning final product. Whether you’re making a low-budget indie for a non-profit or the next summer smash for a Fortune 500, we can learn a thing or two from film.
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
In the last few years, we’ve seen an emergence of a modular way of thinking about code and design. We’ve seen the rise of SMACSS, BEM, and Atomic Design. This talk will look at those modular concepts and how they can streamline development for large and long-running projects. We’ll also look at how these approaches can ease responsive design and development. Lastly, we will look at where the modular approach is going in the future as Web Components slowly make their way into browsers and application frameworks.
Designing an Enterprise CSS Framework is Hard, Stephanie RewisFuture Insights
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
It seems that not a week goes by without a shiny new framework of some type — be it CSS or JS. But no matter how awesome they are, each have shortcomings and idiosyncrasies that invariably make you ask, 'Why?' Now imagine someone gave you the ability to start from scratch to create your own framework. No strings. No preconceptions — well, except that it has to be enterprise scale, platform agnostic, and work in a whole host of disparate situations. In this session, Stephanie will talk about some of the challenges, hurdles, tradeoffs, and unique decisions Salesforce UX made on the way to building an enterprise framework.
Accessibility Is More Than What Lies In The Code, Jennison AsuncionFuture Insights
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
Many associate making a digital product accessible with the guidelines and specifications that address themselves at the code-level. In short, the developers/engineers will take care of it. While the thoughtful implementation of accessible code during the development phase is unquestionable, the truth is accessibility depends heavily on choices made by designers and others involved in determining the user experience, and typically before development begins. Join Jennison as he illustrates this by identifying some of the user interactions and design-related decisions that can pose accessibility challenges. He will also share practical advice for those seeking to scale accessibility and make it a shared responsibility.
Sunny with a Chance of Innovation: A How-To for Product Managers and Designer...Future Insights
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
Growth stage companies need to continue to be as innovative as they were as smaller startups - but how do you actually do it? How can product leaders and designers de-risk valuable new ideas and get the support required to actually execute? From the perspective of a product owner and a designer respectively, Audrey and Alexa will walk through how they ran an innovation team on a recent project. They'll discuss how they rallied a broader group of stakeholders around big and risky ideas, testing the limits of experimentation, and turning small-scale experimental code into real life features. Thinking big and executing in layers is the future of innovation. You will walk away with some easy methods to start launching experiments at your company, regardless of whether you come from a three-person startup or a huge corporation.
Taken from the Future of Web Design, New York 2015 Conference. https://futureofwebdesign.com/nyc-2015/
The future must be universally approachable. In this talk, Andrew looks at designing for dyslexic users. Learn how to create designs that are more universal; designs that not only better fit dyslexics, but are a better fit for everyone regardless of race, religion, national origin, language or ability.
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
Site analytics. The quantified self. Big data. Human activity is creating more and more measurable data. But is more data really helping designers make better decisions? Human problems often require illogical approaches. In order to meet real human needs, we need to approach the data we collect with empathy and find the story in the facts.
Taken from the Future of Web Design, San Francisco 2015 Conference. https://futureofwebdesign.com/san-francisco-2015/
We need to create processes that get us away from nice looking design files to actually shipping our projects into the real world.
FOWA London 2015
In recent years there have been incredible advances in artificial intelligence and deep learning. As a result, powerful technology which used to be rare and expensive has very quickly become easily available and cheap. This will have both positive and negative consequences for web developers. In this talk I will look at how AI will change the development field, and provide techniques that will help designers and developers to work with AI to improve their skills and make better sites and applications for end users.
Digital Manuscripts Toolkit, using IIIF and JavaScript. Monica Messaggi KayaFuture Insights
FOWA London 2015
Monica is part of the DMT project at the Bodleian Libraries (University of Oxford) that aims to create a toolkit using IIIF standard (http://iiif.io) for images, a server solution (to store images of manuscripts and metadata), and a client solution using JavaScript to build an authoring tool that allows editing the manuscript manifest and its metadata. Working specifically on the authoring tool, and on the challenges that different types of manifests presents for the developer. You will have a glimpse of the whole picture and then she taps into the libraries used, choices made, collaboration experiences and lessons learned so far.
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
18. Simple 4-step experimentation process
Data
Figure out what to improve
Hypothesis
Make an educated guess
Experiment
Test your guess
Act
Decide what’s next
30. Increase Conversions
By A/B Testing
Do a Happy Dance
But Watch it Closely
Get More Traffic
So You Can Start A/B Testing
Get More Traffic
So You Can Do a Happy Dance
High Traffic
Low Traffic
High ConversionsLow Conversions
How To Figure Out What To Improve
31. What percentage of your
customers or users have
adopted each feature?
Intercom on Product Management
When you’ve already got usage, analyze it.
32. Where do you have the most
visitors, users, customers,
emails, or notifications?
More volume helps you test faster.
33. ๏ Website Traffic / App Installs
๏ Welcome Emails
๏ Web / Mobile Onboarding
๏ Email Digests
๏ Triggered Notifications
๏ Cart / Funnel Abandonment
๏ Dormant / Churned users
Common areas with large volume and high impact potential
47. Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits
x
conversion
rate
x
order
value
x
li4
=
?
users
x
order
value
x
resurrec7on
rate
=
?
ESTIMATED
ESTIMATED
MODELED
MODELED
48. Assess the potential impact,
difficulty and priority of
each test idea you have.
Prioritize your tests
60. 1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook
61. What learnings do we want to
carry over from the past?
Helps you avoid unknowingly
decreasing your conversions.
62. ✓ Keep the footer links
✓ Autofocus on the first form field
✓ Do not include social proof
✓ Link to the features page
✓ Use “Log in” copy on the bu on
Create experiment checklists based on learnings