AB Testing and UX - a love story with numbers and people
Slides from the NUX6 talk by Craig Sullivan, Friday 27th October 2017.
2017.nuxconf.uk / nuxuk.org
Synopsis:
What’s wrong with the web these days? The mobile experience sucks. The customer experience sucks. It doesn’t work. It’s too hard to use. The text is too small. Nobody measures this happening. The interaction patterns suck. Nobody ever calls up to complain but nobody does anything anyway. Millions of people lose countless days to friction, poor design and frustrating moments on their devices.
There may be thousands of things you can fix that look promising – but how do you know where to start? What if you could measure what sucked, where it sucked and how big the problem was? Using lightweight research methods and tools, you can stop making excuses and start knowing exactly what to do. Life becomes much simpler and easier with a scientific method of optimising growth or delight within your product.
Craig has trained over 500 people on how to measure and optimise their product experience, finding 100M of ‘lost revenue’ using just one of the techniques you will learn. With reports, checklists, downloadable templates and toolkits for every budget and stage of growth – you can stop guessing tomorrow.
3. BACK TO 1999
@OptimiseOrDie
• Now I’m known as ‘that testing guy’
• But I started life as a UX practitioner
• >600 hours moderating in 3 years
• Everything I did started here
• Where did it take me?
4. • UX, Analytics, Tools, blended.
• I coach people and teams on
how to experiment profitably
• I’m a plumber looking for
‘ leaks’ in the delight or revenue
for a business
• Most companies don’t know
• This is my job
@OptimiseOrDie
TODAY
5. Why do I get so angry?
Because We are
F***ing Stupid!
@OptimiseOrDie
9. Reddit Explains…
“If it comes off in one, it's gonna be
banging! If you’re picking plastic out
of your dinner for ten minutes, it's
probably gonna taste like shit.”
@OptimiseOrDie
10. Quick Calculation:
• £4.5Bn Sales @ £3.5 average price
• Assume 60% are failures
• Costs 90 seconds of wasted time
• 72514285714 seconds wasted
• 1208571428 minutes
• 20142857 hours
• 839,285 days of life - wasted on friction!
• What about injuries? Food wastage? Ennui?
@OptimiseOrDie
12. A Packaging Manifesto:
1. Packaging should never require a tool or knife to open
2. Packaging should not require finger control or strength
which discriminates against population groups
3. Any packaging must be capable of being returned to its
original state to preserve any unused contents
@OptimiseOrDie
18. @OptimiseOrDie
“We are a data driven and
customer centric organisation…”
(but none of us have watched,
read or listened to customer
feedback in a year)
19. @OptimiseOrDie
“We have Session Replay,
Heatmaps, Form Analytics and
Voice of Customer Feedback”
(we just don’t look or act
on what they tell us)
20. @OptimiseOrDie
“We have agile product
development teams”
(but we haven’t reformed
anywhere else - signoff
takes a month)
21. @OptimiseOrDie
“We have a huge Conversion
Optimisation Team of 6 people”
(but they’re working
in a Silo, tinkering with button
testing or trivial insights)
22. @OptimiseOrDie
“We apply lean thinking at all
levels of the organisation”
(but we haven’t
changed the way
teams and managers
actually work)
23. @OptimiseOrDie
“We have the best tools,
methodologies and team around”
(but absolutely nothing
useful happens)
37. The Ikea of Psychological Ninja Messaging?
• Like a ‘Franchise’ store
• Every space competing directly
• Maxed out on persuasion
• Point optimised
• Works ‘on average’
• Holistic Experience?
• How do I actually FEEL?
38. The Ikea of Psychological Ninja Messaging?
CMA to probe Hotels: bbc.in/2hfh70r
40. @OptimiseOrDie
• Clothing 1M per month, 2 bugs
• Travel 2M per month, 13 bugs
• Telecoms 1.4M per month, 3 bugs
• Travel 0.6M per month, 1 bug (iPad)
• Event Site 2.5M per month, 11 bugs
– These were only the first bugs we found
– < 5 days time to fix – an ROI of 91,150%
– Larger than the annual IT budget for some!
How much do product defects cost?
44. • We’re swamped with data
• The underlying data is often flawed or unreliable
• We don’t know what’s wrong with our products
• We change stuff without evidence or observation
• We build stuff nobody wants or needs
• Product changes are not measured or tested
• Nobody takes stuff away from the website
• We build what internal people want, not customers
• We don’t remove costly defects and wastage
• Lean & Agile have left us with new types of debt
• Ego is the enemy of all good work
• Much Internet design is infused and driven by Ego
@OptimiseOrDie
What is going wrong?
49. @OptimiseOrDie
• MVP approach
• Launch as alpha, beta, pilot,
phased rollout
• This approach includes
split testing to tune,
optimize or decide!
• 100s of releases a day!
• Like getting in a shower
• Read more at labs.ft.com
FT.COM – Release Methodology
50. @OptimiseOrDie
• Small teams with direct access to publish
• Ability to set and get metrics data directly
• Tools, Autonomy, Lack of interference
• No Project Managers or Business Analysts
• Stakeholders decoupled from work
FT.COM - Corporate Models bit.ly/2mZAxH7
51. @OptimiseOrDie
• Business wants: “15% increase in 12 months”
• Business defines ‘outcomes’ – teams deliver
• No long signoff chain
• No pesky meddling during execution
• Less time wasting meetings with morons!
• Outcome not feature driven optimization
FT.COM - Decoupling bit.ly/2mZAxH7
52. @OptimiseOrDie
• Most product changes we make are a waste of time!
• The Financial Times – 50-70% of product developer time
wasted in a siloed organisational structure
• Minimum project time was “At least 18 months” for anything
• Projects were always late, underperformed, didn’t shift
metrics as expected
• Post changes, every IT project was under time & budget
• Products were iterated to hack their way to success
FT.COM – Advantages
53. @OptimiseOrDie
“Product changes and AB tests are just the
same thing to us. Where we’re moving to is
measuring and testing everything we do.
This approach tells us what works, what
doesn’t, what to keep, what to remove and
what we need to iterate.”
Financial Times – NextFT project
FT.COM – Advantages
54. • Reform the Organisation, not the tools
• Invest in analytics and people skills
• Set people free to learn new disciplines and skills
• Stop delivering ‘features’ and start doing ‘outcomes’
• Validate everything - changes, tests, personalisation
• Test everything that *can* be tested
• Stop Ego from driving product – use insights & data
• Continuous improvement rather than large releases
• Use multiple product ‘horizons’ for strategy
• Blend Lean, Agile, Data, UX and AB testing
• Let’s give you some tips to take away and use!
@OptimiseOrDie
What can we do about it?
55. @OptimiseOrDie
Conversion Research in One Hour:
bit.ly/batmantesting
Session Replay Tools for Research:
bit.ly/decibelWAW & bit.ly/ReplayTools
The List of UX Tools to Rule Them All:
bit.ly/UxTools & bit.ly/uxresourcepack
Lean Research – Further Reading:
56. @OptimiseOrDie
Lean Organisations – resources:
How Booking.com tests & organises teams:
http://bit.ly/SFrisby
Growth Team Models & Explanations:
http://bit.ly/GHmodel
ING video on Lean Corporate:
http://bit.ly/ingvideo
57. 1. Find out your Testing Bandwidth - Can you run any tests at all?
2. Stop copying stuff blindly - Test or validate changes, don’t assume
3. Get with Statistics - Doing testing? You need stats skillz
4. Run a test Calculator - How long it takes and when to stop
5. Prioritise All the Testing - Do the right tests in the right order
6. Get a Decent Hypothesis - Forget this, forget your AB testing
7. Record the device experience - Store the screen & recordings
8. Simplify your Test Decisions - Stop overcomplicating test analysis
9. Avoid Toxic AB Testing - Defects ruin your work and decisions
10. Test Across 3 Horizons of Product - How to test ahead of yourself
@OptimiseOrDie
10 AB Testing Power-ups
58. >1200 tests a year
>1000 tests live
99% chance you’re in 10 tests
7000 tests in 2011
100,000 tests last year
@BartS
1: Find Out your Testing Bandwidth
59. 1: Do you have testing bandwidth?
• This is a little Amoeba
• It hasn’t evolved to ride a bike yet
• Do you have enough traffic?
• < 500 ‘conversions’ a month?
• Work on product/market fit!
• Work that user research!
@OptimiseOrDie
60. 2: Stop Copying Stuff Blindly…
• Your customers are not the same
• Your site is not the same
• Your platform is not the same
• Your advertising isn’t the same
• The traffic isn’t the same
• The UX is not the same
• Your Device Mix is not the same
• Your mistakes are different
bit.ly/BestPUnicorns
bit.ly/BPSuccess
bit.ly/BestpFAIL
@OptimiseOrDie
62. @OptimiseOrDie
“Why every Internet Marketer should be a Statistician”
http://bit.ly/1wMfs1G
“Statistical Significance does not equal Validity”
http://bit.ly/1wMfmY2
“Understanding the Cycles in your site”
http://mklnd.com/1pGSOUP
“Statistics for Dummies”
http://bit.ly/stats4dummies
3: Get With Statistics
63. @OptimiseOrDie
• Run a test calculator first
• If the answer is 9 Million years, do something else instead!
Online Dialogue Suite: bit.ly/CaLC1
VWO : bit.ly/CaLC2
Optimizely : bit.ly/CaLC3
Evan Miller : bit.ly/CaLC5
• Testing is trading time and effort vs. potential reward
4: Run a Test Calculator
65. @OptimiseOrDie
Target Users
per
week
CR % AOV Each 5% 5% li8
detected
sample
Min
Weeks
10% li8
detected
sample
Min
Weeks
15% li8
detected
sample
Min
weeks
Basket 100,000 5% £140 £35,000 48098 1 12301 1 5590 1
Landing 1 60,000 3% £190 £17,100 25566 3 6531 1 9528 1
Landing 2 18,000 0.6% £100 £540 419876 47 107513 12 48913 6
• How long will it take to see a difference of 5, 10 and 15%?
• How much sample are you going to need to ‘detect’ an effect you’re hoping for
• S-ck to tests that have decent sta-s-cal power, to save was-ng -me!
• Read these two ar-cles:
https://inbound.org/discuss/conversionxl-live-notes-slides-and-q-a-
slideshare-s-now-included#paulinemarol
https://conversionxl.com/better-way-prioritize-ab-tests/
5: Statistical Power & Money Model
68. @OptimiseOrDie
• Store the design layout in Google Analytics or other package
• Use this to segment the results by the size of the design seen!
7: Record the Design people ‘see’
72. 1. Get an Analytics Audit - Broken tills mean broken company
2. Make a Device List - Prove you know this please
3. Test and Fix Broken shit - Customers don’t self report problems
4. Do Rapid Usability testing - Low cost and vital for great ideas
5. Add Session Recording - See broken stuff, get evidence, fix it
6. Run a Customer Survey - Regular surveys provide long term insight
7. Install Voice of Customer tools - This asking people shit will never catch on
8. Hire a Copywriter - Best money you’ll ever spend
9. Invest in Analytics - Google Analytics is NOT free
10. Add Form Analytics - Stop guessing and know what happens
11. Check your Web Performance - Performance makes £££
@OptimiseOrDie
11 Optimisation Power-ups
73. @OptimiseOrDie
1: Get an Analytics Audit, NOW!
“Check the tills
are working AND
recording correctly!”
@OptimiseOrDie
@CharlesMeaden
74. • What’s YOUR testing list?
• Anyone checked if it’s true?
• Use Google Analytics to tell you!
• Tip – do these as a minimum:
• iPhone 5/6
• iPad late model
• Samsung late model
• Desktop (Chrome, Edge, Safari)
bit.ly/UltimateXDO
@OptimiseOrDie
2: WHAT do they ACTUALLY use
79. @OptimiseOrDie
4: Shitty User Research – Stop It!
What the f*** is going on here:
• Biased samples from recruitment & screening
• Ineffably bad or biased moderator scripts
• Focus on quantitative questions in qual research
• Biased online panels (serial testers, poor screening)
• Low cost does not mean good fidelity
• Moderation as a skill to me seems ‘less valued’
80. 1.5 Interviewing Tips
• Doing this stuff properly is hard
• Interviewing and testing tips:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/1cQpiIE
“Interviewing Tips” bit.ly/1fKqu03
“More interviewing Tips” bit.ly/1bmvGT
80
85. @OptimiseOrDie
8: The Economics of Copywriting
“Stop asking people to fill-in by writing
copy. Especially don’t ask developers to
do this! Hire staff or train them.
Bad copy on your site costs more money
than hiring someone. Good copy earns
tens or hundreds of times that cost..”
86. @OptimiseOrDie
9: Invest in Analytics
• Google Analytics is NOT free
• Your Car will not run without Oil
• Your Analytics will not run without Love
• Put budget into optimising the analytics!
It’s like writing a love note to your future self!
Do it today. Start writing it.
87. @OptimiseOrDie
10: Add Form Analytics
Formisimo
www.formisimo.com
Heap analytics
www.heapanalytics.com
Hotjar
www.hotjar.com
Google Tag Manager & Analytics
bit.ly/SimoGA
“This is so we can call you in
case there’s a problem
delivering your order.”
88. 11: Check your Web Performance
• Turn off Wi-Fi at all times!
• Use these performance tools:
Mobile & Desktop
Mobile Speed test
Google Pagespeed Insights:
• Google analytics TELLS you what’s slow:
• Google Statistics on Site Performance:
(< 2 seconds good, 2-3 just OK, 3+ bad)
@OptimiseOrDie
89. 1. Get an Analytics Audit - Broken tills mean broken company
2. Make a Device List - Prove you know this please
3. Test and Fix Broken shit - Customers don’t self report problems
4. Do Rapid Usability testing - Low cost and vital for great ideas
5. Add Session Recording - See broken stuff, get evidence, fix it
6. Run a Customer Survey - Regular surveys provide long term insight
7. Install Voice of Customer tools - This asking people shit will never catch on
8. Hire a Copywriter - Best money you’ll ever spend
9. Invest in Analytics - Google Analytics is NOT free
10. Add Form Analytics - Stop guessing and know what happens
11. Check your Web Performance - Performance makes £££
@OptimiseOrDie
11 Optimisation Power-ups
91. SECTION TITLE
1 MOST AB TESTING IS DOG POO
2 CROSS DEVICE EXPERIENCES SUCK
3 DEFECTS AND WASTAGE ARE HUGE
4 EONS ARE WASTED BUILDING USELESS STUFF
5 PEOPLE & CULTURE MAKE PROBLEMS, NOT TOOLS
6 DATA DRIVES NOTHING, PEOPLE ARE THE LENS
7 UX, ANALYTICS, TESTING & TOOLS = SUCCESS
8 TEAM PRODUCTIVITY & QUALITY STILL SUCK
9 REFORM THE COMPANY, NOT THE TECHNOLOGY
10 QUAL WITH QUANT TRIANGULATES & FOCUSES
SUMMARY
93. @OptimiseOrDie
The Experience Optimisation Engine
(1) Pre Model Checklist
Groundwork and common
sense checks before any
modelling or research begins
(2) Rapid Modelling
Basic sketching & modelling to
prioritise focal areas for
research, reduce scope
(3) Research or Heuristic
Use upfront & rapid analytics
insights to prioritise focal areas
for research time
(4) Model
Post research – model the site
around priorities from research,
analytics & more
(5) Streaming
This is where the big pile of
issues gets turned into different
project backlogs
(6) Triage & Triangulate
Where you cluster themes or
issues, triangulate clues from
multiple tools and run sizing
against test targets
(7) Prioritisation
Where you bubble the most
promising test ideas to the top
of the stack bit.ly/CRO4Agencies
95. CRAIG SULLIVAN
Senior Optimisation
Consultant
@OptimiseOrDie
24TH OCTOBER 2013
EMETRICS LONDONOptimiser Of
Everything
THANK YOU!
Manchester Workshop Next Week:
bit.ly/MancClass
20 hours of training for £380:
bit.ly/CRO4Agencies
Get this Deck:
http://bit.ly/ProductPoo
Read my stuff:
medium.com/@optimiseordie
@OptimiseOrDie