Why A/B-testing is not enough, but it works as a complement to user research

1. Is A/B-testing all you need?
The answer is no.
… says Martin Christensen, UX designer & researcher

2. Martin Christensen
RAND()
A
42% conversion 27% conversion
B

3. Martin Christensen
RAND()
A
42% conversion 27% conversion
B

4. Martin Christensen
RAND()
A
27% conversion
B
0% conversion

5. Martin Christensen
What
WHY!?!?!?!?!?!
A B

6. Martin Christensen
A

7. Martin Christensen

8. Martin Christensen

9. Martin Christensen
• Incremental development
• Incremental and iterative development

10. Martin Christensen
User research 101

11. Martin Christensen
User research 101 (measuring for success)
= Learnability issue
”It took me a while to
understand what
to do. That’s because
I’m stupid.”
Do you imagine that most people in the business
would learn to use this app quickly? Rate it on a
scale from 1 to 7, please.
Surely not. Tell me again
why it took a while to
understand.
I did not know what to do
when I opened the app.
There was no clear way
for me to proceed. Great. I understand.
Thanks!
Well, it was hard
for me, so 3.
This is the user
This is you

12. Martin Christensen
User research 101 (measuring for success)
Ease of use
Effectiveness
Efficiency
Satisfaction
Sense of being
in control
Capability
Efficacy
Conviviality
Consistency
Simplicity
Learnability
Usability (mean value)
1
2
3
4
5
6
7

13. Martin Christensen
Summary
• A/B-testing requires a large sample.
• A/B-tests can’t be changed or corrected on-the-fly without
compromising the test.
• A/B-testing does not answer ”why”, only ”what”. The rest is
your assumptions.
• You can not A/B-test if you only have A, or nothing at all.
• Simple user research and A/B-tests complement each other.
• Do enough research to understand.
• Go to the users, shut up, ask why, listen really carefully.
• Dazzle your boss with a nice chart.

14. Thank you!
Bygg klokt med LeanUX
20 maj 2014
Eftersom livet är för kort för att
bygga något som ingen vill ha
11 platser kvar
se.linkedin.com/in/martinuxchristensen/
martin.christensen@crisp.se
martin.ux.christensen
@m8rt
UX open 2014
24 oktober 2014
En knytkonferens inom UX
Super-Early Bird = 600 kr exkl. moms