Usability testing is by far the most widely used usability method. Nonetheless, it's often conducted with poor or unsystematic methodology and thus doesn't always live up to its full potential. This panel will present a lively discussion about a number of controversial beliefs about usability testing and discuss if they are truths or myths.

1. UXPA Conference Panel, London 2014:
"Remote, unmoderated
testing is as reliable as
lab-based testing" — and
other controversial beliefs
about usability testing
Rolf Molich
David Travis
Jakob Biesterfeldt
Steve Krug

2. Presentation of panelists

Jakob Biesterfeldt
• Managing Director of
UserZoom Germany,
provider of UX research tools
• 12+ years of UX research
experience
• Focus on professionalization of
UX research

3. Presentation of panelists

David Travis
• Managing Director of Userfocus, a
London-based UX consultancy
• Author of ‘E-Commerce Usability’
• PhD in Psychology
• Worked as an internal and external
usability consultant since 1989
• Logged over 1500 hours in usability
tests and trained over 500 students
in usability testing

4. Presentation of panelists

Steve Krug
• Author of Don’t Make Me Think
and Rocket Surgery Made Easy
• Usability consultant for 25 years
• Has taught thousands of people
to do their own usability testing
• Thinks people rarely change
their minds about anything

5. Presentation of panelists

Rolf Molich
• Owner, manager of DialogDesign,
a small Danish usability
consultancy
• Managed the Comparative
Usability Evaluations (CUE-1 to 9)
• 30+ years of usability experience
• Focus on Quality and
Professionalism in usability work

6. Please use a few minutes to mark your position on each of the 10 statements

7. 7
Procedure
To keep the discussion lively and cover as many
statements as possible, the procedure will be:
► A statement is presented
► Attendees vote for or against
► The supporting panelist speaks for at most 2 minutes
► The opposing panelist speaks for at most 2 minutes
► 1-2 comments from the audience
Please queue at the microphone. Limit yourself to 30 secs
► Attendees vote again

8. 1
Truth or myth?
Remote, unmoderated testing is
as reliable as lab-based testing
Agree
Both agree & disagree
Disagree

9. 2
Truth or myth?
Five users are enough to catch
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree

10. 2
Truth or myth?
Five users are enough to catch
85% of the usability problems in
practically any product
Agree
Both agree & disagree
Disagree

11. ”The parabola of optimism” (Jared Spool)

12. 12
Real data: CUE studies
CUE = Comparative Usability Evaluation
► 10-17 professional teams test the same website.
► In all CUE-studies, more than 60% of problems
were uniquely reported.
► In CUE-4, more than 300 issues were reported.

13. Five users are
enough to drive a
useful iterative
cycle

14. 3
Truth or myth?
The moderator should say as little
as possible while the test
participant is working with the test
tasks
Agree
Both agree & disagree
Disagree

15. 3
Truth or myth?
The moderator should say as little
as possible while the test
participant is working with the test
tasks
Agree
Both agree & disagree
Disagree

16. 4
Truth or myth?
Report only the 3-5 most serious
problems to the client. Tell them
that you would be happy to give
them more problems to work on,
but ONLY after they had fixed the
most serious ones.
Agree
Both agree & disagree
Disagree

17. 5
Truth or myth?
It's simple to identify disastrous
usability problems
Agree
Both agree & disagree
Disagree

18. 18
Watch two video clips
From Comparative Usability Evaluation 9 (CUE-9)
► Usability test of U-Haul.com
► Participants have been asked to rent a moving truck
► Participants select rental location, then damage
coverage
► Look out for potentially disastrous usability problem

19. 6
Truth or myth?
Observers should be told to keep
quiet if loud sounds from the
observation room can be heard in
the test room
Agree
Both agree & disagree
Disagree

20. 7
Truth or myth?
Usability testing is an iterative
process - one-off tests are useless
Agree
Both agree & disagree
Disagree

21. 8
Truth or myth?
Users have valuable insight into
why they struggle
Agree
Both agree & disagree
Disagree

22. 9
Truth or myth?
Showing picture-in-picture of the
participant’s face is an
unnecessary distraction for
observers
Agree
Both agree & disagree
Disagree

23. 10
Truth or myth?
If you've found 90 serious or
critical problems in a usability test,
it's OK to report them all in the
usability test report
Agree
Both agree & disagree
Disagree

24. Evaluation
Did this presentation change some
of your views of usability testing?
Yes
Don’t know
No
11

25. 25
Main Takeaways
► As usability professionals we must set a good example for
the world. Our products, presentations, doings must be
usable and useful: Correct, relevant, brief, constructive
► Five users are enough to drive a useful iterative cycle.
Never claim that you can find all usability problems
► Moderators should say as little as possible while the test
participant is working with the test tasks in order not to
provide inadvertent clues
► The usability of a test report is influenced by the number
of problems reported