The goal of this presentation is to give attendees a deeper understanding of usability testing so they can leverage it in their own work. The material will shed light on what is important to the research buyer and will help the research provider to better understand how to plan, moderate, and report on a usability study. It will also provide information on where they can go to learn more about this very practical qualitative method. Kay will cover what a usability test is and when to use it, the key planning steps, the language around it, and the unique insights this method produces. She will also discuss the various approaches a market researcher can take when running a usability study at different points in a product’s development (e.g., concept, early prototype, released product).

1. What qualitative researchers should
know about usability testing
QRCA - New York City Chapter
Presenter: Kay Corr y Aubrey
July 20, 2012

2. Background

3. 333
Agenda – what you will learn
1. This presentation will be an “appetizer” on usability
testing (1 hr 15 minutes)
• Overview of how to plan, run, analyze & report on
a usability study*
• Pointers to where you can learn more
2. Hands-on demonstration
• We’ll usability test the Android, iPhone, & iPad
versions of the new QRCA VIEWS app (45
minutes)
3
You will learn the process of usability testing, its language,
and when to use it in a qualitative research project

4. 444
What is a usability test?
• Qualitative study where typical users try to accomplish
typical tasks on their own with the product
• Point is to see how clearly the product “speaks” to
them, meets their expectations, fits into their typical
work and task flow
• Moderator & team watch participant working and keep
score of task success & failure, comments, body
language

5. 555
What can be usability tested?
• Web sites, software applications
• Consumer products (e.g., vacuum cleaners, ovens, mobile
phones – examples from projects I’ve done to illustrate the
range of where you can apply this technique)
• Packaging
• Customer service or ordering procedures
• Training & documentation
• Basically you can usability test any product or service
where there is user interaction

6. 666
Some Good reasons for running a
usability study
• Make interaction with a product as fluid & intuitive as possible
• Avoid embarrassment – expose usability problems
• Test design concepts
• Compare design approaches
• Challenge assumptions
• To compare your product with a competitor’s
• Improve ease of use and learning
• To better understand users
• To understand training and documentation needs
• To increase sales, improve your product’s reputation, decrease need for
technical support
• To save money and time (less need to rework the design, fewer calls to
customer support)

7. 777
Where does usability testing fit in with other
qualitative research methods?
• Similar to in-depth interviews, focus groups, and
ethnographies. Listen to participant, note their
body language, learn their work and task flow,
needs, attitudes, and perceptions
• Different from other forms of qualitative research
because goal is to watch participant perform tasks
against a product design and keep track of success
and failure

8. 888
Where does usability testing fit in with
other qualitative research methods?
Shown with permission

9. Stages of a usability test

10. 101010
Major phases of a usability study
1. Planning the study
2. Running the study
3. Analyzing results
4. Reporting results
A typical soup-to-nuts usability study takes about 70 hours

11. Planning a usability study

12. 121212
Planning the study
• Determine study objectives
• Identify target user characteristics, write recruiting
screener, oversee participant recruiting & scheduling
• Create usability study plan and task list*
• Make sure prototypes & other stimuli are ready
• Manage logistics
*Bolded items are unique to usability testing

13. 131313
Usability objectives should focus on a
specific aspect of the product design
Aspects of an interactive product
design that affect usability
• Are the features, functions,
and content appropriate?
• Does the organization and
navigation align with the user’s
task and work flow, their
“mental model”?
• Do users understand labeling
& terminology?
• Do they know where to start,
what to do next, and the range
of available choices?

14. 141414
Need an analysis framework for an
interactive product

15. 151515
Sync testing approach to product’s
development stage
Test method Project stage Focus
Paper prototype Early Overall product concept,
terminology, navigation
Electronic prototype Design Task flow, visual design, page
layout, specific features,
validate redesign
Functioning product Development & QA Defaults, online help, feature
integration, performance
Comparison Post-release Product features, performance
benchmarks, can be within
your own product or against
competitor’s

16. 161616
Venue choices for a study
Venue Pros Cons
Facility Can invite lots of observers,
fewer logistical headaches,
can test wider range of
products
Artificial environment, costs more
Conference room Saves money - no travel, can
test wider range of products
More logistics, artificial environment,
observers want to sit in same room as
tester
Live online More natural (participant is
in their environment, saves
money, no travel, fewer
logistics, easy for team to
observe
Can only test Web-based products, hear
voice but don’t see body language,
need to recruit more tech savvy
participants, firewall issues
Native habitat (mobile studies) Person is in context of use -
people use mobile phones
when they are “on the go”,
cheap & quick
Cannot easily record, only have 1-2
observers

17. 171717
Tips on recruiting for a usability study
• Usability studies typically involve only a few participants (8-10)
• Choose confident and articulate participants who care about
the product category
• Aim to recruit people with varying levels of experience
• Define the required foundation skills
• Make sure participants understand what it means to be in
usability study

18. 181818
Task list development
EXAMPLE TASK LIST
“You are the owner of a family-owned business that has just deployed the
Acme Web Conferencing product. Today you are working from home but need
to hold several meetings with your managers and employees. For the first
time, you will be using your iPad to participate.”
• Task 1 – Your Administrative Assistant, Kathleen, has set up an Acme Web
conferencing meeting for you. Go to your e-mail and find the invitation.
• Task 2 - Join the conference.
• Task 3 – Enter your information: Chris Doe, DoeChris173@myemail.com,
987-123-9876.
• Task 4 - Enter the meeting room.
• Task 5 - Once you are in the meeting room, connect your phone to the Web
conference so you can hear what is going on.

19. 191919
Planning a study is very involved
Create project plans and check lists to keep your ducks in a row.

20. 202020
Forget a step and you are dead
Just kidding

21. Moderating a usability study

22. 222222
Running the study
Sessions follow a structure that is similar to any qualitative
research session:
• Moderate sessions – steps are the same
• Keep track of participants’ task success and failure
• Note comments, facial expressions, and body language
• Hold debrief discussions with participants and observers
• Administer surveys between tasks and after session has ended
During a usability study you are watching people’s behavior while
listening closely to what they say

23. 232323
What type of data do you collect?
• Collect objective results (can they do it? Define up front what
“success” means for each task)
• Emotional reactions
• Practical information (how does this product design fit into their
world? What kind of training would a person need to be
productive with this technology?)
• Typical measures (task success/failure, time on task)
• When they struggle, note why
• You see patterns after 3-4 sessions but new stuff always
emerges (depending on diversity of participant backgrounds)

24. Analyzing results

25. 252525
Analyzing the study results
• Keep score of task success/failure by participant
and task
• Identify trends – What was easy? What was
hard? How did the product design play into these
results?
• Find verbatim comments and video clips that
encapsulate key findings
• Debrief continuously with clients – gaining
consensus on issues is critical to your success

26. 262626
Use affinity diagramming to help team
reach consensus on issues

27. Reporting results

28. 282828
Writing the report
• Use a conventional approach to structuring the report
(e.g., executive summary, methodology, findings)
• Group findings by product features and functions
• Where possible, tell the story with numbers (e.g.,
number of participants who succeeded at each task)
• Layer the information – place summarized information
in the report, and details in a spreadsheet

29. 292929
Example of a usability test report

30. 303030
Look for opportunities to collect
quantitative data
Product Reaction Cards
 Participants quickly select 5
attributed from among 118 choices
 The attributes are balanced between
positive and negative
System Usability Scale
 Participants answer 10 questions on
key aspects of usability
 Survey produces a score between 0
and 100; a score <60 is considered
poor

31. 313131
Seek results that can be expressed in
pictures
Source: Moxie Software http://www.moxiesoft.com/tal_products/employee-spaces-does-design-matter.aspx

32. 323232
Product Reaction Card results add
human interest
Text size indicates number of times that attribute was chosen

33. 333333
System usability Scale (SUS) scores cut
to the chase
Product ABC Product XYZ
0 0 0 0 0 0 0
10
0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71
1
0
1
0
1 1
3 3
0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71
SUS scores < 60 indicate poor usability. See the study spreadsheet for details.
XYZ’s’ SUS scores from all
10 participants were
between 75 and 100

34. 343434
Reporting – each discipline expects
specific insights from the study results
Executives want a distilled version, an encapsulation of “the problems” and to
understand what is perceived as the premium this product offers
Product managers and marketing people seek insights on segmentation,
product identity, competitive information, participant reaction to feature sets
Product designers want detailed usability feedback to guide refinements to
the product’s interface and behavior. Is the design intuitive? How does it fit
into the user’s work and task flow?
Engineering needs to understand the usability bugs so they can prioritize
them and fix them (often their input is the bottom line)
Training and documentation people want to know which content to include in
their work

35. QRCA VIEWS
for Android
and iPhone
Hands-on exercise
Usability testing a
mobile app

36. 363636
Hands-on exercise – usability test QRCA
VIEWS mobile app
STEPS
1. Translate the objectives into a task list
2. Break into groups of 2-3 people, one person is the administrator
another the participant, others are observers who will take notes
3. Run the study
4. Observers will record impressions on colored sticky notes (each
group will have its own color)
5. We’ll reconvene and do a mini affinity diagramming session to
tabulate and discuss the results and recommendations for
improvement. We’ll hold a debrief.
6. Kay present our results to Laurie and Eddie
7. Then we’ll pat ourselves on the back!!!

37. 373737
Additional resources
• Review a bibliography on user experience plus VIEWS and Quirks articles on
how to run a usability study
• Watch Camille Carlin and my QRCA QCast
• Sign up for my newsletter “Usability Tidbits”

38. The End
Thank You!!

39. 393939
About your presenter
Kay Corry Aubrey - UX researcher and designer
Kay Corry Aubrey is the owner of Usability Resources, which specializes in user-centered
research and design. Kay has over 20 years of experience in applying qualitative research
methods and usability testing to technology-oriented products and collaborative software.
She has led user research, usability, and design efforts for dozens clients including AT&T,
the Broad Institute, Affinnova, Constant Contact, Monster Worldwide, the Massachusetts
Medical Society, the Mayo Clinic, and iRobot.
Kay teaches usability and design for Northeastern University. She is a Managing Editor of
the QRCA VIEWS magazine, a market research journal. Kay has an MSW from Boston
University’s School of Social Work, an MS in information systems from Northeastern
University’s Graduate School of Engineering, and a BA from McGill University. She is a
RIVA-certified Master Moderator who enjoys doing research with both groups and
individuals.
Contact information
e-mail: Kay@UsabilityResources.net phone: 781-275-3020 www.UsabilityResources.net