Slides from my course "Usability Testing and QA" taught at BAVC 7/18/2014

1. Usability Testing & QA
Here early? While we'ʹre waiting for the others...
1. Identify some websites you'ʹd like to test
Best if they are likely to be new for your classmates.
If you have a site (or sites) that you work on, even better!
2. For each website you'ʹve chosen, identify 3-­‐‑5 key tasks
What are the 5 most important things a user should be able
to accomplish on the site?

2. Usability Testing
and QA
A practical approach
towards creating better user interfaces

3. Who am I?
Shilpa Thanawala
@skthana

4. And you?
Name
Role at your company or agency
Background (eg. designer? developer? etc.)
Experience with usability testing
How you think you’ll use what you learn today
Why this class?

5. Housekeeping
Everyone knows where things are at BAVC?
Cell phones on silent, calls outside
Initial and sign the roster
Evaluations at the end of the day
Load the shared Google Doc
(for questions, backchannel interactions, sharing links)
http://bit.ly/ux-­‐‑july-­‐‑18

6. What is Usability Testing?

7. Usability testing refers to evaluating a product or service by testing it
with representative users. Typically, during a test, participants will
try to complete typical tasks while observers watch, listen and takes
notes. The goal is to identify any usability problems, collect qualitative
and quantitative data and determine the participant'ʹs satisfaction
with the product.
usability.gov
Image credit: templatemonster.com

8. Usability testing is a technique used in user-­‐‑centered interaction
design to evaluate a product by testing it on users.
wikipedia.org
Image credit: Mediamatic

9. Usability is a quality attribute that assesses how easy user interfaces
are to use.
nngroup.com
Image credit: Optimum7.com

10. First, some historical
highlights

11. Early 20th Century
Improvements in industrial efficiency
In WW I, reduced work motions for soldiers to assemble &
take apart weapons in the dark.
http://www.measuringusability.com/blog/usability-­‐‑history.php

12. 1947
John Karlin -­‐‑ how short phone cords should be

13. “It is not so much that Mr. Karlin trained
midcentury Americans how to use the telephone.
It is, rather, that by studying the psychological
capabilities and limitations of ordinary people, he
trained the telephone...”
Excerpt from New York Times article

14.
1980’s
New methodologies developed and published
Usability becomes a profession
Defined as a function of efficiency, effectiveness, and
satisfaction
first publishedPsychology of Everyday Things
1990’s
Usability matures
Diminishing returns from testing more than 3-­‐‑5 participants
Books by Jakob Nielsen

15.
2000 -­‐‑ today
Steve Krug’s Don’t Make Me Think
Remote testing
Statistical analysis of usability data
Mobile devices, touch and gestures
http://www.measuringusability.com/blog/usability-­‐‑history.php

16. Why?

17. Why spend valuable time and money doing
usability tests?
It'ʹs the best way to learn how users actually interact with
your website.
Watching > interviewing
You don'ʹt really know your users'ʹ needs. Users are not all the
same. And they'ʹre not your Mom.
All websites have problems, and the serious ones are easy to
find.

18. Why spend valuable time and money doing
usability tests?
You'ʹll learn things you didn'ʹt know that you didn'ʹt know.
Watching users gets everyone on the same page.
Watching users will make you a better designer / developer /
product manager / content writer / etc.
Usability testing doesn'ʹt have to be expensive or time-­‐‑
consuming.

19. What Testing Can'ʹt Tell You

20. Limitations of Usability Testing
Won'ʹt fix your site'ʹs problems (it'ʹs not a user training session)
Results are dependent on the questions you ask (tasks)
Can'ʹt model social interactions very well
Isn'ʹt always representative of real situations
For more on issues testing the social web see Dana Chisnell'ʹs talk

21. When?

22.
When should usability testing be
done?
Before you design or build anything
Before you implement a change
Throughout the project
After implementing a fix
Throughout the life of the website
As soon as possible & repeat often!

23. Demo: A short usability test

24. Qualitative vs. Quantitative

25.
Qualitative vs. Quantitative
Quantitative Usability Studies
Designed and conducted to produce data for statistical analysis
Each measurement represents a count or an amount
Distances, weights, quantities, etc.
Qualitative Usability Studies
Focused on verbal descriptions of users'ʹ experiences
Each measurement is a description or category
Words, sentences, feelings, yes/no
Categories are not quantitative, even if they are assigned
numbers
userfocus.co.uk

26.
“...qualitative methods are much better suited for answering
questions about why or how to fix a problem, whereas quantitative
methods do a much better job answering ‘how many’ and ‘how much’
types of questions.”
More on Quantitative methods at
"ʺWhen to Use Which User Experience Research Methods"ʺ, nngroup.com
measuringusability.com

27. Usability Metrics

28. Effectiveness, Efficiency, and
Satisfaction
Effectiveness
Accuracy, correctness. Are users able to achieve their goals?
Efficiency
Ease of use, speed. How much effort (time) does it take to
complete a task?
Satisfaction
Users'ʹ perception. What does the user think about their
experience?
More at , usability.gov wikipedia.org/wiki/Usability

29.
Some Metrics for Common User Tasks
From Measuring the User Experience – Tullis & Albert

30. Performance-­‐‑based Metrics
Task Success
Commonly used, pass / fail, or degrees of success, should
have clear end-­‐‑state.
Task Time
Particularly important for repetitive tasks
Errors
Incorrect actions leading to significant time loss, additional
costs, or task failure
Efficiency
Effort and time needed to complete the task
Learnability
Effort and time needed for the user to learn how to use the
interface

31. Issue Metrics
Usability findings based on users'ʹ behavior
Consistent / repeatable, or idiosyncratic?
Remain open-­‐‑minded to the unexpected

32. Other Metrics
Self-­‐‑Reporting: Users'ʹ perceptions, feelings
Behavioral / Psychological: Eye-­‐‑tracking, stress,
unprompted verbal expressions
Combinations / Comparisons: Interpretations based on
more than one metric
Card-­‐‑Sorting: How to organize information in a way that
makes sense to users.
A/B Testing: Comparison of two alternate designs
Accessibility: Usability for those with disabilities ( )WCAG

33. Bias
Impossible to eliminate entirely
Participant background, knowledge, comfort level
Task definitions
Methodology (testing process, session length, how much
talking)
Artifacts of the prototype or product
Environment (lighting, noise, cameras, distractions)
The Facilitator (experience, skill, style)
Expectations
Measuring the User Experience – Tullis & Albert

34. Number of Test Participants

35. What'ʹs the right number of users to recruit for
a study?
3-­‐‑5? 4-­‐‑6? 10? 20? 100?
Those who favor a small number feel 80% of the most
important usability problems are detected by the first few
users.
Cases in which more users may be required:
You need to be sure you'ʹve captured as many problems as
possible
You have more than one user group (persona)
You have a large number of screens, complex design, wide
variety of tasks
You'ʹd like to account for evaluator bias

36. Why we'ʹll focus on qualitative testing today
Anyone can do it
No knowledge of statistics or data analysis needed; based on
verbal feedback, observation, and empathy
Cheap, fast, and easy
Great for small budgets, busy teams, minimal resources... and
frequent repeats
It gets the job done
More than sufficient to identify a website'ʹs most serious
usability problems

37. Break
10 minutes

38. DIY Usability Tests
Based largely on Rocket Surgery Made Easy – Steve Krug

39. Establish your Goals
What do you want to learn about your site?
Can users find products easily?
Is it clear to users what the site is about?
Is the information organized intuitively?
Is this new feature distracting?
Keep an open mind to allow for unexpected results.

40. Define your Tasks
Create scenarios based on essential, real-­‐‑world user goals
Tasks should be realistic
Tasks should be actionable
Phrase tasks so as not to give away the answer
Print each scenario on its own piece of paper.
(Avoid labels or numbers.)
"ʺTurn User Goals into Task Scenarios for Usability Testing"ʺ — nngroup.com

41. Recruiting Test Participants
What kinds of participants?
Where to find them?
How many?
How to compensate them?

42. What kinds of participants?
How representative of actual users?
Most serious UI problems will be discovered by non-­‐‑
representative users.
Testing with non-­‐‑representative users is better than obsessing
over finding representative users.
Key requirements: not part of the team, fluent in the UI'ʹs
language

43. Where to find participants?
Where do your users hang out?
Advertise on the site, message boards, social networks
Craigslist

44. How many users?
For most of us doing DIY testing, 3-­‐‑5 participants is fine.
No-­‐‑shows or last-­‐‑minute cancellations are common. Have 1-­‐‑2
backup participants lined up.
Phone-­‐‑screen, scheduling, day-­‐‑before reminders, directions,
parking, NDA'ʹs... use a checklist

45. Compensation
Shows appreciation for participant'ʹs time and effort
Encourages participants to be engaged and enthusiastic.
Gift cards, free product or service.

46. Equipment & Setup
Hint: you don'ʹt need a lab
Bare minimum (if you'ʹre the only one): a computer in a quiet
room, a pen and notepad, and 2 chairs
For a team, add a remote observation room, a good mic, and
screen-­‐‑sharing (Skype or GHangout). And provide
irresistable snacks.
Can use screen-­‐‑capture software (but most of the time it
won'ʹt be watched)
Can record the participant (but it'ʹs not really needed)

47. Make it a Team Effort
Try to get everyone to attend.
Assign a trustworthy team member to manage the
observation room

48. The Session Script
Start with a template & edit to suit your needs
Welcome / introduction
Logistics, release forms
Necessary background information
Test: general questions
Test: scenario activities
Wrapup, follow-­‐‑up questions
Conclude and provide honorarium

49. Some Test Session Script Templates
Free templates at: , , infodesign.com Steve Krug'ʹs site Book site
guide by Rubin, Chisnell & Spool

50. Debrief
Only for those who attended at least one session
Schedule it on the same day (next day latest), 60-­‐‑90 min.
Provide food!
Get all observers to state (or stick) UX problems they
observed
Get general agreement on which are most serious & will be
tackled first (2-­‐‑3 maximum)
Remind everyone when the next round of testing will take
place

51. Run your own usability test
1. Select a website to test
2. Determine goals
3. Identify at least 3 tasks
4. Formulate scenarios for testing each task

52. Lunch
1 hour

53. Run your own usability test
1. Prepare your script
2. Rehearse and refine
3. Print scenarios

54. Run your own usability test
1. Test two participants (~20 min each)
2. Take notes

55. Run your own usability test
1. Debrief within your original group
2. Identify 2-­‐‑3 usability issues

56. Tools
Testing session checklists
, ,
Usability testing software
, ,
Online / Remote services
, , ,
Usability.gov Krug'ʹs checklist checklist on book site by
Rubin, Chisnell & Spool
Silverback (Mac) Morae (Win) Camtasia
Usertesting.com Loop11 many more usabilityhub.com

57. So how do we fix it?

58. Fixing usability issues
The bad news: most issues never get fixed
Make the smallest change possible to "ʺstop the bleeding"ʺ
Large changes might introduce new issues
It'ʹs not fixed until you verify by ... testing again!

59. Testing Accessibility

60. Why is accessibility important?
“The Web is an increasingly important resource in many aspects of
life: education, employment, government, commerce, health care,
recreation, and more. It is essential that the Web be accessible in order
to provide equal access and equal opportunity to people with
disabilities. ”
From The W3C'ʹs Web Accessibility Initiative

61. Is my site accessible?
Unplug your mouse / turn off your trackpad
Why -­‐‑ blind, low-­‐‑vision, compromised motor-­‐‑control, no
hands
How to test -­‐‑ start at url, use tab / shift-­‐‑tab and arrows to see
whether you can access all controls, links, and fields on the
page, in the right order. Popups and dialogs should be
usable.
How to fix -­‐‑ proper HTML heirarchy, tabindex attribute,
javascript plugins

62. Is my site accessible?
Check color contrast
Why -­‐‑ low-­‐‑vision; 1 out of 12 have color deficiency
How to test -­‐‑ online tools ( , ),
browser plugins ( )
How to fix -­‐‑ shift color palette to improve contrast, increase
font sizes for low contrast areas, avoid colorful background
images
Check my Colors Contrast Ratio
Grayscale Tool

63. Is my site accessible?
Turn off images
Why -­‐‑ images useless to low-­‐‑vision or blind users, low
bandwidth
How to test -­‐‑ block images in browser, WebAIM tool
( )
How to fix -­‐‑ proper alt attributes, don'ʹt use images when text
can be used
WAVE

64. Accessibility Resources and Tools
Quick accessibility tests & fixes
WebAIM
W3C'ʹs Web Accessibility Initiative
SitePoint'ʹs Easy accessibility checks

65. Mobile / Touch

66. Mobile Usability
Context / environment (noise, lighting, glare)
Many more contexts beyond desktop
Bandwidth
Mobile devices force us to think about bandwidth again
Touch target size
Average fingertip size = 10mm
Adjustments in spacing, button size
Touch is also being used on large screens
Driving / attention
What usability issues contribute to driver distraction?

67. Mobile Usability Testing Resources
NNGroup recommendations
Article on UXMagazine

68. Break
10 minutes
(back at 3:50)

69. A/B Testing Activity

70.
A Usability Test using Paper
Prototypes
Paper prototype of a kids’ website
Credit: BlueDuckLabs / YouTube.com
View online

71. A/B Testing Activity
1. Select one usability issue from your previous activity
2. Brainstorm at least two ways to fix it
3. Print the page and use the paper, scissors, tape, and pencils
to mockup both cases
4. Test each one on a separate participant
5. Debrief in your group. Which worked better, A or B?

72. Accessibility Testing Activity
1. Click the URL at the top of the browser window
2. Without using the mouse again, try using only the keyboard
to navigate
3. Can you access all the links, menus, and controls?
4. Can you tell where you are?

73. Wrap-­‐‑up
Usability testing is an essential tool for improving your
website or product
Anyone can do a simple usability study
Test as early as possible, as frequently as possible
Tailor the type of testing you conduct (qualitative /
quantitative, metrics, number of participants) to give you the
information you'ʹre after
Use the available tools and resources to help you run your
sessions

74. Wrap-­‐‑up
Get your whole team involved
Small adjustments are preferable (and faster) than a full
redesign
Remember to test your fixes, too
Stay aware of non-­‐‑standard users (accessibility) and the
changing technological landscape (new platforms, new uses)

75. Resources
Websites / Blogs: , ,
,
Quantitative Testing: ,
,
nngroup.com usability.gov sensible.com
(Krug) UIE
measuringusability.com Measuring the
User Experience – Tullis & Albert usability.gov
Thank you!
Shilpa Thanawala | @skthana