Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio. Learn all you need to get started: - Where you can conduct studies (does it have to be in a lab?) - Types of studies (RITE, think aloud, etc.) - Tips for recruiting participants - Tips for Interacting with participants without biasing the study - Preparing for the study (materials needed, forms, etc.) - Guidance for analyzing the study

1. Web SIG ClevelandMay 21, 2011 Usability Testing Basics: What's it All About?

2. Talking About Today #WebSigCLE @carologic @sos_jr

3. Agenda What is Usability? Why we test usability Planning studies Facilitating sessions Analyzing data

4. What is Usability?

5. Usabilityis an important characteristicof what makes a good User Experience

6. Functional Aspects Effective Efficient Learnable

7. User’s Perspective Useful experience Feel in control and supported Supplements and enhances skills and expertise Satisfied Delighted

8. Usability Testing Measures users ability to achieve specific goals of effectiveness, efficiency, and satisfaction.

9. What is it? Real Users doing real tasks Being observed Using prototypes or live products

10. Can Test… Websites, Mobile, Blenders, Airport service Simulations or mockups Early prototypes (paper, low-fi) Production prototypes (html, hi-fi) Help documentation Processes (receipt of materials, purchase)

11. It is not… Quality testing Full accessibility testing System testing Acceptance testing

12. Do I need a lab? Computer / Concept Participant Facilitator Observer Timer Logger Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.

13. Just Do It! Anywhere (conference room, remotely) Any Stage (earlier in process the better) Anytime (un-moderated) Realistic test environment Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/

14. Why we test Usability

15. "The biggest waste of all is building something no one wants" - @ericries #LeanStartupMI via @MelBugai

16. You are not the user Designing for someone else Need to step back - may miss details May be perfect for you, but not the user Honest feedback from users Validate understanding of tasks and context Unforeseen requirements

17. Rationale Goals being met Content & purpose clear Match expectations Verify product meets customer needs Gather information for future product development Comparison testing

18. Save Time & Money Up front by testing prototypes Reduce maintenance issues Reduce Customer Service requests

19. Minimize Human Cost Tiredness Discomfort Embarrassment Frustration Effort

20. Find Design Problems System status available Wording choices clear Placement of content Consistency Recognition, Not Recall

21. Benefits of Good UX Increased Usefulness Increased Efficiency ($$$) Improved Productivity

22. Benefits (continued) Fewer Errors Reduced Training Time Improved Acceptance Happy Users!

23. Planning studies

24. Define Goals Specific Measurable Qualitative Quantitative What you need to learn about the product and its audience

25. Scope Effort Consider budget, resources Time Recruiting Facilitating Analyzing Adding participants increases budget & time

26. Design the Test Select Methodology Based on goals Identify participants Tasks to be completed Team roles

27. Types of Usability Tests Single participant Co-discovery (two) Group usability testing (3 or more) Rapid iterative testing

28. Facilitation Styles Talk aloud/Think aloud Task focused (limited/no discussion) Cooperative usability testing Video review with participant after study (Retrospective)

29. Location Formal Lab On-site Workplace, conference room In home At a conference Remote Moderated Un-moderated

30. Test Roles Facilitator Participant “Computer” (paper prototypes) Note taker / data logger / timer Software, recording operator Product / Technical Expert(s) Observers

31. Use A Script/Guide Memory tool for facilitator Promote consistency Questions Order of questions List out scenarios representative of typical tasks

32. Test Guide Includes Welcome to participants Steps in study(forms, tasks, questions) Notes to yourself Reset/configuration prompts Thank you to participants (incentives if any)

33. Participant Materials Provide with Usernames Passwords Text for forms Images to upload, etc. Tasks in writing if complex (3x5 cards)

34. Task Building Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.

35. Questions Quality of questions correlates to quality of answers: Don’t lead or make assumptions Use participant’s words

36. Stretch & Exercise 36

37. Question 1 Do you regularly book your travel online to save money?

38. Alternates – Question 1 How often do you travel? <listen> What proportion of that do you book online? <listen> Why do you book travel online? <listen>

39. Rationale - Question 1 Do you regularly book your travel online to save money? Address one issue at a time and avoid double-barreled questions.

40. Question 2 What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel?

41. Alternates – Question 2 Would you like to correspond with a travel agent while you are booking travel? <listen> What are some ways that you would like to correspond with a travel agent while you are booking travel? <listen>

42. Rationale – Question 2 What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel? This question asked the participant to predict the future.

43. Schedule & Recruit Schedule location for pilot and study Leave time between sessions Sessions no longer than 2 hours Invite Observers (key stakeholders, project managers, etc.)

44. Pilot Study – Find Problems Verify Tasks are typical Concept is on-track Time needed to complete Practice before going live with participants New ideas for follow-on questions or things to observe Refine script and tasks

45. How Do I Find Participants?

46. Create a Screener List of questions to determines who will participate Describe, then get details: Computer activities Use of product/service People who pass screener = user group

47. Which Student? Rick Connie http://www.flickr.com/photos/mrjkbh/ via http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en http://www.flickr.com/photos/caharley72/ (Christopher Alison Photography) via http://creativecommons.org/licenses/by-nc-nd/2.0/

48. Representative Users Two weeks on average to recruit Primary user population People with disabilities “We are all only temporarily able-bodied. Accessibility is good for us all.” Get to spirit of the law (Section 508, WCAG 2.0) -@mollydotcom at #stirtrek 2011 via @carologic

49. Hire a Recruiter Allows you to focus on activity. Can tell if person will be a good participant. May already have a list they can start with. Good recruiters: find right participants. give regular updates. take care of directions, confirmations, incentives, etc.

50. If You Must Do it Yourself... Go where users go and intercept Online user groups Professional organizations Craigslist Online tools thru your site:http://ethnio.com

51. Final Recruiting Final recruit by phone. Ask questions that force them to talk. Don’t recruit non-talkers. Confirm participant eligibility Note unusual issues Recruit for pilot test

52. Number of Participants As many as possible (rarely statistically significant) Usability Testing Research (in 1990’s) 5 from distinct sub-group of the user population will yield 80% of the findings (Nielsen, Virzi, Lewis) Assumes expert has reviewed concept for obvious issues Recommend: Early tests with 8 – 12 users per user group Iterative testing (3 per day, iterate, 3 new users) Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface. http://www.stcsig.org/usability/newsletter/0301-number.html Retrieved: 20080323

53. Honorarium Pay them or give product credits Amount varies by: Amount of time needed Their role (doctors need more to persuade them) Their interest, devotion to product

54. Facilitating sessions

55. Welcome & Prepare Offer beverage Express appreciation for help Explain purpose of research Sign paperwork Consent Form Non-Disclosure Agreement(s)

56. Participant Reassurance Make sure they are comfortable Not testing them, rather testing… Encourage feedback (positive and negative) “I was not involved in the design of this so you can’t hurt my feelings”

57. During Session Remain passive (body, face) Don’t confirm or reject answers Use participant’s words Listen for vocalizations Watch non-verbal gestures Encourage participant to elaborate Ask your question and let them talk

58. Silence is GoldenUser’s Time to Think! 58

59. Get Them Unstuck Progressively give assistance What are you trying to do right now? What do you think the next step is? What would you do in this situation if you were at work? Hints Do you see anything that might help you? (small hint) Have you checked the Help? (medium hint) What do you think the xxx button does? (large hint)

60. Debrief Participant Post-Study questionnaire Thank for participation Explain honorarium and delivery

61. Analyzing data

62. Look for Patterns Identify repetition Continuation of study Adds cost Delays reporting Low probability of many new findings

63. Measurements Success Time on task % of tasks completed/not completed Number of steps to accomplish task Learning time Number of errors Number of times help consulted Satisfaction

64. Transform Data Create Findings and Recommendations High level or detailed report Think about audience How will it be used?

65. Reporting Includes Executive summary Positive findings Provide solutions for negative findings Provide level of effort and prioritize Examples, screen shots Quotes Appendix with questionnaires, test materials

66. Do UX Early & Often Put it on the Wall as information radiators Test findings Artifacts Competitor info

67. Recommended Readings 67

68. Contact Carol J. Smith @carologic carol@mw-research.com http://www.mw-research.com

69. References Cato, John. User-Centered Web Design. Addison Wesley Longman; 2001. Hackos, JoAnn T., PhD and Redish, Janice C. User and Task Analysis for Interface Design. Wiley; 1998. Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003 UPA Conference. (Activity) Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability. Kuniavsky, Mike. Observing the User Experience: a Practitioner's Guide to User Research. Morgan Kaufmann, 2003. Nielsen, Jakob and Robert L. Mack. Usability Inspection Methods. John Wiley & Sons, Inc. 1994. Redish, Janice (Ginny). Letting Go of the Words: Writing Web Content that Works. Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc.

71. Lab or on-site?

72. Prototype limitations (can it be online?, is it a document or a clickable site?)

73. Number of observers, number of participants?

74. Number of facilitators?

75. Logging and video editing needs (time on task, highlight video creation)?

76. Surveys before or after?

78. Ovo

79. SilverBack (Mac only)

80. UserWorks

81. Noldus

82. Tobii (Eye-tracker)

83. SMI (Eye-tracker)

85. Satisfaction Questionnaires Standard Usability Measurement Inventory (SUMI) office/desktop software, purchase 50 questions Website Analysis and MeasureMent Inventory (WHAMMI) Purchase 20 questions System Usability Scale (SUS) Free 10 questions

86. Recommended Sites Usability.gov W3C Web Accessibility Initiative http://www.w3.org/WAI/ Accessibility Standards in US (Section 508) http://www.access-board.gov/sec508/508standards.htm Jakob Nielsen http://www.useit.com UPA – professional usability organization http://www.usabilityprofessionals.org/