Kristen Davis and Dick Horst from UserWorks presentation slides on the "Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not" from UXPA-DC Conference Redux 2013
Advantages of Hiring UIUX Design Service Providers for Your Business
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt
1. Notetaker's Perspective During Usability
Testing:
Recognizing What's Important, What's Not
UXPA-DC Conference Redux 2013
October 25, 2013
Kristen Davis ● Dick Horst
kdavis@userworks.com
●
dhorst@userworks.com
1738 Elton Rd., Suite 138, Silver Spring, MD 20903
(301) 431-0500 www.userworks.com
2. Presentation Objectives
To explore ways to become a more efficient/effective notetaker
by:
•
•
•
•
•
Developing notetaking and test planning skills
Recognizing what’s important to note and what’s not
Understanding the pros and cons of various styles
Determining what metrics to capture and how to define them
Discussing a list of best practices and tips
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
2
3. Why is Effective Notetaking Important?
To improve the data analysis process by:
•
•
•
•
Capturing the appropriate data the first time
Making analyses more efficient
Increasing accuracy and completeness of data
Identifying trends across participants
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
3
4. Typical Context
One-on-one, moderated sessions (participant in-person or
remote)
Moderator interacting with participant, notetaker elsewhere
Participant attempting typical task scenarios at the direction of
the moderator
Participant thinking aloud
Some usability test principles and best practices may apply to
taking notes in focus groups, user interviews, or ethnographic
settings, but not all
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
4
6. Misconceptions
No preparation needed
Just take notes
Anyone can do it
Fast typing is only key skill necessary
One-size-fits-all approach
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
6
7. Set Goals
Be familiar with the product under test
Understand overall study goals
Understand specific task goals
Set goals prior to data collection
•
•
Record appropriate data
Target type(s) of data
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
7
8. Notetaking Styles/Approaches
Determine appropriate style
•
•
•
Verbatim (court stenographer) – rarely ideal
High level interpretation only, not behavior or comments
Coding behaviors in addition to free form notes
Determine number of notetakers
•
•
•
•
Moderator is only notetaker – not ideal
Multiple notetakers looking for different things
Multiple notetakers covering the same things (redundancy)
Taking notes from playback of a video/audio recording (not live)
Determine data collection tools
•
•
Checklist (word processing or spreadsheet based, or paper-based)
Automated tool (e.g., Morae, Ovo, Silverback)
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
8
9. Data Collection
Define task completion rating scale
•
•
Participant performance to be categorized
Systematic and consistent scoring
Notetaker can quickly record
•
•
•
•
Task scoring (success vs failure)
Errors, wrong paths
Pre-identified issues of interest
Occasional verbatims
Data logging software records
•
•
•
Clickpaths
Time on task
Shorthand codes for behaviors or incidents
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
9
11. Study Context
Product: NIHSeniorHealth.gov website
Participants: Seniors who look for health information online
Study goals:
•
•
•
What navigation path do participants use when looking for specific
pieces of information?
Do participants notice and use the pagination?
Does the visual treatment used in left hand navigation clearly
indicate location within specific health topic?
Task goals:
Are participants able to locate information on the 2nd page?
Are the left hand navigation labels clear?
Task: “Why do people with diabetes need to check their feet?”
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
11
14. Study Context
Product: HealthIT.gov website
Participants: Private physicians
Study goals:
•
•
•
Are navigation labels clear?
Does information architecture make sense?
What suggestions do participants have to improve content?
Task goal:
•
What navigation path do participants use when looking for specific
pieces of information?
Task: “Medical professionals and hospitals are encouraged to
participate in the Government EHR reimbursement programs
early to receive the maximum payment. Find out when the
Government will start or has started these reimbursement
payments.”
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
14
17. Study Context
Product: Blood glucose meter
Participants: People with diabetes who use glucose meter
Rating scale for scoring task completion:
•
•
•
•
Completed with ease: Participant easily completed the task
Completed with minor difficulty: Participant struggled somewhat
to complete the task
(e.g., attempted more than two paths)
Completed with major difficulty: Participant significantly struggled
to complete the task
(e.g., attempted numerous paths and/or used customer service)
Failed to complete: Participant was unable to complete the task
Task: Change meter default settings
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
17
20. Pitfalls
Participant says one thing and does another
Participant thinks they have been successful when they really
haven’t
Participant doesn’t want to look bad, so bluffs
Participant initially struggles, then succeeds, and speaks
highly of product
Participant blames self and rates product highly despite
disastrous task performance
Participant is overly chatty and goes off on tangents
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
20
21. Considerations
Being selective in information noted, without introducing biases
(seeing what you want/expect to see)
Noting participant behavior versus comments
Whether to try to capture participant’s clickpath, menu choices,
data entry
Whether to capture timing
Whether to use a shorthand “code” for observations
Whether to capture participant’s non-verbal behavior
Whether to note good clips for a highlights video
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
21
23. "Take-Aways" for Effective Notetaking (1 of 2)
Ensure the project team values role of notetaker
Choose someone with appropriate education and experience
Include notetaker in test planning discussions and project
debriefs
Establish project-specific notetaking procedures and coding
schemes
Be sure notetaker is thoroughly familiar with:
•
•
•
•
Product
Project goals
Task scenarios
Task goals
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
23
24. "Take-Aways" for Effective Notetaking (2 of 2)
Focus on what participant does, not just what they say
Just don’t note observations; interpret what you are seeing
Review your notes and fill in any gaps as soon as possible
after sessions
Consult with the team and client throughout project
Check available storage space on recording devices before
each session
Spot check recordings throughout study in-between sessions
Discuss findings with notetaker/moderator after each session,
if time allows
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
24
25. Questions/Comments
Kristen Davis
Dick Horst
UserWorks, Inc.
UserWorks, Inc.
Senior User Experience
Specialist/ Note-Taker
Extraordinaire
President/
Principal User Experience
Specialist
kdavis@userworks.com
dhorst@userworks.com
www.userworks.com
(301) 431-0500
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
25
Notas del editor
Not much focus put on notetaking as with moderating during observed research. Based on our practical experience with have learned some tips for notetaking.
Kristen: Today we will explore ways to become more efficient and effective note-takers by…[read bulleted list]We will share some of our lessons learned, but we are also here to learn from your experiences, approaches and methods for note-taking.This presentation will use video clips to facilitate a discussion to explore note-taking best practices and tips. We will conduct a series of short exercises wherein we will show a series of video clips from usability tests. You will take your own notes while watching these short video clips. After each clip, we will lead a discussion of what was important and what wasn’t important to note about the test participant’s performance and/or comments during that clip. We will show four video clips. Before each clip there will be a brief setup commentary, providing the context of the clip and the test that it illustrates. After each clip, there will be a brief discussion to share ideas and thoughts about what you noted, given the contextual information that was provided. We hope to illustrate the skills and planning necessary for effective note-taking, and discuss the pros and cons of various styles. We hope to decide if it is possible and worthwhile to even strive for developing principles, which could then be codified and taught. We want to make this an interactive discussion because note-taking is a skill better learned by doing rather than being lectured to, similar to moderating.By a show of handsHow many have conducted usability tests?How many have been the note-taker?
Kristen: So why is important to be a good notetaker?[After reading bulleted list] All of these help the report basically write itself.Second sub-bullet:Making analyses more efficient by categorizing observations on fly
Kristen:
Kristen: Let’s begin by discussing the planning phase of the project and the necessary steps to play for notetaking.
Second bullet: Just take notes, then figure out how to analyze and data summarizeLast bullet: One-size-fits-all approach will work for all practitioners and for all products under evaluation
First bullet: Important to be familiar with the product and understand the overall goals of the study and the specific goals of the tasks
The method of notetaking impacts the type of data collected during the evaluation and target the type of data that will be useful when summarizing and reporting the findingsDepending on the evaluation objective, one style or a combination of styles may be more appropriate
Identifying what constitutes an assist during the planning phase also allows the team to distinguish the difference between “completing a task with assistance or help” from “completing the task with difficulty.” An “assist” implies the participants would not have completed the task on their own without intervention from the moderator; whereas “completed with difficulty” implies the participant completed the task on his/her own, but struggled to do so, needing several attempts or did not fully understand how the system was working.Helps the notetaker:Know what to look forEfficiently categorize on the fly
Kristen: So let’s see an example were understand the study context and background would help improve notetaking.
Kristen: This is the same study, but a different information retrieval task…[After presenting the bullets] Now, understanding these goals, can anyone think of anything they would have done differently when you were taking notes during the last clip?
Kristen:Point out left menu on page and mention the pagination appears at the bottom of the page[Video Clip: SeniorHealth.gov 1B P1 30:07-32:29]The participantdid not immediately click into the correct section of the site, “Causes and Preventions”. He did use the pagination, but was unable to locate information on the 2nd page of “Causes and Preventions.” He also clicked on the “Symptoms and Diagnosis” link when he was already on that page.
Kristen: Now, let’s move on to the third exercise.
Kristen: Purpose of the site is to provide information to the general public and practitioners about the government’s initiative to increase the use of electronic health records or EHRs.
Kristen: [Video clip: HealthIT.gov]For this study we high level interpretation notetaking style. If we had tried to take verbatim notes, we wouldn’t have collected a lot of meaningful data as related to the study goals.
Kristen: Now, let’s move on to the fourth exercise.This time, we’ll be focusing on metrics. It’s important for all members of the research team to come to a consensus and define all metrics prior to data collection.
Kristen
Kristen: [Glucose meter study: P1 29:50-32:18]We scored this task with a 2 (minor difficulty) because the participant tried two different menu items before reaching the correct item, and as defined in the rating scale, that would be a 2. By a show of hands, how many also gave this participant a 2…But, as you can see, that task would have received various ratings had this scale not been established before data collection began.
Dick
Dick
Dick
Dick
Review your notes and fill in any gaps as soon as possible after sessions while observations are fresh in your mindConsult with the team and client mid-course to be sure your notes reflect project goals and team intentSpot check recordings throughout study to ensure that everything is working as intended