Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Maximizing your moderating_game
Similar a Maximizing your moderating_game (20)
Último
Último (20)
Maximizing your moderating_game
- 1. Maximizing Your Moderating Game! Susan Mercer! Senior Experience Researcher! smercer@madpow.com! @susanamercer!
- 2. What is Usability Testing?! • It is NOT:! • Market Research! • The same as a Focus Group! • About user preferences! • “Fluffy”! • It is:! • A systematic way to observe user behavior! • A method to discover usability issues! • A ongoing field of scientific inquiry! !! 2
- 3. Topics! • What is think-aloud?! • Your roles as a moderator! • 5 rules of great moderating! • Keeping sharp! ! 3!
- 5. Think-aloud Protocol! • Most commonly-used usability testing method! • 1 participant / 1 moderator! • Participant does tasks and thinks aloud! • Origins in Cognitive Psychology! Original protocol:! • To understand short-term memory! • Moderator invisible observer! • Only comments: “Keep talking”, etc.! • Don’t interrupt short-term memory! 5! Source: Ericsson & Simon, 1980!
- 6. Think-aloud Protocol - Problems! • Many practitioners feel the strict no- comments moderation is not ideal:! • Makes participants feel uncomfortable! • Feels “unnatural” not to respond! • Participants may get stuck and not be able to continue! 6!
- 7. Think-aloud Protocols! Original Think-aloud! Usability Testing! Study this…! Study this…! …to understand this ! Short term …to understand this ! memory! System usability 7!
- 8. Modified Think-aloud Protocol! • Based on Conversational Theory! • Set up a conversation! • Participant is main speaker! • Use short, neutral response phrases (continuers) ! • “Mhmm”, “Uh-huh”, “And now…?” ! • Utter phrases like questions ! • Ask for clarifications by repeating a phrase! • Participant: “That was odd…”! • Moderator: “Odd?”! • Provide hints if necessary ! • Neutrality is key! 8! Source: Boren & Ramey, 2000!
- 9. The Reality of Think-aloud! • Many usability professionals get sloppy:! • Ask leading questions! • Participant: “I would share this article with friends.”! • Moderator: “By printing it?”! • Ask closed-ended questions! • Moderator: “Would you print this?”! • Focus on known problems for “ammunition”! • Use biasing response phrases “Good job”, “Yes, that’s a problem.”! ! 9! Source: Norgaard & Hornbaek, 2006!
- 10. The Reality of Think-aloud! • Current practice goes beyond neutrality and includes probing questions! • Some evidence that probing may influence what usability problems are discovered! • Evidence is only suggestive; more research is needed! 10! Source: Krahmer & Ummelen, 2004!
- 11. What does this mean?! • Current moderating practices are not consistent! • Moderating is often taught through mentorship; so bad habits can easily be perpetuated! • Moderators should be willing to examine their practices and adapt if needed! • Therefore, we need to pay attention to our moderation and remain neutral. !! 11!
- 12. Your roles as a moderator!
- 13. Your Roles (Yes, plural)! • Gracious Host! • Guide! • Neutral Observer! • Curious Learner! • Technical Support! ! 13!
- 14. Structure of a Usability Session! Intro! Pre-Test Tasks! Post-Test Wrap-Up! Questions! Questions! Gracious Host! Guide! Curious Learner! Neutral Observer! Technical Support! 14!
- 15. Structure of a Usability Session! Intro! Pre-Test Tasks! Post-Test Wrap-Up! Questions! Questions! Gracious Host! Guide! Curious Learner! Neutral Observer! Technical Support! ?! ?! ?! ?! ?! Challenge: Navigate the Transitions Smoothly! 15!
- 16. 5 Rules of Great Moderating!
- 17. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 17!
- 18. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 18!
- 19. 1. Know Your Goals! • What are you trying to achieve?! • Formative ! • Summative ! • What topics are highest priority?! ! These will influence! • Where you focus! • What you skim when time runs short! • How you fine-tune your moderating style! 19!
- 20. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 20!
- 21. 2. Stay Neutral! • Be Quiet!! • Don’t insinuate they gave a wrong answer! ✘ “Why did you do that?”! ✘ “What made you think to click that link?”! • Don’t put ideas into their heads! ✘ “Does entering your SSN make you feel uncomfortable?”! ü “What do you think about entering your SSN?”! • Use their words! • If they call the dashboard a “chart area”, then call it a “chart area”! • Keep your tone naturally curious, not derisive! • “What did you expect here?”! 21!
- 22. 2. Stay Neutral! • Avoid closed-ended (Yes/No) questions ! • Don’t start with a verb! ✘ Did you see that?! ✘ Was that difficult?! ✘ Is this frustrating?! 22!
- 23. 2. Stay Neutral! • Use Open-ended questions! • Start with What, When, Where, Why, How! ü What would you change to make this easier?! ü When in this process would you be ready to buy?! ü Where did you expect to find that information?! ü Why did you say this was easy?! ü How does that error message make you feel?! 23!
- 24. 2. Stay Neutral! • Use neutral, “continuing” response phrases! • Acknowledge that they gave feedback! • Don’t imply whether it is good or bad! ✘ “Oh”, “Hmm”, “Interesting” - Assessing! ✘ “OK”, “Yeah”, “That’s good” - Agreeing! ü “Uh huh”, “Mhmm”, “Tell me more” - Continuing! 24! Source: Boren & Ramey, 2000
- 25. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 25!
- 26. 3. Be Responsible! You have responsibilities to:! ! • Participant! • Company! • Future Users ! • Study Integrity! 26!
- 27. 3. Be Responsible! • Informed Consent:! 1. Voluntary! 2. Comprehension! 3. Disclosure! • The purpose of the study! • Any reasonably foreseeable risks! • Potential benefits ! • Notification of recording and observers! • Confidentiality protections ! • Ability to withdraw at any time without penalty! 27!
- 28. 3. Be Responsible! Situations happen during testing.! You may need to juggle your responsibilities.! ! What are your priorities?! ! • Study Integrity! • Company! • Future Users! • Participant! 28!
- 29. What to do?! If…! • Participant doesn’t want to be recorded! • You don’t have time to ask every question! • The participant rants about the website and calls it “crap”! 29!
- 30. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 30!
- 31. 4. Be Confident! • Project confidence! • If you make a mistake, keep going! • Don’t over-apologize! • Pause if you need to collect your thoughts! • “Fake it until you make it”! 31!
- 32. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 32!
- 33. 5. Be Genuine! • Don’t play the notes on the page…! …play the music! ! • Don’t just read the questions from the page…! …have a conversation! 33!
- 34. 5. Be Genuine! • LISTEN! • Listen and respond! • Stay in the moment! • Don’t worry about past ! • Or the future! • Create a conversation! ! 34!
- 35. 5 Rules of Great Moderating! 1. Know your goals! 2. Stay neutral ! 3. Be responsible! 4. Be confident! 5. Be genuine! ! 35!
- 36. Keeping sharp!
- 37. Keep Improving! 1. Identify your improvement areas! • Watch your videos! • Have colleagues give you feedback! 37!
- 38. Keep Improving! 2. Learn from others! • Watch others moderate! • Be a participant! • Listen to talk radio interviews! 38!
- 39. Keep Improving! 3. Practice skills in everyday life! • People watch (observe) in public! • Be quiet and listen in everyday conversations! ! 39!
- 40. The most important thing is…! PRACTICE! • No one is perfect the first time! • Do your best and continually improve! • You’ll be awesome in no time!! 40!
- 41. Questions? 41!
- 42. Thank You! (Slides will be posted to Slideshare..follow me on twitter for link) Susan Mercer! Senior Experience Researcher! smercer@madpow.com! @susanamercer! 42!
- 43. References / Photo Credits! Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.! Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.! Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.! Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.! Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.! Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An exploratory study of think-aloud testing. DIS 2006, 209-218.! ! ! ! Slide 4: Samantha Louras Photography ! Slide 5: flickr: torgeaux! Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian! Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls! Slide 32: flickr: bberburb! Slide 34: flickr: Tulane Public Relations! Slide 35: flickr: apdk, flickr: marktristan! Slide 38: flickr: Raphael Quinet! Slide 39: flickr: l-i-n-k! Slide 40: flickr: loppear! ! ! 43!