The document provides guidance on end user testing, including its purpose and key principles. It discusses testing roles like moderator, note taker, and observers. It outlines the testing process from briefing to task-based testing. Key things to watch for include leading questions, asking design questions, feeling too opinionated, and using technical terms. Overall the document aims to help make user testing insightful for improving product design and usability.
2. “User experience is really the whole totality.
Opening the package… good example. It’s
the total experience that matters. And that
starts from when you first hear about a
product… experience is more based upon
memory than reality. If your memory of the
product is wonderful, you will excuse all sorts
of incidental things.”
(Don Norman)
3. Usability Principles
(Jakob Nielson)
• learnability: How easy is it for users to complete basic tasks the
first time they encounter the design? (ex. long press vs. tap)
• efficiency: Once users have learned the design, how quickly can
they perform a task?
• memorability: When users return to a design after a period of not
using it, how easily can they reestablish proficiency?
• errors: How many errors do users make, how severe are these
errors, and how easily can they recover from these errors? (ex.
Does an error message occur? Does it tell the user what to do next? Does it
say what is wrong? )
• satisfaction: How pleasant is it to use the design?
4. Testing Intent
• With all tests you want to discover whether the user:
• Gets the point of the page(s)
• Understands the navigation system
• Can guess where to find things
• In a general test you want to know:
• How do users interact with the device they are testing?
• What is difficult for people to do?
• Where do they get lost?
• What makes sense to them?
• What makes them feel distrustful or insecure?
• What do they like and what do they hate?
• In a specific test you might want to know:
• Can the user accomplish a key task?
• Can the user find something specific?
5. What is the difference?
(functionality vs usability testing)
Usability testing Functionality testing
• Ease of use • Does it work
• Ease of navigation
• Look and feel
6. Testing Roles
• Moderator:
• Person who is guiding the user through the test
• Note Taker:
• Person who is observing (silently) and gathering data
• User:
• Person who is testing the device
• Observers:
• Person or group of people behind a one-way mirror observing the session
7. Test Moderator
• Is friendly
• Makes the user feel comfortable
• Encourages user to comment out loud
• Stays objective and detached
• Watches the user do what comes naturally.
• Sees what the user does alone, without helping
• Knows when to stop a task, but does not rescue user
too soon
• Gives encouraging, non-committal feedback
• Summarizes findings with user
8. Test Note Taker
• Silent observer
• Takes note of user interaction with device. Notes when user:
• hesitates, worries, or is forced to think
• misunderstands something
• gets frustrated or annoyed
• gives up
• Also notes:
• Answers to questions from moderator
• Length of time to complete task
• Difficulty in completing a task
• May ask moderator to ask user additional questions at the
end of the session (if necessary)
9. Observers
• Silent observation of the test session
• Who should be involved?
• Stakeholders
• Designers
• QA Team members
• Executive team
10. Reasons to Observe
• Credibilty: visibility to the test as it occurs vs. belief results are made up/
personal preference or opinion
“That's a big credibility booster. It's not just you saying it, but there's actually someone
else who's saying it as well.” (Jakob Nielson)
• Memorabilty: it’s easier to remember things you have witnessed than to
have glanced through a report
• Empathy: visibility to real end user, rather than test team or a pseudo “user”
• Fewer design mistakes: when design team sees the end user, they are
more likely to use design that works
11. Test Steps
• Briefing with observers/note taker/moderator
• Prepare testing environment
• Greet user
• Interview user
• Distribute any necessary paperwork (legal waiver)
• Conduct test
• Q&A with user
• Summarize findings with user
12. Greeting the User
• Introduction:
• Moderator will put the user at ease
• Try general topics, such as discussing the weather
• Introduce user to self (moderator) & the note taker
• Answer any questions the user might have beforehand
• What to say:
• We are evaluating the device, not you
• You are helping the design team understand what is not working
• You can take a break or stop at anytime during the testing
13. Background Interview
• Things to ask:
• Experience with similar products
• Has the user used a tablet before?
• Has the user used a mobile device before?
• If yes, how often do they use these devices?
• If yes, what do they use these devices for?
14. Task Based Testing
• moderator:
• Give user a task to complete
• Check for understanding of instruction
• Reassure user that the product is being tested, not them
• Watch the user complete task without interference.
• Respond to silence, unclear answers
• Answer user questions at the completion of the task
• note taker:
• Take note of user behavior / interaction with device
15. Sticky Issues
(Things to watch out for)
• leading questions:
• some users may say what they think you want them to say
• asking design questions
• users will know where there is a problem, not how to design the solution.
“Everyday people are not very good designers.” (Donald Norman)
• feeling attached or opinionated:
• be unbiased. All answers are valid and valuable
• using technical terms
• examples:
• bugs vs issues
• device vs product
• launch vs start
16. Still curious?
(recommended reading)
• Don’t Make Me Think by Steve Krug
• The Design of Everyday Things by Donald A. Norman
• iPad Usability Testing Report by Nielson Nelson Group
(slideshow)
• Uxbooth.com
