3,5 hour workshop for CoveyUX “User checks” is an agile way of usability testing with the focus on creating value. With User Checks a design accelerates to a higher level within a very short period and relatively low cost and little resources. User checks maximize the key element of usability testing: getting to empathy. User Checks is closely related to the RITE method: Rapid Iterative Testing and Evaluation.

1. userneeds! @anous
User Checks!
Method for agile usability testing
February 9, 2016
3,5 hour workshop
Anouschka Scholten
userneeds

2. userneeds! @anous
@anous
#ConveyUX

3. userneeds! @anous
What you will learn
You get to know User Checks –
A method for agile usability testing
You experience important facets of the
User Checks method – learning by
doing

4. userneeds! @anous
UX insights!

5. userneeds! @anous
Quantitative methods
What happens
Where
How often
WHY
How better
?

6. userneeds! @anous

7. userneeds! @anous
Make people’s
lives better.

8. userneeds! @anous

9. userneeds! @anous
Focus
group
Enquête
Verify
test
Remote
tracking
Usability
testing
User
driven
Marketing
driven

10. userneeds! @anous
People do not always do what !
you think !
you tell them !
they think !
they say !
they do!
!
!
Observing and asking why makes you
find out what people really !
do and need!

11. userneeds! @anous
Usability testing
is about
observing behaviour
Watch the user do what comes naturally,
while performing a task (thinking aloud).
Don't help.
Not asking about opinion.

12. userneeds! @anous
empathy!
a key function of usability testing
to really see, hear and find out
what moves your users
the ability to see an experience
through another person’s eyes
Getting to

13. userneeds! @anous
Usability ROI
• Increased conversion rates
• Increased productivity
• Reduced need for help desk
intervention
• Reduced fall off rate at checkout

14. userneeds! @anous
No usability testing
• Customer dissatisfaction
• Lose of potential income
• Losing money on marketing
• Higher development costs
• Bad reputation
• Staying behind the competition

15. userneeds! @anous
user checks!
& traditional
usability testing!

16. userneeds! @anous
1
User Check setup

17. userneeds! @anous
17

18. userneeds! @anous
Usability Test traditional
1 round of testing"
In a lab environment, eye tracking
Moderator is a UX researcher
Process based

19. userneeds! @anous
Traditional Usablity test
drawbacks
• 1 time, at end design process
• An extensive report, little or no redesign
• No retest
• Not in user context
• High costs: lab, eye tracking, special UX
researcher
• Product team indirectly involved
• Schedule well in advance (at least 1 month)

20. userneeds! @anous
http://uxpajournal.org/wp-content/uploads/pdf/JUS_Romano_Bergstrom_November_2011.pdf

21. userneeds! @anous
Test and retest within 1-3 weeks
Preferably in the users context, quick and easy set up
Team member (Designer, PO,…) conducts the test, team
directly involved
Output is a better design/ better MVP"
Value based
User Checks Agile Usability Test

22. userneeds! @anous
User Checks Agile Usability Testing
6. Evaluate
Priortize
Improvements
2. User
Checks
round 1
3. Evaluate
Priortize
Improvements
5. User
Checks
round 2
1. Design /
MVP
4. Improved
Design / MVP
7. Improved
Design /
MVP
Repeat
or develop
Based on the RITE-model: https://uxmag.com/articles/the-rite-way-to-prototype

23. userneeds! @anous
0. Prepare

24. userneeds! @anous
Focus the test, informed by data and user
feedback. Make a testplan
What is the goal, what do you want to
improve for whom, where do you want to
find out the WHY
Recruit real users!!
User Checks elements
0. Prepare

25. userneeds! @anous
1. MVP or Prototype

26. userneeds! @anous
The test subject, live or prototyped in
design phase.
With focus on problem area.
User Checks elements
1. The MVP or Prototype

27. userneeds! @anous
User Checks round 1

28. userneeds! @anous
Observe 2 or 3 real users, attempt real
task(s);
Conduct a simple usability test per user,
preferably in context: user check
1 moderator + 1 observer
(UX) designer, PO or other team member
as moderator and a a team member as
observer
User Checks elements
2. User Checks round 1

29. userneeds! @anous
3. Findings & improvements

30. userneeds! @anous
What have you learned? Prioritize.
What are improvements? Brainstorm and
prioritize.
With the team, together
Via KJ-method and Impact/effort matrix
http://www.uie.com/articles/kj_technique/
http://www.innovationgames.com/impact-
effort-matrix/
User Checks elements
3. Findings & Improvements

31. userneeds! @anous
4. Improved design > 5. User Checks Round 2

32. userneeds! @anous
Improve the prototype or MVP
Do a 2nd round of user checks
With the next 2 or 3 real users (not the
same users as round 1!)
User Checks elements
4. Improved design
> 5. User Checks Round 2

33. userneeds! @anous
6. Findings and improvements > 7. Improved design

34. userneeds! @anous
Again with the team:
Collect and group findings, prioritize.
Think of design improvements, prioritize
and implement in better design
Use a backlog for improvements that take
more time
Repeat the process or implement design
User Checks elements
6. Findings & improvements
> 7. Improved design/MVP

35. userneeds! @anous
examples!

36. userneeds! @anous
Design new site ggdhaaglanden.nl
User check with a doctor in his office
GGD Haaglanden project Angi Studio:"
http://angistudio.com/cases/ggd-haaglanden.html

37. userneeds! @anous
GGD Haaglanden project Angi Studio: http://angistudio.com/cases/ggd-haaglanden.html
User Checks during completely new design
I do not know
what OCGZ
stands for…
Is this for
me?
Off course SOA and
child abuse, very
important indeed!

38. userneeds! @anous
“LINDA.Girls..
I expect this is for 14 till
21 year old… but oh,
reading this
subscription I think it
actually does fit me?”
Pop-up information about LINDA.meiden
Project Angi Studio: http://angistudio.com
Redesign
LINDA subscription

39. userneeds! @anous

40. userneeds! @anous
“Very clear and easy to walk through
this way, underneath each other”
Sub home Subscription with incentive -
mobile
Scrolling was no problem at
all on mobile
Footer with next step was used
Navigating via main menu on top
wasn’t used, it was ignored, even
with a sticky menu… dead end alert!
Project van Angi Studio: http://angistudio.com/
2e round

41. userneeds! @anous
User Checks during redesign Greenchoice.nl, project Angi Studio:"
http://angistudio.com/cases/greenchoice.html

42. userneeds! @anous
Subsidie aanvraagsysteem Gemeente Den Haag, project Angi Studio:
http://angistudio.com/cases/gemeentelijk-subsidie-loket.html
User Checks during completely new design

43. userneeds! @anous
Compared
Cost
+/- 10.000 EUR
+/- 5.000 EUR
Test participants
6-8 once
2 rounds with 3 participants
Who performs
UX researchers/specialists
only
Product team (designers, product
owner, developers)
Where
Lab
In context user or meeting room
How often
Once
Twice or more
With
1 MVP/design
2 or more variations of MVP/
design
Equipment
Laptop/mobile, Eytracking,
video
Laptop/mobile, audio- and screen
capture
Plan
At least 1 month upfront
Recruit (target) users and conduct
You get
Report, 1-2 weeks later
A better design and retest in 1-3
weeks
Engaged team
Tradit. usability tests User checks

44. userneeds! @anous
Outsourcing your user research work
is like outsourcing your vacation.
It gets the job done, but probably
won’t have the effects you were
seeking.
Jared Spool
http://www.uie.com/brainsparks/2011/08/02/outsourcing-your-user-research-is-like-outsourcing-your-vacation/
“

45. userneeds! @anous
“Elaborate tests are a waste of resources”
Source: http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
75% of the problems found with 3 or 4 user tests

46. userneeds! @anous
User checks drawbacks
• Too unexperienced biased designer or team
member as moderator
• Heavy pressure on improving the design within
agile process (keeping in pace with development)
• With ‘high fidelity’ testing it demands velocity for
retest with high multidisciplinary efforts: product
owners, IxD, visual design and content
And with a live site: front end, back end…

47. userneeds! @anous
Get down to it!

48. userneeds! @anous
The 4 basics of a 1 on 1
usability check
1. Let your user experience the prototype/
MVP/site
Observe don’t tell. Put your prototype in the user’s hands
(or your user in the prototype) and give just the minimum
context so they understand what to do. Give them a task.
2. Have them talk through their
experience (talk aloud)
For example, when appropriate, as the host, ask “Tell me
what you are thinking as you are doing this.”

49. userneeds! @anous
The 4 basics of a 1 on 1
usability check
3. Actively observe
Watch how they use (and misuse!) what you have given
them. Don’t immediately “correct” what your user tester is
doing.
4. Follow up with questions
This is important; often this is the most valuable part of
testing. “Show me why this would [not] work for you.”
“Can you tell me more about how this made you feel?”
“Why?”
Answer questions with questions (i.e “well, what do you
think that button does”)

50. userneeds! @anous
Key: moderating the test
If they…
Say…
Are not talking
“What are you thinking?”
Ask you a question (e.g. “Is that
what I should do here?”)
Rephrase the question (e.g “ What do
you think you should do?”)
Get a task right or wrong
“Thank you , that is very helpful”
“Thanks for the feedback”
Mess up
“Remember, you can’t make any
mistakes. You’re doing a great job.”
Are unsure if they have
completed a task and ask you
“Is this what you would do if you were
doing X at home?”
Criticize the design
“Thanks for the feedback”

51. userneeds! @anous
Observe the user’s
behaviour
The observer(s) note every occasion the user:
• hesitates, worries, or is forced to think
• misunderstands something
• gets frustrated or annoyed
• gives up
• is surprised

52. userneeds! @anous
User Checks set up
• Test script with real tasks
• Real target users as participants
• Ideally conducted in context user on their own
device
• Record audio and screen capture
• 1 test moderator, 1 observer
(more observers are possible, but not too much
and really on the background)

53. userneeds! @anous
Test script-agenda
1. Introduction and questions up front
2. Situation sketch + think aloud instruction
3. Think aloud session
4. Evaluate session, open questions
5. End session - thank participant

54. userneeds! @anous
1. Example Introduction
Thank you for participating, we really appreciate it!
I’m <name> and there is one person besides me <name> is also here.
We’re helping a company to develop a new service / to optimize a website
This company cares about their customers and thinks it’s very important
to involve potential customers when developing new services. They want
the new service to be valuable for you.
That’s why we want to ask you some questions and along the way we’d
like show you the service/site. You can walk through it.
The interview takes about a 20 minutes. If you do not understand
something, or you want to add something, do not hesitate to interrupt me!
Do you mind if we record the session for future reference within our
organisation? Everything we discuss and record is confidential; we will not
share things without your permission!
Any questions?

55. userneeds! @anous
Task-think aloud instruction
You are going to see: a draft proposal of the service, which has
not yet been developed/ the site now.
<if a proptotype> The proposal is only images. It looks finished
and you can click through it, but most of it is not working yet.
It’s also not as fast as a normal app and you can’t do anything
wrong.
<Situation sketch and task>
Please open the link on your mobile phone and walk through
the app.
Can you please say things aloud? Not to me, but for
yourself? Everything you do or say is fine, eg. Also "what the
hell is this ?! “.
You can start now by clicking the link

56. userneeds! @anous
Evaluation questions
After the think aloud session, zoom in to what you observed, like:
• You mentioned… what did you mean with that?
• I saw you doing… why? What did you expect? How would you call
it?
• You didn’t go to … why not? What do you expect to find?
Also ask service, product or content specific questions, e.g.:
• You read this text thoroughly, why?
• You were going back and fourth between product X and Y, why?
• You choose X and said something about .., what did you mean
with that ?

57. userneeds! @anous
a case!

58. userneeds! @anous
Volunteers
• 2 moderators
• 2 users

59. userneeds! @anous
Tell us
• The main goal(s) of the site/application
• Target users
• Main tasks, main questions of the users
• Context of use
Do not tell us how you think the site works now!!!

60. userneeds! @anous
Observe the user’s
behaviour
Note every occasion the user:
• hesitates, worries, or is forced to think
• misunderstands something
• gets frustrated or annoyed
• gives up
• is surprised

61. userneeds! @anous
1st user check session!
20 min
!
2nd user check session!
20 min
!
Findings & improvements!
20 min
!

62. userneeds! @anous
The findings, issues
• Collect Findings
• Prioritize issues to be
solved: prioritize from the
business/user goals
perspective, rate them
Do this with thewhole team

63. userneeds! @anous
Improvements
• Improvements on every level, note down
Labeling, content, categorization, visual assets,
interaction, concept, flow, persuasion > but keep the
business/user goals perspective in mind (focus)!
• Prioritize by Impact & Effort (matrix) or dot-
vote
Choose improvement(s) to be re-tested
Do this with thewhole team

64. userneeds! @anous
Impact & Effort matrix
Impact: creates high/low value for users
effort: high/low effort in terms of time, knowledge, tools, people…
Source: http://www.innovationgames.com/impact-effort-matrix/
Do this with the
whole team
incl. decision
makers!

65. userneeds! @anous
Prepare to retest
Improve the design
Set up the hypothesis (or more than one, focus for the retest)
We believe that
[this statement is true (e.g adding social proof, better
labeling, visual cues… will…)]
We will know we are right/wrong when we have the
following result
[quantitative feedback; qualitative feedback; KPI;…}

66. userneeds! @anous
A better design
fast!

67. userneeds! @anous
With User Checks
• Checking with users on a regular bases
Effective problem detection
• Ensures that users can successfully reach their goals and shows where
their satisfaction can be improved
Boosts optimalization
• Offers insights about the proposition or concept weaknesses and
unknown opportunities
Enables innovation
• Ensures the whole team understands how users use their product
Empathy and commitment
• Emphasizes where a design change is needed, reduces waste
Brings focus, avoids feature creep

68. userneeds! @anous
A holistic focus on a
good UX!
Understand!
Find out!Build!

69. userneeds! @anous
Be outrageous
efficacious and !
make people’s lives
better!
!
Just do it!!

70. userneeds! @anous
Thank you!
anouschka.scholten@userneeds.nl
www.userneeds.nl
Twitter: @anous
LinkedIn: https://nl.linkedin.com/in/anouschkascholten