In October 2013, I gave the industry element of the lecture for the User Experience Design module at Manchester Metropolitan University. To fit in with the overall programme, this covered thinking about objectives for an evaluation, goals, questions to be answered, choosing a method, arranging participants, and thinking about practicalities. It also touched on running a test, preparing students for future lectures covering specific methods.

1. Usability testing
Planning & running
Chris Collingridge (@ccollingridge)
22 October 2013
Manchester Metropolitan University
1

2. 1.0
2.0
3.0
4.0
Who are we and who am I?
What type of test?
Fitting it in
Planning and doing
Manchester Metropolitan University
2

3. Sage and me
Who we are and what we
do
Manchester Metropolitan University
3

4. Sage – Global
• 6 million customers
• 13,300 employees
• Major offices in UK, Ireland,
France, Germany, Spain, USA,
Canada, Austrailia, & Brazil
Manchester Metropolitan University
– Small business accounting
– Payroll
– Customer relationship management
(CRM)
– Taxation and accountancy
– Electronic payments
4

5. Sage – UK
• Only software company in the
FTSE 100
• 800,000 UK businesses use
Sage
• #1 in small business
accounting
• 1 in 4 people in the UK are
paid by Sage Payroll
Manchester Metropolitan University
5

6. Sage – Manchester
• Software for accountants in
practice
• On-premise and online
software
• 300,000 sets of company
accounts filed using Sage
each year
• 200,000 corporate tax
submissions
• 520,000 personal tax
submissions
Manchester Metropolitan University
– Final accounts production
– Corporate and personal taxation
– Practice management
– Time recording and billing
– Accountant/client collaboration
6

7. Summary
We’re big
Manchester Metropolitan University
7

8. Me
– Degree in economics (obviously!)
– Worked in a shop
– Decided there must be a career in
computers
…and mainly self-taught 15 years
later…
– Senior User Experience Specialist!
Manchester Metropolitan University
8

9. Day-to-day
•
•
•
•
•
•
User research – understanding the
problem
What do people know?
What are they trying to do?
Where do they do things?
What do they value?
What troubles them?
•
•
•
•
Interaction design – solving the
problem
Information architecture
User flows
Patterns
Low-level interaction (controls etc.)
•
•
Usability testing – evaluating
solutions
Manchester Metropolitan University
9

10. Day-to-day
•
•
•
•
•
•
User research – understanding the
problem
What do people know?
What are they trying to do?
Where do they do things?
What do they value?
What troubles them?
•
•
•
•
Interaction design – solving the
problem
Information architecture
User flows
Patterns
Low-level interaction (controls etc.)
•
•
Usability testing – evaluating
solutions
Manchester Metropolitan University
10

11. What type of test?
Manchester Metropolitan University
11

12. Formative vs. Summative
•
Summative
• Formative
“evaluation of a product with
“a type of usability evaluation that helps to
representative users and tasks designed
"form" the design for a product or service.
to measure the usability (defined as
Formative evaluations involve evaluating
effectiveness, efficiency and satisfaction)
a product or service during
of the complete product…the main
development, often iteratively, with the
purpose of a summative test is to evaluate
goal of detecting and eliminating
a product through defined
usability problems.”
measures, rather than diagnosis and
correction of specific design problems”
Ref: Usability Body of Knowledge
Manchester Metropolitan University
12

13. Formative vs. Summative
•
Summative
Analytics
Customers surveying and feedback
“Voice of customer”
Usability test = large scale,
expensive, scientific, resource
intensive, low ROI
Manchester Metropolitan University
•
Formative
Most of our usability testing is
formative
Blended with research
Often conceptual
Usability test = small scale,
iterative, pragmatic, high ROI
13

14. Fitting it in
Manchester Metropolitan University
14

15. Overall process
Plan
Prepare
Do
Analyse
Act
This is the usability
test bit
Manchester Metropolitan University
15

16. Planning
Manchester Metropolitan University
16

17. Fact*
96% of #### usability tests are
a result of poor planning*
* Entirely made up
Manchester Metropolitan University
17

18. Failure
•
•
•
•
•
•
•
Unclear goals
Test not addressing goals
Method produces unusable results
Too many variables
Unrepresentative participants
Inconsistent moderation
Inconsistent/incomplete
notes/recording
= a whole load of rubbish
Let’s plan!
Manchester Metropolitan University
18

19. Planning
Manchester Metropolitan University
19

20. My template for planning
Objectives
Goals
Questions
Method
Participants
Practicalities
Manchester Metropolitan University
20

21. 1 – Objectives
What will you be able to do as a consequence of this
test?
The point is always to stimulate valuable action
Example: Redesign the delivery elements of the shopping
cart
Manchester Metropolitan University
21

22. 1 – Objectives
Your objective is not a report
Manchester Metropolitan University
22

23. 2 – Goals
What will you know about or understand?
Examples:
• Know how people enter addresses for other people if
they’re sending gifts
• Understand how people react to different default delivery
prices
Manchester Metropolitan University
23

24. 3 – Questions
What specific questions will you have answers to?
Examples:
• When asked to enter the address of a friend or
sibling, where do they get the address from?
• Do postcode lookups help people enter addresses other
than their own?
• Are people less willing to continue if higher priced delivery
options are the default?
Manchester Metropolitan University
24

25. 4 – Method
How will you get answers to the questions?
Hint:
• Don’t think only about checking something
• Compare one thing to another (but control your
variables)
• Evaluate the perceived value of something
• Gain contextualise insight into something
Manchester Metropolitan University
25

26. 4 – Method
Some things to think about when defining your method
(1)
• Do you need to be able to see the user?
• Do they need to be using their own equipment?
• (PC vs. Apple, desktop vs. laptop, multi-screen, browser, etc.)
•
•
•
•
How much time will they have?
How much time do you have?
What equipment/software will you be able to use?
How much help might they need to access the software?
Manchester Metropolitan University
26

27. 4 – Method
Some things to think about when defining your method
(2)
• Where can you run it?
• What do you need to be able to measure?
• Time*, success rate, errors, comments?
• How will you avoid order effects?
• Are you able to access/build software to support the
tasks?
* Using “talk aloud” invalidates objective measures of time. But does it matter?
Manchester Metropolitan University
27

28. 4 – Method
Your requirements drive your method
Onsite
usability
test
In-person
usability
test
Remote
moderated
test
Or something else
entirely…
Manchester Metropolitan University
Remote
unmoderated
test
Survey
Tip: The answer almost certainly
is *not* a focus group
28

29. 5 – Participants
Who will you get to participate in your test?
Manchester Metropolitan University
29

30. 5 – Participants – How many?
Manchester Metropolitan University
30

31. 5 – Participants – How many?
5 is the classic answer, and it may or may not be right
for your situation.
• Practical experience is that you do start to see a lot of
repetition about here
• Repeating “like” tasks gives you a lot more data
• An explicit assumption is that you will iterate – this is not
a 1-time activity never to be repeated
Manchester Metropolitan University
31

32. 5 – Participants – Representativeness
“Just go out to a coffee shop, buy someone a
coffee, and sit down with them for 5 minutes”
– Almost everyone on every blog on the internet
Manchester Metropolitan University
32

33. 5 – Participants – Representativeness
Manchester Metropolitan University
33

34. 5 – Participants – Representativeness
Do people you will encounter in a coffee shop represent
the important characteristics of your users?
• If so, great! Head down to the coffee shop of your choice
• If they are owner-managers of scrap metal merchants,
consider whether they are likely to be hanging out in
Starbucks
Manchester Metropolitan University
34

35. 5 – Participants – Representativeness
One of the single most important things you can do to
generate valid insight is recruit people who represent
your target users
• What are the important characteristics of your users (for
this test)?
• Where can you find people like that?
• When will people like that be available?
• Will you need to incentivise them?
• Are there any special ethics? (Children, etc)
Manchester Metropolitan University
35

36. 6 – Practicalities
What do you need to sort out to make it happen? (1)
• A space somewhere
• A time
• Equipment and software
• Script
• Note recording sheets and/or recording software
• Task sheets for people to follow
• Participant availability (and overbooking)
• Prototypes, live sites, login details…
Manchester Metropolitan University
36

37. 6 – Practicalities
What do you need to sort out to make it happen? (2)
• Attendees to help you
• Remember you are trying to stimulate valuable action.
Who can help make action come about? (A: People who
set priorities and choose what work to do)
• Development Managers
• Product Managers
• Developers
• Etc…
Manchester Metropolitan University
37

38. Example
1. A plan
2. A “method” (prototype)
Manchester Metropolitan University
38

39. Doing
Manchester Metropolitan University
39

40. This morning, 9am
– Remote, moderated
test
– Web-hosted prototype
plus alpha build
– Skype with video, +
Evaer to record
– Note recording sheets
(formatted)
– Participant sent task
sheets and URLs in
advance
– Chris – pictured – comoderating & notetaking
Manchester Metropolitan University
40

41. Running a test - Prepare
Be ready
Do run-throughs in advance to check everything works
Make sure all co-moderators know what’s going to
happen and what they need to do
Think about if you need to do “resets” between
participants if data or state can be persisted
Allow yourself time to set things up
Manchester Metropolitan University
41

42. Running a test - Introduce
Usability tests can be stressful for the participant
• It’s not them on test – it’s the product or site
• Everything that goes “wrong” is the fault of the designers,
and the most useful bits of the session
• People just like them are particularly interesting – you
want to know how easy they find it to use
• Reprise what to expect, and how long the session will take
• Get any permissions (e.g. informed consent, consent for
recording etc…)
Manchester Metropolitan University
42

43. Running a test - Prepare
Recommendations
• 1 moderator (talker), 1 note taker
• Lets one person remain engaged and keep the
conversation going
• Should result in better notes
• Notes vs. audio/video
• Need to be clear on what makes good notes
• Notes are faster to analyse than video, if they’re good
• Video is good to go back to if the notes are lacking
Manchester Metropolitan University
43

44. Running a test – Run!
Talk aloud?
• Probably – good insight
• Completely invalidates any objective measures of
efficiency (but does it matter?)
• Might not be appropriate for tasks where “flow” is
important
Manchester Metropolitan University
44

45. Running a test – Dealing with
questions
Manchester Metropolitan University
45

46. Running a test – Questions
DO NOT ANSWER
QUESTIONS
Manchester Metropolitan University
46

47. Running a test – Questions
Some suggestion for avoidance
• “What do you expect that to do?”
• “What do you think you could do next?”
• “If I wasn’t here, what would you do?”
• “Try doing whatever you think might help”
• If necessary, admit you’re being awkward and unhelpful,
but you need to know what they’d do if you weren’t here
• Let people struggle for a while, but rescue them before
they’re suicidal
Manchester Metropolitan University
47

48. Running a test – Finish
Show your appreciation
• You should be grateful
• Be grateful
• Be genuine
Manchester Metropolitan University
48

49. Analysis and reporting
Manchester Metropolitan University
49

50. Analysis and reporting
For another day…but remember
We are doing this to stimulate valuable action.
NOT:
Manchester Metropolitan University
50

51. Thanks
Chris Collingridge
@ccollingridge
Manchester Metropolitan University
51