Much of our work in UX research focuses on usability – evaluating products and interfaces to ensure they are easy-to-use. However, in today’s digital world, they are no longer enough. Consumers also have come to expect entertaining and engaging experiences. Web and mobile applications need to be usable, useful and engaging.
So, how do we evaluate web interfaces to determine how useful and engaging they are? Desirability has been evaluated in recent years by the use of the Product Reaction Card technique, originated by folks at Microsoft. However, there are many other techniques used in market and industrial design research that we can borrow to complement this technique. Likewise, we can use standard usability testing techniques with lines of questioning with a slightly different focus to evaluate the relative usefulness of different solutions for a particular user group.
In this talk, I discuss several techniques that I have used in recent months to evaluate the usefulness and desirability of interfaces The best techniques I have discovered to evaluate usefulness involve open-ended interview questions regarding current processes and pain points, followed by a usability evaluation of the interface and then a reflective interview discussing the benefits and drawbacks of that solution to their personal situation. To evaluate desirability, I will discuss the product reaction card technique and variations using more defined vocabularies for emotional responses and product personalities. In addition I will show results from techniques borrowed from psychology and marketing research - sentence completion, collaging, and the use of dyad rating scales. These techniques offer a variety of both qualitative and quantitative data that can be used to compare different interface options.
16. Sentence Comple,on
• This app makes me feel…
• If I used this all day, I would…
• Reading on this website is…
• Compared to other apps, this app is…
• I want to use this because…
• I do not want to use this because…
17. Product Reac,on Cards
• 118 words
• 58% posi,ve
• 36% nega,ve
• 6% unclear
• 21% synonyms with 1+ other words
• Mostly posi,ve words
18. Product Reac,on Cards – Alternate List
• Removed synonyms
• Added opposites to create Posi,ve / Nega,ve Pairs
• Compared against two other emo,onal word vocabularies for
Product Appeal and Emo,ons
• Marc Hassenzhahl, Darmstadt University of Technology, Germany
• Geneva Emo,on Wheel
19. Annoying
Costly
Expected
Interes=ng
Poor
quality
Trivial
Apathe=c
Crea=ve
Familiar
Irrelevant
Predictable
UnaJrac=ve
AJrac=ve
Desirable
Fast
Managable
Professional
Uncontrollable
Boring
Difficult
Flexible
Meaningful
Relaxing
Understandable
Busy
Difficult
Fragile
Mo=va=ng
Relevant
Undesirable
Calm
Disappointed
Fresh
Nondescript
Reliable
Unexci=ng
Cheap
Discouaraging
Friendly
Not
Secure
Rigid
Unimagina=ve
Clean
Disorganized
Frustra=ng
Not
Valuable
Sa=sfying
Unimportant
Clear
Dull
Fun
Novel
Secure
Unnecessary
CluJered
Easy
to
use
Hard
to
Use
Obstruc=ng
Simple
Unpredictable
Common
Effec=ve
High
quality
Old
Slow
Unprofessional
Compelling
Efficient
Impersonal
Op=mis=c
Sophis=cated
Unrefined
Complex
Effortless
Impressive
Ordinary
Stable
Unreliable
Comprehensive
Empowering
Incomplete
Organized
Standard
Unusable
Confident
Encouraging
Incomprehensible
Original
Strange
Unwelcoming
Confusing
Enthusias=c
Inconsistent
Overwhelming
Stressful
Usable
Conserva=ve
Essen=al
Ineffec=ve
Patronizing
Suppor=ng
Useful
Consistent
Excep=onal
Inefficient
Personal
Tedious
Useless
Controllable
Exci=ng
Innova=ve
Pessimis=c
Time-‐consuming
Valuable
Convenient
Exclusive
Inspiring
Pleasant
Time-‐Saving
Weak
20. AJrac=ve
UnaJrac=ve
Excep=onal
Common
Organized
Disorganized
Calm
Busy
Exci=ng
Dull
Original
Ordinary
Clean
CluJered
Exclusive
Standard
Personal
Impersonal
Clear
Confusing
Familiar
Strange
Pleasant
Annoying
Compelling
Unimportant
Fast
Slow
Predictable
Unpredictable
Comprehensive
Incomplete
Flexible
Rigid
Professional
Unprofessional
Confident
Weak
Fresh
Old
Relaxing
Stressful
Consistent
Inconsistent
Friendly
Unwelcoming
Relevant
Irrelevant
Controllable
Uncontrollable
Frustra=ng
Encouraging
Reliable
Unreliable
Convenient
Difficult
Fun
Tedious
Sa=sfying
Disappointed
Costly
Cheap
High
quality
Poor
quality
Secure
Not
Secure
Crea=ve
Unimagina=ve
Impressive
Nondescript
Simple
Complex
Desirable
Undesirable
Innova=ve
Conserva=ve
Sophis=cated
Unrefined
Easy
to
use
Hard
to
use
Inspiring
Unexci=ng
Stable
Fragile
Effec=ve
Ineffec=ve
Interes=ng
Boring
Suppor=ng
Obstruc=ng
Efficient
Inefficient
Manageable
Overwhelming
Time-‐saving
Time-‐consuming
Effortless
Difficult
Meaningful
Trivial
Understandable
Incomprehensible
Empowering
Patronizing
Mo=va=ng
Discouraging
Usable
Unusable
Enthusias=c
Apathe=c
Novel
Expected
Useful
Useless
Essen=al
Unnecessary
Op=mis=c
Pessimis=c
Valuable
Not
Valuable
21.
22. Reliable
Accessible
Appealing
Confusing
Comfortable
Organized
Professional
Busy
Confident
Straight
Forward
Approachable
Complex
Effec=ve
Understandable
Efficient
Dull
Friendly
Easy
to
Use
Flexible
Hard
to
Use
Invi=ng
Clear
Helpful
Impersonal
Time-‐saving
Convenient
Overwhelming
Trustworthy
Clean
Boring
Useful
Familiar
Frustra=ng
Simplis=c
In=mida=ng
Valuable
Stressful
Usable
Time-‐consuming
Customers
Customers
&
Non-‐Customers
Non-‐Customers
23. 68
67
62
42
20
22
17
31
0
10
20
30
40
50
60
70
Exis1ng
Design
A
Design
B
Design
C
Count
Number
of
Words
Posi=ve
Nega=ve
24. Exis1ng
Design
A
Design
B
Design
C
Appealing
Appealing
Appealing
Appealing
Clean
Clean
Clean
Clean
Efficient
Efficient
Efficient
Efficient
Engaging
Engaging
Engaging
Intui=ve
Professional
Professional
Professional
Useful
Useful
Useful
Useful
Total
9
20
16
9
34. Pre-‐Usability Interview
• How do you reconcile invoices today…?
• Who needs to approve these and why…?
• Where do you track which claims have been processed…?
• When a new pa,ent arrives, what do you do?
35. During Usability Tes,ng
Watch for comments about usefulness
• “Wow. This would really make it easier to reorder the same things.”
• “I don’t like this. It would take me too long to do this part.”
Watch for neutral reac,ons
• “Eh. This is OK.”
• Probe: “How would this help, or not help, you do your job?”
36. Post-‐Usability Interview
Show screens again and ask:
• “How would this work for you in your job?”
• “What about this works for you? What doesn’t work?”