1. Evaluating the User Experience
Sónia Sousa & Mati Mõttus

2. How to evaluate UX?

3. Evaluation
• Before thinking on EVALUATION,
– We should consider reasoning upon what we
should evaluate, right?
Where should
evaluation practices
are valuable?
2016 3@ Sónia Sousa

4. ISO 9241-210
iterative design
early focus on
users and tasks
empirical
measurement
http://www.system-concepts.com/assets/images/usability/usability%20diagram%20for%20blog.jpg
John D. Gould and Clayton Lewis. 1985. Designing for usability: key principles and what designers think. Communications of the ACM 28(3), 300-11
2016 4@ Sónia Sousa

5. Three main design paradigms
2016 5
Purpose
Evaluat
i-on
Artefact
Benefic
i-aries
User centered
design
Engineering Applied Arts
Purpose
Evalu-
ation
Benefici-
aries
Purpose
Evalu-
ation
Artefact
Benefici-
aries
@ Sónia Sousa

6. Purpose
Evalu-
ation
Artefact
Benefici-
aries
Interaction design
2016 6@ Sónia Sousa

7. Lesson 1
• Let’s then take a look into Evaluation related
concepts such as…
• Usefulness, Usability and User experience
• Evaluation Methods and Tools
– Objects for Evaluation
– Methods & categories & metrics
– Data & Scales
2016 7@ Sónia Sousa

8. Evaluation Concepts:
usefulness, usability and user experience

9. a good user experience depends
on both pragmatic and hedonic
qualities

10. usefulness and usability are
pragmatic

11. pleasure is hedonic

12. a user experience occurs when a
person interacts with a product…

13. Usability and user experience
• Product
– Behaviour
– Do goal
• Metrics
– Efficiency
– Effectiveness
– User satisfaction
Usability testing is… a systematic experimental
evaluation of the interactions
Is easy to use?
Is easy to learn?
Is satisfying to use?
132016 @ Sónia Sousa

14. Systematic measure
• Nowadays when a person interacts with a
product… they look for more than just
– Efficient, effective and cheap products.
• They look for products that stimulate
pleasure; and
– Address our needs for social and individual
differentiation.
2016 14@ Sónia Sousa

15. Hanssenzahl and
Tractinsky (2006)
User
Context
Content
Kuniavsky (2003)
Identity
Jetter and Gerken (2010)
Design results and
marketing
Marketing
Nielsen, Norman and Tognazzini (2011)
All aspects of the end-user’s interaction
Services
Roto, Law, Vermeeren &
Hoonhout (2011)
Interface
Usability
Information, interaction
and identity
An active and passive encounter
with a system
Characteristics
of the system
Time factor
Information
Interaction
Product
company
152016 @ Sónia Sousa

16. Usability
• According to the ISO standard
– Usability is the extent to which a product or
service can be used by specific users to achieve
predefined goals in a specified context of use
– The standard defines three main usability
dimensions:
• Effectiveness,
• Efficiency, and
• Satisfaction
2016 @ Sónia Sousa 16

17. Usability
• Anyway, when looking into usability, we
normally account for:
– Effectiveness
– Efficiency
– Satisfaction
– Learnability
– Memorability
• We name these the pragmatic qualities of a
product or service
2016 @ Sónia Sousa 17

18. User Experience
• According to its ISO standard, user experience
is a person’s perception and responses that
result from the use or anticipated use of a
product or service
2016 18@ Sónia Sousa

19. User Experience
• User experience subsumes usability and
includes includes
– emotions, beliefs, preferences, perceptions,
physical and psychological responses, behaviors
and accomplishments that occur before, during
and after use
• We name these the effects of the hedonic, or
pleasure related, qualities of a product or
service
2016 19@ Sónia Sousa

20. Usefulness…
• The degree to which a product enables a user
to achieve his or her goals: and
– Assesses user’s willingness to use the product
• If a system is easy to use, easy to learn, and even
satisfying
• But, this vector is most often overlooked
during experiments and studies in the lab
• As it addresses system-oriented design takes
– Jeffrey Rubin
202016 @ Sónia Sousa

21. Efficiency…
 How quickly can users perform tasks?
– How fast It can be accomplished (accurately and
completely)
– Quantitative measure
• Associated to % of total users
• Represents:
• Product behavior
• Users expectations
• Do exactly what user wont
212016 @ Sónia Sousa

22. Learnability…
• How easy is it for users to accomplish basic
tasks
– For the first time
– After stop using for a while
• Represents
– user’s competence
• ability to operate the system
– after some predetermined amount and period of training
• ability of infrequent users to relearn the system
– after periods of inactivity
222016 @ Sónia Sousa

23. Satisfaction…
• Does the product meets or satisfy users needs
– Quantitative and qualitative measures
• captured through both written and oral questioning
• Represents
– user’s perceptions
– user’s feelings
– User’s opinions
• Typically, users are asked
– to rate and rank products that they try,
– Asked to reveal causes and reasons for problems that
occur
232016 @ Sónia Sousa

24. Accessibility
• Describe the ability to access and benefit of
something
• The degree to which a product, device, service, or
environment is accessible by as many people as possible,
without modification
– The degree it can accommodate things that
people can't easily change…
• Focus often
– on people with disabilities and their right of
access to entities
242016 @ Sónia Sousa

25. How to…evaluate?

26. Evaluation
• Before addressing the evaluation protocol,
– we should consider:
1. What is the purpose of evaluation?
2. What metrics will I need?
– Is a measurements: a method to measuring something
3. What methods I can use?
– a particular procedure for accomplishing or approaching
something
4. What type of data we will need?
2016 26@ Sónia Sousa

27. Evaluation
• UX evaluation depends on what we focus on…
– The most significant issues preventing users from
accomplishing their goals
– What works and what do users find frustrating
– What are the most common errors or mistakes
users are making
– Assessing the improvements being made from one
design interaction to the next
– Identifying issues that are expected to remain
even when the product is launched
2016 27@ Sónia Sousa

28. 2016 28
What is the usage intention?
Positive outcomes
Motivation aspects
Granularity measure methods
Value Vs importance of use
Low level experience measure
Overall experience measure
+ +
AttitudesIntentions
What is the Usage attitude
@ Sónia Sousa

29. 2016 29
EmotionsValue Perceptions
User AttitudesProduct Qualities Usage IntentionsUser Beliefs
Before interaction During interaction After use
+
First impressions Mood State
Why people like and use certain products What make people use a certain product
@ Sónia Sousa

30. What to evaluate?
2016 30
Product
Layout, utility, functionalityFirst impressions
Hedonic qualities
Emotions
Mood state
Value
Perceptions
Pleasure, Affective arousal
symbolic aspects of the product
User perceptions
@ Sónia Sousa

31. Then…choose
a specific
metric or method?

32. Metrics and methods
• Issues-based metrics
– What is an issue?
• Anything that prevents task completion
• Anything that takes someone off-course
• Anything that creates some level of confusion
• Anything that produces an error
• Not seeing something that should be noticed
• Assuming something is correct when it is not
• Performing the wrong action
• Misinterpreting some piece of content
• Not understanding the navigation
– How are they identified?
• Using an inspection method
• Eventually combined with some performance data analysis
2016 @ Sónia Sousa 32

33. 2016 @ Sónia Sousa 33
Utility
Usability Pleasure
Intentions Social Value
Efficiency Perceptions of
interactions
The product
Affection
Satifaction
Social Value
Mood State
Relevance of the product
User intention to use
usefulness Trust
Learnability
Accessibility
UX
Asthetics
Social Links

34. Metrics and methods
• Issues-based metrics
– After identified, issues usually classified according to their
severity
• Small impact on user experience, few users experiencing issue
– Low severity
• Small impact on user experience, many users experiencing issue
– Medium severity
• Large impact on user experience, few users experiencing issue
– Medium severity
• Large impact on user experience, many users experiencing issue
– High severity
2016 @ Sónia Sousa 34

35. Metrics and methods
• And others exist such as
– Self-reported metrics, used for assessing satisfaction
among other participant perceived measures
– Behavioral and physiological metrics
• of which eye-tracking is one of the most used ones as far as Web
usability testing is concerned
– Combined and comparative metrics
• based on combinations of the previously mentioned siblings
– And others such as…
• Server logs
• Card-sorting data
– Open card sorting
– Closed card sorting
• Accessibility indicators
2016 @ Sónia Sousa 35

36. Metrics and methods
2016 36
using
Task
oriented
Perceived
usage
Usability UX
Prevent Experience Reflection
Concern with product
attributes and
prevent errors
Concern in build
positive experiences
resulted from the
interaction with the
product
@ Sónia Sousa

37. How to…choose
a specific method?

38. Evaluation methods
• Inspection methods
– These are methods where an expert evaluator
inspects a product or service
• Testing methods
– These are methods where products and services
are evaluated by testing them on real users
2016 38@ Sónia Sousa

39. Inspection methods
• There are also several but we will address:
– Cognitive walk-through
– Heuristic evaluation
2016 39@ Sónia Sousa

40. Cognitive walk-through
• The purpose is to verify if the paper prototype
actually allows the fulfillment of the selected
user stories
– This is a within team activity that ensures that
your prototype complies with the identified user
stories
2016 40@ Sónia Sousa

41. Cognitive walk-through
• Designers and developers of the product or
service then walk through the steps as a
group, asking themselves a set of questions at
each step
– Data is gathered during the walk-through, and
afterwards a report of potential issues is compiled
– Finally the evaluated proposition is redesigned to
address the issues identified
2016 41@ Sónia Sousa

42. Heuristic evaluation
• This is a type of evaluation ideally carried out
by an expert.
– It specifically involves evaluators examining the
design and judging its compliance with recognized
principles, such as Jakob Nielsen’s 10 usability
heuristics
• These evaluation methods are now widely taught and
practiced in the new media sector, where products and
services are often designed in a short space of time on
a budget that may restrict the amount of money
available to provide for other types of interface testing
2016 42@ Sónia Sousa

43. Testing methods
• There are several but we will address:
– Co-discovery
– Wizard of Oz
2016 43@ Sónia Sousa

44. Co-discovery
• Two users attempt to perform tasks together
while being observed
– They are to help each other in the same manner
as they would if they were working together to
accomplish a common goal using the product
– They are encouraged to explain what they are
thinking about while working on the tasks
2016 44@ Sónia Sousa

45. Co-discovery
• The designers and developers should refrain
from explaining the design decisions and
rather focus on getting the most of the pair of
users tacking with your prototype
– Note taking is fundamental and you should run
this kind of test until no significant additional
information is feed back into the design process
2016 45@ Sónia Sousa

46. Wizard of OZ
• This is a testing approach built upon a paper
device instead of using a working technological
artifact
– This king of testing involves systematic observation
under controlled conditions to determine how well
people can use a product or service
– Rather than showing users a rough draft and asking,
Do you understand this?, this kind of testing involves
watching people trying to use something for its
intended purpose
2016 46@ Sónia Sousa

47. Wizard of OZ
• Setting up such a test involves asking the test
subjects to recreate a set of user stories after
being introduced to the underlying scenario
– For example, to test the attachment function of an
e-mail program, a scenario would describe a
situation where a person needs to send an e-mail
attachment, and ask him or her to undertake this
task
– The aim is to observe how people function in a
realistic manner, so that developers can see
problem areas, and what people like.
2016 47@ Sónia Sousa

48. Testing methods
• This kind of evaluation should be repeated
until no significant added value comes from
bring in an additional subject
– The usual number is 5, but other claim otherwise
2016 48@ Sónia Sousa

49. Pre and post-tests
• When applying testing methods, pre-test and
post-test questionnaires are also used to
gather feedback on the product being tested
– A common questionnaire is the 25 years old
System Usability Scale
2016 49@ Sónia Sousa

50. How to…design an
evaluation protocol?

51. Evaluation protocol
• But these methods are useless without an
adequate evaluation protocol
• While designing your evaluation protocol, you
should take into account…
– A mixed selection of inspection and testing
methods;
– The user stories supported by your
concepts/prototype; and
– The affordances of your concepts/prototype.
2016 51@ Sónia Sousa

52. Evaluation protocol
• We will work on perfection our evaluation
protocol throughout the course…
– In all cases, evaluation protocols should be piloted
to ensure that once they are being applied, you
are actually focusing in assessing the prototype or
concepts and not on solving evaluation protocol
issues
2016 52@ Sónia Sousa

53. Now it’s up to you to
make it happen

54. ???How???
• Answer to
– what you wont to evaluate?
– What are your evaluation
needs?
• Decide for instance
– Product name
• Digital e-Textbook service
– Product Description
• Web service, mobile, desktop
and tablet application
– Design stage
• Concept design
– Product representation
• Functional prototypes
– Purpose of evaluation
• Find the best concept idea for
implementing the service
– Participants
• Students, teachers and
publishers
– Time restriction
• 3 weeks
– Equipment and tools
• Audio recorder, screen
recording, assistant for note
taking
– Skills of researcher
• HCI students
2016 4@ Sónia Sousa

55. …and that is why
it should be carefully addressed

56. evaluating user experiences is
tricky but possible

57. and we will now address possible
approaches

58. Most known Methods…
• Usability Methods
• System Usability scale [SUS]
• Methods that were transposed from Usability
– Cognitive Walkthrough
– Think aloud
– Diaries
2016 58@ Sónia Sousa

59. Most known Methods…
• Others are adaptation of
– Extended usability testing
– TRUE Tracking Realtime User Experience
– UX Expert evaluation
– Property checklists
• Others are new
– AttrakDiff -
– Emocards, Emotion Cards, Emofaces
– Reaction Cards
2016 59@ Sónia Sousa

60. SUS is a quick and dirty scale
popular for end-of-test
subjective assessments

61. System Usability Scale
2016 61@ Sónia Sousa

62. Affect Grid

63. 2016 63@ Sónia Sousa

64. AttrackDiff

65. AttrackDiff assesses the user's
feelings about a system with a
questionnaire addressing both
hedonic and pragmatic
dimensions of UX

67. More…

68. How user perceive product features
• Aesthetics and interaction features
• Layout, content, functionality
– When before and during
• Personal expectation and needs
– Record of mouse activity and mood state
– AttractDiff and EMG values
– Kleiss, Lavies questionnaire
2016 68@ Sónia Sousa

69. What is the intention to use
• Judgments of the product based on
– Feelings towards the product
2016 69@ Sónia Sousa

70. First impressions
• How user perceive product features
– Aesthetic and symbolic aspects of products
• Measurements
– Visual and verbal tests
2016 70@ Sónia Sousa

71. Values & perceptions
• How relevant users see the product?
– Pleasure, challenge, needs, expressions
• Mood boards
• Perceived quality
• Efficiency
• Effectiveness
• sensors
2016 71@ Sónia Sousa

72. Emotions & Mood state
– Positive outcomes
• Emotions, joy, excitement, value
– Affects
– BMIS
– MIS
– RAS
– Affect grid
– SAM
– PreEmo
2016 72@ Sónia Sousa