1. EPFL, spring 2010 - week 12!
UX evaluation

2. today’s schedule
➝ Review of UX evaluation plans
➝ Test run of scripts
➝ Guest talk
➝ Individual walk throughs

3. overview
➝ about UX evaluations
➝ UX evaluation criteria
➝ UX evaluation methods
➝ tips and hints
➝ for more information
Attribution: parts of this presentation are based on Forum
Nokias ÚX Evaluation series

4. why UX evaluation?
Success factor
➝ UX is a critical success factor for products and services
➝ We need confirmation during design and development
that we are on the right track
➝ It is challenging for the software developer to keep an
The designer is
objective view of the developed software
not the user
➝ It is human nature to become blind of your own
outcomes
➝ Thus it is important to get a second opinion of the UX,
from outside of the development team
Simulate real
➝ Preferably from real users trying it in a realistic
use situations
environment

5. usability and product quality
SW Product Quality (ISO-9126)
Functionality Reliability Efficiency Usability Portability Maintainability
Suitability Maturity Time behavior Operability Installability Testability
Accuracy Fault tolerance Resource Learnability Adaptability Stability
utilization
… … ... … … …
Implemented Code
Number of sw Function point Source lines of
functional coverage
bugs
analysis
code
requirements
➝ there are many metrics to measure software performance and quality

6. UX and product quality
In this definition, UX is not a
part of product quality.
A SW
SW Product Quality (ISO-9126)
engineering
point-of-view?
Functionality Reliability Efficiency Usability Portability Maintainability
Suitability Maturity Time behavior Operability Installability Testability
Accuracy Fault tolerance Resource Learnability Adaptability Stability
utilization
… … ... … … …
➝ product quality Expectations, motives, ➝ measuring UX is not straight-
has a strong actions, interpretations,
forward
impact on UX
…
➝ there are many intervening
issues
User Experience

7. UX evaluation criteria 1/5
Drivers tailored Issues
As such
for your detailed for
product
your product
basic usability heuristic
UX drivers derived from the UX elements

8. UX evaluation criteria 2/5
➝ Evaluate the design against the specified UX artefacts

9. UX evaluation criteria 3/5
➝ evaluate against benchmark
➝ products and experiences

10. UX evaluation criteria 4/5
Bar
Slide
Full hw qwerty
Full sw qwerty
Touch screen
Touch only screen
Resistive screen
Capacitative screen
Clamshell
Swivel
➝ Run your tests with all the target devices and form factors

11. UX evaluation criteria 5/5
Indirect, unspoken criteria
Direct, spoken criteria
- originating from the
context
➝ the ultimate criterion:
➝ potential users try the product in their mobile context

12. planning the UX evaluations 1/2
A step-by-step
process would
be easy to
manage.
Specify needs Define the UX
Create UI designs In practice,
and context of use concept design and
evaluation are
often in a parallel
and iterative
relationship.
Evaluate concepts and designs
➝ for any UX artifacts, you can think can and should it be evaluated, somehow

13. planning the UX evaluations 2/2
My cool travel-
mate concept
➝ What artefacts do we want to evaluate
➝ Sketches, visualisations, prototypes,...
➝ What issues are we evaluating?
➝ Concept, interaction, grahical design, ...
➝ Who will do the evaluation?
➝ Experts, end users,...
Evaluate concepts and designs

14. UX evalution: expert evaluation
➝ Usability experts evaluate a system based on
common heuristics (i.e. design principles) and
personal expertise
➝ Common heuristics are based on knowledge gained
through expertise and experience
➝ Knowledge about perceptional and cognitive processes
- such as the function of memory – are utilized
➝ At least two experts take part in the evaluation to
ensure reliable results
➝ At the beginning of an expert evaluation, the value of
the system to its user and buyer/provider is
determined
➝ Based on these values, usability criteria are defined
➝ At the end, the findings documented and prioritized.
Solutions proposals are made
➝ Also user tasks can be defined and walkthroughs
conducted as part of the evaluation

15. UX evaluation: focus groups
➝ a group of people (4-7) that has
something in common will be invited
to semi-structured group discussion
session to share their views on
certain topic. Lasts usually 2 hours
➝ face-to-face sessions
➝ Sometimes on-line
➝ variations of the method
➝ Exploratory
➝ Feature prioritization
➝ Competitive analysis
➝ Trend explanation

16. UX evaluation: usability testing
➝ the test user is asked to conduct certain tasks with the
user interface under design
➝ paper prototypes can be used
➝ users are asked to think aloud
Special equipment can
Moderator
Observer
be useful to catch the
actions on mobile
Tested sw
devices
Test user
Test cases
➝ sometimes good to test the UI with pairs of users to
trigger more discussion or to test social aspects
➝ usability problems are identified and prioritized based on
the user data gathered during the usability tests

17. UX evaluation: observation
➝ the researcher does not interrupt the user
but just observes the user s actions
➝ the user is followed usually shorter time
than when shadowing in her/his own
environment, and usually only certain
activities/time period are observed
➝ good for studying users that cannot be
interrupted
➝ reveals task flows and possible detectable
If observation is not possible
(e.g. in hospital) video
problems in them and behavioral patterns
recording can be alternative. but not motivations, attitudes, values,
Video can be gone through
with the participant(s) concerns
afterwards

18. UX evaluation: make a mix
➝ an expert evaluation and a usability
test can be used as
complementary methods
➝ an expert evaluation require less
time and preparation than a
usability test
➝ expert evaluations are
recommended before usability
tests in order to support designing
the test tasks
➝ a satisfaction questionnaire can be
combined with a usability test

19. UX evaluation: tips and hints
➝ the UX evaluation criteria should be in-line with
the defined UX drivers, targets and requirements
➝ make sure to share your UX evaluation criteria
with the designers before they start their work
➝ do evaluations with real users
➝ expert evaluations can provide quick useful findings
➝ try to have a end user mindset
➝ ideally your evaluation methods should scale to weekly (or
even daily) evaluations
➝ design & evaluation iteration loops do not have to be
big monolithic steps
➝ UX evaluation(s) should be an integral part of
software development process, just like
software testing