The document discusses evaluation frameworks in interaction design. It introduces key concepts like prototypes, evaluation paradigms, and techniques. Low and high fidelity prototyping are described. Evaluation paradigms include quick and dirty evaluations, usability testing, field studies, and predictive evaluation. Common techniques involve observing, asking, and testing users. The DECIDE framework is presented as a process for planning evaluations by determining goals, exploring questions, choosing techniques, and addressing practical and ethical concerns. Pilot studies are recommended to test evaluation plans.

1. Interaction Design
ISB36403
Evaluating Framework
in Interaction Design

2. The aims
 Explain key evaluation concepts & terms based on
prototyping.
 Describe the evaluation paradigms & techniques used
in interaction design.
 Discuss the conceptual, practical and ethical issues
that must be considered when planning evaluations.
 Introduce the DECIDE framework.

3. Introduction
 Evaluation helps ensure that product meets the users’
needs
 What to evaluate?
 Usability
 User Experience

4. What is a prototype?
In other design fields a prototype is a
small-scale model:
a miniature car
a miniature building or town

5. What is a prototype?
In interaction design it can be (among other things):
a series of screen sketches
a storyboard, i.e. a cartoon-like series of scenes
a Powerpoint slide show
a video simulating the use of a system
a cardboard mock-up
a piece of software with limited functionality
written in the target language or in another
language

6. Why prototype?
•Evaluation and feedback are very important to
interaction design
•Stakeholders/client can see, hold, interact with a
prototype more easily than a document or a drawing
•Team members can communicate effectively
•You can test out ideas for yourself
•Prototypes answer questions, and support
designers in choosing between alternatives

7. Different kinds of prototyping
•Low fidelity
•High fidelity

8. Low-fidelity Prototyping
•Uses a medium which is unlike the final
medium, e.g. paper, cardboard
•Is quick, cheap and easily changed
•Examples:
sketches of screens, task sequences,
etc
index cards
storyboards
‘Wizard-of-Oz’

9. Storyboards
•Often used with scenarios, bringing more
detail, and a chance to role play
•It is a series of sketches showing how a
user might progress through a task using
the device
•Used early in design

10. Sketching
•Sketching is important to low-fidelity
prototyping
•Don’t be inhibited about drawing ability.
Practice simple symbols

11. •Index cards (3 X 5 inches)
•Each card represents one screen
•Often used in website development
Using index cards

12. ‘Wizard-of-Oz’ prototyping
•The user thinks they are interacting with a
computer, but a developer is responding to
output rather than the system.
•Usually done early in design to understand
users’ expectations

13. High-fidelity prototyping
•Uses materials that you would expect to be in
the final product.
•Prototype looks more like the final system than
a low-fidelity version.
•For a high-fidelity software prototype common
environments include Macromedia Director, Flash,
Visual Basic
•Danger that users think they have a full system

14. Low vs High Fidelity
Type Advantages Disadvantages
Low fidelity prototype •Lower development cost •Limited error checking
•Evaluate multiple design
concept
•Poor detail specification
to code to.
•Address screen layout
issues
•Limited usefulness for
usability test
•Navigational and flow
limitation
High fidelity prototype •Fully interactive •More expensive to
develop
•User driven •Time-consuming to
create
•Clearly defines
navigational scheme
•Developers are reluctant
to change something they
have done for hours
•Look and feel of final
product

15. Compromises in prototyping
•All prototypes involve compromises (give and
take)
•For software-based prototyping maybe there is a
slow response? sketchy icons? limited
functionality?
•Two common types of compromise
• ‘horizontal’: provide a wide range of
functions, but with little detail
• ‘vertical’: provide a lot of detail for only a
few functions
•Compromises in prototypes mustn’t be ignored.

16. Evaluation Techniques
Copyright © 2014 Pearson Education, Inc. Publishing as Prentice Hall Slide 5-16

17. Evaluation Paradigms
 4 core evaluation paradigms:
 “Quick and dirty” evaluation
 Usability Testing
 Field Studies
 Predictive Evaluation

18. “Quick and Dirty”
 A common practice in which designers informally get
feedback from users or consultants to confirm that
their ideas are in-line with users’ needs and are liked.
 Done any time.
 The emphasis is on fast input to the design process
rather than carefully documented findings.

19. Usability Testing
 Usability testing involves recording typical users’
performance on typical tasks in controlled settings.
 As the users perform these tasks they are watched &
recorded on video & their key presses are logged.
 This data is used to calculate performance times,
identify errors & help explain why the users did what
they did.
 User satisfaction questionnaires & interviews are
used to elicit users’ opinions.

20. Field Studies
 Field studies are done in natural settings
 The aim is to understand what users do naturally and
how technology impacts them.
 In product design field, field studies can be used to:
 Identify opportunities for new technology
 Determine design requirements
 Decide how best to introduce new technology
 Evaluate technology in use.

21. Field Studies
 Two Approaches:
 Outsider: observing and recording what happens as an
outsider looking in.
 Insider: participant in study that explores the details of
what happens in a particular setting.

22. Predictive Evaluation
 Experts apply their knowledge of typical users, often
guided by heuristics, to predict usability problems.
 Note: heuristics- design principles used in practice
 Another approach involves theoretically based
models.
 A key feature of predictive evaluation is that users
need not be present
 Relatively quick & inexpensive

23. Evaluation Techniques
 Observing users
 Asking users their opinions
 Asking experts their opinions
 Testing users’ performance
 Modeling users’ task performance to predict
the efficacy of a user interface

24. Observing Users
 Techniques
Notes
Audio
Video
Interaction Log

25. Asking users their opinions
 Questions like:
What do you think about the product?
Does it do what you want?
Do you like it?
Does the aesthetic design appeal to you?
Did you encounter problems?
Would you use it again?

26. Asking experts their opinions
 Use heuristics to step through tasks
 Typically use role-playing to identify problems
 It is inexpensive and quick to ask experts
rather than perform laboratory and field
evaluations

27. User Testing
 Usually conducted in a controlled environment
 Users perform well-defined tasks
 Data can be collected and statistically analyzed

28. Modeling users’ task performance
 Model human-computer interaction to predict
the efficiency and problems in the design
 This is successful for systems with limited
functionality

29. DECIDE: Framework
 Determine the goals the evaluation addresses.
 Explore the specific questions to be answered.
 Choose the evaluation paradigm and
techniques to answer the questions.
 Identify the practical issues.
 Decide how to deal with the ethical issues.
 Evaluate, interpret and present the data.

30. Determining the Goals
 What are the goals of the evaluation?
 Who wants it and why?
 Goals influence the paradigm for the study.
 Some examples of goals:
 Check that evaluators have understood user needs
 Check to ensure that the final interface is consistent.
 Investigate how technology affects working practices.
 Improve the usability of an existing product .

31. Explore the Questions
 All evaluations need goals & questions to guide them
so time is not wasted on ill-defined studies.
 For example, the goal of finding out why many
customers prefer to purchase paper airline tickets
rather than e-tickets can be broken down into sub-
questions:
 What are customers’ attitudes to these new tickets?
 Are they concerned about security?
 Is the interface for obtaining them poor?

32. Choose Evaluation Paradigm and
Techniques
 Evaluation Paradigms determine which type of
techniques will be used.
 Trade-Offs
 Combinations of Techniques

33. Identifying Practical Issues
 For example, how to:
 Select users
 Stay on budget
 Staying on schedule
 Select evaluators
 Select equipment

34. Decide on Ethical Issues
 Consideration for peoples rights.
 Develop an informed consent form.
 Participants have a right to:
 know the goals of the study
 know what will happen to the findings
 know privacy of personal information
 not to be quoted without their agreement
 leave when they wish
 “do unto others only what you would not mind being
done to you”

35. Evaluate, Interpret, and Present Data
 How data is analyzed & presented depends on the
paradigm and techniques used.
 The following also need to be considered:
 Reliability: can the study be replicated?
 Validity: is it measuring what you thought?
 Biases: is the process creating biases?
 Scope: can the findings be generalized?
 Ecological validity: is the environment of the
study influencing it

36. Pilot Studies
 Pilot Study is a small trial run of the main
study.
 Pilot studies are always useful for testing
plans for an evaluation, before launching the
main study
 Often evaluators run several pilot studies.

37. Summary
 An evaluation paradigm is an approach that is influenced
by particular theories and philosophies.
 Five categories of techniques were identified: observing
users, asking users, asking experts, user testing, modeling
users.
 The DECIDE framework has six parts:
- Determine the overall goals
- Explore the questions that satisfy the goals
- Choose the paradigm and techniques
- Identify the practical issues
- Decide on the ethical issues
- Evaluate ways to analyze & present data
 Do a pilot study

38. Q&A