The document discusses various methods for evaluating gestural user interfaces (UIs), including comparing a UI to a reference model, collecting evaluation data on usability criteria, and using standardized scales and metrics. Common dimensions for evaluation are goals, utility, usability, and factors like system acceptance, ease of use, and cost. Methods mentioned include observations, questionnaires, heuristic evaluations, and measuring task performance and preferences using standardized scales. Guidelines are provided for designing and assessing the usability of different gestures.

1. Lecture 4: Evaluating Gestural Interaction:
Models, Methods, and Measures?
Jean Vanderdonckt, UCLouvain,
Vrije Universiteit Brussel, on-line, 18 May 2021
Pleinlaan 9, B-1050 Brussels

2. 2
• UI Evaluation = comparing a UI of an interactive
system with respect to a reference model
Interactive system
to be evaluated
Reference
Model
Compare
Evaluation dimensions: goals, utility, usability
Evaluation data: ergonomic criteria
Data collection methods:
observations, metrics, analyzes

3. 3
• Evaluation dimensions
• System acceptance
• Social acceptability
• Practical acceptability
• Ease of use
• Utility
• Usability
• Cost
• Compatibility
• Robustness
• ISO 9241, ISO 2100, ISO 25000 Square
• Quality Management

4. Sampling
error
Recognition
rate
Recognition
time
.9***
.3**
Effective-
ness
?
Gesture
recognition
performance
Other performance
properties

5. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Error #3
Error TLX
.9***
.3**
?
?
Effective-
ness
Efficiency
?
?
?
Gesture
recognition
performance
Other performance
properties

6. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Error #3
Error TLX
.9***
.3**
?
?
Effective-
ness
Efficiency
?
?
?
Gesture
recognition
performance
Other performance
properties
Source: https://arxiv.org/abs/2001.09963

7. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Error #3
Error TLX
.9***
.3**
?
?
Effective-
ness
Efficiency
?
?
?
Gesture
recognition
performance
Other performance
properties
Reproductibility = of the motions with task intent, the rate of gestures achieved as intended
Relevance = of the tasks, the rate of gestures performed as intended for the task

8. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Intuitiveness
Naturality
Social
acceptability
Adjustment
quality
Error #3
Error TLX
…
.9***
.3**
.3*
?
?
?
Effective-
ness
Efficiency
Satisfaction
?
?
?
?
?
Gesture
recognition
performance
Other performance
properties

9. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Intuitiveness
Naturality
Social
acceptability
Adjustment
quality
Attractiveness
Playability
Pleasurability
Error #3
Error TLX
Error #m
…
.9***
.3**
.3*
?
?
?
?
?
?
Effective-
ness
Efficiency
Satisfaction
Gestural
Experience
Other UX
properties
?
?
?
?
?
?
?
End user’s
gesture
preference
Gesture
recognition
performance
Other performance
properties
Other preference
properties
?
?

10. Sampling
error
Error #2
Discoverability
Recognition
rate
Recognition
time
Learnability
Memorability
Workload
(NASA-TLX: mental,
physical, temporal, perfor-
mance, effort, frustration)
Intuitiveness
Naturality
Social
acceptability
Adjustment
quality
Attractiveness
Playability
Pleasurability
Error #3
Error TLX
Error #m
Quality of
Gestural user
experience
…
GUESS
(Gesture User Experience
Special Scale)
.9***
.3**
.3*
?
?
?
?
?
?
Effective-
ness
Efficiency
Satisfaction
Gestural
Experience
Other UX
properties
?
?
?
?
?
?
?
End user’s
gesture
preference
Gesture
recognition
performance
Other performance
properties
Other preference
properties
?
?
?
?

11. End
user’s
preference
Gesture recognition performance
Superfluous Too system-
oriented
Desired
System
oriented
Neutral
User-
centered
Too
user-
centered
Any GUESS
evaluation
Confidence
interval
low medium high
low
medium
high

12. Source: Evaluation of Gesture-Based In-Vehicle Interaction: User Experience and the Potential to Reduce Driver Distraction
Lisa Graichen, Matthias Graichen, and Josef F. Krems

13. Source: Evaluation of Gesture-Based In-Vehicle Interaction: User Experience and the Potential to Reduce Driver Distraction
Lisa Graichen, Matthias Graichen, and Josef F. Krems, Human Factors, V61, N5, 2019

14. 14
• Whitefield classification of evaluation methods
Analytical methods
User
reports
Expert
reports
Observational methods
Represented Real
User
Represented
Real
Application
Source: A. Whitefield, F. Wilson, J. Dowell, A framework for human factors evaluation,
Behaviour and Information Technology 10(1), January 1991, pp. 65-79

15. 15
• Whitefield classification of evaluation methods
Analytical methods
Observational methods
Represented Real
User
Represented
Real
Application
Questionnaires
Design tests
Alternative designs
Iterative evaluation
Code static analysis
Source: A. Whitefield, F. Wilson, J. Dowell, A framework for human factors evaluation,
Behaviour and Information Technology 10(1), January 1991, pp. 65-79

16. 16
• Nielsen classification of evaluation methods
Source: https://www.nngroup.com/articles/which-ux-research-methods/

17. 17
• Nielsen classification of evaluation methods
Source: https://www.nngroup.com/articles/which-ux-research-methods/

18. 18
• Nielsen classification of evaluation methods
Preference
Performance

19. 19
• Nielsen classification of evaluation methods
Source: https://www.nngroup.com/articles/which-ux-research-methods/

20. 20
• Nielsen classification of evaluation methods
Product Development Phase
Strategize Execute Assess
Goal: Inspire, explore and
choose new directions
and opportunities
Inform and optimize
designs in order to
reduce risk and improve
usability
Measure product
performance against itself
or its competition
Approach: Qualitative and
Quantitative
Mainly Qualitative
(formative)
Mainly Quantitative
(summative)
Typical
methods:
Field studies, diary
studies, surveys, data
mining, or analytics
Card sorting, field
studies, participatory
design, paper prototype,
and usability studies,
desirability studies,
customer emails
Usability benchmarking,
online assessments,
surveys, A/B testing
Source: https://www.nngroup.com/articles/which-ux-research-methods/

21. • Evaluation of gesture UIs: guideline review
Gesture Gesture name Suggested behavior
Fixed or application-
specific
Hot point Notes
Scratch-out Erase content Fixed Starting point Make the strokes as horizontal as possible, and draw at least
three strokes. If the height of the gesture increases, the
number of back and forth strokes also needs to increase.
Triangle Insert Application-specific Starting point Draw the triangle in a single stroke, without lifting the pen.
Make sure that the top of the triangle points upward.
Square Action item Application-specific Starting point Draw the square starting at the upper left corner. Draw the
square with a single stroke, without lifting the pen.
Star Action item Application-specific Starting point Draw the star with exactly five points. Do this in a single stroke
without lifting the pen.
Check Check-off Application-specific Corner The upward stroke of the check must be two to four times as
long as the smaller downward stroke.
Curlicue Cut Fixed Starting point is
distinguishing
hot point
Draw the curlicue at an angle, from lower left to upper right.
Start the curlicue on the word that you intend to cut.
Double-
Curlicue
Copy Fixed Starting point is
distinguishing
hot point
Draw the double-curlicue at an angle, from the lower left to
the upper right. Start the double-curlicue on the word that you
intend to copy.

22. • Evaluation of gesture UIs: heuristic evaluation
• Heuristic: Visibility of System Status
• Violation: Originally, the system does not tell the user whether a
Kinect was connected to the system. So, if the system goes un-
responsive, the user might wonder what is going wrong, “did I click a
wrong button?” or “how to fix this problem?”
• Severity: Major usability problem: important to fix, so should be given
high priority
• Improvement: I added a Kinect status viewer that is able to show any
problem the Kinect may be experience. In addition, if the Kinect is
working as expected, it will act as a mirror, showing the user’s gesture
so the user knows exactly what his gesture looks like to the system
• Heuristic: Recognition rather than recall
• Violation: while the user is doing an exercise, if he wants to pause the exercise,
he can simply say “pause”. However, there is no way to for new users to know
that this command even exists. Also, it is not a good idea to rely on user’s
memory to let them remember a particular feature.
Source: Zhaochen Liu, Design a Natural User Interface for Gesture Recognition Application, University Of California,
Berkeley, 5/1/2013

23. • Evaluation of gesture UIs: guideline review
0
20
40
60
80
100
120
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Distribution des règles en pondéré
Résultats globaux en linéaire
29%
71%
Nombre de règles ergonomiques
enfreintes
Nombre de règles ergonomiques
respectées
Résultats globaux en pondéré
39%
61%
Nombre de règles ergonomiques
enfreintes
Nombre de règles ergonomiques
respectées
Nielsen’s scale for usability problems: 0 – it is not a usability problem; 1 – it is a superficial problem, it must be fixed unless extra time is
available; 2 – it is a minor usability problem; 3 – it is a major usability problem. It is important fix the problem; 4 – It is usability
catastrophic.

24. • Evaluation of gesture UIs: by questionnaire
Post-experiment questionnaire:
Rating vs. Ranking (absolute vs. relative)
Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of
INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106

25. • Evaluation of gesture UIs: by questionnaire
Post-experiment questionnaire:
Rating vs. Ranking (absolute vs. relative)
Source: Jean Vanderdonckt, Sara Bouzit, Gaëlle Calvary, Denis Chêne, Exploring a Design Space of Graphical Adaptive Menus:
Normal vs. Small Screens. ACM Trans. Interact. Intell. Syst. 10(1): 2:1-2:40 (2020

26. Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of
INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106
• Evaluation of gesture UIs: by questionnaire
Post-experiment questionnaire:
Rating vs. Ranking (absolute vs. relative)

27. Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of
INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106
• Evaluation of gesture difficulty
Rule #1: Relative Difficulty Ranking
Gesture A is likely to be perceived as more difficult
to execute than gesture B if the production time of
A is greater than that of B:
time(A) > time(B) suggests
Ranking(A) > Ranking(B)
93% accuracy

28. Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of
INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106
• Evaluation of gesture difficulty
Rule #2: Classifying Difficulty Rating
Mapping from production time to difficulty class (very
easy, easy, moderate, difficult, and very difficult)
83% accuracy

29. Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of
INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106
• Evaluation of multi-touch gesture difficulty

30. • Evaluation of gesture UIs: Gesture Usability Scale
(GUS)
Source: Manipulating 3D Content using Gestures in Design Review Scenarios
October 2009, Daniel Wickeroth, Paul Benoelken, Ulrich Lang
GUS = 67.2
SUS = 72.8
Diff = 7.1

31. • Evaluation of gesture UIs: Gesture Usability Scale
(GUS)
Source: Manipulating 3D Content using Gestures in Design Review Scenarios
October 2009, Daniel Wickeroth, Paul Benoelken, Ulrich Lang
GUS = 67.2
SUS = 72.8
Diff = 7.1

32. • Evaluation of gesture UIs: by metrics
Source: Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law
Diana Carvalho, Maximino Bessa, Luís Magalhães, Eurico Carrapatoso, Proc. of Interaccion ‘2015
According to MacKenzie, three-dimensional
movements may follow the same predictive model as a one directional
Task (A=amplitude, W=width)

33. • Gestures offer many opportunities, but…
“When developing a gesture interface, the
objective should not be *to make a gesture
interface*. A gesture interface is not universally
the best interface for any application. The
objective is to *develop a more efficient interface*
to a given application.”
[Nielsen et al., 2004]
Source: Nielsen, M., Störring, M., Moeslund, T.B., Granum, E. (2004). A procedure for developing intuitive and ergonomic gesture
interfaces for HCI. GW 2003. LNCS (LNAI) 2915, Springer 2004