Several interaction techniques have been proposed to enable transfer of information between different displays in heterogeneous multi-display environments. However, it is not clear whether subjective user preference for these different techniques depends on the nature of the displays between which information is transferred. We explore subjective usability of speech, touch and gesture for moving information between various displays in a heterogeneous multi-display environment, consisting of a multi-touch table, a wall-mounted display and a smartphone. We find that subjective user evaluation of the various interaction techniques depends on the combination of displays being used. This implies that the type of display combination should be taken into consideration when designing interaction techniques for the transfer of items between displays in a heterogeneous multi-display environment. Also, gesture based interactions were judged more acceptable when they involved holding a mobile phone, probably since this provided a cue explaining the action.
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Achi 2013
1. Subjective Usability of Speech, Touch
and Gesture
in a Heterogeneous Multi-Display
Environment
Arnoud de Jong Susanne Tak Alexander Toet Sven Schultz
Jan-Pieter Wijbenga Jan van Erp
TNO, The Netherlands
2. Overview
‒ Introduction
‒ Related work
‒ User Study
‒ Tasks & Results
‒ Conclusions
‒ Future Work
4. Introduction:
Using different interaction
techniques
Touch Speech Gesture
- Hand contact - Spoken commands - In-air body movements
- Physical interaction - Direct hands-free interaction - Interaction over
at close range at close to medium range longer ranges
5. Related Work
− New techniques have been proposed to move objects between heterogeneous
devices.
− Typically keyboard, touch, gesture, speech based.
− Single-user single-display paradigm still dominates.
− Available techniques complement each other:
6. Related Work
− New techniques have been proposed to move objects between heterogeneous
devices.
− Typically keyboard, touch, gesture, speech based.
− Single-user single-display paradigm still dominates.
− Available techniques complement each other:
− Direct touch is fast - but only feasible at close range.
7. Related Work
− New techniques have been proposed to move objects between heterogeneous
devices.
− Typically keyboard, touch, gesture, speech based.
− Single-user single-display paradigm still dominates.
− Available techniques complement each other:
− Direct touch is fast - but only feasible at close range.
− Speech suitable for descriptive techniques
and in conditions with occlusion.
8. Related Work
− New techniques have been proposed to move objects between heterogeneous
devices.
− Typically keyboard, touch, gesture, speech based.
− Single-user single-display paradigm still dominates.
− Available techniques complement each other:
− Direct touch is fast - but only feasible at close range.
− Speech suitable for descriptive techniques and in conditions with occlusion.
− Gestural interaction enables remote manipulation of objects.
9. Related Work
− New techniques have been proposed to move objects between heterogeneous
devices.
− Typically keyboard, touch, gesture, speech based.
− Single-user single-display paradigm still dominates.
− Available techniques complement each other:
− Direct touch is fast - but only feasible at close range.
− Speech suitable for descriptive techniques and in conditions with occlusion.
− Gestural interaction enables remote manipulation of objects.
− Users prefer combination of speech and gestural interaction
when handling graphics.
10. User Study:
Usability and Preference
Blabla
Participants: Display Pairs: Interaction Modes: Usability Questionnaire:
21 participants Table Screen − Speech − without thinking
12 male Table Mobile − Touch − intuitive
9 female Screen Mobile − Gesture − unnatural
27 average age Mobile Screen − tiring
− responsive
− complex
− error-prone
11. Interaction Techniques
Table to screen
Speech: Select photo and say “send to screen”.
Touch: Drag photo to a window entitled ‘Screen’.
Gesture: Select photo and point at the screen.
Table to mobile
Speech: Select photo and say “send to Harry”.
Touch: Drag photo to a window entitled ‘Harry’.
Tangible: Place mobile on table and drag photo to it.
Screen to mobile
Speech: Start voice command by saying “screen” then say “send to Harry”.
Gesture: Hold phone as if taking a photo of the screen.
Mobile to screen
Speech: Start voice command by dragging finger downwards over screen and
say “send to screen”.
Touch: Press send button below photo and select the ‘Screen’ menu item.
Gesture: Point phone at the screen.
12.
13. Tasks:
Table to Screen
Results
• Most participants preferred touch
technique
• 5 participants thought speaking
commands out loud was awkward
Speech Touch Gesture
Select photo and say Drag photo to window Select photo and point
“send to screen”. entitled ‘Screen’. at screen.
14. Tasks:
Table to Screen
Preference Usability
Speech Touch Gesture
Select photo and say Drag photo to window Select photo and point
“send to screen”. entitled ‘Screen’. at screen.
15. Tasks:
Table to Mobile
Results
• Most participants preferred the tangible
technique
• The tangible technique was very well received,
with 10 participants calling it “fun” or “cool”.
Speech Touch Tangible
Select photo and say Drag photo to window Place mobile on table
“send to Harry”. entitled ‘Harry’. and drag photo to it.
16. Tasks:
Table to Mobile
Preference Usability
Speech Touch Tangible
Select photo and say Drag photo to window Place mobile on table
“send to Harry”. entitled ‘Harry’. and drag photo to it.
17. Tasks:
Screen to Mobile
Results
• No significant preference for either of the
techniques
• 12 participants preferred speech the most,
9 participants preferred gesture the most
Speech Gesture
Say “screen” followed Hold phone as if taking
by “send to Harry”. photo of screen.
18. Tasks:
Screen to Mobile
Preference Usability
Speech Gesture
Say “screen” followed Hold phone as if taking
by “send to Harry”. photo of screen.
19. Tasks
Mobile to Screen
Results
• Most participants preferred the gesture
technique
Speech Touch Gesture
Drag finger and say Select ‘Screen’ menu Point phone at screen.
“send to screen”. item and press ‘Send’.
20. Tasks
Mobile to Screen
Preference Usability
Speech Touch Gesture
Drag finger and say Select ‘Screen’ menu Point phone at screen.
“send to screen”. item and press ‘Send’.
21. Conclusions
‒User preference for interaction techniques depends
on the nature of the displays.
‒Speech technique is generally disliked: people feel
embarrassed when speaking commands out loud.
‒Holding an object that provides a clear visual cue
indicating the user’s actions makes gesture-based
interaction more acceptable.
22. Future Work
‒Explicitly address embarrassment:
test whether participants are primarily
spatial, verbal or object oriented
‒Register objective performance measures
(e.g. time required to perform different actions).
‒Investigate multiple users settings
and/or more complex display
combinations.
Several interaction techniques have been proposed to enable transfer of information between different displays in heterogeneous multi-display environments. However, it is not clear whether subjective user preference for these different techniques depends on the nature of the displays between which information is transferred. In this study we explored subjective usability of speech, touch and gesture for moving information between various displays in a heterogeneous multi-display environment, consisting of a multi-touch table, a wall-mounted display and a smartphone.
First I will introduce the problem of interactive information transfer between multiple heterogeneous displays.Then I will discuss some related work on interaction and navigation techniques used in multi-display environments. Next, I will describe the user study that we performed to assess the subjective usability of gesture, touch and speech techniques for this purpose, and I will present the results of this study. Finally, I will present our conclusions and provide suggestions for future research.
Distributed computing environments (e.g. meeting rooms, collaborative work spaces) are increasingly populated with many heterogeneous display devices like smartphones and tablets (providing small personalized displays), tabletop displays (facilitating collaboration between small groups), and large size displays (for information presentation to larger groups). A frequent task in these heterogeneous multi-display environments is moving objects between displays .
Although several cross-display interaction techniques have been proposed, it is still unknown if subjective user preference for these different techniques depends on the nature of the selected display pairs. The current study assesses subjective user preferences for gesture, touch and speech based interaction techniques in a setting that integrates a multi-touch table with a wall-mounted display and a smartphone.
The single-user single-display paradigm: still dominates: the user interacts with one display at a time, using an interaction technique that is considered most appropriate for that particular type of display. Touch based interaction has become popular for devices like mobile phones, tablets and interactive tabletops. Gestural interaction has gained popularity through interactive computer games (Wii, Microsoft’s Kinect system) and is appropriate for direct interaction with large displays that can be operated from a distance.Speech based interaction is typically used for in-car navigation devices and hands-free phone systems, and might gain in popularity with the increasing availability of voice operated smartphone apps (e.g. via Siri on the iPhone). Similar to touch, speech interaction is only suitable for direct interaction at close range.
To assess user preference for different interaction techniques in a heterogeneous multi-display environment, we performed a study in which users transferred items (photographs) between different types of displays, using gesture, touch and speech techniques. Subjective user experience was quantified through semantic questionnaires. Participants rated their agreement on a 5-point Likert scale ranging from “completely disagree” to “complete agree”. For each display pair, participants ranked the 3 (2for the screen to mobile display pair) techniques from most to least preferred. On average, participants completed the experiment in 60 minutes.
Participants were requested to send a photograph from one display to another using various techniques. The 4display pairs were (touch)table to screen, table to mobile (phone), screen to mobile, and mobile to screen.
This movie illustrates the different interaction techniques.
Even though most participants preferred the touch technique, the subjective usability scores of the touch and gesture techniques are not so different.
Even though most participants preferred the tangible technique, the subjective usability scores of the tangible and touch techniques are nearly equal.
Even though most participants preferred the gesture technique, the subjective usability scores of the gesture and touch techniques are nearly equal.
Speech was only preferred when the large screen was the target of the speech command, probably because it was the only viable interaction mode in this condition.Participants felt uncomfortable or embarrassed using a gesture for the table to screen task, but not for the mobile to screen task. Possibly, holding an object that provides a clear visual cue explaining the user’s actions makes gesture-based interaction more acceptable.
Subjective usability scores showed no difference between the various gesture conditions, but the user preference rankings did.Future user evaluations should explicitly address embarrassment to resolve this issue.
