Just a few years ago, our common understanding of “online” was tightly linked to the notion of a yellowish desktop computer with a tube monitor, a keyboard and a wired mouse. Today, novel user interfaces and devices are rapidly entering the digital sphere. Speech recognition, gesture and touch input promise a more natural interaction between a human and a machine. These input modes pose new challenges to the UX community. Which mode is suitable for which context and task? How do user knowledge and culture interfere with gestures? Is it a good idea to steer services with natural speech? Should we eventually wave our old mice goodbye with a gesture? Michael Wörmann discussed these questions from a user perspective at UX Poland 2014, drawing on a series of recent Facit Digital studies on natural user interfaces from different industries.

1. Speak, wave, touch:
How to do it right.
Michael Wörmann
UX Poland 2014
Warsaw, 10. April 2014

2. 2
Facit Digital

3. 3
Joint quality levels.
Local culture and UX expertise: UX Fellows

4. Natural user interfaces

5. 5
A popular development model of UIs
CLI GUI NUI
Metaphors
Exploration & recognition
Natural behaviour
Intuition
Commands
Learning & recall
Base: Hinman (2012)

6. 6
Natural (direct) user interfaces,
metaphor-free computing
Steve Mann
Many different modes:
» Speech
» Gestures
» Touch
» Real-world UIs (wearables)
» Immersive UIs (e.g. Oculus
Rift)

7. 7
Pic: Michael Henke
» “Invisible”: No mice, no
keyboard, sometimes no screen
» No “artificial” control devices have
to be learned to use
» Often simplified compared to classic
computers
» Intuitive: Just try it out
» Interaction comes naturally
(e.g. grasp, manipulate objects, speak)
» Steep learning curve
The NUI Promise

8. 8
» NUIs are not intuitive per se: Cultural conventions still need to be
followed, e.g. edge swipe gestures.
» We still have to abstract and transfer metaphors from the real world
to the machine: You cannot pinch a paper photo.
» Many NUIs still need a GUI – NUI and GUI is not mutually exclusive
in the development model.
» Is using a commonplace WIMP UI* so unnatural?
Some NUI Controversy
* Windows, Icons, Menus, Pointer

9. 9
Sometimes it appears more natural using
a tool rather than fingers.

11. 11
» Research started in the 1960s
» 1st voice recognition system from
IBM: 1984
» Automatic information systems (since
1980s)
» Dictation, speech to text (since
1990s)
» Automobile (since 2010)
» “Mobile helpers” (since 2011)
Voice Recognition and Synthesis

12. 12
Voice Control at any price?

13. 13
» Our auditory system processes information more slowly than we
can scan text and images with our eyes
» However, humans talk faster than they type
» Speech is fleeting: We easily forget what has been said a while
before.
Thus, not suited for large amounts of information
Difficult to explain all possible commands of a complex system
» Unlike as in popular touch interfaces, there are no widespread UI
conventions regarding voice operation
What do we know about speech?

14. 14
» Large amounts of data better rely on vision
» Speech recognition can be more efficient for large text input
» Voice output suitable for brief prompts and confirmations.
Otherwise inefficient.
» The user must be aware of possible commands for voice-controlled
UIs.
» Good application examples:
1) Extremely large lists where the user knows what he wants
2) Recurrent voice commands (that need to be learned once)
What does that mean for UIs?

15. 15
“Where do you want to go today?”
“Where do you want
to go today?”teamdochnoch.de

16. 16
“Where do you want to go today?”

17. 17
“Where do you want to go today?”
EASY BANK
online banking made easy
We do everything!

18. 18
Voice Control at any price?

19. Healthcare

20. 20
» Voice or gestures could be a means to
reduce disinfection efforts.
» Better durability of devices if less exposed
to detergents.
» In a nurse’s routine, there are potentially
many recurrent commands.
» Hence, potential to skip menus. Use voice
as a shortcut.
» Potential problem: Privacy.
Voice control in Healthcare

21. Wave

22. 22
Gesture input has been last year’s hype
and continues to go strong.

23. 23
Source: Infineon Technologies
Infineon 3D Image Sensor

24. Gesture Alphabet Test

25. Application Test

26. 26
Gesture control is embraced by potential users
However,
» which are the killer apps for the 3rd dimension?
(advantages in efficiency and joyfulness of gesture instead of touch)
» which gestures go beyond touch replacement?
» which are the most suitable devices?
(AIO computers, laptops, tablets, TVs?, other?)
Great Technology, still many questions to
answer.

27. Arm in the airElbow rested modeWrist rested mode
27
Source: Facit Digital, Infineon
Ergonomic questions
Little helper

28. 28
Source: Facit Digital, Infineon
Touch replacement: Lack of a surface is a
challenge.
air button depth click
virtual
depth
barrier
click per
Key
gun click
horizontal
gun click
vertical
+ Ctrl / Space
Examples how to trigger a click

29. Pointing Gestures

30. 30
» Gestures with an associated meaning
» Potentially more efficient
Challenges
» Presumably influenced by local cultures
» Some cultures might be more predisposed to
the use of gestures than others
Semantic Gestures
Semantic gesture
for mission control
on a Mac

31. C: I like!
B: I guess I ate
something wrong
A: I think I need to
see a dentist
D: That meal is
really tasty
What are these people trying to tell us?
D: That meal is
really tasty

32. 32
UX Fellows compared spontaneous
gesture behaviour in 18 different countries.
RU
CN
IN
AU
AR
CA
US
MX
ES
FR
UK
IT
FI
DE
NL
BE
KRTR

33. 33
UXF Gesture Study

34. 34
Source: UX Fellows (2013), No. of countries.
For some commands a common sign
language already exists across cultures.
0
2
4
6
8
10
12
14
16
18

36. 36
Which device is best for TV zapping?
A BB

37. 37
Which device is best for finding a nice
movie in an online video library?
A BA

38. 38
Smart TV Ethnography

39. Source: Facit Digital, Die Medienanstalten (2014)
Small Screen (aka Second Screen)
» Research, choose and purchase
VOD movies
» EPG for advanced planning and
recording
» Research additional information for
TV broadcast
» Social Media
“Lean forward”
Big Screen (aka TV & remote)
» Linear TV, Zapping
» EPG for fast information
» Moving image apps (VOD)
» AV output for second screens
“Lean back”

40. 40
NUIs can reach new levels
of efficiency & joy of use if
adapted to human
psychology and the
usecase. #uxpoland
@wrombel

41. Michael.Woermann@facit-digital.com
@wrombel
www.facit-digital.com/en
www.uxfellows.com
Thank you.
Dziękuję.

42. 42
» Rachel Hinman (2012). The Mobile Frontier: A Guide to Creating Mobile Experiences.
Rosenfeld Media
» Die Medienanstalten – ALM GbR (2014): Wie smart ist die Konvergenz? Markt und
Nutzung Connected TV.
» http://en.wikipedia.org/wiki/Natural_user_interface
» http://www.uxfellows.com/gesture.php
» http://www.infineon.com/cms/de/product/sensor-ics/time-of-flight-tof-imager-infineon-
registered-3d-image-
sensor/channel.html?channel=db3a30433e82b1cf013e84770c2403e7
References