1. The State of Touch Technologies: Educational
Opportunities and Challenges
Touch, Motion Gesture National Conference
Opening Address December 7, 2011
Paul Resta
The University of Texas at Austin

2. Touch, motion and gesture (TMG) technologies
will engage more of the sensorium for learning
Sight
Hearing
Touch

3. Convergence of TMG with Other Trends
Impacting Learning
Multimodal
learning
Touch, motion,
Visualization gesture
technologies
Learning
Analytics
Cloud computing

4. TMG and Embodied Cognition

5. Embodied Cognition
Embodied cognition
does not isolate brain
from body and
environment.
Our bodies shape how
we sense and process
information in our
interactions with the
world.

6. Touch offers more intuitive way to
interact with digital tools
Sourece: http://www.youtube.com/watch?v=pT4EbM7dCMs

7. Multitouch offers new ways to
support collaborative learning

8. Touch feedback on screens
Source: Senseg

9. Gesture-Based Computing
• Gestural systems offer a path for a more
holistic, human interaction of people with
technology
• Gesture-based computing is potentially
transformative technology, because it
changes not only the physical and
mechanical aspects of interaction with
computers, but also our perception of
collaborating with computers.

11. Gestural Computing Challenges
• Need for standards (Different systems may
use different conventions)
• gestures are ephemeral, they do not leave
behind any record of their path
• Gestures also vary in meaning and use by
culture

12. Importance of interaction design principles
in creating new gestural interfaces
• Visibility (also called perceived affordances or signifiers)
• Feedback
• Consistency (also known as standards)
• Non-destructive operations (hence the importance of
undo)
• Discoverability: All operations can be discovered by
systematic exploration of menus.
• Scalability: The operation should work on all screen
sizes, small and large.
• Reliability: Operations should work.
Norman (2010)

13. Recognizing the
power of gestures for learning
Source: Power Posing – Carney et al, 2010

14. Mixed Reality
Using TMG to fuse video game
with real world interaction
Source: Marta Ferraz, Doctoral research University of Texas at Austin

15. Moving into a multimodal world
Source: Nokia Research Center Future Technologies

17. A Day Made of Glass
Source: A Day Made of Glass Corning

19. Wearable Technology
ViconRevue – Life Logs
Sony razor thin
display that bends
like paper
LED display you can wear as contact lens

20. TMG provides new ways to
support Multimodal Learning
• Multiple means of
representation
• Multiple means of
action and
expression
• Multiple means of
engagement

21. Questions for Multimodal Learning Design
• Scaffolding needed to prepare students to
effectively use multimedia visual
representations and touch, motion and
gesture technologies for learning
• How to minimize cognitive overload for
students in interacting in a multimodal
environment

22. Now able to capture and analyze a
student’s interactions on touch devices

23. Will require new ways to analyze and visualize
the new massive data sets
Source: Taylor Martin :Analysis of student actions with virtual manipulatives
on interactive whiteboard

24. Education Visualization Lab
University of Texas at Austin Learning Technology Center and
Texas Advanced Computing Center

25. Implications of Touch and Gesture
Technologies
• The future trajectory of touch and gesture
technologies have profound implications for:
– the creation of new ways of interacting with
computers and with each other
– the design of new learning environments that
engage more of the sensorium and are more
authentic and engaging for students

26. The Future of Touch and Gesture
Technologies for Learning
• To achieve this goal, we need to:
– break out of disciplinary silos
– Form new interdisciplinary teams of kinesiology
experts. learning scientists, neuroscientists,
software designers, engineers, computer
scientists and artists working together

27. The new TMG technologies offer exciting
opportunities to move toward a new vision of
learning built upon embodied cognition.