Presentation held at the conference "Multimodality and Learning: New Perspectives on Knowledge, Representation and Communication", London, 19-20 June 2008. More information: http://www.navimationresearch.net/2008/multimodality-and-learning-in-london/

1. The times are a-changing
in the interface
Andrew Morrison
InterMedia, University of Oslo
Oslo School of Architecture & Design
and
Jon Olav Eikenes
Oslo School of Architecture & Design

2. Introduction

5. Social semiotics as frame (Kress & van Leeuwen)
Socio-cultural perspective on design
RECORD project: text-production-use study
Discourse in action (Norris & Jones 2005)
‘ … multimodal refers to communicative artifacts and processes which
combine various sign systems (modes) and whose production and
reception calls upon the communicators to semantically interrelate all
sign repertoires present.’ (Stöckl 2004: 9)

6. Interface

8. ‘movement in the interface’ (Skjulstad & Morrison 2005)
multimodal, multiply, mediated activity
navigational features are kinetic
‘transtextuality’, discourse across links (Kolb 1996)

9. Navimation

10. Navimation:
kinesis in navigation is intertwined with animation in the interface

11. Navimation & multiliteracies
• situated and multimediational multiliteracies (Snyder 1998, Ulmer 1998, Cope
& Kalantzis 2000, Hawisher & Selfe 2000, Kress 2003, Buckingham 2003)
• multimodal literacies in schools (Jewitt & Kress 2003, Jewitt 2005)
• metamedia literacy (Lemke 1998)
• part of the participatory flux of popular culture (Anderson 2003, Jenkins 2006)
• social semiotic frame, desinging affordances, social networking

12. Concepts

13. Selection of sites in a social semiotic frame:
• web usage, web design practice, web analysis
• trawling web for instances of navimation
• identification of c. 100 sites with generic navimational features
• categorisation of core kinetic features
(informed by concepts from interface design, hypertext, information visualisation,
film, animation, motion graphics, multimodal discourse, electronic art, etc)
• selection of sites with clear yet challenging features for textual analysis

14. Core concepts:
• Multimodal interaction
• Situated motion
• Transformative motion
• Non-sequential navigation
• Screened spatialisation

15. Analysis

16. Multimodal interaction

17. Multimodal interaction
• physical context physical mobile device in urban setting vs. stationary home PC
device touchscreen vs. mouse
• degree of interaction
sporadic vs. continuous
• form of interaction
body gestures vs. direct mapping
• interface (HCI): ‘any part of the computer system that the user comes in contact
with’ (Moran 1981)
• interaction modes (info.vis.): continuous, stepped, passive, and composite (Spence
2007)
• embodied interaction (Dourish 2004)
• human factors, ergonomic design, haptic design etc.

18. Situated motion

19. Situated motion
• Motion that is generated and screened in real time, in the use situation
• Not predefined in detail, but allowed for by the designer
• Can be affected by parameters such as user interaction
• Machinima (machine cinema) : production techniques where computer-
generated imagery is rendered using real-time, interactive 3D engines
(Burn & Parker 2001)
• Software art, computer games, Virtual Reality, Augmented reality, CGI in
Computer Graphics

20. Transformative motion

21. Transformative motion
• ‘Shapes themselves are transformed gradually over time, thus changing their
inherent nature’ (Woolman 2004)
• Widely used in linear media
• Related terms are ‘shape transformation’, ‘image morphing’ and
‘tweening’ (in-betweening), used in traditional animation, computer
animation and motion graphics
• A motion graphics textbook describes the process of transformation as
either reductive, elaborative, or distortive (Woolman 2004)

22. Non-sequential navigation

23. Non-sequential navigation
• Traditional hypertext systems are based on websites (or nodes) connected
by links. The navigation may be non-linear, but is still sequential
• If a user continuously affects the navigation, there are no longer any
predefined nodes. Thus, the navigation is non-sequential.
• Wayfinding in architecture (Lynch 1960)
• Structural patterns in hypertext: cycle, counterpoint, mirrorworld, tangle, sieve,
montage, split/join, missing link, feint (Bernstein 1998)

24. Screened spatialisation

25. Screened spatialisation
• Motion is used to create, enhance, manipulate, or distort the sense of two
or three-dimensional space as we know it
• Interface becomes a virtual camera, moving in virtual space
• Interface in physical space
• Camera movement in film, such as zoom, pan, dolly (Zettl 1990)
• Also used in electroacoustic music to denote sounds’ different sources in
space or sounds’ spatial movement
• Embodied interaction (Dourish 2004)

26. Reflections

27. Some implications for design & media education
• textually, interfaces and navigation are becoming more dynamic as multimodal
‘compositions’
• in designing there is a need to know new computational features and
communicative functions
• a working vocabulary for navimation may be informed by design & through
terms from developers
• multimodal discourse that includes navimation may be built in and through
action, via production with analysis
• design and media students may be drawn together in collaborative production
and reflection on navimation as part of their wider critical social semiotic
analyses and emerging practices of mediated meaning making

28. Further research in navimation
• RECORD: a multi-level research by design project
• Phase 1: features, characteristics, concepts, consultations, ‘sketching’
• Phase 2: collaborative design, creativ eindustry parrtners and users
• Phase 3: collaborative design with greater user iteration and evaluation
• research publications, including an article based PhD

29. Design examples

32. References

33. Bernstein, M. (1998). ‘Patterns of hypertext’. Hypertext ‘98. ACM.
Burn, Andrew & Parker, David. (2001). ’Making your mark: digital inscription, animation and a new visual semiotic’. Education,
Communication and Information.Vol. 1, No. 2.
Cope, B. & Kalantzis, M. 2000. (Eds). Multiliteracies: Literacy Learning and the Design of Social Futures. London: Routledge.
Dourish, P. (2001). Embodied Interaction. The MIT Press: Cambridge MA.
Fishwick, P. (2006). (Ed.). Aesthetic Computing. The MIT Press: Cambridge.
Ivarsson, J. & Saljö, R. (2005). ‘Seeing through the screen: human reasoning and the development of representational technologies’.
In Gärdenfors, P. & Johansson, P. (Eds). Cognition, Education and Communication Technology. Lawrence Erlbaum Associates: Mahwah.
203-222.
Jewitt, C. & Kress, G. (2003). Multimodal Literacy. New York: Peter Lang.
Jewitt, C. (2005). Technology, Literacy, Learning: A Multimodal Approach. Routledge London.
Kress, G. (2003). Literacy in the New Media Age. London: Routledge.
Kress, G. & van Leeuwen, T. (1996). Reading Images: the grammar of visual design. London: Routledge.
Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse: the Modes and Media of Contemporary Communication. London: Arnold.
Lemke, J. (1998). ‘Metamedia literacy: transforming meanings and media.’ In Reinking, D.; McKenna, M., Labbo, L. & Kieffer, R.. (Eds).
Handbook of Literacy and Technology: transformations in a post-typographic world. Mahwah: Lawrence Erlbaum. 283-301.
Lemke, J. (2005). ‘Place, pace, and meaning: multimedia chronotopes’. In Norris, S. & Jones, R.. (Eds). Discourse in Action: Introducing
Mediated Discourse Analysis. Routledge: London. 110-122.
Löwgren, J. & Stolterman, E. (2004). Thoughtful Interaction Design. The MIT Press: Cambridge MA.
Manovich, L. (2001). The Language of New Media. The MIT Press: Cambridge MA.
Morrison, A. (2008 in press). Inside Multimodal Composition. Hampton Press: Cresskill.
Shneiderman, B., Bederson, B. & Drucker, S. (2006). ‘Find that photo! Interface strategies to to annotate, browse and share’.
Communications of the ACM. Vol. 49, no. 4. 69-71.
Skjulstad, S. (2007). 'Motion graphics and communication design on the web'. Journal of Media Practice.
Skjulstad, S & Morrison, A. (2005). ‘Movement in the interface’. Computers & Composition. Vol. 22, No. 4. 413-433.
Stöckl, H. (2004). ‘In between modes: language and image in printed media’. In Ventola E. Charles, C. & Kaltenbacher, M. (Eds).
Perspectives on Multimodality. Amsterdam: John Benjamins. 9-30.
Ulmer, G. (1994). Heuretics. The Johns Hopkins Press: Baltimore.
Ulmer, G. (2003). Internet Invention: From Literacy to Electracy. Longman: New York.
Wood, A. 2007. Digital Encounters. Routledge. London.
Woolman, M. 2004. Motion Design: Moving Graphics for Television, Music Video, Cinema, and Digital Interfaces: RotoVision.
Zettl, H. 1990. Sight, sound, motion, applied media aesthetics. 2nd ed. Belmont, CA: Wadsworth.