PhD Student Conference at the OU's Centre fro Research in Computing

1. 2010 CRC PhD Student Conference
Analysis of conceptual metaphors to inform music
interaction designs
Katie Wilkie
k.l.wilkie@open.ac.uk
Supervisors Dr Simon Holland
Dr Paul Mulholland
Department/Institute Music Computing
Status Part-time
Probation viva After
Starting date April 2008
Music is interwoven through many facets of our daily lives and experiences,
from the deep to the trivial. It can be both an art form providing a conduit by
which emotions and ideas can be communicated, and a means to
communicate personal tastes through, for example, the choice of a mobile
ringtone. Despite the ubiquity of music, opportunities for non-experts to
interact with music in meaningful ways, to understand it and to affect it by,
for example creating and manipulating melodies and harmonic progressions,
are limited.
Popular and pervasive though music is, understanding and analysing the
structural properties of musical artifacts often requires knowledge of domain
terminology and notation. Such specialist knowledge is generally restricted to
highly trained domain experts who have pursued a path of detailed academic
study. In particular, musical concepts such as harmonic progression and voice
leading, which make use of a number of different terms and notations to
describe various parameters and aspects, can be difficult to understand and
describe. Furthermore, providing ways of interacting with music that are
sufficiently expressive for experts whilst still being usable by non-experts
remains an open challenge. We hypothesise that if we can represent this
specialist knowledge in a form that exploits pre-existing and universally held
sensory-motor experiences, we will be able to lower some of the barriers to
musical expression. Thus we believe that music interactions designed in this
manner would lessen the requirement for specialist domain knowledge and
be more intuitive to both domain experts and novices alike.
The identification of image schemas, exposed through linguistic constructs,
provides a promising foundation for this work. Image schemas are defined by
Johnson (2005) as “recurring patterns of our sensory-motor experience” where
the experiences Johnson is referring to are those of interacting with other
bodies, space and forces within our environment. Johnson further
hypothesises that these image schemas can be applied to other, often abstract,
domains through the creation of conceptual metaphors, enabling us to
develop our understanding of more complex abstract concepts.
Image schema and conceptual metaphor theories have already been applied
to a number of different domains such as arithmetic (Lakoff, Nunez 2007),
musical concepts (Saslaw 1996, 1997, Zbikowski 1997a, 1997b, Brower 2000,
Larson 1997, Johnson 1997, Johnson, Larson 2003, Eitan, Granot 2006, Eitan,
Timmers 2006), user interface design (Hurtienne, Blessing 2007) and music
Page 114 of 125

2. 2010 CRC PhD Student Conference
interaction design (Antle et al. 2008, 2009). In the domain of user interface
design for example, Hurtienne and Blessing (2007) carried out experiments
attempting to determine whether user interface controls which were
configured to support simple conceptual metaphors such as MORE IS UP, a
metaphorical extension of the UP-DOWN image schema, would be more
intuitive to use. Their results do appear to support this hypothesis to an
extent, however only a small number of user interface controls and
conceptual metaphors were tested.
In the domain of music theory, work by Saslaw (1996, 1997), Zbikowski
(1997a, 1997b), Brower (2000), Larson (1997), Johnson (1997, Johnson, Larson
2003) and Eitan et al. (Eitan, Granot 2006, Eitan, Timmers 2006) has used
image schemas and conceptual metaphors in an attempt to increase our
theoretical understanding of musical concepts. This has yielded promising
results indicating that musical concepts can be understood in terms of image
schemas and conceptual metaphors.
Antle et al. (2008, 2009) designed an interactive sound generation system
based on embodied metaphors that allowed users to generate sounds and
modify simple sound parameters through body movement. They ran a series
of experiments attempting to establish whether this approach to interaction
design enhanced the ability of children to learn about sound concepts.
Although the results were inconclusive, they did highlight the importance of
discoverability of the embodied metaphors used in the interaction model.
This research draws upon these works, aiming to establish if the conceptual
metaphors elicited from dialogues between musicians discussing various
musical concepts can be used to inform interaction designs for
communicating information about, expressing and manipulating complex
musical concepts such as harmony and melody. Thus, the specific questions
this research aims to address are as follows:
1. How can conceptual metaphors aid our understanding of the musical
concepts of pitch, melody and harmony?
2. How can the conceptual metaphors identified be used to inform and
evaluate the design of music interactions for communicating
information about and manipulating pitch, melody and harmony?
Methodology
In order to address the question of the ways in which conceptual metaphors
aid our understanding of the musical concepts of pitch, melody and harmony,
we must first identify the conceptual metaphors that experienced musicians
use to understand, define and describe such phenomena. A series of studies
have been planned involving musicians from both classical and popular
traditions. The participants will be provided with musical artifacts in different
representation formats (e.g. musical score, audio file and piano roll) and
asked to discuss aspects of the artifacts in order to elicit a dialogue which can
then be analysed to identify the conceptual metaphors in use. Once a
collection of commonly used musical conceptual metaphors has been
identified, it is planned to validate these with a wider audience through the
use of an online questionnaire.
The second research question, regarding the use of conceptual metaphors to
evaluate and inform music interaction designs, will be addressed by firstly
evaluating a number of existing music interaction designs using the identified
Page 115 of 125

3. 2010 CRC PhD Student Conference
musical conceptual metaphors. The results of these evaluations will be used to
generate a series of guidelines for designing music interactions. In order to
validate the guidelines, example music interactions will be developed based
on the guidelines and subsequently evaluated with participants to establish
their suitability.
A summary of the work plan for these tasks is provided in the table below.
Dates Task
May 2010 – Dec 2010 Identify and validate the musical conceptual
metaphors used by musicians through a series
of studies and an online questionnaire.
Jan 2011 – Apr 2011 Evaluate existing music interaction designs
using the identified musical conceptual
metaphors and establish a series of design
guidelines/patterns for designing future music
interactions.
May 2011 – Dec 2011 Implement a number of small-scale solutions
based on the defined design guidelines and
evaluate these solutions to further improve the
guidelines.
Jan 2012 – Jun 2013 Write-up.
At this stage, one study has already been completed (Wilkie, Holland and
Mulholland 2009) and further studies are in the process of planning.
Contributions
It is envisaged that this research will provide the following contributions to
the field:
1. Increase knowledge of how conceptual metaphors aid understanding
of musical concepts such as pitch, melody and harmony. This will be
achieved through identifying and validating the conceptual
metaphors used by musicians when discussing various aspects of
music.
2. Some preliminary indication of how different musical representation
formats affect and align with the conceptual metaphors elicited
during discussions.
3. Improve knowledge of how musical conceptual metaphors can be
used to evaluate and inform the designs of intuitive music
interactions. This will be achieved through the development of a
series of design guidelines aimed at assisting designers to make
decisions about the most appropriate manner for communicating
information about and manipulating specific musical parameters.
References
ANTLE, A.N., CORNESS, G. and DROUMEVA, M., 2009. Human-computer-
intuition? Exploring the cognitive basis for intuition in embodied interaction.
International Journal of Arts and Technology, 2(3), 235-254.
ANTLE, A.N., DROUMEVA, M. and CORNESS, G., 2008. Playing with the
Page 116 of 125

4. 2010 CRC PhD Student Conference
sound maker: do embodied metaphors help children learn? Proceedings of the
7th international conference on Interaction design and children, 2008, ACM pp178-
185.
BROWER, C., 2000. A cognitive theory of musical meaning. Journal of Music
Theory, 44(2), 323-379.
EITAN, Z. and GRANOT, R.Y., 2006. How Music Moves: Musical Parameters
and Listeners' Images of Motion. Music Perception, 23(3), 221-247.
EITAN, Z. and TIMMERS, R., 2006. Beethoven’s last piano sonata and those
who follow crocodiles: Cross-domain mappings of auditory pitch in a musical
context, Proceedings of the 9th International Conference on Music Perception and
Cognition, 2006, pp875-882.
HURTIENNE, J. and BLESSING, L., 2007. Design for Intuitive Use - Testing
Image Schema Theory for User Interface Design, Proceedings of the 16th
International Conference on Engineering Design, 2007, pp1-12.
JOHNSON, M., 2005. The philosophical significance of image schemas. In: B.
HAMPE and J. GRADY, eds, From Perception to Meaning: Image Schemas in
Cognitive Linguistics. Berlin: Walter de Gruyter, pp. 15-33.
JOHNSON, M., 1997. Embodied Musical Meaning. Theory and Practice, 22-23,
95-102.
JOHNSON, M.L. and LARSON, S., 2003. Something in the Way She Moves-
Metaphors of Musical Motion. Metaphor and Symbol, 18(2), 63-84.
LAKOFF, G. and NUNEZ, R.E., 2000. Where Mathematics Comes From. Basic
Books.
LARSON, S., 1997. Musical forces and melodic patterns. Theory and Practice,
22-23, 55-71.
SASLAW, J., 1996. Forces, Containers, and Paths: The Role of Body-Derived
Image Schemas in the Conceptualization of Music. Journal of Music Theory,
40(2), 217-243.
SASLAW, J.K., 1997. Life Forces: Conceptual Structures in Schenker’s Free
Composition and Schoenberg's The Musical Idea. Theory and Practice, 22-23,
17-34.
WILKIE, K., HOLLAND, S. and MULHOLLAND, P., 2009. Evaluating
Musical Software Using Conceptual Metaphors, Proceedings of the 23rd British
Computer Society Conference on Human Computer Interaction, 2009, British
Computer Society pp232-237.
ZBIKOWSKI, L.M., 1997a. Conceptual Models and Cross-Domain Mapping:
New Perspective on Theories of Music and Hierarchy. Journal of Music Theory,
41(2), 193-225.
ZBIKOWSKI, L.M., 1997b. Des Herzraums Abschied: Mark Johnson's Theory
of Embodied Knowledge and Music Theory. Theory and Practice, 22-23, 1-16.
Page 117 of 125