Heat Seeker is an interactive audiovisual project created between 2003-2009 by Nuno N. Correia and André Carrilho. It began as an audiovisual performance combining visuals manipulated in real-time with electronic music compositions. Videos were later created and screened in festivals. An interactive web version was also developed. The project aims to reunite the separated experiences of sound and image in performances by making the manipulation of visuals transparent to audiences. Future developments could explore incorporating the project onto mobile and gaming devices.

1. Heat Seeker
An Interactive Audio-Visual Project
for Performance, Video and Web
Nuno N. Correia
Aalto University,
School of Art and Design, Media Lab
IADIS Visual Communication Conference
Albert-Ludwigs-Universität Freiburg, 27.7.2010

2. 1. Introduction, Context, Objectives

3. Introduction – Heat Seeker
• An audio-visual project by Video Jack
– Nuno N. Correia and André Carrilho
• Developed between 2003 and 2009
• Audio-visual performance (2004-2008)
– in Portugal, Poland, Germany and Russia
• Videos: DVD (2006), YouTube/Vimeo (2007)
– DVD screened in festivals (2007-2008)
– in the UK, Brazil, China and France
• Interactive web version (2009)
– http://www.videojackstudios.com/heatseeker

4. Contextualization
• Heat Seeker is related to similar projects combining
sound and visuals.
• In the 1920s, Oskar Fischinger and Walther Ruttman
created “visual music” films in Germany – a
combination of tinted animation with live music
(Moritz 1997).
• Fischinger was an inspiration to a younger generation
of visual music artists, such as brothers John and
James Whitney (Moritz 1995).

5. Contextualization
• Digital artists such as Golan Levin have explored
interconnected audio-visual creative expression
(example: Audiovisual Environment Suite,
1998-2000).
• John Klima’s Glasbead (1999), an “online art work
that enables up to 20 simultaneous participants to
make music collaboratively via a colorful three-
dimensional interface” (Tribe and Jana 2007, p. 54).
• Toshio Iwai’s Electroplankton (2005) for Nintendo DS,
a group of ten interactive audio-visual games.

6. Golan Levin, Audiovisual Environment Suite
(http://www.flong.com/projects/aves/)

7. Problem
• The enjoyment of music has always been linked with
the experience of “watching a performer physically
produce musical sound”. (Austerlitz 2007, p. 11)
• In laptop-based electronic music performances, the
visual impact of physical musical manipulation is
usually limited.

8. Objectives
• The main objective:
– Combine visuals with sound in an electronic music
performance,
– creating an engaging hypermediated experience for the
audience.
• In other words:
– to “reunite the separated segments of the musical
experience” (sound and image) realizing “Wagner’s dream
of gesamtkunstwerk” (Austerlitz 2007, pp. 11-12 )

9. Additional Objectives
• Create a tool for manipulating visuals with adequate
flexibility and expression
• Make the act of manipulating the visuals apparent to
the audience
• Explore other channels to present the project (beyond
performances), such as DVD and Web

10. Additional Objectives
• Create coherent audio-visual experiences.
• Meaning resides “not in musical sound, (…) nor in the
media with which it is aligned, but in the encounter
between them” (Cook 1998, p. 270).
• Having created the visuals and music of Heat Seeker
in articulation with each other, Video Jack hoped to
provide the elements for the construction of a
coherent audio-visual meaning.

11. Methodology
• Practice-based research
• Researcher as developer and user
• Questionnaires will soon be conducted to users

12. 2. Description, Development, Spin-Offs

13. Description of the Software
• The Heat Seeker software was created originally for
performances.
• It was meant to be used as a visual content
management and manipulation system for
performers, and also to be projected to the audience.

14. Description of the Software
• The graphical interface of Heat Seeker is mainly
situated in the edges of the screen, in order to
emphasize the animated content in the central area.
• The interface is visible to the audience, and is part of
the visual experience, in order for the audience to see
how the visuals are being manipulated in real time.

15. Content Development
• Once the software tool was ready, Video Jack started
preparing the audio-visual material for their first Heat
Seeker performance (in 2004).
• Nuno N. Correia had already composed part of the
music.
• The preparation of visual content to be used with the
software involved a discussion regarding the themes
and inspiration behind the music.

16. Content Development
• André Carrilho developed animations for use in Heat
Seeker based on that discussion, and his own
interpretation of the music.
• He produced additional animations, which in turn
served as inspiration for more music.
• The genres “film noir” and “nouvelle vague” were
particularly emphasized, as were concepts related to
“heat”.

17. Content - Similarity with Music Videos
• Most music videos “do not embody complete
narratives or convey finely wrought stories”;
otherwise “the song would recede into the
background, like film music” (Vernallis 2004, pp. 3-4).
• The viewer becomes a participant in the music videos,
as it is up to him/her to determine the ultimate
meaning (Vernallis 2004, p. 10).
• The meaning of a music video is a “puzzle” for the
viewer to solve – “stories are suggested but not given
in full” (Vernallis 2004, p. 37).

18. Project Spin-Offs
• Performances (2004/2008)
– Two performers, with two computers running different
software: Heat Seeker for visuals; Ableton Live for audio.
• DVD, YouTube/Vimeo and screenings (2006/2008)
• Web (2009)
– Motivated by the experience gathered from the following
Video Jack project, AVOL (2007), which was developed as a
web-based project.
– As Heat Seeker was built using Adobe Flash, which is also a
web development platform, it could be functional online
with some adaptations.

19. Heat Seeker Performance

20. Heat Seeker Online
• In Heat Seeker online, the distinction between user
and audience becomes blurred.
• The user is also audience to his/her own interactions
with the software and audio-visual material.
• It is not possible to manipulate the music.

21. Heat Seeker Online - Menu

22. 3. Conclusions and Future Developments

23. Conclusions
• The aims of conveying a hypermediated experience
– by creating a tool allowing for flexibility of expression,
– and for transparency of content operation,
• were achieved (though partly, more could be done).
• Questionnaires conducted with users will assess
conclusions in more detail.
• With its spin-offs, the project achieved further
exposure, and explored different means of
involvement with the audience.

24. Conclusions - Performances
• In performances, audio and visual manipulation is
conducted by two different users, with different
software.
• The integration of sound manipulation with the
visuals and GUI would make the sound manipulation
more apparent to the viewer.
• More could be done to better integrate the actions of
the performers with the visuals.

25. Conclusions - Video
• The video format is very portable and easily
distributed on the web.
• Different audiences want different levels of
interaction.
• Image capture quality could be improved.

26. Conclusions - Web
• Sound manipulation capabilities should be
implemented, to achieve a higher level of
engagement and transparency.
• Some visual manipulation capabilities, which relied on
keyboard shortcuts, were removed for the web
version. These functionalities could be implemented
in the GUI.
• A non-interactive, generative version could be
implemented online.

27. Conclusions - Content
• The integrated development of sound and visuals
contributes to a coherent result within each of the
“chapters”.
• However, the visual and music content is not
homogenous enough.
• The project appears more as a “collection of short
stories”, and not as a “novel”.

28. Future Developments
• Possible additional spin-offs
– Mobile devices
– Games consoles
– Interactive TV
• Follow-up project, addressing limitations identified in
Heat Seeker
– Master and Margarita (2009)
– http://www.videojackstudios.com/masterandmargarita

29. References
• Austerlitz, S., 2007. Money For Nothing: A History of the Music Video
from the Beatles to the White Stripes. Continuum Books, New York
• Cook, N., 1998. Analysing Musical Multimedia. Oxford University
Press, Oxford
• Moritz, W., 1995. Color Music – Integral Cinema. In Poétique de la
Couleur. Musée du Louvre, Paris.
– http://www.centerforvisualmusic.org/WMCM_IC.htm
– Referenced January 24, 2010.
• Moritz, W., 1997. The Dream of Color Music and the Machines that
Made it Possible. Animation World Magazine, Apr. 1997.
– http://www.awn.com/mag/issue2.1/articles/moritz2.1.html
– Referenced 24 January 2010.
• Tribe, M. and Jana, R., 2007. New Media Art. Taschen, Köln.

30. Thank you!
• Any questions?
• www.videojackstudios.com
• www.nunocorreia.com
• mail@nunocorreia.com