Architecture and Motion: Ideas on Fluidity in Sound, Image and Space

Yolande Harris Metronom Electronic Arts Studio Fundacio Rafael Tous d'Art Contemporani Carrer de Fusina 9 08003 Barcelona, Spain [email protected]

Introduction
The relationship between music, image and architecture through space, time and movement has become a very significant theme in contemporary thought. This can be seen to stretch back to the Futurist manifesto and the emergence of film and the moving image at the beginning of the twentieth century (Marinetti 1910, Carra 1913). In this paper I would like to present three complementary strands of research with relation to this central theme - a brief historical overview of the emergence of movement as a vital idea permeating all disciplines; the practical performance experiments with a new instrument called the Video-Organ; the relation to issues brought up by notation as the visualisation and communication of ideas across the disciplines. The historical overview in the section Architecture and Motion is based on research carried out at Cambridge University in 2000, the Video-Organ as developed at Metronom Barcelona with Bert Bongers since 2000, the Notation section has been researched over the last seven years by the author and further consolidated in Barcelona in 2002. From the large number of possible examples I have chosen ones personal to me - either art or architecture that I have visited or musical scores that I have performed. It is not intended as an inclusive document but rather a trace through a dense maze of ideas. I have a practical, empirical approach to theory and the majority of my work grows out of a symbiotic attitude towards theory and practice where the academic and artistic approaches inform each other. This paper describes a number of different practical attempts of coming to terms with the theoretical ideas and the performances in turn provide insights into relevant strands for future research. With increasing frequency exhibition catalogues, arts magazines, reviews, calls for conferences, discuss the coming together of music and moving image in a new audio-visual language enabled by the digital revolution. This is to a great extent true, the real-time image capabilities of a personal computer are only recently becoming as flexible as that which could be achieved with audio ten years ago. Live video and live image generation have started a scene of digital image and sound performances in both the VJ and DJ scene and arts events in recent years. At the same time architectural methods of design and production have dramatically transformed as architects have absorbed new generative computer-based design tools that have introduced dynamic geometries and helped to bring alive the discourse of movement within architecture (Imperiale 1999). I have chosen the first word of the title to be architecture because of its unusual relationship with movement. Perhaps out of all three disciplines of music, visual arts and architecture, it is architecture that is at present coming to terms with the implications of movement. In my opinion architecture is offering some of the richest experiments and thorough discourse on the topic, although historically furthest from a time-based foundation. Through my performances and compositions I inherently find myself within this multi-disciplinary contemporary context in a variety of ways. Although my chosen approach is not to make explicit references and comments on this context, I am very aware of the need to learn from and try to understand the basis of contemporary developments. Architecture and Motion is an attempt to create a balance between an historical grounding and an active participation in the general discussion, and although it is essentially a personal account I hope to contribute to the development of this multi-disciplinary field.

Notation as an Interface
The visualisation of a sound, or the temporalisation of a static image, are examples of the boundary area between the different modes of image and sound, time and space. At its most basic a musical notation can be described as a spatial (visual) realisation of a temporal (musical) event. Paul Klee, by famously taking his "line for a walk", explicitly introduced the idea of time within the static graphic image (Klee, 1930's). Both examples can be called 'notations' in the usual sense of the term - that they consist of a symbolic system for representing something notation coming from the Latin notatio meaning a marking. In my opinion the area of notations presents one of the most relevant existing examples of the inherent crossing and combinations of the disciplines and as such is an important area of discussion in this paper. In this first section I will mainly discuss examples from a historical musical background while the final section will suggest new approaches and relevance for notation through its connection to movement. Notation is either a visualisation of ideas in the mind or a translation of qualities or information from another sense. The simplest example of a 'notation revolution', one with the most profound consequences, is the writing down of

speech (McLuhan 1967, 1980). The separation of word and meaning from speech and the human voice - the result of the visualisation of sound - has a counterpart in the visualisation of musical sound at the emergence of Western musical notation in the Middle Ages. In the sixth century Isodore of Seville wrote in Etymologies Book 3 "unless sounds are remembered by man, they perish for they cannot be written down" (Livermore 1972). The emergence of musical notation in Mediaeval Europe illustrates a shift in attitude away from Isodore's position as it became possible to capture, represent and encode the temporal art of music within a static visual medium. Although the precise system of neumic notation is not relevant here, it is important that the two major parameters represented, at the exclusion of all potential others, were pitch and later rhythm. The development of Western classical music, based on this early system of a flexible and relative portrayal of pitch and rhythm, fixed and refined these parameters into a rigid system capable of conveying considerably more complex musical ideas. It was not until the mid twentieth century that a need became inescapable for musical notation to communicate other parameters than pitch and rhythm. There are numerous examples of graphic scores from the sixties, from composers as diverse as Cage and Stockhausen, signifying a dissatisfaction with the Western system as incapable of conveying the suitable communications (Cage 1969). It is interesting to consider the emerging use of 'scores' by visual artists at this time to guide their performances - the Fluxus scores of George Brecht or those of Audio Arts in England are examples. (Lippard 1973, Furlong 1994) A significant number of the experiments with graphic notations in the sixties were built on the idea that the performers had their own recognised and vital influence over the interpretation. For example, the composer, instead of trying to dictate the musical ideas as precisely as possible, may have suggested a number of possible paths for the performer to choose from, explore and combine. In an extreme case like Cornelius Cardew's Treatise consisting of purely graphic symbols with no instructions, the performers fill the role of interpreters arguably to the point of being composers. (Cardew 1971) These experiments are early examples of today's increased conflation of composerperformer-improviser. Comparable in a significant way to the separation of words from speech is the separation that has come about through digital technology - that of a direct cause and effect understanding between our actions and the result. A simple example is that of traditional acoustic musical instrument where the player is in direct physical contact with and hopefully control over the sound produced, as compared to electronic musical instruments based on the computer where the action of the player is without the same direct link to the sound produced. The fact that the same physical action can produce entirely different sonic results, is an example of the separation of the control action from the sound production (Bongers 1994, Paradiso 2002). Traditional concepts of notation become increasingly challenged in this situation, as will be seen in the Video-Organ work, and the role of notation need to be reconsidered. With this in mind I would propose that 'notation' is an interface (or perhaps an inter-surface) that lies between all forms of the realisation of ideas into objects - whether architectural, musical, visual or linguistic. It is in the most general sense a form of communication between people or as suggested above between human input and computer output. For example, the musical score is an artefact of the process of communication between composer and performer. It codifies communications, visualising these through notation and re-interpreting them into sound or action. One can consider the score and its corresponding notations both from the level of a systematic method for precisely conveying instructions, to a broader concept as an 'interface' between people, music, image and space. In such a situation notation becomes an interface between music, visual arts and architecture. I will illustrate the above with two examples of my own scores. In my composition Invisible Cities (1999), inspired by one of Italo Calvino's cities that visualises (in the mind, through text (Calvino 1985)) the relationships between the cities inhabitants by stretching coloured strings (Calvino 1974), I took the parameter of the performers relationships to one another as the main notation feature, which has an audible effect on the music and tension of the performance. However, the image and sound are still distinct, one static and visual, the other temporal and audible, and both as important as each other in the final performance. In trying to reconcile this I looked towards movement and the possibility of creating moving notations which naturally led to video and animation, and to the computer as the main tool. For the piece Explorations in Movement (2000) I took the different types of movement inherent in the video and extracted a graphic line as a flat representation of these movements. The live performer has to interpret as a score the elements video, sound and graphic line. The different forms of movement become very apparent when projected next to each other - the video presents the illusion of depth and overlaying of images, while the movement of the graphic score is, flat, linear and horizontal - one has internal movement while the other is moved as if by a carrier or what may be called external movement. This leads to a discussion of my research at Cambridge into the analysis of different kinds of movement and the relationship to moving image, sound and architecture.

Two pages from Invisible Cities (1999)

Four stills from Explorations in Movement (2000)

Architecture and Motion

The fundamental premise upon which this research rests is that apparent temporal and spatial contradictions between image and sound come together through movement. The inherent differences between space and time were reconciled in the idea of 'space-time' during the last century. Both Einstein's Special Theory of Relativity and Seigfried Giedion's Space, Time and Architecture (Giedion 1941) reveal how our individual perception of space and time are intimately linked and cannot be treated as separate entities. Movement inherently occupies both space and time, as does music, visual arts and architecture, particularly when related to the perception of an individual. (Stasis in this case is a form of movement as described by its position and trajectory in space-time.) This research was carried out at Cambridge University Moving Image Studio and the Department of Architecture in 2000. The thesis 'Explorations in Movement: Towards the Symbiosis of Architecture, Moving Image and Music' has drawn up a theoretical outline for the analysis of movement as the central point of conjunction between the three distinct disciplines by discussing their positions in space-time. The research is described in three distinct sections: 1. Architecture gains movement through its representation in the moving image screen, analysed through dynamic cameras, editing and sound. 2. Architecture and Motion: which formulated a series of examples including movement within architectural spaces and kinetic architecture. 3. The dynamic placing of moving image and sound within architectural space, taking examples from the cinema setting, sound diffusion and liquid architecture theory. Architectural spaces are translated and re-constructed within filmic space and a film language has evolved that enables the viewer to understands jumps in location or viewpoint that would not be possible to experience in real space. I suggest that the first stage of this translation begins at the camera, the instrument of capturing space with its different possibilities for movement. At the most basic level the are two approaches to camera movement, either the camera moves or the object or space it films moves, or any combination of these two. The use of the camera itself can become an expressive gesture that can be 'played' in time, reminiscent of how one articulates time through a musical instrument even though one captures image and the other gives out sound. The next stage in the process of defining and shaping an architectural space within the screen lies in the editing process, both at transitions between clips and special effects which transform or overlay the material. The difference in editing styles was illustrated by a comparison between the rhythm created by the Vertov's montage technique which was entirely about the editing (Vertov 1929, Michelson 1984), and that created through inherent movements within the original camera work described by Tarkovsky as 'time pressure' (Tarkovsky 1986). The two extremes show very different ways of describing a quality of a three dimensional space through the window of a two dimensional screen. They also bring in a notion of qualities of movement that are very familiar to a musician who works with the variety of placing elements in time. Recent digital editing techniques, particularly morphing, allow entirely different kinds of internal movements to be generated. At the most basic level sound is used in film to describe and augment the image of a space and is vital to the viewer's understanding of the space being presented. It can also join otherwise disjunct images by for example introducing a sound before we are shown the image which reveals the sound source to create an illusion of flow. Sounds external to the image give a sense of an extensive space providing a role comparable to peripheral vision to the focused cinema screen. Film music of course adds another layer altogether and has become a genre in itself. In summary it is hard to overstate the importance of a balanced dialogue between image and sound which, if used well together, create an entirely whole audio-visual experience where the image and sound can no longer be separated without loosing the essence. (Chion 1994) It is fair to say that the combination of the sound with the image in the representation of space in film is one of the most fundamental examples of an audio-visual discipline and is of primary relevance to creating in a multi-disciplinary format. The section Architecture and Motion concentrates on the spatio-temporal aspects of architecture by exploring at its inherent dynamic possibilities. It looks at how architecture, outside of the screen and the techniques of moving image rhetoric, has inherent qualities of motion. Architecture has an ambiguous relationship to movement. The material of architecture - the building - occupies one physical location in space, yet the function of architecture is characterised by the process with which it contains and directs the movements of its occupants. In one respect it is static - an artefact - in another it is dynamic - a process. Architecture can be described not purely as a spatial construction, but also temporal, and it is the combination of the two that provides the basis for the discussion of movement in architecture in this chapter.

Architecture and motion, the conflict between the static and the dynamic, comes increasingly into focus in the context of technological developments at the beginning of this century. The mass reality of globalised instantaneous communications, and the increasing normality of travel as a way of life, are driving the traditionally stable discipline of architecture to reflect this attitude of flexibility and movement through time and location (Paul Virilio 1984, Sola Morales 1997) It could be said that the unifying feature of architecture today lies in the various approaches to this central theme of change, fluidity, movement. Centering around the term "dynamic" this section maps some existing forms in which the emphasis on fluidity can be perceived using various contrasting architectural examples of both theory and practice to illustrate the following: Dynamic inhabitation - flows of people, traffic, services Dynamic materials - light, sound, water Dynamic structures - kinetic architecture and engineering Dynamic generation and decay - design (computers), construction, decay Dynamic connections - local to world-wide communication and reputation Dynamic associations - nature, organic flows of movement If there is to be any symbiotic relationship between architecture, moving image and music it must lie in what I termed the 'dynamic placing' of image and sound in space. After analysing different approaches to movement through film and in architecture in the first two sections, the third investigates the combination of the two. It contrasted a number of existing situations where image and sound are placed in space. From the television within a house, to the cinema with one screen, surround sound and fixed seats, to multiple screen shows like those of the architects Charles and Ray Eames (Neuhart, Nuehart & Eames 1989), to spatial sound diffusion techniques pioneered by composers like Stockhausen (Stockhausen 1991), to experiments with physically moving interactive environments such as those of Kas Oosterhuis associates (Oosterhuis 1999). I proposed that moving image and sound will inevitably become architectural materials and, like traditional known building materials, they need to be understood in their own right.

Eames' Film Think IBM pavilion, New York 1964-65

potentially inside Trans_Port 2001 by Kas Oosterhuis

The Video-Organ
The desire to practically realise and develop these ideas, particularly those presented on dynamic placing, formed the basis for the development of the Video-Organ. It consists of a specially developed set of modular instruments for live performances of video and sound. The performances concentrate on augmenting the architectural qualities of specific spaces with moving sounds and images in an attempt to create a unified environment where none of the three disciplines overpowers the others. The design approach is described in a separate paper 'A Structured Instrument Design Approach: The Video-Organ' presented at the NIME (New Interfaces for Musical Expression) conference held at MediaLabEurope in Dublin 2002 (Bongers and Harris 2002) . Here I will discuss the theoretical issues behind its development. The Video-Organ is a modular instrument for the live performance of audio-visual material which has been developed over the last eighteen months through performances and residencies in Spain, Ireland and Holland. It is a collaborative project between Bert Bongers and Yolande Harris and has played a key role in their development of the Metronom Electronic Arts Studio in Barcelona. It enables us, as the performers, to control the progression, speed, direction and placement in space of both image and sound simultaneously. The research focuses on establishing relationships between the movement parameters of the sounds and images and those of human gestures and actions which are read by the computer through sensors. The interface consists of sensor elements that we call 'instrumentlets', individually designed, each with their own character. Each video or sound clip is mapped to an instrumentlet that has a similar characteristic with the aim of enabling an intuitive and expressive control over the audio-visual material. The performances explore the potentials of transforming specific spaces through the dynamic placing of image (light) and sound using multiple screens and speakers.

Examples of dynamic placing from Video-Organ performances, Amsterdam and Barcelona 2001-2002

The self-imposed limitations of the video-organ as an instrument enabled or forced us to focus on clarity and simplicity in compositional ideas. By the later performances, video effects are absent. The palette is a combination of the original material - image, sound, image + sound clips - and the manipulation possibilities of the instrumentlets. We have confined this now to variable clip speed, variable direction, access to individual frames, and most importantly a general play function that when played against provides feelings of inertia. The palette is limited to one image at a time, although rapid changes and cuts between images are used. This model is roughly the same for the sound manipulation providing a general compatibility and consistency between the use of image and sound. The only significant difference in treatment of image and sound material is that the sound can be built up in a number of layers simultaneously. In order to achieve sufficient depth in the image we use multiple projections from different sources to compose and play with tensions of similarity and difference between the screens. The simplicity of the set up enables us to focus on one key issue - movement. Movement is developed from the original material in the following steps: movement of the camera/microphone or capturing movement within the raw image pre-editing (before live editing) whether in morphs or simple dissolve effects, and selection of clips mapping of certain movements to gestures in the instrumentlets, for example, a circular movement is mapped to a circular turning gesture placing in space by movement between screens and speakers, for example, by panning the sounds in relation to the four points of the Squeezamin or moving the image between two screens The issue of narrative is unavoidable as, however abstract the images and sounds are, when placed together in a sequence they will be naturally understood by the audience as in some way a "narrative". Through our experiences of playing and composing with the video-organ in different circumstances an approach towards structure and narrative has naturally evolved, whereby the general direction and sequence of events is "noted" before performance, leaving a flexibility in precise timings and allowing a certain degree of improvisation into the performances. This flexibility in live performance is one of the main virtues of the video-organ, encouraging an interaction with audience, atmosphere, the two players. The narrative or journey that the audience is taken through can be described as 'elastic' narrative, giving a sense that the time, movement and position in space can be stretched, spring back and be malleably alive. To achieve and use this elasticity is the performers task as much as it is central to the compositional issues of gathering material and mapping to the control surface. The relationship between the Video-Organ instrument and the composition is not comparable to a traditional instrument in that ideas are not communicated through notation (except for the small notes that order the outline). The composition is not 'notated' but is contained within the material and the process of composition can be said to encompass the apparently different tasks of collecting material, editing the movements, designing and making the instrumentlets, to finally performing. The traditional concept of notation is not useful in these circumstances as the content of the composition/performance is made visible by our chosen mapping from clip to instrumentlet. This raises a number of questions. Where is the notation? Does the notation lie externally to the composition as coded instructions, or internally as the content of the performance? Is it possible to realise notation not as a set of instructions, or simply a visualisation (marking), but as an essential element lying within the content? Does the idea of notation as an interface address similar issues to those of the Video-Organ as an interface? The composition MediaEval, performed as part of the Metronom Experimental Music Festival in Barcelona, January 2002, explicitly addressed these questions, and in other performances these ideas are implicit in the work.

Performing MediaEval Barcelona 2002

Notational Dreaming
So, taking the above as an example, in the understanding of today's increasingly technological society what relevance is notation? We are in a period of notational experimentation and exploration in all fields - what I would call notational dreaming - which suggests at least an expansion, if not a turning over, of the role of meaning through its

representations. My idea of notation now includes moving images, virtual spaces, evolving spaces, forms and structures that change and develop over time. Western society in general is becoming increasingly 'literate' in these new structures - McLuhan's "extensions of man" are developing with profound implications for our understanding of ourselves (McLuhan 1964). I am suggesting a need to redefine notation from something that was one step removed from reality, an externalisation of an idea, a symbolic representation that needed a translation from sound to image and back to sound, to a working idea of notation that incorporates movement. A famous early example of the effect of notational fluidity on space is the composer and architect Iannis Xenakis, particularly the correlation between the plastic shapes depicted in the orchestral score of Metastasis (1954) and the physical reality of the curved concrete planes built of hyperbolic paraboloids (used even earlier by Gaudi) in the Philips Pavilion at the Brussels World Fair (1958), itself designed as an audio-visual exhibition (Matossian 1986, Treib 1996). Architectural examples of the influence of movement extend throughout the century to present-day experiments with computer generated graphical tools that encompass a notational fluidity that cannot yet be achieved in the solid plastic reality of an actual building, although attempts are made and it is closer than ever.

Iannis Xenakis string glissandi in Metastasis 1954

Philips Pavilion, Le Corbusier, Varese, Xenakis 1958

The rethinking of the concept of space becomes central to this discussion of notation and its potential direction. A Renaissance representation of three dimensional space used the two dimensional technique of perspective which centres on a fixed, immovable point. The contemporary realisation of space rests on the individual experience of space and time, or space described by what moves within it rather than by its physical characteristics. The attitude to space becomes a place where a viewer can explore and create their own individual experiences - an open structure where the audience 'interacts' with the space (Bongers 2002). The contemporary discourse of liquid architecture, resides in notations that communicate these spaces as a fluid, temporal, even foldable, individual experience. These and more recent developments suggest that notations must also be changeable and flexible, able to grow and develop, fluid, moving and spatial. Once our form of communication and way of understanding the world relies totally on a basis of many dimensional spaces that are fluid, that change in real time, that cannot be related to a static fixed point, what is a notation? Is it a 'marking' that can transform itself and if so what relation does it have to the object it 'communicates'? Does notation become autonomous? I would suggest that we have to realise how notation is becoming the subject itself, an architectural material, an integral part of experience not a detached representation of that. This suggests an element of notational dreaming that may be comparable to the emergence of musical notation in Mediaeval times. It is my hope to build an understanding of these recent experiments on a basis of historical notations and recent movement based ideas.

Stills from Paradise Dublin 2002

References:
Bongers, Bert 'The Use of Active Tactile and Force Feedback in Timbre Controlling Musical Instruments'. Proceedings of the International Computer Music Conference (ICMC), rhus, Denmark Bongers, Bert (2002); 'Interactivating Spaces'. Proceedings of the Symposium on Systems Research in the Arts, Baden-Baden, Germany Bongers, Bert and Harris, Yolande (2002); 'A Structured Instrument Design Approach: The Video-Organ'. Proceedings of the NIME conference on New Interfaces for Musical Expression Dublin, Ireland Cage, John Notations (Something Else Press, New York, 1969) Calvino, Italo (1974) Invisible Cities (Vintage, London 1997) Calvino, Italo (1985) Six Memos for the Next Millennium . (Vintage, London, 1992) Cardew, Cornelius Treatise and Treatise Handbook (Peters, London 1971) Carr, Carlo D 'From Czanne to Us, the Futurists' , (Florence: Lacerba, 1913) cited in (Chipp, 1968), p304 Chion, Michel Audio-Vision: Sound on Screen (Columbia University Press, 1994) ISBN 0-231-07899-4. Translated from French, L'Audio-Vision (Editions Nathan, Paris 1990) by Cludia Gorbman. Furlong, William Audio Arts: Discourse and Practice in Contemporary Art (Academy Editions, London 1994) Giedion, Sigfried Space, Time and Architecture, the growth of a new tradition. 1941. Fifth edition, (Harvard University Press, Cambridge MA, 1967) Harris, Yolande (2000); 'Explorations in Movement, Towards the Symbiosis of Architecture, Moving Image and Music'. MPhil Thesis, Cambridge University Architecture Department Imperiale, Alicia New Flatness: Surface Tension in Digital Architecture (Birkhuser, Basel 1999?) part of The information Technology Revolution in Architecture series. Klee, Paul (!930's) Notebooks Volume 1, The Thinking Eye. Translated from German, Das Bildnerische Denken, (Schwabe & Co. Verlag, Basel, 1956). (Lund Humphries Publishers, London, 1961) Lippard, Lucy R. Six Years: The Dematerialization of the Art Object from 1966 to 1972 (Studio Vista, London 1973) Livermore, Ann A Short History of Spanish Music (Duckworth, London 1972) Marinetti, F.T. Futurist Painting: Technical Manifesto. Originally published as a pamphlet in Milan, 11 April 1910. Cited in (Chipp, 1968) p289 Matossian, Nouritza Xenakis (Kahn and Averill, London 1986) McLuhan, Marshall (1964)Understanding Media: the extensions of man (1994 Routledge, London) McLuhan, Marshall Verbi-Voco-Visual Explorations (Something Else Press, New York 1967) McLuhan, Marshall Essays: Media Research: Technology, Art, Communication ed. Moos, Michael A. (G+B Arts International, Amsterdam 1980) Michelson, Annette (ed.) Kino-Eye: The Writings of Dziga Vertov (Pluto Press, London 1984) ISBN 0-86104-767-2 Neuhart, Nuehart & Eames Eames Design, The Work of the Office of Charles and Ray Eames (Thames and Hudson, London 1989) Oosterhuis, Kas 'The Wild Side of Architecture' 1999 interview with Marco Brizzi www.architettura.it/interview/19991100/index_en.htm Paradiso, Joe (2002); 'The Edge of NIME - from Cult to Conference'. Proceedings of the NIME conference on New Interfaces for Musical Expression Dublin, Ireland de Sola-Morales, Ignasi 'Liquid Architecture' from the conference Anyhow held in Netherlands Architecture Institute, Rotterdam, Holland, June 1997. Published in Davidson, Cynthia C.(ed.) Anyhow, (Cambridge: MIT Press with Anyone Corporation, 1998) website: www.anycorp.com/index2.html Stockhausen, Karlheinz Oktophonie: Electronic Music of Tuesday from Light , 69'30 minutes, realised at the Studio for Electronic Music of the WDR Cologne 1991, Compact Disc and notes by Stockhausen (Gema 1994) Tarkovsky, Andrei Sculpting in Time: Reflections on Cinema translated from the Russian by Kitty Hunter-Blair (University of Texas Press, Austin 1986) ISBN 0-292-77624-1 Treib, Mark Space Calculated in Seconds: The Philips Pavilion, Le Corbusier, Edgard Varese (1996, Princeton University Press) Vertov, Dziga Man with a Movie Camera (USSR 1929) black & white, silent 137mins Virilio, Paul 'The Overexposed City', "La Ville Surexpose", from L'espace critique (Paris, Christian Bourgeois, 1984): translated in Zone 1-2 (New York, Urzone, 1986) by Astrid Hustvedt. From (Hays, 1998) pp540-550