ENGAGEMENT AND INTERACTION IN PARTICIPATORY SOUND ART

ENGAGEMENT AND INTERACTION IN PARTICIPATORY SOUND ART Visda Goudarzi Institute of Electronic Music and Acoustics University of Music and Performing Arts Graz, Austria [email protected] ABSTRACT This paper explores a variety of existing interactive and participatory sound systems and the role of different actors in them. In human computer interaction (HCI), the focal point on studying interactive systems has been the usability and functionality of the systems. We are trying to shift the focus more towards creative aspects of interaction in both technology development and sound creation. In participatory sound art, the roles of technology creator, composer, performer, and spectator are not always distinct but may overlap. We examine some challenges in such systems, like the ownership of technical and aesthetic components and balancing engagement and interaction among different stakeholders (designer, composer, spectator, etc). Finally, we propose a discussion on participation, human-computer and human-human interaction within the process of creation and interaction with the system. 1. INTERACTIVE SOUND SYSTEMS The process of design and development of interactive sound systems used to be a separate task from sound creation. From design and development to performance, there used to be a linear flow (Figure 1). New interfaces for sound creation, however, allow sound artists and composers to engage themselves more in the process of system design and development. Artemi-Maria Gioti Institute of Electronic Music and Acoustics University of Music and Performing Arts Graz, Austria [email protected] approach aims to direct attention away from the system’s accuracy and efficiency. We are more intrigued by nonquantifiable goals and notions such as creativity and engagement. In terms of interaction, we find Suchman’s perspective [7] appropriate for interactive and particularly participatory sound systems. She moves away from a goal-oriented interactivity and meaningful action towards a concept of interactivity in which action is central and goals are emergent. Furthermore, the iterative development of software engineering is not necessarily transferrable to artistic creation. In artistic production iterative processes are part of the creative experimentation and not part of the evaluation of a completed artwork [8]. 1.1 Stakeholders and the scope of interaction In traditional sound making systems (e.g. musical instruments), the designer of the sound system was usually a different person than the musicians (composers and performers). Due to rapid technological advancements, the gap between designers and sound makers is getting smaller. Examples include the democratization of sound and musical instruments, the evolution of Internet, open source software, community based design and DIY (Do It Yourself) instruments. However, sound making is usually left to composers and musically trained people. There are only a few sound artworks that allow people with little or no experience in sound or music to participate in the creative process and potentially increase their awareness about sound in their surroundings improving their analytical listening skills [9]. In HCI and software development, an iterative approach of design and development is common where evaluations allow to improve the system in successive iterations. Applications of adapted iterative HCI methods in sound creation range from interaction design to creativity support in sound and technological domains. Studies which incorporate HCI methods to evaluate sound creation systems are mostly focused on how musical tasks are performed. Aspects evaluated might be related to performance [1], the quality of the user experience and the degree of expressiveness [2, 3, 4], the usefulness of the system [5] or participant’s/audience’s engagement [6]. Our Copyright: © 2016 Visda Goudarzi and Artemi-Maria Gioti. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Figure 1. An overview of the flow of actions in a traditional interactive sound system. Furthermore, in the last decade computing systems such as tablets and mobile devices, which used to be specifi- cally deployed by engineers and experts, have become more popular among the general public. This has led to a different media ecology, extending the cultural context for interactions through consumer devices and creating a new platform for engagement in participatory art, by using one’s own devices. Having such powerful -potentially sonic- devices in hand, the space becomes the instrument and the stage of creativity that has so far been confined to a small elite of educated musicians expands [10]. In the last decade, research has focused on the development of methods to assist music composition by hiding part of the complexity of creation from the user [11]. This is on the one hand a democratization of sound or any content creation, but on the other hand it can add to the sonic pollution in public spaces. You may hear up to 20 different sounds within a minute coming out of a mobile phone because of different apps. At the same time this has created a new condition for artistic creation: art is not presented only in museums and galleries; it has rather expanded to public spaces much easier and faster than before. 1.2 Design and Development HCI design process could be a goal-oriented problem solving activity informed by intended use, a creative activity, or a decision-making activity to balance trade-offs (e.g. requirements of product compatibility and ease of use may be contradicting.) In a HCI design process there is usually a plan for development and a set of alternatives and successive elaborations. Whether the main design decisions are made by the composer or the designer depends on the goals and complexity of the system. Available computer languages could also have a huge influence on the focus of the design process. McCartney [12] describes the early computer music languages as strong in abstractions but providing very few control and data structures and little or no user functions. Later on, computer music languages such as Max and Pd allowed abstractions, so that in some cases the users are not even noticing that they are programming. These languages allowed for more participation of composers and performers in the process of programming sound systems, without requiring extensive software engineering training. Most importantly, the orientation of these programming languages towards live interaction brought HCI into the focus of both composition and performance, bringing about important changes in both fields. 1.3 Composition and Performance The growing availability of electronic devices and audio programming languages has lead not only to a merging of instrument/interface design and composition, but also to a new understanding of composition and performance as interactive processes. Human-computer interaction in the composition and performance of live electronic music ranges from interface-based interaction, employing human-computer interfaces that translate the performer’s actions into sound, to responsive and autonomous agent- based systems, able to interact with human performers in real-time. The latter approach in particular has broadened our understanding of composition and the traditional roles of composer and performer. By delegating some responsibility to machine agency, the real-time interaction between the performer and the computer is transformed into a dynamical and reciprocal process of communication between human and machine agency. Due to this integration of human and machine agency the spectrum between "fixed" composition and free improvisation is becoming “increasingly densely populated” [14], while the borders between performer and composer, as well as instrument and composition are becoming obscure. An example of the merging of the traditional roles of composer and performer is interactive composing. Chadabe describes interactive composing as ‘a two-stage process that consists of (1) creating an interactive composing system and (2) simultaneously composing and performing by interacting with that system as it functions’ [15]. In interactive composing systems, also known as real-time composition systems [16], the composer is also the performer, while it is impossible to distinguish between composition and performance, since they occur simultaneously. A different approach to interaction in the composition of live electronic music are Di Scipio’s audible ecosystems, in which the field of interaction is defined as the triangular connection among a human agent, a DSP unit and the sonic ambience [17]. In Di Scipio’s audible ecosystemics interaction is expanded to include not only the human agent and the computer, but also the ambience itself as a performance agent. Further examples of interaction strategies in live electronic music performance are performance networks and virtual player systems. Performance networks are technology-based systems that enable remote or co-located collaborative musical performance. Weinberg [18] categorises interconnected musical networks in three different approaches: the server approach, in which the network is limited to the individual interaction between each player and the system, the bridge approach, that enables collaborative performances among performers that are in different locations, and the shaper approach, in which the system uses algorithmic processes in order to generate musical material that the performers can modify collaboratively. Virtual player systems are systems that fall into the player category of Rowe’s taxonomy [19]. In these systems, a virtual player learns from one or more human improvisers and responds to them in real-time. Examples of virtual player systems are those developed by Lewis [20], Bakht and Barlow [21] and Dubnov et. al [22]. Such systems usually employ machine learning in order to produce musical material that is similar to that played by the human improviser. For this reason, the material generated by the virtual player is in most cases pitch-based. This, in addition to their machine learning based approach, makes such systems more suitable for improvised performances, than for compositional applications. Finally, a machine learning, but not pitch-based approach was followed by the creators of the Wekinator software, designed for end-user interactive machine learning [23]. With the Wekinator, the user (performer/composer) can create desired gesture-sound mappings by training a learning algorithm. The software is end-user oriented, enabling musicians to work with intelligent systems without requiring any programming knowledge. However, its approach is in fact interface-based: even though it uses machine learning, its functionality is restricted to gesture-sound mappings. 1.4 Audience participation and engagement Expanding the field of interaction beyond composition and performance to audience participation has usually overlapped with the creation of multi-user instruments, which have switched the role of a passive audience (or spectator) to an active player. Dixon [24] identifies four types of interaction based on different levels of engagement: navigation, participation, conversation and collaboration. which is a sonification of tweets from the audience played by a laptop ensemble. The piece is not as precisely choreographed as Telesymphony, which gives the audience a certain freedom to “compose” (at least composing their own tweets) within the framework of the piece. Still conducting the piece is left in the hands of the designer. The individual cannot change the direction of the whole structure of the ensemble, but has at least control over his/her own sounds (or tweets). Ximena Alarcon’s [30] Sounding Underground is an example of leaving the creative aspects to the user. It’s an online interactive sonic environment which links sound experts from metros of London, Paris, and Mexico City. She translates the public transport into interactive multimedia using interactive ethnography to involve participants’ perception of space. 2. ASPECTS OF AUDIENCE PARTICIPATION From these examples it is clear that participatory sound systems display a wide range of both participation and interaction strategies. Some of the most important parameters of participatory sound systems design are discussed in detail below: 2.1 Audience engagement Engaging the audience is not a new approach in sound art, but engaging them to the extent of being the main creators has not been explored in depth. E.g. John Klima’s Glasbead [25] is a great example of multi-user collaborative musical interface used to engage 20 or more remote players with each other. Another example is Golan Levin’s Telesymphony [26], in which he choreographed ringing of audiences’ mobile phones. In this example, the audience doesn’t have any control over the structure of the piece or the creation of the sounds. They are almost passive users with an active instrument in hand that is mostly controlled by the composer and creator of the piece. In recent years, since crowd sourcing and creating content by users have become more common, interfaces that take advantage of that also entered the music world, such as Kruge’s MadPad [27]. He uses the audio/visual content that the audience sends before the performance, during the concert. In this approach, the users create the whole content and the performer only uses algorithms to compose with it. However, the audience is not participating in real time. Other real-time applications are: Tweetscapes [28] and TweetDreams [29]. Tweetscapes is a project of sonification experts, media artists, and a radio broadcaster. Online data from German Twitter streams is sonified and visualized in real-time. The sounds are based on a large sound database and randomly – but reproducibly fixed – assigned to different semantic terms (hashtags). These sounds are then modified according to metadata, e.g. from which location in Germany the tweet was sent. Another example is TweetDreams by Dahl et.al. Harries [31] refers to authorship, performance and spectatorship as different types of audience participation using the terms performance and authorship as interchangeable. We would like to distinguish between three types of audience participation, with increasing level of engagement: crowdsourcing, performance agency and co-authorship. Crowdsourcing, in general, is a paradigm for the use of human processing power to solve problems. Computational systems where the crowd performs tasks to solve problems in the context of computer music are very common, especially in the field of music information retrieval. To name a few: Mechanical Turk that uses people’s opinion to find similarities between songs [32], Last.fm [33] and Freesound [34] that use the crowd sound sample collection and music library management. In participatory systems crowdsourcing refers to audiencecomputer interaction systems that allow a large crowd to participate in the process of sound making mainly by functioning as a source of data, such as TweetDreams [29], Flock [35] and One Man Band [36]. For instance, in TweetDreams, audience members use their personal mobile devices to tweet. Tweets containing hash-tags (chosen by performers) are sonified and visualized into a dynamic network. In performance agency the role of the audience is similar to that of a performer. Unlike crowdsourcing, in performance agency audience engagement is active and involves real-time control over sound parameters that are determined by the composer/designer. Even though com- positional decisions are made by the creator of the system, the audience is able to explore the ‘space’ defined by the composer/designer and interact with it by setting runtime control data. However, despite the active participation, performance agency as a form of audience engagement implies a hierarchically structured interaction model, based on the dichotomy between the creator/designer and spectator/performer. An example of performance agency as an audience engagement strategy is Auracle. In Auracle the users can control a synthesized instrument with their voice, while interacting with other users in real time over the Internet [37]. The most active form of audience participation is coauthorship. In this case, the spectator is not just a performer, but co-author in the process of sound creation. Instead of setting the values of a fixed set of run-time control variables, the audience is invited to participate in the creative process, by making compositional decisions regarding both the sound material and the processes applied to it. In this case, the designer’s role is limited to the creation of a platform that enables collaborative sound creation, while it involves little to no compositional responsibility at all. Co-authorship is the most democratic form of audience participation and the one that has been explored the least by participatory systems so far. 2.2 Human-computer interaction Our discussion of human-computer interaction in participatory sound systems focuses on the aspects of control and mapping. Instead of discussing technical aspects of HCI, like functionality or usability, we prefer to focus on different strategies in the design of audience-system interaction in participatory systems. The aspects of HCI that we examine here are: ‘multiplicity of control’ [14], type of control, mapping of control actions, control parameters and ‘control modality’ [14]. - Multiplicity of control: By multiplicity of control we refer mainly to the differentiation between single-user and multi-user systems. In single-user systems, only one user can interact with the system at a given moment, while in multi-user systems more than one users can interact with the system simultaneously. The concept of multiplicity of control is not necessarily limited to human agency, but can also include machine agency, meaning that the system itself can perform control actions. An example of an interactive sound system with multiple control channels is the reacTable [27]. - Type of control: The type of control refers to the different human-computer interfaces that can be used as part of the audience-system interaction. Like in live electronic music performance, this interaction can be tangible and embodied (e.g. Michael Waiswisz, The Hands) or disembodied (e.g. Alvin Lucier, Music for solo performer), non-tactile etc. - Mapping of control actions: Mapping control actions to sound parameters is perhaps the most important part of sonic human-computer interaction design. Mapping processes can be linear (a simple scaling of input values to control values) or non-linear and based on dynamical processes (e.g. dynamical systems modeling, machine learning etc.). An example of a linear mapping process is assigning the keys of a MIDI keyboard to pitches, while an example of a dynamical mapping process, based on machine learning, is the software Wekinator [23]. The type of the mapping process affects the level of perceived control and transparency (i.e. how perceivable the relationship between control actions and sound output is). - Control parameters: Sound parameters, the value of which can be set by the user. - Control modality: Control modality refers to the type of control value (discrete or continuous) [14] and depends on the control parameter itself, as well as the type of the interface. For example, faders allow for continuous control and are more suitable for controlling a parameter like amplitude, while buttons could be used for triggering prerecorded samples. 2.3 Human-human interaction Human-human interaction in participatory sound systems has evolved excessively with the growth of internet and social media. Server-based cloud computing has enabled the audience to participate in performances in active or passive roles without the need for special technical background or pre-configuring hardware/software. The cost effectiveness, familiarity, and ease of use of some technological devices have also made the entrance to participation easier. Furthermore, the roles of performer, composer and audience have become more interchangeable and the participation has added more uncertainty factor to performances which is on the one hand technologically and artistically challenging, and on the other hand compelling. Some factors that influence the communication and interaction between audience members, or the audience and other performers are: - Location: according to Barbosa [38] collaboration in participatory performances could take place within remote or co-located network music systems. In the former, people could participate in the performance from different parts of the world (e.g. SoundWIRE group [39]) or could be even in the same room using different computers connected to local networks. In both cases they share the same sonic environment. Since these systems create a platform of synchronous improvisations for a broad audience, participants usually don’t need to be experts. Another possibility is that participants even share the same physical interface or device. (e.g. in table-top instruments such as the reacTable. [27]) - Levels of communication: some collaborative environments only allow participation of expert musicians in the participatory performance whereas some others encour- age collaboration between experts and novices. In the latter, the communication is more focused on a performative involvement of the audience. In other cases a master performer reacts or communicates back to the audience [40] during the performance by shaping sounds or data received from the audience (hierarchical collaboration). Another level of communication is the interaction among audience members. However, what determines a successful sonic Human-Computer Interaction and how can participatory design encourage audience engagement are questions that still need to be answered. Especially, in the case of co-authorship participatory design seems to be compelling, both aesthetically and technologically. 3. DISCUSSION [4] O. Bau, A. Tanaka and W.E. Mackay, “The A20: Musical Metaphors for Interface Design,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2008. [5] T. Coughlan and P. Johnson, “Interaction in Creative Tasks: Ideation, Representation and Evaluation in Composition,” in Proceedings of the ACM Human Factors in Computing Systems, CHI 2006. [6] N. Bryan-Kinns and F. Hamilton, “Identifying mutual engagement,” Behaviour & Information Technology, 31(2), pp. 101-125, 2009. [7] L.A. Suchman, Plans and Situated Actions: The Problem of Human-machine Communication. Cambridge: Cambridge University Press, 1987. In this paper we discussed some challenges in participatory and interactive sound systems, focusing on audience participation and engagement. Technological advancements and new artistic concepts have lead to a closer collaboration among the traditionally distinct fields of design, composition and performance and enabled various forms of audience participation. Participatory strategies in sound art can vary from a passive participation in which the audience functions simply as a source of data (crowdsourcing) to active participation in the performance and creation of an artwork (performance agency and co-authorship). The last two approaches are as radical as they are challenging, bearing implications for both the spectator and the creator/author. Questions regarding authorship inevitably arise as a result of this shift of creative responsibility: is a participatory sound work the work of the artist who designed it or is it a creation of the participants? How does this democratization of the creative process affect its goal? Is the goal of the creative process still the artifact or does the goal shift from the aesthetic artifact to the interaction itself? And, finally, how does collective creative responsibility affect the aesthetics and perception of the artwork? [8] Z. Bilda, E. Edmonds and L. Candy, “Designing for creative engagement,” Design Studies, 29(6), pp. 525-540, 2008. 4. REFERENCES [15] J. Chadabe, “Interactive Composing: An Overview,” in Computer Music Journal, 8 (1), pp. 22-27, 1984. [1] M. M. Wanderley and N. Orio. “Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI,” Computer Music Journal, 26(3), pp. 6276, 2002. [2] D. Stowell, M.D. Plumbley and N. Bryan-Kinns, “Discourse analysis evaluation method for expressive musical interfaces,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2008. [3] C. Kiefer, N. Collins and G. Fitzpatrick, “HCI Methodology For Evaluating Musical Controllers: A Case Study,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2008. [9] R.I. Godøy, “Gestural-Sonorous Objects: embodied extensions of Schaeffer's conceptual apparatus,” Organised Sound, 11(02), pp.149-157, 2006. [10] D. Cope, Computer models of musical creativity. Cambridge: MIT Press, 2005. [11] H.C. Jetter, F. Geyer, T. Schwarz, and H. Reiterer “Blended Interaction-Toward a Framework for the Design of Interactive Spaces,” in Workshop DCIS. Vol. 12, 2012. [12] J. McCartney, “Rethinking the computer music language: SuperCollider,” Computer Music Journal, 26(4), pp. 61-68, 2002. [13] M. Wooldridge and N. Jennings, “Agent Theories, Architectures, and Languages: a Survey,” in Intelligent Agents, M. Wooldridge and N. Jennings, Ed. Berlin: Springer-Verlag, pp. 1-22, 1995. [14] J. Pressing, “Cybernetic Issues in Interactive Performance Systems,” in Computer Music Journal, 14 (1), pp. 12-25, 1990. [16] A. Eigenfeldt, “Real-time Composition as Performance Ecosystem,” in Organised Sound, 16 (2), pp. 145-153, 2011. [17] A. Di Scipio, “Sound is the interface: From interactive to ecosystemic signal processing,” in Organised Sound, 8 (3), pp. 269-277, 2003. [18] G. Weinberg, “Interconnected Musical Networks: Toward a Theoretical Framework,” in Computer Music Journal, 29 (2), pp 23-39, 2005. [19] R. Rowe, Interactive Music Systems: Machine Listening and Composing. London: MIT Press. 1993. [20] G.E. Lewis, “Too Many Notes: Computers, Complexity and Culture in "Voyager",” Leonardo Music Journal, 10, pp. 33-39, 2000. [21] S. Bakht, and C. Barlow, “PAPAGEI: An Extensible Automatic Accompaniment System for Live Instrumental Improvisation,” in Proceedings of the International Computer Music Conference, ICMC 2009 pp. 521-523, 2009. [22] S. Dubnov, G. Assayag, O. Lartillot and G. Bejerano, “Using Machine-Learning Methods for Musical Style Modeling,” IEEE Computer, 10 (38), 2003. [23] R. Fiebrink and D. Trueman, “End-User Machine Learning in Music Composition and Performance,” Present. CHI 2012 Work. End-User Interact. with Intell. Auton. Syst. Austin, Texas, May 6, 2012, pp. 14–17, 2012. [33] X. Hu, M. Bay and J. S. Downie, “Creating a Simplified Music Mood Classification Ground-Truth Set,” in Proceedings of the International Society for Music Information Retrieval Conference, ISMIR 2007, pp. 309-310, 2007. [34] F. Font, G. Roma, and X. Serra, “Freesound technical demo,” in Proceedings of the 21st ACM International Conference on Multimedia, pp. 411-412, 2013. [35] J. Freeman, and M. Godfrey, “Creative collaboration between audiences and musicians in Flock,” Digital Creativity, 21(2), pp. 85-99, 2010. [36] J. N. Bott, J. G. Crowley and J. J. LaViola Jr., “One Man Band: A 3D Gestural Interface for Collaborative Music Creation,” in Proccedings of the Virtual Reality Conference, VR 2009, pp. 273-274, 2009. [24] S. Dixon, Digital Performance: A History of New Media in Theater, Dance, Performance Art, and Installation. Cambridge, MA and London: MIT Press, 2007. [37] J. Freeman, K. Varnik, C. Ramakrishnan, M. Neuhaus, P. Burk and D. Birchfield, “Auracle: a voicecontrolled, networked sound instrument,” Organised Sound, 10(03), pp. 221-231, 2005. [25] C. Paul, “Renderings of digital art,” Leonardo, 35(5), pp. 471-484, 2002. [38] Á. Barbosa, “Displaced soundscapes: A survey of network systems for music and sonic art creation,” Leonardo Music Journal, 13, pp. 53-59, 2003. [26] G. Levin, “A personal chronology of audiovisual systems research,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2005, pp. 2-3, 2005. [27] S. Jordà, “Multi-user instruments: models, examples and promises,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2005, pp. 23-26, 2005. [28] T. Hermann, A. V. Nehls, F. Eitel, T. Barri, and M. Gammel, “Tweetscapes - real-time sonification of twitter data streams for radio broadcasting,” in Proceedings of the International Conference on Auditory Display, ICAD 2012. [29] L. Dahl, J. Herrera, and C. Wilkerson, “Tweetdreams: Making music with the audience and the world using real-time twitter data,” in Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2011, pp. 272-275 2011. [30] X. Alarcón Díaz, “An interactive sonic environment derived from commuters' memories of the soundscape: a case study of the London Underground,” Ph.D. dissertation, Music, Technology and Innovation Research Center, De Montfort University, Leicester, United Kingdom, 2007. [31] J. H. Lee, “Crowdsourcing Music Similarity Judgments using Mechanical Turk,” in Proceedings of the International Society for Music Information Retrieval Conference, ISMIR 2010, pp. 183-188, 2010. [32] G. Harries, “‘The open work’: ecologies of participation,” Organised Sound, 18(01), pp. 3-13, 2013. [39] J. P. Cáceres and C. Chafe, “JackTrip: Under the hood of an engine for network audio,” Journal of New Music Research, 39(3), pp. 183-187, 2010. [40] G. Paine, “Interactivity, where to from here?,” Organised Sound, 7(03), pp. 295-304, 2002.