Art and Affect in the Predictive Mind - Book of Abstracts

Art and Affect in the Predictive Mind 30 June - 2 July 2021, Online Book of Abstracts Art and Affect in the Predictive Mind 30 June - 2 July 2021 Book of Abstracts Edited by: Jacopo Frascaroli Tracey Davison Genevieve Stegner-Freitag Organising Committee Jacopo Frascaroli Firat Altun Monika Axmannova Tracey Davison Genevieve Stegner-Freitag Conference website: https://sites.google.com/view/artandaffectinpp Supported by: Contents Keynote Talks Sander Van de Cruys Preferences Need Inferences…………………………………………………………………………………………………………..5 Ladislav Kesner ‘Hole in a Cardboard´ and Bayesian Brain. Misunderstanding of Modern Art in Light of the Predictive Processing Paradigm………………………………………………………………………………………………………6 Karl J. Friston In the I of the Beholder……………………………………………………………………………………………………………………7 Claudia Muth Unsolvable, yet Insightful - Semantic Instability in Art…………………………………………………………………….8 Diana Omigie “That the Attention Not Be Suffered to Dwell Too Long”: Or Why We Value Information in Music…9 Peter Vuust Groove on the Brain: Predictive Brain Processes Underlying Musical Rhythm and Interaction………10 Main Sessions Session 1 Joerg Fingerhut Wonder, Play, and Radical Predictive Processing of Art………………………………………………………………..11 Blanca T. M. Spee Art as Hypotheses-Exploring Game-Space: An Enactive Predictive Processing Perspective on Art Engagement…………………………………………………………………………………………………………………………………12 Session 2 María Jimena Clavel Vázquez and Adriana Clavel-Vázquez The Rhythm of the Eye: Expectations, Imagination, and Aesthetic Perception…………………….………..13 Mathijs Tratsaert Integrating Decorum with PP Accounts of Vision and Visual Art………………………………….………………..14 3 Session 3 Mark Miller The Joy of Horror Films: Active Inference, Aesthetic Pleasure and the Paradox of Horror……………..16 Kathrine Cuccuru “A Dinner Reservation for 12”: John Wick and Predictive Processing…………………………………………….17 Session 4 Max Jones and Sam Wilkinson Creative Imagination in the Enculturated Predictive Mind…………………………………………………………….18 Federico Pianzola and Karin Kukkonen Presence, Flow, and Narrative Absorption: A New Integrated Model based on Predictive Processing…………………………………………………………………………………………………………………………….………20 Session 5 Vincent Cheung Connecting Musical Expectancy and Pleasure: Insights from Cognitive Neuroscience……………………22 Pietro Sarasso The Pleasure of Learning: A Perceptual Learning Hypothesis for Aesthetic Appreciation………………23 Session 6 Iris Mencke Exploration Revisited – New Music as a Model for Exploratory Behavior Under High Uncertainty Conditions…………………………………………………………………………………………………………………………………....24 Renee Timmers Using Bayes’ Rule to Model Emotional and Sensory-Motor Associations with Music………………….…25 4 Keynote Talks Preferences Need Inferences Sander Van de Cruys University of Antwerp [email protected] More than 40 years ago, pioneering social psychologist Robert Zajonc (1980) published his seminal work titled “Preferences need no inferences” in which he argued for the primacy of affect over cognition. Affective evaluation (the preference) comes first, he claimed, and only then do cognitive processes (the inferences) kick in. The view is, of course, untenable in light of recent predictive processing accounts which hold that all mental functioning is built from (approximate) Bayesian inference. Predictive processing casts perception, action, and learning as inference, but, perhaps counterintuitively, valuation too. I will discuss specifically how valuation — understood as the process of how we come to value, prefer, or like things — emerges as a function of inference, and how this conception may help us resolve traditional conundrums in the science of aesthetic experience, such as the nature of the beholder's share, the link between curiosity and appreciation, and the tension between the mere exposure principle and the goldilocks principle. 5 ‘Hole in a Cardboard´ and Bayesian Brain. Misunderstanding of Modern Art in Light of the Predictive Processing Paradigm Ladislav Kesner Masaryk University, Brno; National Institute of Mental Health, Klecany [email protected] My talk is meant to bring a rather sceptical tone into the topic of art in the predictive mind. Current attempts to link the predictive processing framework and visual arts appear to be akin to what has been called “Bayesian just-so stories.” In considering the way forward, a critical question concerns the level of granularity at which it makes sense to apply predictive processing to art phenomena. I shall argue that anything like a ´Predictive processing account of art experience´ is unlikely to succeed, and a better strategy might be to focus on narrowly circumscribed issues. In the main part of my talk, I shall attempt to test whether the prediction error minimization framework can be productively applied to a specific problem of art history and theory, namely misunderstanding of modern and contemporary art. This will require a struggle with the thorny problem of how highlevel (cultural) hyperpriors operating in art perception are to be defined and related to sensory predictions. Whether this will yield more than another just-so story remains to be seen. 6 In the I of the Beholder Karl J. Friston Wellcome Centre for Human Neuroimaging, University College London [email protected] This presentation attempts a first principle account of sentience – and how we make sense of the world. It starts with basic considerations from statistical physics to suggest that all self-organising – or perhaps self-evidencing – systems (i.e., particles or people) are compelled to minimise a certain kind of free energy. This free energy can be construed as prediction error, leading to a formulation of sentience in terms of actively minimising prediction error (i.e., surprise) by selectively sampling those sensations that we predict: in other words, creating our own sensorium. This line of argument can be taken further to include planning our (visual) palpation of the world to resolve uncertainty (i.e., expected surprise), leading to a quintessentially constructivist and enactive view of perception. I hope to illustrate these ideas using formalisms from statistical physics and in silico simulations of active inference. 7 Unsolvable, yet Insightful - Semantic Instability in Art Claudia Muth University of Bamberg [email protected] We do not passively receive information but interact with our environment, explore structures and look for meaningful patterns. Predictive Processing provides a powerful theoretical framework that explains these processes as a constant matching between sensations and predictions about the cause of the stimulation. Organisms seek to reduce uncertainty by updating these predictions. But while we seem to strive for semantic stability, we sometimes produce and expose ourselves to objects and situations that challenge sense-making, break perceptual habits, or offer a variety of possible meanings. The term of Semantic Instability describes such phenomena as multi-stability, dichotomy, visual indeterminacy and delayed recognition. This talk attempts to show how artworks challenge perceptual habits and at the same time (or because of that) promote active sense-making that can eventually lead to pleasurable insights. These insights, however, do not necessarily resolve Semantic Instability. They may even reveal ambiguity or integrate contradictory elements. The theory of Predictive Processing might help to understand what motivates such unstable yet insightful sense-making if we take a process-oriented view of art perception. 8 “That the Attention Not Be Suffered to Dwell Too Long”: Or Why We Value Information in Music Diana Omigie Goldsmiths, University of London [email protected] A middle point between us identifying a gap in our knowledge and acting in order to fill that gap, curiosity plays an important role in driving learning in many everyday situations. However, while research into this epistemic state is increasing in intensity across different domains, it remains to be established how, or in fact if, it should be operationalised or studied in the context of music listening. Drawing on theory and empirical data, I will use this talk to argue in support of music as a testbed for studying such human information seeking states. Amongst others, I will present a study that highlights the role of individual differences in trait curiosity – alongside music-induced state curiosity – in explaining how and when reward is derived from musical events. I will argue for the importance of invoking the information seeking framework if we are to better understand complex music listening behaviours in everyday life, and will close with recommendations as to how musical stimuli might, in turn, be of yet further use to the cognitive neurosciences. 9 Groove on the Brain: Predictive Brain Processes Underlying Musical Rhythm and Interaction Peter Vuust Center for Music in the Brain; Aarhus University; Royal Academy of Music, Aarhus [email protected] Musical rhythm has a remarkable capacity to move our minds and bodies. When we listen to “Blame it on the Boogie” by The Jacksons, it is difficult to refrain from tapping a foot or bobbing the head to the beat. Here, I will describe how the theory of predictive processing can be used as a framework for understanding how rhythm is processed and why we move to certain kinds of music more than others. Importantly, music is fundamentally a social phenomenon: we listen to, synchronize to, and make music together. This interaction with music is typically based on the agreement on predictive structures such as meter or tonality. I will end the presentation by presenting a new line of studies showing how predictive coding can be applied to understand the dynamics involved in interpersonal synchronization using a minimal tapping paradigm, where two individuals are placed in separate rooms with headphones and EEG equipment and asked to tap together in different conditions. 10 Main Sessions Wonder, Play, and Radical Predictive Processing of Art Joerg Fingerhut Humboldt-Universität zu Berlin [email protected] Predictive Processing accounts claim that the pleasure we experience in engaging with the arts could be explained by negatively-valenced prediction error and a pleasurable re-instantiation of predictability that is amplified in art-specific ways by the delay of prediction confirmation (Van de Cruys & Wagemans 2011). In this paper we will explore this hypothesis by proposing possible extensions under a more general predictive mind view. In particular, we shall argue that: (1) It is crucial to define some “normal conditions” of our engagement with artifacts (pictures and moving images, language and texts, the built environment) to develop an understanding of how artworks in those domains might challenge our perceptual habits. We will make use of Andy Clark’s notion of “design environments” within a Radical Predictive Processing account that sees cognitive-cultural niches as providing us with pre-structured information regimes. Material culture co-constitutes our expansive habits of exploring the world and provides external, intersubjective models that could themselves be described as predicting the world. (2) Art builds upon such models of the world but it also aims to deviates from them by providing access to what could be called the “extraordinary". This can be achieved by a. providing a plurality of hypothesis (a game space of exploring hypothesis); b. posing particularly challenging hypotheses (by generally twisting our expectations in more radical ways); c. providing novel means of engaging a hypothesis or model of the world. Under all these descriptions, art’s value is related to its ability to provide challenging models of the world and ourselves. (3) Finally, we will relate this view to two philosophical accounts of what we appreciate in the arts. The “wonder hypothesis”, on the one hand, claims that aesthetic appreciation of art as art is directly correlated with the amount of wonder experienced (Fingerhut & Prinz 2018). Several elements of the PC account of art speak to the wonder hypothesis. Wonder as an emotion is mixed in valence (corresponding to the transformation from prediction error and ensuing resolution), and wonder’s subemotional components of strong sensory engagement and cognitive perplexity fit nicely the prediction error minimization strategy required by art stimuli. Wonder also helps to explain the aspects (a) and (b) mentioned above. The cognitive profile of wonder relates to an expansive information processing style and a lower need for cognitive closure (i.e. the tolerance of higher prediction error). The concept of “play", on the other hand, could be instrumental in explaining how we integrate art engagement in predictive routines. It has been recently suggested that we experience a form of facilitating pleasure related to play (Matthen 2017) when engaging with art, which can be employed to explain aspect (c). 11 Art as Hypotheses-Exploring Game-Space: An Enactive Predictive Processing Perspective on Art Engagement Blanca T. M. Spee University of Vienna [email protected] In the last 10 years, Van de Cruys and Wagemans (2011), Friston (2012), and Clark (2016) among others have put forward a pioneering account of humans’ interaction with unpredictable stimuli – such as visual artworks – in the context of the predictive processing (PP) framework. Although the PP approach ultimately argues that our mind aims to maximize predictability or minimize prediction error, the hierarchical architecture of our neural network seems to require a certain amount of unpredictable stimulation. With a certain degree of update in our active inference protocols, we not only realize the potential diversity of predictions used in our generative models, but we also gain confidence in our representations. Our goal in this talk is to add some aspects to this framework and to introduce the idea that art engagement is a dynamical action within a hypotheses-exploring game-space. Guided by former theories (Dewey, Heidegger, Berlyne), we claim that: (1) artworks are symbolic image-using systems that help us see things differently. As agents, they break through the restrictions of habits and prejudices, restructuring our previous understanding in all kinds of areas. Artworks are not only ambiguous, but embody a variety of ideas and a plurality of hypotheses. (2) Humans’ ability to adopt an "artistic modus operandi" is based on culturally learned, habitual routines, hyperpriors anchored in the multi-level brain-body architecture. (3) Such an artistic modus operandi allows an active reciprocal process balancing divergent and convergent thinking (Robinson, 2010) and is guided by precision-weighting mechanisms; by enabling a higher gain in prediction error signalling, we explore earlier expectations, and gain new ways to design our active inference protocols in a space that, (4) can be considered to be safe. In sum, art engagement is an attentive precision-weighting exercise, in which stereotypes are bypassed and new, even deviant, appealing images (Gombrich, 1960) are created. Art engagement seems to be a key to transforming probabilistic models, enabling us to understand environment, emotions, time, space, evaluations, and even the creative process itself (Pelowski et al., 2017). As an example, we present a study describing task-related differences in behaviour and pupillary responses during the observation of ambiguous abstract paintings. Interestingly, the change between an exploitation and an exploration mode seems not only to challenge or enhance processability (fluency effects), but also to influence each mode: depending on the accuracy of our predictions, effort-motivating factors seem to influence the way we deal with art. Finally, we will emphasize the need for a neuroendocrinological, network-oriented model for art engagement within a PP framework. Such a model could not only describe neurodegenerative diseases (e.g., Parkinson's disease) as partial loss of viability in updating generative models, but it could possibly explain why art engagement is precarious – namely because our predictive mind requires the construction of new problem spaces, enabling – in its pure form – the act of participating in the negotiations of meaning. 12 The Rhythm of the Eye: Expectations, Imagination, and Aesthetic Perception Adriana Clavel-Vázquez University of Oxford [email protected] María Jimena Clavel Vázquez University of Tartu [email protected] Perceptualism, the claim that we can perceive high-level, evaluative properties, is controversial. However, aesthetic properties have long been thought to be perceptible in that they are directly perceived. We do not reason out of principles that Picasso’s Guernica is violent, but are struck by its violence all at once. This paper argues for aesthetic perceptualism. Building on an anticipatory view of perception that draws on the sensorimotor theory and predictive approaches to perception, we argue that aesthetic properties are brought into perceptual presence when low-level properties of objects are organized in virtue of perceivers’ sensorimotor and affective expectations. Our thesis is that the rich expectations posited by anticipatory views of perception to account for perceptual content explain that aesthetic properties feature in perceptual experience. Aesthetic and nonaesthetic cases, however, are distinguished by the precise role played by our imaginative capacities. While in non-aesthetic cases current sensory stimuli plays a more determinant role, in aesthetic cases these stimuli serve only as an anchor and our imaginative capacities are given more free reign. 13 Integrating Decorum with PP Accounts of Vision and Visual Art Mathijs Tratsaert Independent scholar [email protected] How do neural events in the brain relate to the experience of perceiving a work of art? An increasingly established framework within which this question is examined is that of predictive processing (PP). According to PP, the brain anticipates sensory input with a flow of multi-layered predictions trying to account for incoming signals. In tandem with energetic stimulations provided by external stimuli, conscious and unconscious expectations constitute our experience of the world. Both artists and art theorists have been aware of the importance of patterns of expectation in experiencing a work of art. This awareness lies accumulated in the notion of decorum, a key art historical concept denoting a sensibility to tacit expectations in viewers of works of art. Until now, little attention has been paid to what this concept has to offer to the neuroscience of vision and visual art. Following the recent overview provided by De Lange, Heilbron & Kok (2018), I first consider different ways of mapping decorum onto a PP understanding of perception. Subsequently, I show how decorum might prove valuable for current research practices working towards an understanding of visual art within PP. As my review of the literature shows, there has been a decade-long bias towards nonfigurative art (Escher, Picasso and Magritte being favourites), leaving figurative and pre-20th-century works of art underrepresented in models of perception of visual art. I argue that this focus on non-figurative art is the expression of an underlying lack of integration of neuroscientific theories of perception, with art historical research into the social and cultural behaviours that gave rise to the patterns of expectation in historical viewers. Building on the work of Lee (1940), Baxandall (1972), Williams (2008) and Kesner (2014), I propose decorum as a concept and tool that will help us integrate art historical knowledge of the patterns of expectation within a PP understanding of the perception of art. Decorum exists in many guises throughout the history of western art as a conceptual point of convergence of formal, social, and cultural expectations and, as such, lets art historians trace historical changes in both expectations and affective response. Consequently, operationalizing decorum as a research tool could prove to be invaluable not only to a PP understanding of the perception of visual art, but to PP accounts of visual perception in general. To the former, decorum offers the possibility of taking into account patterns of expectation in a way that is nondiscriminating relative to different artistic styles. To the latter, it proposes a historical dimension to the patterns of expectation constitutive of visual perception, although methodological compatibility with current research efforts will have to be proven. As a carrot at the end of my stick, I suggest further conceptual integration between art historical research and the neuroscience of perception as an answer to what in neuroscience has been dubbed ‘the question of ecological validity’, or the necessary constraint of decontextualization and subsequent lack of data from real-life environments when conducting lab-based research. References Baxandall, M. (1972). Painting and Experience in Fifteenth Century Italy, Oxford: Clarendon Press. 14 De Lange, F. P., Heilbron, M., & Kok, P. (2018). How do expectations shape perception?. Trends in Cognitive Sciences, 22(9), 764-779. Lee, R. W. (1940). Ut pictura poesis: The humanistic theory of painting. The Art Bulletin, 22(4), 197269. Kesner, L. (2014). The predictive mind and the experience of visual art work. Frontiers in Psychology, 5, 1417. Williams, R., (2008). Italian Renaissance Art and the Systematicity of Representation, in Elkins and Williams (eds.), Renaissance Theory, London: Routledge, 159-184. 15 The Joy of Horror Films: Active Inference, Aesthetic Pleasure and the Paradox of Horror Mark Miller Hokkaido University [email protected] Starting from the active inference framework, I will explore why an agent striving to minimize uncertainty would find pleasure in engaging with horror films – a rich source of uncertainty. The answer, I propose, lies in the nature of consumable errors: prediction errors with just the right amount of complexity and novelty for the agent to make good progress in its learning. Films of all kinds carefully curate these sorts of consumable errors – they play at frustrating and confirming our expectations in just the right quantities over the course of the film. It is these curated changes in the rate at which error is reduced that produces the primary source of aesthetic pleasure of films for the error minimizing agents like us. It is also the key to understanding why so many of us enjoy watching horror films in particular. 16 “A Dinner Reservation for 12”: John Wick and Predictive Processing Kathrine Cuccuru University of Sussex [email protected] When John Wick utters down the phone “I’d like to make a dinner reservation for 12” it transpires that he is requesting the disposal of 12 dead bodies; the entirety of the professional hit-squad that he has single-handedly killed in their attempt to kill him. Following the well-established cinematic trope of action films, at the beginning of this scene the audience can reliably guess that Wick will survive this unsurvivable hit on his life. The excitement, the action of the film, is instead driven by the audiences’ anticipation of just how he will survive the unsurvivable; the thrill we get from not being able to guess how he inevitably out-guesses his attackers. Importantly, it is his characterisation as the ‘Baba Yaga’ (the boogeyman) of assassins that makes Wick’s survival satisfyingly thrilling rather than predictably mundane or absurdly comic. I suggest that the satisfying thrill of the action in John Wick can be best explained in terms of predictive processing (PP). In this paper, I propose that the audience’s (what I call) aesthetic satisfaction arises out of the pleasing tension and interplay of the modes of prediction involved in both the viewing of the onscreen action and portrayal of the predictive processes in it. My basic claim is that the viewing audience’s primary mode of prediction is anticipatory, that is, the conscious guessing of what will happen next — how will Wick survive? However, in that mode, the audience is attempting to track the non-conscious probabilistic guessing of the portrayed characters and action; the reliable guess that gives the audience the expectation that Wick is highly likely to out-guess — as it turns out near perfectly so — a 12 person hit-squad. Yet, at the same time, the audience cannot reliably guess how. This is further complicated by the ways in which the action of the imagined world converges and diverges from everyday PP. Specifically, the ways in which the cinematic portrayal of Wick plays on and goes against what might happen if he existed or were encountered in the world we ordinarily perceive and act in. I argue that the audience is aesthetically satisfied when these layers of prediction are pleasingly balanced and resolved. To demonstrate my view, explaining what makes the film John Wick aesthetic satisfying, I shall carefully set out how these complex layers and processes of prediction are in tension, balanced, and resolved throughout the particular scene beginning with the assault on Wick’s house and ends with him phoning through his ‘reservation.’ From this example, it will be seen that the tools of PP offer a robust account of the aesthetic elements of surprise, plausibility, suspense, consistency, and success. Finally, I speculate on how my description of aesthetic satisfaction in terms of PP might be generalised across other aesthetic works and genres, and also how it might enrich and expand upon existing PP theories of action and imagination. 17 Creative Imagination in the Enculturated Predictive Mind Sam Wilkinson University of Exeter [email protected] Max Jones University of Bristol [email protected] Proponents of the Predictive Processing Framework (PP) argue that it suggests a deep continuity between perception and imagination (Clark 2015; Kirchhoff 2018). Since PP is acknowledged to apply well to perception, it is assumed that imagination is just an “offline” or “de-coupled” version of this. However, in (Jones and Wilkinson 2020) we argue that, despite the apparent close connections between perception and imagination, PP struggles to explain (i) how imagination differs from other forms of offline cognition and, (ii) how we are able to engage in deliberate drastic departures from reality that are nonetheless constrained. This in turn creates problems for PP in explaining the role of imagination in the creation of art and fiction, where creating content that radically departs from and transcends our everyday experience of the world is often the precise goal. If perception is controlled hallucination, and hallucination is uncontrolled perception, then what keeps the distinctive process of creative imagination under control? In Jones and Wilkinson (2020), we speculated that manipulation of external linguistic symbols (such as written or spoken words) and engaging in artistic practices, such as sketching, may allow us to transcend the constraints on imagination that seem to follow from PP. By actively playing with words and images we can steer the mind to surprising content that goes beyond the everyday. An interesting upshot of this hypothesis is that creative imagination is a cultural achievement rather than a natural evolutionary endowment. Creative imagination depends on having inherited an environment rich in cognitive technologies, such as language and tools for creating art. Moreover, it depends on cognitive practices and norms that govern their use, as well as the kinds of social norms that encourage and positively evaluate such endeavours. In short, imagination is a result of enculturation (see Fabry 2017, 2018a; Menary 2013, 2015; Menary & Gillett 2017, Veissière et al. 2019) This hypothesis about the nature of creative imagination in PP has a number of interesting consequences. First, the presumed direction of causation between expressive use of language and artistic creativity, on the one hand, and creative imagination, on the other, is reversed. Rather than creative imagination being a precursor to creative practices, it is a product of them (at least initially, and then, of course, scaffolding looping effects ensue). Second, this suggests that, to the extent that the imagination can truly be described as imaginative, it may be a uniquely human capacity that emerged in relatively recent history. While many non-human animals may engage in tasks that recruit mental imagery (such as memory or planning), not all engagement with mental imagery qualifies as imagination in the full-blooded creative sense, and not all imagination involves mental imagery, with creative imagination being a uniquely human cultural achievement. Third, this perspective may support some extended cognition approaches to the predictive mind, in the sense that the cognitive process of creative imagination can only be accomplished through interacting with and integrating external cognitive technologies and practices (Fabry 2018b; Menary 2015). 18 References Clark, A. (2015). Surfing uncertainty: Prediction, action, and the embodied mind. Oxford University Press. Fabry, R. E. (2017). Cognitive innovation, cumulative cultural evolution, and enculturation. Journal of Cognition and Culture, 17(5), 375-395. Fabry, R. E. (2018a). Enculturation and narrative practices. Phenomenology and the Cognitive Sciences, 17(5), 911-937. Fabry, R. E. (2018b). Betwixt and between: the enculturated predictive processing approach to cognition. Synthese, 195(6), 2483-2518. Kirchhoff, M. D. (2018). Predictive processing, perceiving and imagining: Is to perceive to imagine, or something close to it?. Philosophical Studies, 175(3), 751-767. Menary, R. (2013). The Enculturated Hand. In Radman (Ed.) The hand, an organ of the mind: What the manual tells the mental, (pp. 349-368). MIT Press Menary, R. (2015). What? Now. Predictive Coding and Enculturation. In Open MIND. Open MIND. Frankfurt am Main: MIND Group. Menary, R., & Gillett, A. (2017). Embodying culture: integrated cognitive systems and cultural evolution. In Kiverstein (Ed.) The Routledge Handbook of Philosophy of the Social Mind (pp. 7287). Routledge, Taylor and Francis Group. Veissière, S. P., Constant, A., Ramstead, M. J., Friston, K. J., & Kirmayer, L. J. (2019). Thinking through other minds: A variational approach to cognition and culture. Behavioral and Brain Sciences, 197. Jones, Max & Wilkinson, S. (2020). From Prediction to Imagination. In Abraham (Ed.) The Cambridge Handbook of the Imagination (pp. 94-110). Cambridge University Press. 19 Presence, Flow, and Narrative Absorption: A New Integrated Model based on Predictive Processing Karin Kukkonen University of Oslo [email protected] Federico Pianzola University of Milano-Bicocca [email protected] Experiences mediated by technology (e.g. printed books, screens, and virtual reality) are studied across a variety of disciplines, often with little cooperation. Different theorizations, models, and empirical tools have been developed, resulting in a fuzzy agglomerate of related and overlapping concepts, like presence (Lombard et al., 2015), flow (Csikszentmihalyi, 1990; Harmat et al., 2016), and narrative absorption (Hakemulder et al., 2017). After thirty years of empirical research on experiences described by such concepts, we attempt here to organize knowledge from various fields – ranging from neuropsychology, cognitive narratology, empirical aesthetics, communication studies, and media psychology (Kukkonen, 2020; Riva et al., 2015). We investigate the continuities and differences between mediated and unmediated experience with particular attention to presence, that is, the feeling of being there with the imagined action while engaging with virtual reality, playing video games and reading literary fiction. The outcome is a new integrated model of presence and related concepts, based on predictive embodied cognition (Clark, 2016; Friston, 2012; Hohwy, 2013), which is valid for both mediated and unmediated experience. In brief, subjects are present if they are able to enact intuitively (i.e. without the involvement of reasoning) their implicit (predictive coding) and explicit (intentions) predictions while moving through real or virtual spaces, or while reading (Riva et al., 2015). The enactment of predictions can occur (i) through the body in a non-mediated way, (ii) through a proximal physical or symbolic tool (an artifact present and directly manipulable by the body, including language) to exert an action upon an external object (first-order mediation), (iii) through a proximal physical or symbolic tool (an artifact present and directly manipulable by the body, including language) that controls a different distal one (i.e. a tool present and visible in the extrapersonal space or a narrative) to exert an action upon an external (i.e. using a joystick to control an avatar in a videogame) or internal (i.e. making sense of a narrative or solving a thought experiment) object (second-order mediation) (Riva & Mantovani, 2014). The main difference in the enactment of predictions related to physical/digital and symbolic tools is related to the way used to verify their efficacy: predictions related to physical/digital tools and physical/digital objects are assessed using mainly exteroceptive information. Instead, predictions related to symbolic tools and symbolic objects are assessed using mainly interoceptive information. Moreover, in narrative the temporal unfolding of an experience has a prominent role in sensorimotor perception and affects the enactment of intentions, consequently influencing the sense of presence (Kukkonen 2020). Last, flow is a concept related to the balance between a person’s skills and the complexity of the stimulus, that is between a person’s intentions and the effort required by predictive processing. Predictive processing offers new perspectives on long-standing discussions of presence, immersion, absorption and transportation into fictional worlds, while in turn, these cases from VR, 20 computer games and literature could also contribute to the ongoing discussions around agency and consciousness in unmediated contexts. References Clark, A. (2016). Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford University Press. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Collins. Friston, K. (2012). Embodied inference and spatial cognition. Cognitive Processing, 13(S1), 171– 177. https://doi.org/10.1007/s10339-012-0519-z Hakemulder, F., Kuijpers, M. M., Tan, E. S., Bálint, K., & Doicaru, M. M. (Eds.). (2017). Narrative Absorption. John Benjamins. Harmat, L., Ørsted Andersen, F., Ullén, F., Wright, J., & Sadlo, G. (Eds.). (2016). Flow Experience: Empirical Research and Applications. Springer International Publishing. https://doi.org/10.1007/978-3-319-28634-1 Hohwy, J. (2013). The Predictive Mind. Oxford University Press. Kukkonen, K. (2020). Probability Designs: Literature and Predictive Processing. Oxford University Press. Lombard, M., Biocca, F., Freeman, J., Ijsselsteijn, W., & Schaevitz, R. J. (Eds.). (2015). Immersed in Media: Telepresence Theory, Measurement & Technology. Springer. Riva, G., Mantovani, F., Waterworth, E. L., & Waterworth, J. A. (2015). Intention, Action, Self and Other: An Evolutionary Model of Presence. In M. Lombard, F. Biocca, J. Freeman, W. IJsselsteijn, & R. J. Schaevitz (Eds.), Immersed in Media (pp. 73–99). Springer International Publishing. https://doi.org/10.1007/978-3-319-10190-3_5 21 Connecting Musical Expectancy and Pleasure: Insights from Cognitive Neuroscience Vincent Cheung Academia Sinica, Taiwan [email protected] Many people enjoy listening to music because of the emotions it evokes. What differentiates music from other sounds is its inherent structure. Our ability to perceive and make sense of this structure is argued to be a key reason for music’s irresistible appeal, for it enables us to make predictions about upcoming musical events. Two dominant neurocognitive accounts explain how musical expectancy drives pleasure. The reward prediction account proposes that listeners make predictions about the value of upcoming musical events. Musical pleasure arises when the arrival or omission of such events are better than expected. This account builds on findings showing that pleasant music activates dopaminergic circuits in the mesolimbic reward network. In particular, musical reward value is thought to be related to the nucleus accumbens, whose activity is thought to encode reward prediction errors. However, a critical flaw in the reward prediction account is that it is unclear what establishes a musical reward. In contrast, the content prediction account argues that pleasure is one of the many emotions evoked when listeners’ predictions on a specific musical event (such as a note) are confirmed, deviated, delayed, or hastened. Despite its intuitiveness, previous empirical evidence did not seem to support this account. Although musical violations were found to be regarded as unpleasant, people do not enjoy music that is perfectly predictable (with repetition being the best example). How musical expectancy relates to pleasure has thus remained elusive. Recent evidence shows that pleasure is not only a function of how a prediction is violated, but also how specific that prediction was. For example, in our work, we have demonstrated that listeners’ pleasantness rating to chord progressions taken from commercially successful pop songs depends on the dynamic interaction between the uncertainty a chord is predicted and the extent it deviates from expectations. These predictions are assumed to be formed through long-term exposure to a musical style. Interestingly, we observed that the same interaction modulated activity in the amygdala but not the nucleus accumbens, which was only sensitive to chord uncertainty. This led us to propose that the nucleus accumbens may instead be involved in reorienting listeners’ attention to the music until the uncertainty is resolved. However, our predictions are not exclusive to abstract representations of music structure. Taking a model comparison approach, we found that low-level sensory mechanisms and abstract representations contribute distinct variance in explaining listeners’ surprise. This suggests that the object of musical expectancy is multifaceted and contingent on not only what is learnt, but also the way the auditory perceptual system is organised. Taken together, neurocognitive evidence has shown that our appreciation for music depends on prospective and retrospective states of prediction. These predictions are both learnt and innate, and represented at multiple levels of abstraction. The beauty of music is thus perhaps not only in the ear of the beholder, but also in the minds of every listener. 22 The Pleasure of Learning: A Perceptual Learning Hypothesis for Aesthetic Appreciation Pietro Sarasso University of Turin [email protected] According to a century-old philosophical debate, aesthetic emotions are tightly connected to knowledge acquisition. The German philosopher Alexander Gottlieb Baumgarten, who founded modern aesthetics with his book Aesthetica (1750), described aesthetics as an “alternative approach to the philosophy of knowledge, experience and perception.” Recent evidence, collected across different disciplinary domains, from empirical aesthetics to neuroscience, seems to confirm this ancient intuition. Namely, neurocomputational models of cognition have framed aesthetic appreciation within the domain of knowledge acquisition and learning, suggesting that aesthetic appreciation might be considered as a hedonic feedback on successful perceptual learning dynamics. Such hypothesis, however, has never been empirically demonstrated. Given that the mismatch negativity (MMN) registered with EEG is commonly considered as a reliable index of implicit perceptual learning, we tested the above hypothesis by investigating the relationship between MMN responses and the perception of beauty in a roving paradigm with auditory stimuli. To this end, we registered EEG responses to frequency (Hz) standard and deviant musical intervals. Moreover, for each single trial, we computed an information-theoretic index of perceptual learning (Bayesian surprise) indicating the magnitude of the update of predictive representations following each stimulus. Participants were then asked to judge the beauty of the same musical intervals. We found that higher subjective aesthetic appreciation was associated with larger MMN responses, which in turn correlated with trial-by-trial fluctuations in Bayesian surprise. Consistent with previous results, Bayesian surprise was also found to correlate with slower RTs in a detection task, evidencing an inhibition of motor behaviour induced by surprising sensory states. Our results provide the first empirical evidence of a positive relation between subjective aesthetic appreciation and EEG indexes of implicit perceptual learning (i.e. the update of predictive representations). We interpret this finding in the light of an original perceptual learning hypothesis for aesthetic appreciation framed within the predictive coding theory. We propose that the sense of beauty might have evolved to signal the nervous system new sensory knowledge acquisition, thus intrinsically motivating the search for highly informative stimuli. We present evidence from previous studies further supporting our perceptual learning hypothesis. Recently, neuroimaging and neuropsychological studies indicated a strong link between meta-learning feedbacks (selfgenerated rewards in response to information gains) and the activation of the dopaminergic rewardrelated circuits which are also correlated to subjective aesthetic pleasure. Moreover, we show that aesthetic emotions emerge in correspondence of input processing enhancement at the level of sensory cortices (i.e., optimizing learning), and the inhibition of motor behavior (i.e., minimizing action to concentrate all the available resources on perception). Finally, we present some possible clinical and technological applications of the perceptual learning hypothesis. 23 Exploration Revisited – New Music as a Model for Exploratory Behavior Under High Uncertainty Conditions Iris Mencke Max Planck Institute for Empirical Aesthetics, Frankfurt [email protected] Western art music from the 20th and 21st centuries (New Music) is one of the most perceptually challenging artistic styles. Evolving as an antithesis to Western tonal music around 1910, it is characterized by a lack of tonal hierarchies and an irregularity of meter that results in a high degree of predictive uncertainty. Up to a century later, New Music pieces are still perceptually challenging, so that listeners are confronted with music that is not only unpredictable in terms of tonality and metre, but also entails a large variety of different timbres, unpredictable dynamics, and a high degree of dissonance. Thus, two questions arise: what are the unique strategies or attitudes that listening to New Music afford and, more generally, how can the cognitive state induced by such music be characterized? In the context of an interview study, we compared reports of listening experiences with New Music and classical-romantic music. Employing qualitative content analysis, we found that New Music experts adopt an exploratory listening attitude, driven by a curiosity for novelty and an enjoyment of the insights stemming from a recognition of underlying patterns. Further, a strong attentional focus on low-level acoustic features like ‘sound design’ and ‘sound transfer’ was found in the New Music, but not in the classical-romantic, expert group. The findings provide novel insights into how an unpredictable musical language may shape the experience of hearing. Indeed, we propose that conscious engagement with New Music puts the listener in a state that requires a specific form of exploratory behaviour. This behaviour is afforded by the confrontation with a completely unknown auditory terrain, entailing high entropy in which existing predictive models are no longer applicable. Thus, listeners must discard those and generate new, more useful, predictive models. While an agent in a more utilitarian non-art context would withdraw from such a task, an art context provides a situation in which this model building task can be taken on. We hypothesize that a cognitive state induced by New Music is therefore one that may be characterized by oscillating between states of pattern seeking, and momentary states of insight that ultimately lead to such new models. Taken together, we propose that New Music provides researchers with a valuable, ecologically valid model of exploratory behaviour in high uncertainty situations. A question worth investigating is whether the state that is induced by New Music can influence the ways in which the brain processes acoustic events. Preliminary neuroimaging findings (including one that shows that tonal uncertainty affects our conscious awareness of sound events more than our implicit perception of low-level acoustic features), will be presented along with a theoretical account of the relationship between predictive uncertainty of musical styles and the “aesthetic states” they induce. 24 Using Bayes’ Rule to Model Emotional and Sensory-Motor Associations with Music Renee Timmers University of Sheffield [email protected] One of the pleasures of music listening relates to evoked affective, emotional as well as sensorimotor associations. Although at its core an auditory experience, music listening may in fact engage the whole body including various sense-modalities (i.e. vision, touch). Previous research has made great progress in documenting and demonstrating the range of listeners’ emotional and crosssensory associations with music. This has outlined what emotions are frequently evoked in concert as well as everyday situations, what processes may underlie perception and feelings of emotions, and also what environmental, neural and developmental bases may be responsible for cross-sensory correspondences. While assumptions of associative or statistical learning are regularly employed in models of music listening and affective experiences, in this presentation I will investigate the effectiveness of a more contextually sensitive adoption of Bayes’ rule. According to this rule, the perception of a concept does not only relate to supporting evidence in the context (i.e. feature characteristics), but also to the prior probability of the concept in the particular context, and to the distinctiveness of the supporting characteristics. This seems particularly well suited to model listeners’ associations with music: what sensorimotor and emotional associations listeners may perceive depends on their prior uses and engagements with music and its constituting elements (instruments, sounds, harmonies, pitches and rhythms), as well as on the properties and the distinctiveness of the musical characteristics. Using existing data, I investigate the use of Bayes’ rule to model emotional associations with music. For example, cultures may differ in expectations related to the prior probability of emotions in musical contexts. Reanalysing data from Laukka et al. (2013) suggests that western listeners (from Sweden) show a higher expectation for ‘longing’ to occur in music than eastern listeners (from India and Japan). Furthermore, following Bayes’ rule allows for fine-tuning of the weight of tonal mode for emotion perception, in accordance with the distinctiveness of a mode in a musical context, and listeners’ sensitivity to this distinctiveness. A full prediction of emotion perception is trialled in the context of the 24 Preludes of Bach’s Well-Tempered Clavier Book I. A similar approach can be used to model sensorimotor associations with music. Prior probabilities of associations with e.g. weight, movement, size, and colour may depend on listeners’ experiences and engagement histories with sound and music, and listening context, while specific associations are informed by musical characteristics and their distinctiveness. By modelling both types of associations in similar ways, relationships between the affective and sensorimotor experiences are explored. 25