English language and multimodal narrative

English language and multimodal narrative This is a pre-publication version of the following article Thompson, R. (2020). English Language and Multimodal Narrative. In S. Adolphs & D. Knight (Eds.), The Routledge Handbook of English Language and Digital Humanities. Routledge. https://www.routledge.com/The-Routledge-Handbook-of-EnglishLanguage-and-Digital-Humanities/Adolphs-Knight/p/book/9781138901766 When citing from or referring to this article please use the final publisher version. Abstract: We are in an age of media transition in which the textual is increasingly integrated with other semiotic modes and old and new media interact in a complex relationship. This chapter examines the evolving technology of whiteboard animation as a relatively new digitally mediated multimodal storytelling form that exemplifies the complexity of the convergence between old and new media. Some whiteboard animation videos rely on the material practices of drawing and photography, whereas others mimic the material, using digital animation software and video to generate a story. The most common elements of whiteboard animation are the visualisation of a whiteboard as the canvas, the illustrator’s hand in the field of view and a marker drawing out the story. In the last decade, do-it-yourself (DIY) software programmes have emerged allowing ordinary people to tell their stories through whiteboard animation using libraries that include pre-drawn characters, stock images, music, drawing implements and hands. Whiteboard animation is a multimodal storytelling form that is increasingly used to educate, inform, advertise and entertain. This chapter investigates the little-studied form of whiteboard animation to uncover the persuasive power of digital storytelling and reveal how technological transformations can inform storytelling practices. To develop frameworks for understanding the affordances that digital technologies bring to stories, this research explores narrativity and narrative discourse as social, rhetorical and multimodal. Through the lens of multimodality and rhetorical narratology, setting, time, point of view and voice are analysed in commercial and DIY whiteboard animation examples, paying attention to material practices of production and multimodal narrative development, as well as the complex relationship between speaker, audience, message and medium. This research on whiteboard animation looks to understand the intersection between technology and storytelling and contributes to digital humanities scholarship that looks to understand the implications of technology, writing and narrative. Section 1 Page 1 of 28 English language and multimodal narrative 24 English language and multimodal narrative Riki Thompson Introduction Definitions of the digital humanities (DH) are numerous and expansive, with competing and complementary views on the topic (Borek et al. 2016; Fails and Kelton 2016). Moves to define the field and figure out what ‘counts’ as DH have been going on at conferences and in journals for the last decade. At THATCamp Seattle 2010, the opening session was dedicated to unpacking the various definitions and identifying scholarly areas of interest to manage the logistics of creating sessions around research affinities. As ‘unconference’ participants attempted to define the field, concerns were raised about narrow definitions that potentially excluded scholars whose research brought together the humanities and technology. A year later, Melissa Terras (2011) gave a plenary talk at Interface where she addressed this continuing crisis of inclusion: It’s something to do with the Humanities, and digital technologies, but what exactly, we are slow – and even reluctant – to pinpoint. The latest definition, ‘Big Tent Digital Humanities’, deliberately obfuscates the focus of the field. . . . Big Tent DH, then, is an ecumenical approach, whilst giving the freedom for individual scholars to explore their own interests, wherever in the research and teaching spectrum they lie. In the digital age, storytellers have been able to harness technologies to present narratives in new ways. Unlike previous eras that showed a preference for ‘monomodality’ in communication, there is now a movement towards multimodality in which the textual is increasingly integrated with other semiotic modes rather than segregated and used in isolation (Kress and Leeuwen Section 1 Page 2 of 28 24 English language and multimodal narrative 2001; Machin 2007; Kress 2009). According to Jenkins (2006), we are in an age of media transition and instability, a ‘convergence culture’ in which new and old media interact in a complex relationship. An example of a narrative form that exemplifies the complexity of old and new media is whiteboard animation. This relatively new genre, also known as sketchboard animation, video scribing or fast draw video, is a computer-mediated form in which a sequence of events is animated on a white surface in time with an audio voice-over. With the development of whiteboard animation comes a new mode of narrative communication in need of examination. This chapter contributes to Big Tent DH scholarship by examining the evolving technology of whiteboard animation as a relatively new digitally mediated storytelling form. Background Many of the components of this relatively new narrative form have been around for a very long time. The use of stop-motion photography can be seen in clay-animation films (also known as Claymation) that have been around since 1908, and the process of animating cartoons and combining them with film was pioneered by Disney in the 1930s. Though it is not clear when the first whiteboard animation was developed, the emergence of the genre can be connected to the popularity of the RSAnimate series that was published online in 2010 when Cognitive Media adapted the public lecture series of Royal Society for the Encouragement of Arts, Manufactures and Commerce (RSA) to ‘help people to discover and learn new stuff through storytelling’ (Cognitive Media 2017). Some whiteboard animation videos rely on the material practices of drawing and photography, whereas others mimic the material, using digital animation software and video to generate a story. In early whiteboard animation, a cartoonist drew the story frame by frame directly on a whiteboard, and then time-adjusted photography techniques were applied to the final video. Most of these types of whiteboard videos combine stop-motion animation, bringing together sequential drawings in similar ways to traditional animation and flip books, with time- Section 1 Page 3 of 28 24 English language and multimodal narrative lapse photography to play back the recording of the stop-motion photography at a faster speed and then adding a voice-over to tell the story. The most common elements of whiteboard animation are the visualisation of a whiteboard as the canvas, the illustrator’s hand in the field of view, and a marker drawing out the story. With advancements in digital technology, these elements may be digital simulations that attempt to replicate the analogue. The representation of these elements is central to a rhetorical view of narrative in the ways that setting, voice and style are signified. Although the medium is known by a variety of names that do not signal any sort of canvas (e.g. video scribing and fast draw video), as well as names that signal a different sort of canvas (e.g. sketchboard animation), whiteboard animation appears to be the dominant descriptor regardless of the true nature of the material practices. Since RSAnimate videos gained popularity, other creative studios have found ways to simulate the appearance of this narrative form and their own art form as a version of whiteboard animation similar to the RSAnimate in their menu of products. In fact, a Google search for RSAnimate brings up links to a number of creative studios that replicate Cognitive Media’s whiteboard animation style; the official RSA site ranks fourth on the search results, with Whiteboard Animation, Idea Rocket Animation and Video Explainers showing up as the top three search results. Though the final product may look similar to the audience, the process of whiteboard animation differs across studios and do-it-yourself (DIY) software tools. Sparksight combines old and new media by using hand-drawn illustrations, which are created on white paper with a black Sharpie marker and carefully photographed at a high shutter speed, then imported into an after-effects programme, spliced together and replayed at high speed to give the impression of movement (Sparksight Inc. 2015). Idea Rocket, on the other hand, takes a high-tech approach to simulate the analogue by using digital animation software for the illustration process, then compiling sketches into a movie, speeding up the film, adding complementary audio and, finally, integrating a green-screened hand as an additional video layer of the final movie (Idea Rocket 2017). Section 1 Page 4 of 28 24 English language and multimodal narrative The production of these sorts of multimodal narratives are not limited to professional creative studios, as video scribing software and apps have made it possible for animators to participate in the evolution of digital animation and present stories in new ways. Though professional animators have been using computer input devices, such as graphic tablets with a stylus, to capture data and digitise it for years, the mass production of touch screen tablets coupled with animation software and apps allow ordinary people to draw on their digital devices and produce these sorts of narratives. Critical issues and topics With the visual and digital turn came an increase in adaptations of narrative forms that communicate in different ways. Desktop publishing, animation programmes, video editing, tools for digital illustration and social media allowed the ordinary person to become a self-published digital storyteller. In online media environments, YouTube provided space for producers and consumers of content to teach and learn from each another through video sharing and discussion spaces. With the surfacing of digital technologies and social media, user-generated content has proliferated with the rise of ‘prosumers,’ blurring the lines between amateurs and professionals (Toffler 1984). Tapscott and Williams (2010) later applied the term to describe the increased sharing of content and know-how between professionals and enthusiastic hobbyists that came with the emergence of Web 2.0 technologies. With these shifts, Jenkins (2007) suggests we are experiencing ‘the emergence of a new form of participatory culture (a contemporary version of folk culture) as consumers take media in their own hands, reworking its content to serve their personal and collective interests’ (para 5) such that the relationships between ordinary users, producers and content are more complex as old and new media converge. Innovation is happening from multiple directions, both top-down and bottom-up, as users experiment with a variety of techniques and devices for multimodal storytelling. Section 1 Page 5 of 28 24 English language and multimodal narrative With cartoons serving as a primary tool to teach and engage people, this digital storytelling genre provides an example in which the mundane is engineered through the visual rhetoric of cartoons to be more appealing, and hence, persuasive. McCloud (1994) argued that comics and graphic novels are often thought of as simpler versions of their text-only counterparts, but should be considered as a significant storytelling form, deserving of respect. Similarly, Tiemensma (2009) pointed out that comics serve as an important medium for improving literacy as they combine printed words and pictures to represent narrative in today’s increasingly visual world. Drawing from cognitive literary theory, Nikolajeva (2014: 711) has shown how picture books are ‘a narrative art form that creates meaning through the synergy of two media, the verbal and the visual.’ Cartoons have a long history of teaching and captivating audiences. In the United States, political cartoons have been a vehicle to engage citizens in public discourse since the 1700s (Medhurst and Desousa 1981; Edwards 1999). In the 1970s, the animated musical series School House Rock! was used to teach children about grammar and civics, while adults were introduced to intellectual ideas via comic books in the For Beginners series. Educators have used cartoonbased video games to engage learning around serious topics such as the Middle Eastern crisis with Darfur Is Dying (Peng et al. 2010). The publication of the graphic doctoral dissertation Unflattening (Sousanis 2015) and a graphic special issue of Digital Humanities Quarterly (Whitson and Salter 2015) point to a significant shift in the level of acceptance for cartoons as an intellectually rigorous medium for teaching and presenting complex ideas through comic storytelling. Main research methods Through the lens of multimodality and rhetorical narratology, this chapter will investigate this little-studied form to reveal how technological transformations can inform storytelling practices, paying attention to material practices of production and multimodal narrative development. Section 1 Page 6 of 28 24 English language and multimodal narrative Narrative Early sociolinguistic research by Labov and Waletsky (1997) suggested that oral narratives of personal experience tend to have structural components (abstract, orientation, complicating action, resolution, evaluation and a coda) and could be understood through these formal frameworks. Scholars have interrogated traditional notions of narrative and call for research that develops frameworks for understanding the affordances that digital technologies bring to stories (Bamberg and Georgakopolou 2008; Page 2009, 2012). Georgakopoulou (2006: 1) suggests that narrative research has ‘definitively moved from the study of narrative as text (first wave) to the study of narrative-in-context’ as the field enters a second wave. This chapter reflects this second wave and explores narrativity and narrative discourse as social, rhetorical, and multimodal. Narrativity informs ‘almost any verbal utterance . . . ranging from fragmentary reports and abortive anecdotes to those more distinctly framed and conventionally marked tellings that we are inclined to call “tales” or “stories”’ (Smith 1980: 228). The social aspect of narrative discourse focuses on the act in which someone tells someone else something that happened, with each party motivated to participate in the telling of and listening to the narrative (Smith 1980; Bal 2004). Narrative is not just about the genre, but about who tells the story and how it’s perceived. Rhetorical narratology has been most commonly applied to literary texts, although there is space to move beyond this. This narrative framework combines speech-act theory with rhetoric and narratology to consider the relationship between tellers and audiences as it occurs in specific contexts (Kearns 1999). Phelan (1996: 19) argued that a rhetorical approach to narrative ‘shifts emphasis from author as controller to the recursive relationships among authorial agency, textual phenomena, and reader response, to the way in which our attention to each of these elements both influences and can be influenced by the other two’. Rhetorical approaches, regardless of their specific emphases, are ultimately concerned with the how, the what and the why of narrative (Phelan 1996). From a multimodal standpoint, Section 1 Page 7 of 28 24 English language and multimodal narrative narratives that are told through a combination of discursive resources that complicate meaning potentials, especially when resources are contradictory. This chapter considers the relationship between narrator and audience with consideration given to the multimodal dimensions of audio, visual and textual. Multimodality In combination with narrative theory, multimodal discourse studies provide an additional scaffolding to understand how whiteboard animation develops meaning through verbal and visual means. Semiotic resources carry their own small fragments of meaning-making that, when combined into a single presentation within an image, transmit a rhetorical message to the targeted audience (Jewitt et al. 2016). These fragments include design elements such as gestures, visuals and linguistic features within the image itself. Any combination of these elements builds a multimodal experience of meaning-making. Those who are the subject of the communication are considered as represented participants and both interactive and represented participants shape the position of the viewer (Kress and van Leeuwen 2006). Multimodality decentres language and extends meaning making to a variety of modes of learning (Adami 2017; Jewitt 2013). Emphasising that meaning is not made in isolation from other modes of learning, multimodality challenges the primacy of verbal language (Cope and Kalantzis 2009; Adami 2017). That people express language from available resources or modes, such as writing, image, sound, speech, gesture, gaze and posture, and are active in its day-to-day production, is an underlying assumption of multimodality (Machin 2007). Current contributions and research Whiteboard animation is a multimodal storytelling form that is increasingly used to educate, inform, advertise and entertain. To uncover the persuasive power of digital storytelling in Section 1 Page 8 of 28 24 English language and multimodal narrative whiteboard animation, this chapter examines the complex relationship between speaker, audience, message and medium using a multimodal narrative analysis approach. Specifically, through a multimodal narrative framework, setting, time, point of view and voice will be analysed in commercial and DIY whiteboard animation examples. Setting Structure and text can provide a useful apparatus for considering the medium and mode of presenting a message – specifically comics animated on a whiteboard to tell a story. The rhetorical choice to situate animated stories on a whiteboard creates a context that influences message delivery. From a sociolinguistic perspective, the whiteboard can be seen as an orienting feature of the narrative (Labov and Waletsky 1997), providing a semiotic resource that signals the scene similar to the ‘setting’ in a literary narratives. Over time, the semiotic meaning of the whiteboard has shifted from primarily denotative to far more connotative in these animated stories. Multimodal theorists point to the importance of props, such a whiteboard and marker, to impart meaning, especially in generic scenes (Machin 2007). At the beginning of each RSAnimate video, for example, the series name is presented on a whiteboard. A hand holding a marker enters the field of view and writes the letters RSA on the white background while the sound of a marker squeaking across the slick surface of a whiteboard is audible with each stroke. This sensory information from multiple sources provides important cognitive information such that when the audience ‘sees’ a dry erase marker moving across a whiteboard when they ‘hear’ the squeaky marker. From a multimodal viewpoint, the use of sound complements the visual, such that the audio signals the white background as a specific type of space – a whiteboard. With the literal visualisation of the whiteboard and thick marker denoting the scene, the symbolic meaning attached to the canvas imbues it with meaning potentials. To consider the possible connotative values associated with the whiteboard as an object, we may ask: where do Section 1 Page 9 of 28 24 English language and multimodal narrative we find whiteboards? what are whiteboards used for? and who uses whiteboards? Because ‘images are never free of connotative power’ (Machin 2007: 23), the visualisation of the whiteboard, be it authentic or synthetic, carries meaning that will be interpreted in specific ways. The associations of the medium as a note-taking space often used in classrooms and conference rooms to teach and share ideas carries connotative value such that the visual cues signal to the audience that the story produced will be educational and/or informational. Moreover, the visualisation of the whiteboard context signals a virtual learning space, infusing the message with authority and credibility. As the genre of ‘whiteboard’ animation has evolved, the representation of the whiteboard continues to serve as an important prop, but it has become more conceptual than physical, such that the actual whiteboard rarely exists – and if it does, it is merely a synthetic representation of it. The explicit move to simulate the whiteboard is present with the DIY app Doceri, which is described as ‘the interactive whiteboard for iPad’ (SP Controls, Inc. 2015). While Doceri provides animators with a variety of background options, including a chalkboard or blank background (in customisable colours), the default setting for background is a slide of a whiteboard, complete with the aluminium frame around the edges. For some creative studios, the whiteboard is implied through complementary tools and previous videos that have normalised an understanding of the genre as ‘whiteboard animation.’ Sparksight uses old school techniques of paper, Sharpie markers and stop-motion photography, whereas Idea Rocket uses a white background with digital illustration software. The whiteboard is now taken for granted as the composing canvas, and viewers no longer need to see an authentic version. The final video produced with these tools is understood by the audience as a whiteboard video regardless of whether a whiteboard exists. Time Section 1 Page 10 of 28 24 English language and multimodal narrative Temporality is another orientating device that is inextricably tied with narrative (Ricoeur 1984). Multimodal approaches provide tools for critiquing types of images and presentation in terms of meaning potentials, a multimodal narrative approach allows for consideration of sequencing and temporality in the creation process. With whiteboard animation, time is managed through the telling and visualisation of events in a story. As the audience listens to the narrator and watches the animator draw on whiteboard, the story develops in front of the viewer’s eyes. With each marker stroke on the canvas, the audience anticipates story elements as the animator takes the viewer through the narrative. While temporal sequencing is integral to narrative, linear disruption is commonly employed for effect. For example, the narrator may refer to events out of order or return to an earlier time in a story to add details. In literary narrative, authors may do this by presenting story scenes out of order, such as presenting the climax of the story as an opening scene, or through reading order, as seen with books like the ‘choose your adventure’ franchise of the 1980s and the digital novel Arcadia (Pears 2015), both of which allow the audience to drive the sequencing of events via choices about which path to follow in the reading process. With whiteboard animation, linear disruption may be realised both through the telling of the story and/or through the animation process – and not always simultaneously. For example, while the story may be told with a forward moving linear progression, the visualisation of the story may be recursive when a visual story element is brought back into the frame, bringing previously presented illustrations back into the audience’s field of view, and the illustration is revised. Ewert (2010: 72) has pointed out that narratology involving comics must consider textual and graphic elements, which is often ‘a challenge for critics habituated to text-based narrative’. Narrative time is manipulated in comics through visual space in a variety of ways (Eisner 2008). For example, animators may integrate symbolic resources that connote time (like images of a clock), or they may use space between and within panels to represent time as compressed or extended by using more or less white space. The concept of ‘closure’ describes the phenomena Section 1 Page 11 of 28 24 English language and multimodal narrative when a reader is able to imagine the action that has taken place between drawn panels and ‘close the gap’ about what is expected to happen in the blank space between panels (McCloud 1994). The concept of closure is visible in the DIY whiteboard animation Have You Met Olivia? (Childhaven 2014). In a scene that depicts the child as a picky eater, the audience watches as the pen strokes draw Olivia with arms crossed. Initially, the face includes only eyes and a nose, leaving the face virtually expressionless. Next, a plate and food items are drawn into the sequence as if to present the audience with the conflict that the main character must overcome – eating the food in front of her. Finally, as the third-person narrator describes how Olivia had outbursts when food was placed in front of her, the main character’s mood shifts from an emotionless state to anger as the animator fills in the details of the story one pen stroke at a time. Initially, the only indicator that the audience has of Olivia’s mood is relayed through her crossed arms. It isn’t until the animator draws a screaming mouth, furrowed eyebrows and beads of sweat flying into the air that the audience learns of the extent of Olivia’s upset, revealing the narrative’s conflict (Figure 24.1). In this way, stroke order serves as a temporal narrative technique, with each stroke of the marker foreshadowing the completed image to come. Figure 24.1 Have You Met Olivia? Source: Childhaven 2014 Section 1 Page 12 of 28 24 English language and multimodal narrative Point of view: who sees the story It has been suggested that few narrative features have been discussed as extensively as point of view – the physical, psychological, and ideological position in terms of which narrated situations and events are presented, the perspective through which they are filtered – and few have been associated with as rich a terminology (from central intelligence, vision, and focalization to filter and slant). (Prince 2010: 442) Point of view (POV) describes the position of the narrator in relation to the story being told and provides the audience with a perspective on events, which can emanate from a character or a nonanthropomorphic existence such as a camera. In this way, POV is the lens through which an audience is influenced to ‘see’ how events of a story unfold. Thus, POV is a narrative device used to give readers insight into events in the story and is achieved differently depending on the media. In literary genres and oral narratives, linguistic devices are the tools that provide perspective. First person and third person are the most common point of view techniques used in literary narratives, with pronouns doing the work of signalling to the audience. In first person, the narrator tells the audience how ‘I’ progressed through the story, whereas with third person, all characters are spoken of in an objective manner through pronouns such as he, she and they. The lesser-used second-person POV, in which the narrator speaks directly to ‘you,’ the audience, tends to be found most often in self-help, instructional and technical communication story forms. POV is an important tool for negotiating social distance between the narrator and audience. Because first-person narrative provides a window into the main character’s mind as they narrate the story through their POV, it closes social distance by giving readers a sense that they know the narrator and are with them on the narrative journey. While third person does give Section 1 Page 13 of 28 24 English language and multimodal narrative the audience access to the narrator’s mind and keeps them close as the narrator tells the audience what they want them to know at any given moment, first person puts the audience directly into the character’s mind, unlike third person which moves the audience out of it. With second-person POV, the audience does not have access to the narrator’s thoughts, yet the direct address works to include them into the narrative. In film and video games, POV is achieved through camera angles that allow the audience to see events through the perspective of particular characters. With first person, the audience sees events through the eyes of the main character as represented via the camera lens. In third person, the audience still sees what the main character sees, but the audience does not have a window into the main character’s thoughts as the camera appears to be looking over the shoulder of, or even including, the main character in the field of view. Second-person POV is used when the narrator speaks directly to the camera and uses the pronoun ‘you’ to address the audience. POV can refer to the visualisation of what a character sees, or more abstractly, as used with narratology approaches, POV can reference attitudes and tendencies of the cinematic narrator (Keating 2010). With whiteboard videos, POV is complicated by the multimodality of the story form as layers of textual and cinematic narration are combined to tell a story. First, there is the spoken narrative that the audience hears as the lecturer talks to the audience, the perspective of the camera that films the animator drawing the story and, finally, the POV is presented through cartooning techniques. This is not to say that POV is the same across whiteboard animation. Rather, POV in these narratives often reflects stylistic diversity through software and animation tools used to tell the story. As creative studios have experimented with this storytelling form, shifts in POV can be seen even within a set of seemingly similar narratives. Across the 21 RSAnimates created between December 2009 and July 2016, visual POV varies. While the verbal narrative POV has largely remained constant, with narrators switching between first-, second-, and third-person Section 1 Page 14 of 28 24 English language and multimodal narrative perspective throughout the course of their lecture, the visual perspective has evolved with technological and cinematic shifts. RSAnimate provides an example of how POV can be used to push the audience away or bring them closer to the story through camera angles. Early whiteboard animations primarily used over-the-shoulder camera angles with periodic medium long shots. Visually, the use of close-up over the shoulder camera angle gives the audience a first-person perspective as they watch the animator’s thought process with each stroke drawn. As the camera angles move from the over-the-shoulder to the medium long shot, the camera captures the animator’s shoulder, neck and back of the head as well as the back of the body from the knees up, giving the audience an opportunity to step back and watch the animator telling the visual story. Thus, the camera utilises a first- and third-person visual perspective, showing the story through the eyes of the animator, as well as stepping away to show the animator. As the RSAnimate series evolved, the medium long shot was abandoned and only the over-the-shoulder and close-up perspective remained, with only the drawing hand marker strokes visible. With this evolution of perspective, in which the camera lens moves into to focus on the drawing hand and the animation in process, the boundary between the animator and the speaking narrator becomes blurred. An underlying assumption of multimodality is that people do language from available resources or modes such as writing, image, moving image, sound, speech, gesture, gaze and posture (Jewitt 2013: 253). Image types and photographic perspectives interact with the audience to perform visual speech acts and simulate social interaction. Body parts, such as arms, legs and fingers, or held objects, like hand tools, can be used to establish connectivity, as can the gaze that visually addresses and demands an action of the viewer (Kress and Leeuwen 2006). Moreover, perspective angles within images can establish vector patterns between semiotic resources. Specifically, a vector pattern is an interaction that ‘[does] something to or for each other’, such as a character pointing a finger at another person or object (Kress and Leeuwen 2006: 59). For the animator visualising the narrative, the shifts in POV need to be managed through cartooning techniques that prompt interaction between characters and the audience. In cartooning, visual Section 1 Page 15 of 28 24 English language and multimodal narrative ‘demands’ that simulate social interaction may include gestures, eye contact and speech bubbles in which the character directly addresses the audience. In Changing the Education Paradigm (2010), POV is realised linguistically through the use of the first-person pronoun ‘we’ and second-person pronoun ‘you’ and visualised through strategic integration of a caricature of the RSA lecturer Sir Ken Robinson. In responding to his own question of whether we should keep academic standards high, the caricature of Robinson is drawn facing the audience and speaking via the cartooning device of the speech bubble. The caricature of the narrator points his finger in the air to accentuate his point and speech bubbles represent the linguistic engagement moves; Robinson uses a solidarity move by saying, ‘Yes! We should!’ and then asks the audience ‘why would you lower [standards]?’ speaking directly to the audience while making eye contact. In a follow-up frame, the caricature of Robinson points directly at the audience while telling them, ‘If you are interested in the model of education, you don’t start from production line mentality’. In these examples, POV is enacted on both linguistic and semiotic levels, with second-person pronouns, speech bubbles, the demand gaze and pointing gestures to draw the audience into the story (Figure 24.2). Figure 24.2 RSAnimate – Changing the Education Paradigm (2010) Similarly, in Economics Is for Everyone! (2016), the narrator, RSA lecturer Ha-Joon Chang, is also integrated into the animation to shift POV. When Chang linguistically shifts between first-person, second-person and third person POV in his lecture, it is also visually captured through speech bubbles and hand gestures. As Chang uses second-person pronouns to Section 1 Page 16 of 28 24 English language and multimodal narrative talk to the audience, saying ‘This is the attitude you have to have in debating economic issues’, visual perspective is achieved through caricatures of Chang gesturing with an open hand and looking at the audience. In another frame, where the narrator attempts to create solidarity by stating that ‘We all need these diverse approaches to economics’, the pronoun ‘we’ is operationalised along with the demand gaze and outreached arms that invite the audience to be part of the larger group. Additionally, special effects are used to show Chang blinking, making the cartoon version of the narrator seem more real and rhetorically amplifying the demand gaze (Figure 24.3). This move to enhance authenticity increases modality and, following social semiotic theory, images that act with higher modality are deemed more credible and hold more persuasive power with audiences. Figure 24.3 RSAnimate – Economics Is for Everyone! (2016) One way in which digital composers can appeal to the emotions is by manufacturing a sense of intimacy between the narrator and the audience. This sort of synthetic personalisation can be seen when a message for a mass audience is engineered in such a way that individuals feel as if they are personally addressed (Fairclough 1996). By employing visual resources to befriend an online audience through linguistic and visual speech acts that rhetorically manufacture a sense of intimacy and closeness, digital composers engage in the business of virtual visual synthetic personalisation to sell ideas and products (Thompson 2012). Multimodality is helpful for making sense of rhetorical appeals that bring together a complex web of audio and visual modes for Section 1 Page 17 of 28 24 English language and multimodal narrative communicating casually with online audiences. In the RSAnimate series, caricatures of narrators speak and gesture directly to the audience, with camera angles allowing the audience a sense that they are sitting with the animator and watching the narrative process from the same perspective – these are just some of the ways in which whiteboard animators manipulate POV to connect with the audience. While the story is narrated by the lecturer with an audio voice-over, it is simultaneously told through a cartoon caricature of the lecturer. Synthetic personalisation, be it verbal or visual, is a rhetorical device that appeals to the emotions of the audience (pathos). Aristotle (2004: 9) pointed to such appeals as one of the most important tools of persuasion such that ‘the emotions are all those feelings that so change men as to affect their judgements, and that are also attended by pain or pleasure’. Voice: who tells the story POV and voice are narrative devices that are often conflated. The best way to differentiate these elements is to think of POV as ‘who sees the story’ and voice as ‘who tells the story’. This is not unlike Smith and Watson’s (2010) discussion of autobiographical ‘I’ as multiple with a distinction between the narrator and the narrated. In whiteboard animation, these devices of authorial control can be complicated by the levels of narration and POV that work on the audience simultaneously. There is the narrator telling the story verbally, as well as an animating narrator who tells the story visually, so the audience ‘sees’ the visual voice. Thus, a speaker provides one level of voice and an animator adds another, making whiteboard animation rich with multivocality. In literature, multiple narrators are employed to represent a story from different perspectives with distinct voices. In whiteboard animation, however, the voices of the narrator and animator are combined to create the impression that they are one and the same. This is further complicated when a caricature of a narrator is drawn into the story to serve as an orator who speaks to the audience, such that authorial control can be a combination of the speaking narrator, an illustrating cartoonist (via some form of embodiment) and a cartoon version of the Section 1 Page 18 of 28 24 English language and multimodal narrative narrator telling the story. By taking a multimodal narrative approach, these layers of voice can be examined. As whiteboard animation has evolved, the drawing hand has come to represent a significant element of authorial control. In the earliest RSAnimates, camera angle alternated between mid-shot and close-ups of the back of the animator and his drawing hand. A comparison between early and recent videos demonstrates a clear move from a process in which the animator is recorded drawing on a sketchboard to a technologically simulated version of a hand animating on a whiteboard. As the RSAnimate series has evolved, camera angles have moved in and the animator’s presence has moved out with only the drawing hand and forearm visible through close-up shots. In the most recent RSAnimates, the sporadic integration of the hand appears to be a visual effect of animation software that marks the genre rather than an authentic representation of the hand in motion. With the new process, the body of the animator is no longer present, and the hand is only sporadically visible (if at all). For RSAnimate, the visual narrator has slowly been erased, with only the marker strokes left to follow as the story unfolds. In a tutorial about the process used by Sparksight, the animator discusses the importance of the hand and how aftereffects editing is used to make the drawing hand appear uninterrupted and constant (Sparksight 2013). From a rhetorical narratology approach, ‘voice is the fusion of style, tone, and values’ that link to attitudes, ways of talking and ideology for the audience (Phelan 1996: 45). An analysis of RSAnimate shows that audio voices narrating the series are overwhelming male, at 86 percent, and while 14 percent of RSA lectures are given by women, all the corresponding whiteboard animation versions are visually voiced through what RSA describes as ‘everyone’s favourite hairy hand’ on their website (Figure 24.4). Phelan (1996: 43) has pointed out that ‘To listen to the narrative is, in part, to listen to values associated with a given way of talking’, such that the visual re-voicing of female narrators means that the audience only sees the narrative through the gendered male body, problematically eliding the voices of female intellectuals. Since there has been a historical lack of representation of women in academia (Winslow and Davis Section 1 Page 19 of 28 24 English language and multimodal narrative 2016) and comics (McCloud 2000), this sort of visual re-voicing through the animation reinforces the problem of gender inequity in subtle ways. One may apply Bakhtin (1984) conceptions of ‘polyphony’ (where more than one voice can be heard) and ‘double-voicing’ (where a different voice is adopted to the discourse of another) to whiteboard animation to understand how narrative voice can be ‘overpopulated’ with the voices of others (Aczel 2010b). And while the RSAnimate examples of double-voicing can be described as ‘uni-directional’ in that the goal is to maintain the original discourse through the stylisation of animation (Aczel 2010a), the stylistic choice to re-gender the animator, or ignore the option to maintain the original gender, politicises multimodal narrative. Figure 24.4 RSAnimates: Smile or Die (2010); The Paradox of Choice (2011); How to Help Every Child Reach Their Potential (2015) The inclusion of the hand, as well as a specific type of hand, is a rhetorical choice. DIY programmes like Sketch Book Pro 3.0 allow composers to choose from drawing hands or even import their own hand. Though the ‘hand’ library for VideoScribe is limited to pre-set categories of gender (male and female), age (adult or child) and race represented by colour (white, brown and black), it is progressive in representing the embodied narrator voice in diverse ways. DIY users have called for more diverse choices in the hand library on the help forum and VideoScribe has responded with additions. However, at the time of this writing, there was only one blackskinned hand. Thus, the hand that is a hallmark of the genre is rarely the true embodiment of the illustrator, but rather has become a digital element to be manipulated and customised for stylistic effect, with the hand serving as a rhetorical device that visually constructs the animator based on Section 1 Page 20 of 28 24 English language and multimodal narrative a limited number of options. Similarly, software programmes often allow composers to choose the writing implement that will be shown in the drawing hand; options include a pen, pencil, crayon, paint brush and, of course, a whiteboard marker. In this way, the combination of the hand and the writing tool represent an aspect of authorial control that works in concert with the voice of the narrator and the visual imagery of a cartoon character talking to the audience. Future directions With the continuing emergence of new forms of digital storytelling, methods that move beyond linguistic tools of narrative analysis are needed to adequately examine these complex multimodal narrative forms. As the tools of production for digital composing continue to become more readily available and user-friendly, avenues for distribution and consumption become more accessible, making research that takes a critical approach to these elements vital. In the last decade, DIY software programmes have emerged allowing ordinary people to tell their stories through whiteboard animation through libraries that include pre-drawn characters, stock images, music, drawing implements and hands. Just as noted in this chapter, future research must continue to examine issues of representation in professional and DIY whiteboard animation. As the genre of whiteboard animation has evolved, the ways in which narrative elements work on the audience has also shifted. The current use of the whiteboard as a canvas carries with it connotations specific to material practices associated with the writing context, whereas innovations to this context have the potential to create new ‘settings’ for audiences to imagine in the future. Narrative time can be managed multimodally through audio narration, visual space and stroke order, as shown in these examples, but also through devices such as music, iconic cartoon devices and after-effects in the post-production phase. Research that examines whiteboard animation through the lens of multimodal narrative fits within a Big Tent DH framework that looks to understand the intersection between technology and storytelling. At the annual Day in the Life of the Digital Humanities, Selber Section 1 Page 21 of 28 24 English language and multimodal narrative argued that DH is ‘the study of the role of technology in helping to structure the work of writers and writing contexts’ (TAPoR 2011). With a similar concern for rhetorical forms, Piez (2008: 9) called for ‘media consciousness’ and pointed out that ‘not only do we study digital media and the cultures and cultural impacts of digital media; also we are concerned with designing and making them’. Thus, future research that considers other aspects of multimodal narrative devices would also contribute to DH scholarship that looks to understand the implications of technology, writing and narrative. Further reading 1 Comer, K. (2015). Illustrating praxis: Comic composition, narrative rhetoric, and critical multiliteracies. Composition Studies 43(1): 75–104. This essay proposes using narrative theory alongside comic studies, with a focus on narrative gaps, narration and focalisation as a heuristic for rhetorical design to foster critical multiliteracies. 2 Machin, D. (2013). What is multimodal critical discourse studies? Critical Discourse Studies 10(4): 347–355. This introduction to a special issue of Critical Discourse Studies looks to bring a critical lens to multimodal discourse studies, illustrating how different levels of communicative activity are infused by, and shaped by, power relations and ideologies through semiotic resources. 3 Royce, T.D. (2015). Intersemiotic complementarity in legal cartoons: An ideational multimodal analysis. International Journal for the Semiotics of Law – Revue Internationale de Sémiotique Juridique 28(4): 719–774. This article applies multimodal discourse analysis to legal cartoons and argues to extend the framework to address more complex layers of represented semiotic meaning. Section 1 Page 22 of 28 24 English language and multimodal narrative 4 Souto-Manning, M. (2014). Critical narrative analysis: The interplay of critical discourse and narrative analyses. International Journal of Qualitative Studies in Education 27(2): 159–180. This article looks to establish critical narrative analysis by bringing together critical discourse analysis and narrative analysis to address theoretical and methodological dilemmas to examine power inequities in narratives. References Aczel, R. (2010a). Polyphony. In D. Herman, M. Jahna and M.-L. Ryan (eds.), Routledge encyclopedia of narrative theory. London: Routledge. Aczel, R. (2010b). Voice. In D. Herman, M. Jahn and M.-L. Ryan (eds.), Routledge encyclopedia of narrative theory. London: Routledge. Adami, E. (2017). Multimodality. In The Oxford handbook of language and society. Oxford: Oxford University Press. Aristotle. (2004). Rhetoric (D. Thrift, eds.). New York: Dover Publications. Bakhtin, M. (1984). Problems of Dostoevsky’s poetics, 1st ed. Minneapolis: University of Minnesota Press. Bal, M. (2004.) Narrative theory: Major issues in narrative theory. Taylor & Francis. Bamberg, M. and Georgakopolou, A. (2008). Small stories as a new perspective in narrative and identity analysis. Text & Talk 28(3): 377–396. Borek, L., Dombrowski, Q., Perkins, J. and Schöch, C. (2016). TaDiRAH: A case study in pragmatic classification. Digital Humanities Quarterly 10(1). Childhaven. (2014). Have you met Olivia? Available at: www.youtube.com/watch?v=GhrIYJ_s6Xw (Accessed 5 March 2017). Cognitive Media. (2017). Explainer videos & digital storytelling – previous work. Available at: www.wearecognitive.com/our-work (Accessed 18 February 2017). Section 1 Page 23 of 28 24 English language and multimodal narrative Comer, K. (2015). Illustrating praxis: Comic composition, narrative rhetoric, and critical multiliteracies. Composition Studies 43(1): 75–104. Cope, B. and Kalantzis, M. (2009). “Multiliteracies”: New literacies, new learning. Pedagogies: An International Journal 4(3): 164–195. doi:10.1080/15544800903076044. Edwards, R. (1999). Politics as social history: Political cartoons in the gilded age. OAH Magazine of History 13(4): 11–15. Eisner, W. (2008). Comics and sequential art: Principles and practices from the legendary cartoonist. New York: W. W. Norton & Company. Ewert, J.C. (2010). Comics and graphic novel. In D. Herman et al. (eds.), Routledge encyclopedia of narrative theory. London: Routledge. Fails, J.A. and Kelton, A. (2016). Call for papers. The Journal of Emerging Learning Design. Special Issue: The Digital Humanities. Available at: www.hastac.org/opportunities/callpapers-journal-emerging-learning-design-special-issue-digital-humanities (Accessed 13 February 2017). Fairclough, N. (1996). Technologisation of discourse. In C.R. Caldas-Coulthard and M. Coulthard (eds.), Texts and practices: Readings in critical discourse analysis. London: Routledge, pp. 71–83. Georgakopoulou, A. (2006). Thinking big with small stories in narrative and identity analysis. Narrative Inquiry 16(1): 122–130. doi:10.1075/ni.16.1.16geo. Idea Rocket. (2017). IdeaRocket – animated video production services: 2d, 3d, whiteboard. Available at: http://idearocketanimation.com/ (Accessed 19 February 2017). Jenkins, H. (2006). Convergence culture: Where old and new media collide. New York: New York University Press. Jenkins, H. (2007). From participatatory culture to participatory democracy (Part Two). Confessions of an Aca-Fan. Available at: http://henryjenkins.org/blog/2007/03/from_participatatory_culture_t_1.html (Accessed 21 February 2018). Section 1 Page 24 of 28 24 English language and multimodal narrative Jewitt, C. (2013). Multimodal methods for researching digital technology. In S. Price et al. (eds.), The Sage handbook of digital technology research. Los Angeles: Sage Publications, pp. 250–265. Jewitt, C., Bezemer, J. and O’Halloran, K. (2016). Introducing multimodality. London: Routledge. Kearns, M.S. (1999). Rhetorical narratology. Lincoln, NE: University of Nebraska Press. Keating, P. (2010). Point of view (Cinematic). In D. Herman et al. (eds.), Routledge encyclopedia of narrative theory. New York: Routledge. Kress, G.R. (2009). Multimodality: Exploring contemporary methods of communication. Abingdon and New York: Routledge. Kress, G.R. and Leeuwen, T. van (2001). Multimodal discourse. London and New York: Bloomsbury Academic. Kress, G.R. and Leeuwen, T.V. (2006). Reading images: The grammar of visual design. Abingdon: Routledge. Labov, W. and Waletsky, J. (1997). Narrative analysis: Oral versions of personal experience. Journal of Narrative and Life History 7(1–4): 3–38. Machin, D. (2007). Introduction to multimodal analysis. New York: Oxford University Press. Machin, D. (2013). What is multimodal critical discourse studies? Critical Discourse Studies 10(4): 347–355. doi:10.1080/17405904.2013.813770. McCloud, S. (1994). Understanding comics: The invisible art, reprint ed. New York: William Morrow Paperbacks. McCloud, S. (2000). Reinventing comics: How imagination and technology are revolutionizing an art form. New York: William Morrow Paperbacks. Medhurst, M.J. and Desousa, M.A. (1981). Political cartoons as rhetorical form: A taxonomy of graphic discourse. Communication Monographs 48(3): 197–236. doi:10.1080/03637758109376059. Section 1 Page 25 of 28 24 English language and multimodal narrative Nikolajeva, M. (2014). Picturebooks and emotional literacy. The Reading Teacher 67(4): 249– 254. doi:10.1002/trtr.1229. Page, R. (2009). New perspectives on narrative and multimodality. New York: Routledge. Page, R. (2012). Stories and social media: Identities and interaction. New York: Routledge. Available at: www.routledge.com/books/details/9780415889810/ (Accessed 19 April 2014). Pears, I. (2015). Arcadia: A novel. New York: Knopf. Peng, W., Lee, M. and Heeter, C. (2010). Les effets d’un jeu sérieux sur la prise de rôle et la volonté d’aider. Journal of Communication 60(4): 723–742. https://doi.org/10.1111/j.1460-2466.2010.01511.x Phelan, J. (1996.) Narrative as rhetoric: Technique, audiences, ethics, ideology. Columbus: Ohio State University Press. Piez, W. (2008). Something called digital humanities. Digital Humanities Quarterly 2(1). Prince, G. (2010). Point of view (Literary). In D. Herman et al. (eds.), Routledge encyclopedia of narrative theory. London: Routledge. Ricoeur, P. (1984). Time and narrative, vol. 1. Chicago, IL: University of Chicago Press. Royce, T.D. (2015). Intersemiotic complementarity in legal cartoons: An ideational multimodal analysis. International Journal for the Semiotics of Law – Revue internationale de Sémiotique juridique 28(4): 719–744. doi:10.1007/s11196-015-9421-1. RSA: The royal society for the encouragement of arts, manufactures and commerce. (2010). Changing education paradigms: Sir Ken Robinson. Available at: www.youtube.com/watch?v=zDZFcDGpL4U (Accessed 17 February 2017). RSA: The royal society for the encouragement of arts, manufactures and commerce. (2016). Economics is for everyone! Available at: www.youtube.com/watch?v=NdbbcO35arw (Accessed 2 February 2017). Section 1 Page 26 of 28 24 English language and multimodal narrative RSA: The royal society for the encouragement of arts, manufactures and commerce. (2010). Smile or die: Barbara Ehrenreich. Available at: www.thersa.org/discover/videos/rsaanimate/2010/03/rsa-animate-smile-or-die (Accessed 17 February 2017). RSA: The royal society for the encouragement of arts, manufactures and commerce. (2011). The paradox of choice: Renata Salecl. Available at: www.thersa.org/discover/videos/rsaanimate/2011/06/rsa-animate-choice (Accessed 17 February 2017). RSA: The royal society for the encouragement of arts, manufactures and commerce. (2015). How to help every child fulfil their potential: Carol Dweck. Available at: www.thersa.org/discover/videos/rsa-animate/2015/how-to-help-every-child-fulfil-theirpotential (Accessed 24 July 2018). Smith, B.H. (1980). Narrative versions, narrative theories. Critical Inquiry 7(1): 213–236. Smith, S. and Watson, J. (2010). Reading autobiography: A guide for interpreting life narratives. Minneapolis: University of Minnesota Press. Sousanis, N. (2015). Unflattening. Cambridge, MA: Harvard University Press. Souto-Manning, M. (2014). Critical narrative analysis: The interplay of critical discourse and narrative analyses. International Journal of Qualitative Studies in Education 27(2): 159– 180. doi:10.1080/09518398.2012.737046. Sparksight. (2013). How to do a fast draw animated video. Available at: www.youtube.com/watch?v=QvyziGZJGM0 (Accessed 3 March 2017). Sparksight Inc. (2015). Corporate video: Animation vs. live action. Sparksight Blog, 22 September. Available at: https://sparksight.com/blog/corporate-video-animation-vs-liveaction/ (Accessed 22 April 2018). SP Controls, Inc. (2015). Doceri – the interactive whiteboard for iPad. Available at: https://doceri.com/ (Accessed 3 March 2017). TAPoR. (2011). How do you define humanities computing/digital humanities? Available at: www.artsrn.ualberta.ca/taporwiki/index.php/How_do_you_define_Humanities_Computin g_/_Digital_Humanities%3F (Accessed 21 July 2016). Section 1 Page 27 of 28 24 English language and multimodal narrative Tapscott, D. and Williams, A.D. (2010). Wikinomics: How mass collaboration changes everything, expanded ed. New York: Portfolio. Terras, M. (2011). Peering inside the big tent: Digital humanities and the crisis of inclusion. Melissa Terras’ Blog, 26 July. Available at: http://melissaterras.blogspot.com/2011/07/peering-inside-big-tent-digital.html (Accessed 16 July 2018). Thompson, R. (2012). Looking healthy: Visualizing mental health and illness online. Visual Communication 11(4): 395–420. doi:10.1177/1470357212453978. Tiemensma, L. (2009). Visual literacy: To comics or not to comics? Promoting literacy using comics. In World library and information congress: 75th IFLA general conference and council. Milan, Italy. Available at: www.ifla.org/past-wlic/2009/94-tiemensma-en.pdf (Accessed 1 June 2016). Toffler, A. (1984). The third wave. New York: Bantam. Whitson, R.T. and Salter, A. (2015). Introduction. Comics and the Digital Humanities 9(4). Available at: www.digitalhumanities.org/dhq/vol/9/4/000210/000210.html (Accessed 28 February 2017). Winslow, S. and Davis, S.N. (2016). Gender inequality across the academic life course. Sociology Compass 10(5): 404–416. doi:10.1111/soc4.12372. Section 1 Page 28 of 28