Digital Puppetry

Abstracto: Dentro del contexto de la generación de recursos multimedia para e-learning, crear animaciones resulta una tarea poco accesible para el usuario inexperto, además der ser un proceso costoso y laborioso. Se presenta el diseño e implementación de una herramienta como alternativa para animar de manera simple, mediante una interfaz natural, un personaje 2D articulado por medio de la captura de movimiento en tiempo-real usando una cámara de profundidad de bajo costo tipo Kinect.

The end of the 20th and beginning of the 21st centuries brought heightened visibility of puppets and performing objects in various fields of cultural performance, so that we might call these last decades ‘a puppet moment’ – as Claudia Orenstein points out in the introduction to the book "The Routledge Companion to the Puppetry and Material Performance" (London, 2014). We are profoundly convinced of the truth of this reflection and we see in this point of view an important research challenge, one that leads toward a discussion about the processes, tendencies, and influences shaping contemporary puppetry in different countries. The intention of our monograph is to present theoretical and practical ideas, analyses and questions which have arisen since the turn
of the century under the influence of the latest puppet performances and works inspired by puppet art. The collection of articles naturally represents only a few of the possible approaches to these topics, but we hope it provides a glimpse of multidirectional contemporary reflection and different perspectives of research now being applied to (and demanded by) puppet art.

The human hand dexterity provides a natural and rich interface for computer interaction. Technologies for tracking hand and finger motion are now accessible and can be used with a minimum computation power for the manipulation of virtual puppets in real-time. However creating an intuitive hand-based motion control interface presents problems such as, mapping the hand to an object that demands more degrees of freedom or to assign gestures that are difficult to memorize or to execute. We propose an ergonomic hand-mapping model for digital puppetry, based on the human hand anatomy and biomechanics, adapting traditional puppetry methods. A cinematic virtual puppetry application was developed supporting distinct interaction styles based on the hand dexterity skills. An experiment using the Leap Motion controller was conducted to evaluate the hand mapping feasibility. The participants considered that the proposed interface provides a good level of directness. Furthermore, a custom interface for smartphones was combined with the application to extend the puppet manipulation and show control operations.

Puppets can be great storytellers when performed in a dramatic way. They create the illusion of life, making the audience believe in the story. But animating puppets using traditional key frame animation is not a trivial task taking too much time and practice, in particular for the non-expert artists. Digital puppetry presents performance-driven animation, making the puppet reactive to the motion of the performer in real-time. Motion capture methods makes the puppet animation fast and simple, based on the acting of the performer, but they are out of reach for the major consumers. We present a low-cost performance-driven technique that allows real-time puppet animation based on the inter-acting, which can be used in live storytelling. In this paper we study how users can interpret simple actions like walking, using their bodies as puppet controllers. The system was deployed using the Microsoft Kinect and by assuming the marionette aesthetics and constraints, showing how low-cost devices can provide a new mean for motion capture representation. We extend the previous study by presenting another method to interact with puppets using indirect mapping by connecting the puppet to the puppeteer with virtual strings. Last, we performed a pilot experiment animating silhouettes and 3D puppets, to better understand differences in the interaction. An audience had to identify by the output final animation, the actions performed by non-expert artists using their bodies to drive the puppets. We conclude, inter-acting with 2D puppets is similar to the marionette manipulation and needs more interpretation than with 3D puppets.

The interactive animation of articulated puppets in real time has been a great challenge to many researchers. Despite of many new hardware and software solutions guided in this di- rection, they are still presented in a limited level of use or high cost. On the other hand, the articulated models for animation with a great degree of freedom of movement can have a large number of points of control making it difficult to manipulate in real time.
One of the main issues to control the articulated puppets concerns the interaction devices. Some of them offers great control but are less accessible, on the other hand devices that are very accessible like the mouse have many limitations in manipulating several control points simultaneously.
We propose to develop a prototype of an articulated puppet for real time animation with a great degree of control using accessible devices like the mouse or the game pad. To simplify the puppet manipulation without diminish the degree of freedom we propose the adaptation of a marionette to the virtual world reducing the number of control points.
The underlying question to this adaptation is to know if the virtual marionette can have the same behaviours and similar features of the real marionette and if this can be considered one marionette from the point of view of the proximity with its operator.

Si foisonnante soit la recherche actuelle, notamment en France, autour de la marionnette, aucun ouvrage collectif n’a jusqu’alors proposé, à notre connaissance, une tentative de saisie synthétique des aspects philosophiques de la marionnette et de son théâtre. Il s’agit avec cet ouvrage de commencer à combler ce vide et de poser des jalons sur un sujet très vaste. En effet, si les liens entre théâtre et philosophie ont fait l’objet de nombreux travaux et sont aujourd’hui un terrain d’investigation très dynamique, la densité métaphorique et symbolique de la forme particulière de théâtre qu’implique la marionnette engage d’autres problématiques : en mettant à distance la présence humaine, le théâtre de marionnettes fait du questionnement sur la nature et la condition humaine ainsi que sur les frontières entre le vivant et le non vivant le centre de la représentation. Par ailleurs, les récents développements de la pensée scientifique, philosophique et anthropologique, associée aux évolutions technologiques récentes, autour des questions touchant au « post‑humain », rendent nécessaire de réactiver le modèle de la marionnette aux côtés d’autres figures comme l’automate, le robot, l’avatar numérique ou le cyborg, particulièrement présents sur les scènes actuelles. On le verra largement dans ce livre, la marionnette nous aide depuis longtemps à penser ce qu’il y a en nous d’« autre » que l’humain et constitue en cela un outil précieux pour appréhender nos avatars ou prothèses contemporaines.

“Haz Makami”, which can be translated as “Modes of Pleasure” into English, was a unique event in Turkish theatre field. It was a production of 6’dan Sonra Tiyatro, who also owns the blackbox stage Kumbaraci50 in central Istanbul. “Modes of Pleasure” was first staged in 2012 and continued for two theatre seasons. “Modes of Pleasure” starts with the daily interactions of four women who live in the same apartment in Istanbul, but story takes a wild turn after midnight when each of them start exploring their very kinky sexual fantasies, separately in their own homes. The rhythm that each character catches is unique, but they find unisons immediately so it forms a “makam”, a specific definition of “mode” in tone and timber scaling system of classical Ottoman music, which results in a rhythm (and melody) that is unique to Istanbul and women of Turkey. Communication takes place among the characters through reduced body language of the puppets and musical rhythms they find throughout the play. This paper describes and analyzes the play theoretically through the layers of estrangement effects it brought to the audiences of Istanbul (estrangement via the technique of puppetry and feminist subversion of bringing women’s sexuality on stage) through placing the play in the broader social context it belongs to.

Lo spettacolo teatrale Sayonara è un dialogo di venticinque minuti tra un androide e un’attrice. Il nuovo robot del Professor Ishiguro è di genere femminile, è stato chiamato Geminoid F, in riferimento al precedente Geminoid (HI-1), una copia identica del suo creatore, il Professor Ishiguro. Come la precedente, anche questa nuova creatura è destinata ad un uso “teleoperato”. La Gem F è la copia di una donna giapponese di una ventina d’anni la cui identità non è stata rivelata per preservarne la privacy. Essendo per un quarto di origine straniera, ha lineamenti atipici. Ha lunghi capelli castano scuro. Le istruzioni dicono che può ridere, sorridere ed avere altre espressioni facciali più naturali e dolci rispetto al modello precedente. Una particolare attenzione è stata dedicata all’abbigliamento, sobrio ma elegante, disegnato dallo stilista Junko Koshino. Indossa un abito di pizzo nero, corto, con un’ampia scollatura arrotondata e delle maniche svasate a tre quarti, porta anche dei collant neri e un paio di scarpe col tacco in camoscio grigio antracite. Tra i modelli precedenti alla Gem F, ricordiamo le “attroidi” femminili Repliee R1, il clone della figlia del Professor Ishiguro con un abito a fiorellini, e poi Repliee Q1, spesso presentata in T-shirt, e il clone dell’annunciatrice televisiva di NHK, Repliee Q2, entrambe in pantaloni. Mentre Repliee R1 è solo una bambina a piedi nudi e Repliee Q1 è stata hostess all’Aichi World Expo nel 2005, Repliee Q2 e Gem HI-1 hanno un capitale simbolico: una come annunciatrice televisiva nota al grande pubblico e l’altro come insegnante, a cui i media hanno dato altrettanto risalto. Grazie alle loro rappresentazioni sociali, la prima ha goduto dello status di star della televisione e l’atro ha assunto una posizione di autorità scientifica. Tutti questi androidi sono il risultato di una stretta collaborazione tra l’Università di Osaka, il centro di ricerca ATR  di Kyoto e l’azienda giapponese Kokoro Compagny Ltd.  con sede a Tokyo e specializzata nelle animatroniche chiamate actroids, cioè degli androidi ultra-realistici per l’intrattenimento.

The Vari House virtual world is a reconstruction of an ancient Greek farmhouse excavated fifty years ago in southern Greece. Implemented in Unity3D, the virtual world features a digital
puppet, an avatar representing the teenage son of the farming family that lives there. We project Vari House onto a large screen, so that the house and puppet are life-sized to enhance audience engagement. Under the control of a teacher or puppeteer, the avatar communicates through voice and gesture, moving freely through the virtual space. He discusses the house itself and daily life, but any relevant topic is accessible. For depth of conversation, a human puppeteer is superior to any artificial intelligence. This version of Vari House works well for museum audiences of all ages and fits the ancient history curriculum mandated in most states for middle school.

«How do you prove to be human?» It is the head of an Amletic robot that interrogates the audience at the end of the show Robot Dreams by the German company Meinhardt Krauss Feigl, a performance that includes dancers, automata, robots and animatronics. The title is a clear reference to the work of Asimov, whose main themes are present within the show with references and quotations, such as the constant reflection on «What do robots dream of?».
The show was performed at the Festival mondial des théâtres de marionnettes in Charleville-Mézières in 2019, confirming its belonging, in the section related to new experiments, to the world of puppetry. This is not an episode in itself, increasingly in the world of puppetry we can see the appearance of new technologies and robotic elements in performances. From Puppet Robots to mechanical gloves, from animatronics to great puppets: puppet theatre is moving much faster than other theatrical techniques towards the discovery of new technological means.
The present study aims to investigate the relationship between puppeteer and robot within the contemporary scene, exploring the interaction between the artist and the object manipulated through sensors and new technologies. In puppet theatre, the puppeteer, wearing a glove, gives life to the puppet, the body of the puppet is formed by the hand of its manipulator. What happens when the puppet is made through a prosthesis created in animatronics? How is the manipulation handled during the performance? Who manages the machine in relation to the puppet master's work? The aim of this research will be precisely to try to answer these questions, starting from some embryonic examples, such as Bit the electronic puppet animated through a cyberglove created by Giacomo Verde to the prostheses created as animatronics in the show Robot Dreams by the company Meinhardt Krauss Feigl.

On 12th August 2021, supported by Asia Europe Foundation / Mobility First, Harkat Studios, India, held a virtual International Open House connecting European and Indian artists to the Indian arts community, with the aim of sharing technical knowledge, creative experiences and small technical and creative victories from the worldwide lockdowns. Over the course of the day, there were panels and fireside chats, master classes and presentations in the realms of film, theatre and music. This article is dedicated to the theatre part.
In hands-on, very personal conversations, Mahesh Dattani (Playpen Mumbai, India), Baraba Kölling (Helios Theatre, Hamm, Germany) and Anurupa Roy (Katkatha, Delhi, India) share how they have been using the internet and online mediums to collaborate, create and mount performative works. They open up about workflows, learnings and collaboration methods from their experience of working online.

The outcome was an optimistic look into the future. Instead of merely succumbing grudgingly to the technically challenging character of this new medium of ‘the digital’ while waiting for better days when the world opens again, they see themselves at the beginning of a new movement or even an art form. And after nonetheless painful birth pangs, they are now very excited to contribute to the language of this new ‘2D theatre’ movement.

To access the recordings of all conversations email to [email protected] .

The main issue of the article is to present multimedia practices in contemporary puppet theatre, which were discussed during the International Conference "Puppetry and Multimedia" in Bratyslawa (Slovakia),  2017, October. The field of research interest are: mediatization of the stage design wht the screen aesthetic, coexistance the animate and inanimate, the human and non-human in multimedia puppetry projects, digital puppetry, cinema as the source of inspiration for puppeteers, multimedia techniques as the tools of experiments with perception of puppets and creation of puppet theatre world.

By drawing parallels from theatre, this paper examines new paradigms in interaction models for the viewer (user) of interactive virtual worlds. Studying the changes in the aesthetic experience of the viewer from theatre to cinema to interactive virtual worlds, changing dynamics of the author-viewer relationship across these media are highlighted. Agency of both, the author and the viewer are discussed, establishing a case for exploring interaction models based on ideas of co-authorship.
Puppetree, a digital puppetry platform, has been developed as means to develop the ideas of co-authorship, taking inspiration from the position and agency of a traditional puppeteer. The plat- form is built with Intel’s perceptual computing that uses a motion sensing technology to detect a user’s hand and translate the movement to a puppet in a 3D environment. The direct hand- control allows the user to experience the virtual environment as an extension of his immediate physical reality vs. immersing into an environment as an avatar. Exploring narrative structures that shape user-experience in such environments is in further scope of this work which requires an understanding of the logic of interactive storytelling in digital media.

Animation and special effects studios today are making increasing use of motion capture, computer-generated imagery (CGI), and digital puppetry in television, theatre, commercial films, and videogames. As CGI grows ever more capable and pliant, it is not only changing the way actors and puppets show up in various media, it is enabling animated characters to perform like puppets and vice versa. Actors are controlling cartoon characters, while visual-effects teams are seamlessly mingling objects with images, blurring the lines between acting, animation, puppetry, and special effects. Hybrids are appearing in a range of environments, including live theatre, theme parks, and - most commonly - fantasy films. In the latter, animation and puppetry are sometimes combined to achieve human likeness through motion capture technology. In this essay, I will argue that successful hybrids are radically transforming what puppetry and animation can do. In some cases, however, where these fusions are used to achieve naturalistic human likeness, their creators are setting aside what is most advantageous about puppetry and animation, thus diminishing their creative impact.
Published in The Routledge Companion to Puppetry and Material Performance (2014), ed. Dassia N. Posner, Claudia Orenstein, and John Bell.

Character animation has traditionally relied on complex and timeconsuming key frame animation or expensive motion capture techniques. These methods are out of reach for low budget animation productions. We present a low-cost performance-driven technique that allows real-time interactive control of puppets for performance or film animation. In this paper we study how users can interpret simple actions like, walking, with different puppets. The system was deployed for the Xbox Kinect and shows how low-cost equipment's can provide a new mean for motion capture representation. Last, we performed a pilot experiment animating silhouettes and 3D puppets in real-time, based on body movements interpreted by non-expert artists, representing different Olympic Sports that vary on visual representation for other participants to identify. As a result, interaction with 2D puppets needs more interpretation than with 3D puppets.

Animation and special effects studios today are making increasing use of motion capture, computer-generated imagery (CGI), and digital puppetry in television, theatre, commercial films, and videogames. As CGI grows ever more capable and pliant, it is not only changing the way actors and puppets show up in various media, it is enabling animated characters to perform like puppets and vice versa. Actors are controlling cartoon characters, while visual-effects teams are seamlessly mingling objects with images, blurring the lines between acting, animation, puppetry, and special effects. Hybrids are appearing in a range of environments, including live theatre, theme parks, and - most commonly - fantasy films. In the latter, animation and puppetry are sometimes combined to achieve human likeness through motion capture technology. In this essay, I will argue that successful hybrids are radically transforming what puppetry and animation can do. In some cases, however, where these fusions are used to achieve naturalistic human likeness, their creators are setting aside what is most advantageous about puppetry and animation, thus diminishing their creative impact.
Published in The Routledge Companion to Puppetry and Material Performance (2014), ed. Dassia N. Posner, Claudia Orenstein, and John Bell.
ebook: http://www.vlebooks.com/vleweb/product/openreader?id=none&isbn=9781317911722.

Virtual Marionette is a research on digital puppetry, an interdisciplinary approach that brings the art of puppetry into the world of digital animation. Inspired in the traditional marionette technology our intention is to study novel interfaces as an interaction platform for creating artistic contents based on computer animated puppets. The overall goal of this thesis is to research and deploy techniques and methods for the manipulation of articulated puppets in real-time with low-cost interfaces to establish an interaction model for digital puppetry.

Pull-the-Strings presents a mapping model for digital puppetry based on a transparent framework to support generic device controllers and generic tools. Digital puppetry requires a creative interaction design, in particular in the way designers map the puppet to the puppeteer using specific devices. This process depends on a constantly changing interface technology, which limits the reuse of devices and mappings. This paper proposes a methodology and a set of tools that facilitate the mapping process, and promote the recycling of technologies. A flexible and generic environment independent from device specifications. By abstracting the hardware layer, the artist is motivated to think in terms of signal flow, establishing relations through meaningful mappings instead of handling the diverse specifications of each device and application. Pull-the-Strings is a data-flow ecosystem that focus on the functional usage of control signals. It provides a scalable environment for building semantic blocks that connect, transform and generate signals for the manipulation of virtual objects. Its goal is to make technology as transparent as possible, facilitating connections and reducing the obstacles between the performer and the performing object. On the other hand, it proposes an interaction design space that takes into account the manipulation and perception distance, responding to the specifications of the digital puppetry medium. This model was evaluated comparing a set of tools and methods with experienced and non-experienced users.

The Vari House virtual world is a reconstruction of an ancient Greek farmhouse excavated fifty years ago in southern Greece. Implemented in Unity3D, the virtual world features a digital puppet, an avatar representing the teenage son of the farming family that lives there. We project Vari House onto a large screen, so that the house and puppet are life-sized to enhance audience engagement. Under the control of a teacher or puppeteer, the avatar communicates through voice and gesture, moving freely through the virtual space. He discusses the house itself and daily life, but any relevant topic is accessible. For depth of conversation, a human puppeteer is superior to any artificial intelligence. This version of Vari House works well for museum audiences of all ages and fits the ancient history curriculum mandated in most states for middle school.

By drawing parallels from theatre, this paper examines new paradigms in interaction models for the viewer (user) of interactive virtual worlds. Studying the changes in the aesthetic experience of the viewer from theatre to cinema to interactive virtual worlds, changing dynamics of the author-viewer relationship across these media are highlighted. Agency of both, the author and the viewer are discussed, establishing a case for exploring interaction models based on ideas of co-authorship. Puppetree, a digital puppetry platform, has been developed as means to develop the ideas of co-authorship, taking inspiration from the position and agency of a traditional puppeteer. The platform is built with Intel’s perceptual computing that uses a motion sensing technology to detect a user’s hand and translate the movement to a puppet in a 3D environment. The direct handcontrol allows the user to experience the virtual environment as an extension of his immediate physical reality vs. immersing int...

In this paper we propose to use our body as a puppetry controller, giving life to a virtual silhouette through acting. A framework was deployed based on Microsoft Kinect using OpenNI and Unity to animate in real-time a silhouette. This was used to perform a set of experiments related to the user’s interaction with human and non-human like puppets. We believe that a performance-driven silhouette can be just as expressive as a traditional shadow puppet with a high degree of freedom, making use of our entire body as an input. We describe our solution that allows real-time interactive control of virtual shadow puppets for performance animation based on body motion. We show through our experiment, performed by non- expert artists, that using our body to control puppets is like mixing the performance of an actor with the manipulation of a puppeteer.