The Processed Video Body as a Supernatural Being

Todd Winkler
Brown University
Providence, RI USA
[email protected]

Abstract familiar experience of mirrors and shadows paves the way

Realtime video processing enables a live sensing body to be for the believability of two different but simultaneous
depicted on screen endowed with supernatural qualities. An bodies. Even when a process leans so far towards
exploration of these ideas through the perspective of multiple abstraction as to blur the connection to the original, the
art forms shows common techniques that enhance the gestures are still felt and read as tethered to a
experience for participants and viewers. Close attention paid to
spontaneously acting person.
the altered self starts a feedback loop that creates new sensation
and kinesthetic response. The mystical depiction of the We have a great ability for recognizing and
expressive human body, by itself and in relation with others, understanding human movement, even from the most
drives the research and content of this study. Video processing minimal of detail. Perception experiments that abstract the
serves as deep content and inspiration for the work. moving body image down to just a few point lights
(illuminated dots representing the joints of the body) show
Keywords that the distinctive timing, articulation and pathways of
Video processing, supernatural, digital double, performance human movement are easily recognizable, even when
distorted by masking, timing or viewing angle. Point-light
Introduction experiments have also shown the inherent readability of
Digital doubles representing spiritual emanations or the expressive body, with our ability to recognize such
incarnations of the body relate to notions of ghosts, astral things as gender, intent and emotions solely from
bodies, out-of-body experiences, and soul projection. abstracted movement information [2]. These studies
The double may be depicted as a gaseous figure confirm the expressivity of human movement, even in a
composed of particles, or in another, more liquid-like highly abstracted supernatural form.
ethereal form, luminous and transparent. Steve Dixon Along with the proper scale, synchronization is
In his book, Digital Performance (2006), Steve Dixon important in assuring that participants locate their body
suggests several different theatrical functions of the digital simultaneously in the real world and in the virtual world.
double. Most pertinent to the work discussed here is his Even small time delays will disrupt this effect. This not
notion of the digital double as a spiritualized or only allows the participants to see their processed image
supernatural being [1]. Live video processing offers a glowing, shimmering or erupting in fire, but to become that
transformation and a transcendence of the physical body. image to feel what it is like to have that body. This is the
Even though the video projection is a result of computation one of the most interesting facets of this work: the
and manipulation of pixels, it's uncanny attachment to a interactive loop of observing and identifying with the
performer imparts it with the expressive language of a processed body informs the actual movement and
moving, sensing being. sensations in the real body. Full attention focused on the
The author has explored these concepts in several digital double produces new sensations and a new
artistic disciplines and offers a comparison study here kinesthetic vocabulary refined and directed by the process.
gleamed from insights creating audio/video installations, The author has written elsewhere about studies in the
dance pieces and theatre productions that all share mislocation of the body and its relevance to video
similarities. Careful attention is paid to creating a life-sized installations - under the proper conditions, participants may
processed body and to its relationship with the living body. perceive the location of their bodies shift from the actual
Even in more abstract works, the processing has meaning, location into the projection world [3]. Especially in
suggesting a narrative to participants as they negotiate their interactive work that allows free fully-body movement, the
newfound status as mystical beings. These techniques also actions of the user provides a coherent sense of bodily
find their purpose as characters in storytelling found in self in the projected image [4]. This sense of body
theatrical work. ownership encourages installation participants and
The life-sized processed image is important to obtain performers to explore the physical potential of the virtual
balance and parity between the two performers. The worlds, portraying unnatural physics, strange colors, or
space and time distortions.

1
 Common Techniques
The first step in the process is to isolate a participants
figure from the background. This is done with a single
camera input using some form of background subtraction,
either lumakey (selecting a lit figure in front of a black
background) or difference sensing (taking a shot of the
empty space and looking for any pixel changes when a
subject enters). Using visible light captures more details of
facial expression and clothing. Infrared lighting with an
infrared camera yields an excellent silhouette, with the
added feature of being able to ignore any lighting changes
on a stage or projected video images that might otherwise
be picked up by the camera and cause video feedback (the
projected video is outside of the IR range, and is thus
ignored). The IR technique on stage can be used to meld
the performers real bodies and their supernatural Fig. 1 A processed figure immersed in moving light.
processed bodies into a unified scene. Video processing
and output is handled using Jitter (Cycling 74), with To enhance and represent connections between friends
simple projection mapping and additional effects handled and strangers, a special effect is activated when there is
by the OpenGL component. With work featuring multiple more than one participant; a fine webbing attaches to the
video outputs, the hardware devices used are extremities of two or more silhouettes when they get close
TripleHead2Go (3 projectors) and DataPath wall (within 3 feet), shooting out from one body to connect with
controllers (5 projectors). and touch another. Sometimes traces of the body are left
Three primary techniques will be examined here: visible on screen, encouraging users to paint the walls
keying, connecting, and traces (painting). In keying, the with their interactions and gestures. A hypnotic soundtrack
silhouette is filled with a moving video image, showing the completes the world and assists in pulling viewers in to a
body as light, water, clouds, or fire. Connecting uses some more complete and immersive experience.
objects in the cv.jit collection (computer vision) to create With the loss of physical detail that might create a
rays of light, an energy body, and webbing that connects feeling of separation or body self-consciousness,
two or more participants. Traces allows users to draw participants notice their similarity and connection with
pathways of their movements, simply allowing each frame other freely moving, and sensing, virtual bodies. This
to build up without erasing the previous, or using pixel shared experience leads to the dissolution of the barrier
averaging over time to capture an image with stillness, or a between self and others. Everyone is in the same boat
ghostlike figure with movement. and is made up of the same material. The relationships are
These techniques and ideas are now followed over the complex: people isolated with friends or strangers in a
course of several years of development in three different darkened room, a feeling of being in two bodies at once,
projects. the location or mislocation of sensation, the intense images
of flowing water and light, and the overlap and interaction
Glint (2010) Audio/Video Installation with real and virtual bodies.
Glint imagines an ethereal space where participants as
spiritualized beings meet and interact. The original video
source material for Glint is high-contrast shots of various
forms of sunlight reflected off of the ocean. The long shots
capture a shimmer of light, whereas close-ups produce an
intense ray or large sparkles of changing light. Viewers see
their life-sized bodies filled with this intense light, and/or
floating on top of it, or portrayed as shimmering rays of
energy. Some report feelngs of movements when filled
with moving water, or a feeling of warmth in their body
from the light filling the torso.
One or more people enter a specially built 10 x 14
room with large projections completely filling three walls,
and an infrared camera used to capture participants
silhouettes. There, they are free to move and explore their
spiritualized bodies made of this magical light.
Fig. 2 Bright reflected light inside the torso.

2
 Bali Ghost Dance (2013) Dance Improvisation infrared techniques, the color camera has to be oriented
Bali Ghost Dance was a collaboration with choreographer away from the projection screen or video feedback would
Nyoman Catra to explore improvised Balinese dance occur, but the tradeoff is that is can pick up and process a
movements and live video processing. A painting mode full colored image (the IR technique is good for getting a
produced a colorful collaborative image by accumulating silhouette, and possible some facial features, but results in
successive video frames of an energy body image, rather odd black and white images).
than just showing a single current frame. This effect shows
the traces of dance movements, a Muybridge-type result
with the addition of fine filigree textures and changing
colors. This technique gave the dancers agency and
creativity inside the virtual world, where they developed
methods for realizing amazingly complex collaborative
pictures from their gestures and movements.

Fig. 4 Traces of the head, hands and feet.

Time's Up: Love, Friendship, and

Transformation Across the Fourth Dimension
(2015) Multimedia Theatre
Fig. 3 Collective painting showing body traces. Times Up is the evening-length theatre piece that is the
culmination of three years of research and development
One section feature a more abstract movement of just with writer/director Kym Moore, presented by the Anti-
the extremities of the limbs by having a solo dancer wear Gravity Theatre Project. The author created the video
all black except for his hands, feet and head. With the design and original music for the production. The script
difference sensing technique, using a black background, and the technology evolved organically together through
any black clothing worn by a dancer becomes invisible numerous workshops, so that the depiction of supernatural
to the process. In this section the dancers paints a beautiful and spiritualized beings became an integral aspect of the
counterpoint of head, hands and feet as clean trails on the storytelling. A complete account of this large project is
screen, clearly differentiated by height, without the torso beyond the scope of this paper, but knowledge of the story
there to erase the previous images. Other sections used and set design is necessary for understanding the essential
black cloth to show various dancers magically appear or role video processing played in creating characters and
disappear on screen. action.
Similar techniques of transparency and invisibility
were used in an abstract connected-line environment. This The Story (Press release by Kym Moore)
technique translated the body and its movements into long Across moments of time - from medieval France to
lines, rendering ever-changing geometrical forms. Finally, present-day Harlem - and even in spaces outside of time,
a technique was discovered whereby a dancers image two souls keep brushing up against one another, intuitively
could be frozen on screen, but the dancer could continue to sensing their past and future lives together. They
interact with the frozen image, making the two come to life alternately fight to remember the intimacy of their souls
with connected webbing. This is an especially useful and yearn to forget in the hopes of skirting pain.
technique for a solo artist, whereby they can interact in a Throughout, conflicting cosmological forces look on and
duet or an ensemble of their previous selves. sometimes assume physical bodies, with which they thread
The video capturing process used a color camera to these two souls through discord and unity. What is
separate the dancers from a black background. The color revealed is a relationship existing outside of time and
image was essential for many of the effects. Unlike the space, in a place without culture, gender or politics.

3
 The two souls, Jason and Sienna, are dropped into A recent iteration of the piece adds two large
three different historical settings and their idealized projections to the side walls of the theatre, expanding the
relationship must encounter the assumptions, limitations, on-stage images into the entire theatre space to immerse
and prejudices of each time and place. The piece weaves the audience. In the opening scene in Medieval France, for
between a feudal village during the Cathar crusade in example, Sienna is accused of witchcraft and is burned at
medieval France where religious dissenters are burned at the stake. A burst of fire onstage grows in intensity,
the stake; a whites only diner in 1960s Selma during the spreading into the audience and engulfing the theatre. The
chaos of police attacks on civil-rights demonstrators; and a Selma riots, as another example, moves from the stage to
trendy soul food restaurant in contemporary Harlem where the walls of the theatre with archival footage and intense
Sienna (the owner) is reluctant to hire Jason because he is sound. Still images are also used to create scenography in
white. Gradually Jason and Siennas ethereal bodies each real world era, differentiated by image quality and
become unstuck in time and space, laying bare the arbitrary color: grainy black and white imagery for 1965 Selma;
yet very real devices of racism, politics and religion. This intense high-saturation colors for present day Harlem; and
true connection between their souls is manifested by gauzy and darkened images for Medieval France.
transformational live video processing of the actors The video projections support the story with three
bodies, projected throughout the space. Times Up seeks to main functions: establishing each of the three historical
reconsider "otherness and disrupt our habitual methods of eras; depicting a place outside of time; and showing the
perceiving one another that prevent us from connecting supernatural transformation of the actors bodies, the
with each other in meaningful and productive ways. primary topic here, of which a just a few examples follow.
The main body of the piece explores the relationship
of Jason and Sienna in three historical eras in the real Processing Examples
world, interspersed with sections where their ethereal Selma Dream
bodies become unstuck in time and space. These sections During a riot in Selma (1965), Jason has protected Sienna
rely on video processing to capture and alter the actors live (a stranger) by taking her into his parents whites-only
images, placing them inside a fantastical screen diner, where they hide in a storage room while the riots
environment. The processing furthers the conceptual goals rage. After some tense moments, they open up to each
of the piece by literally stripping away superficial other with revelatory conversation. Since it is too
differences. These poetic worlds are filled with intense dangerous for Sienna to walk home, they both fall asleep in
sound, music and color, with the actors onscreen bodies the storage room. They dream together in a place where
appearing to float, fracture and dissipate. Towards the end, they can be connected, happy, and free as children playing.
it is through these sequences that the couple remembers The video effect in this section shows a pulsating golden
and their spiritualized bodies become unstuck in time, able outline of their bodies, with webbing that stretches to
to go back and forth to traverse the previous real world connect them together whenever they get close to one
scenes. This realization not only transforms their past another. They play freely with this elastic connection. In
relationships but also opens them up to a whole new way this world, the processing shows what they could be to
of living in the current moment. each other without the constraints of politics, race and the
historical era in which they are caught. The spiritualized
Set Design, Projection Design body is displayed not only as a separate being, but also as a
The set for Times Up, designed by Sara Brown, features combined entity that connects and separates as the
two grey vinyl rear-projection screens in a V formation at relationship and dance unfolds. Just before dawn, they
the back of the stage, and a curved scrim on rollers at the return to their sleeping positions as one unified body, but
front of the stage (front projection on a split curtain). This they must confront the reality of their dangerous situation
allows for the actors to be cocooned in projections with a once they awaken.
more typical video backdrop in back, and the semi-
transparent scrim providing a curved projection surface
that the actors can be viewed through in front (with proper
lighting). Above the scrim is a long white banner, also used
for projection from the main front projector. Using
projection mapping, the front projector serves multiple
purposes: projecting onto the actors bodies, projecting
onto the banner with the scrim open, projecting only on the
scrim, and all of the above. This flexible projection design
offers numerous opportunities to enhance the story by
quickly establishing unique relationships between the
images, the actors and the audience.

4
 material (fire, air, water), and imbued with supernatural
powers. In one scene, she appears as a warrior, doing a
fierce dance to the sounds of pounding drums. As she stops
with her hands overhead, she leaves traces of her arms on
screen (using video averaging), depicting the multi-arm
goddess Kali, filled with the images of explosions on the
sun (taken from NASA public domain footage).

Fig. 5 Selma Dream with lines connecting actors.

Afterlife
After Sienna is burned at the stake in the Medieval France
scene, she discovers that she exist as a body of sparkling
light, floating in a star field. She is finally free and marvels
at her new ethereal body. She discovers that Jason is there,
too.

Fig. 7 Sun explosions filling the body of Kali

Another scene ends after a huge storm and a struggle,

with the clearing blue sky and light clouds finding their
way inside of her body, and her hand elongated to connect
to the others by extending across the entire back wall.

Fig. 6 Afterlife showing sparkling ghostly image.

Kali, The Goddess

A character encountered outside of time, The Goddess, Fig. 8 Cloud body reaching a long hand to others.
appears in several scenes, her body filled with elemental

5
Jason and Sienna Remember References
At some point towards the end of the performance, Jason 1. S. Dixon, Digital Performance (Cambridge, MA: MIT Press,
and Sienna remember all of their pasts together, and all 2007), 253 254.
time collapses. Two scenes show this transformation by
combining the spiritualized bodies, seen previously 2. M. Shiffrar, People Watching: Visual, Motor, and Social
isolated, composited with the scenography of the real Processes in the Perception of Human Movement, Cognitive
Science, 2 (2011): 68 78.
world scenes. The first such moment takes place in the
Selma diner, where Siennas colorful pulsing light body is
3. T. Winkler, ``Flying, Spinning, and Breaking Apart: Live
mixed with the previously seen black and white diner
Video Processing and the Altered Self,'' Proceedings for the
image. Jason walks through the wall to greet her in the International Symposium and Electronic Art (2010).
diner, also appearing as an energy body with saturated blue
and deep brown colors There, they embrace and they 4. M. Tsakiris, G. Prabhu, and P. Haggard, Having a body
remember everything. At that moment, they are able to versus moving your body: how agency structures body-
travel through time at will, and they jump to Harlem. There ownership, Conscious and Cognition, 15 (2006): 423 432.
they test the reality of the video backdrop and find that
by touching it, it turns to liquid, revealing its transient
nature and the falseness of the fixed world. Authors Biography
Todd Winkler is a composer and video artist, whose work over
the past 30 years has explored ways in which human actions can
influence sound and images produced by computers, in
multimedia dance and theatre productions, interactive video
installations, and concert pieces for computers and instruments.
He is the author of Composing Interactive Music (MIT Press), as
well as papers that bridge the fields of music, computer science,
video art, cognitive science and dance/theatre. Winkler's creative
projects have received international attention at festivals and
venues throughout the US, Australia, Europe, and Asia. He is the
recipient of awards from National Endowment for the Arts,
American Composers Forum, Irish Arts Council, Fulbright
Scholar Program, Meet the Composer, and National Endowment
for the Arts. He is a professor in the Brown University
Department of Music, where he is Director of the PhD program in
Multimedia and Electronic Music Experiments (MEME).

Fig. 9 A translucent spiritualized body in the Selma diner.

Conclusion
Video processing offers unlimited malleability and a
wildly expanded range of possible meanings of human
movement. The realtime and life-size aspects of these
works insures that participants feel an ownership and
participation in the expressive force of their newfound
supernatural bodies. A concerted effort has been made to
make these effects absolutely essential to each piece it is
the material from which they are made. Storytelling,
spiritual beings, connection to others, and magic are all
revealed in this one very personal perspective of this
readily available technology.