Comics-Like Motion Depiction from Stereo

Danijela Marković and Margrit Gelautz

Institute for Software Technology and Interactive Systems, Vienna University of Technology
Favoritenstrasse 9-11/188/2, A-1040 Vienna, Austria
e-mail: {markovic, gelautz}@ims.tuwien.ac.at

ABSTRACT
In this paper, we present an algorithm to depict motion in comics-like form, with an artistic drawn-like representation of the
scene, from a stereo image sequence. The input to the algorithm is a natural scene, along with a user-defined set of parameters
that define the tone and stylistic properties of the image to be produced. The algorithm uses a dense disparity map, computed
from the input stereo video, to preserve the perspective perception of the stylized image by drawing each stroke in a direction
determined by the stereo derived disparity layers. To outline important features in the image, we utilize the contour edges
provided by the Edge Combination algorithm. In the next step, we detect motion of the objects in the scene by tracking points
that are close to the dominant edges of the Edge Combination image. The extracted dominant structure is further used to obtain
a larger variety of styles for visualizing the motion trajectories. The output of the algorithm is a drawn-like form of the original
scene with the motion highlighted in imitation of comics produced by hand.
Keywords: Real scene, stereo, Edge Combination, image-based rendering, motion depiction, tracking.

1 INTRODUCTION skills. Contrary to this, comic books are usually created

by skilled and creative artists who use a style of draw-
The heart of comics lies in the space between the ing unique for this type of art.
panels where the reader’s imagination makes still Comic books can be compared to silent movies, hav-
pictures come alive! [McC00] ing dialogs in a written form and telling a story through
sequences of images with a very important difference:
Although there is an ongoing discussion about the comics use one image to present a sequence of frames.
historical roots of comics and their artistic contribu- The form to visually depict the motion in just one im-
tion, reputed to be the first comic strip magazine is age has been a big challenge for artists through history
“Ally Sloper’s Half Holiday” published in 1884. It was and resulted in different methods like: dynamic bal-
one of the most famous and most popular of all Victo- ance, multiple stroboscopic images, affine shear, pho-
rian comics, featuring a regular character, Ally Sloper. tographic blur, and action lines [Cut02]. Today, picto-
However, the great boom that prompted the beginning rial description of the motion is particularly interesting
of comics as an ongoing popular art form was Richard for image stylization and initiated the development of a
Fenton Outcault’s cartoon series “Hogan’s Alley” for number of systems [MSS99, BE01, CRH03, KE05].
Joseph Pulitzer’s New York World in 1895. The Yel- In this work we concentrate on a comic books’ style
low Kid, as he later came to be known for his yellow of motion expression, where few lines suggest an action
nightshirt, became the first comic character to serve as in the scene through motion lines and multiple contours.
a marketing tool for the sale of newspapers. This set Similar to our work, Masuch et al. [MSS99] depict mo-
the bases for a new kind of art with the adventures of tion, but different to our system they use a 3D scene
superheroes and monsters in stories written in a power- as input, having access to the 3D geometry and pre-
ful language of visual symbols. [Sab01, Har99] cise motion information of objects. The overall goal of
Comic books lost a lot of their popularity with the our system is to enable a user to generate effective and
development and expansion of 3D animation and new attractive illustrations of dynamic natural scene record-
media. Today a great variety of commercial applica- ings from stereo as input.
tions give to the user the ability to create interesting
artwork even without the requirement of good drawing 2 ALGORITHM
Permission to make digital or hard copies of all or part of this Images of a natural, real scene usually contain texture,
work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or commercial noise, or other features such as shadowing that set the
advantage and that copies bear this notice and the full citation on the accurate extraction of object boundaries to a complex
first page. To copy otherwise, or republish, to post on servers or to task. A simple approach of using only the output of
redistribute to lists, requires prior specific permission and/or a fee.
an edge detector for most real scene’s images induces
WSCG 2006 conference proceedings, ISBN 80-86943-03-8 the problem of distinguishing object boundaries from
WSCG’2006, January 30 – February 3, 2006
Plzen, Czech Republic. the other features edges. To overcome this problem,
Copyright UNION Agency – Science Press one of the most valuable sources of information about
 keep the aspect of real.
To present an image in a drawn-like form, we make
use of our previous work [MG05]. The computed stroke
density varies according to the tone of the intensity im-
age region, from dense strokes in dark to no strokes
in light areas. To better convey the shape of objects
in natural scene images, which usually suffer from a
bad contrast, we first perform a local image contrast en-
hancement, which is driven by the color segmentation
result of the original image. This produces an image
with newly revealed details, as presented in Fig. 2. We
then draw each stroke in a direction determined by the
stereo derived disparity layers with the stroke density
and distribution derived from a Poisson disc distribu-
Figure 1: Example 1 from the “Alan Ford” comic book
tion. The disparity maps that we use in this work are re-
series. c Copyright 2005 by Max Bunker Press. Used
sults of the graph-cut based stereo-matching algorithm
by permission.
[BG05], which represents disparity by a set of planar
layers. Such a representation of enhanced images using
the objects present in the scene is the depth informa- disparity to orient strokes results in a good distinction
tion. To obtain this information, from the images of the and shape depiction of objects in the scene.
real scene, we use the depth-from-stereo approach. The
output of the stereo matching process delivers a dis-
parity map, which is directly related to the depth. In
most cases, however, the object contours cannot be per-
fectly recovered from the disparity map alone, due to
matching errors along the depth discontinuities. To sup-
press errors produced in the matching process we sug-
gest the Edge Combination algorithm [MSG05], which
uses disparity edges to identify corresponding edges in
the original edge image to obtain more accurate infor-
mation of the object boundaries. The result of this pro-
cess, dominant edges that highlight the principal struc-
ture of the scene, is further used to stylize the image
in a drawn-like form and to depict the motion through
motion lines and multiple contours. Generally, our ap-
proach consists of two major steps: image representa-
tion in a drawn-like form and comics-like depiction of
motion. These steps are further explained in the follow- (a)
ing sections.

2.1 Drawn-like image representation

As we previously mentioned, comic books are usually
created by skilled and creative artists who use a lan-
guage of visual symbols for drawing, unique for this
type of art. A very strong distinctive feature of comics
is a tight connection between a story and a style of
drawing: this results in funny stories having charac-
ters drawn like caricatures or “saving the world” stories
with superheroes drawn in a way to show the power.
Our system does not intend to replace an artist, although
the result can be used as final artwork, but to assist an
artist in technical aspects and support him in routine (b)
tasks. It can also be very useful in a learning process Figure 2: (a) Original image, (b) enhanced original im-
because the final output preserves the position, propor- age, prepared for further stylization.
tion and shape of objects. Since we are dealing with
recordings of a natural scene, the drawn representations
2.2 Tracking the motion
For tracking the motion of objects in the scene we em-
ployed the KLT tracker [ST94]. To depict motion in
comics-like style, we need only information of distinct
discontinuities in motion, which usually go along with
depth discontinuities. By following this idea, among
the point set selected automatically by the KLT tracker,
we track only those points found to be close to edges in
the Edge Combination image through all the frames of
the user selected sequence.

2.3 Depicting motion

In comics, motion is usually represented using either
motion lines, multiple contours or their combination, Figure 3: Example 2 from the “Alan Ford” comic book
which results in a variety of styles. An example of mo- series. c Copyright 2005 by Max Bunker Press. Used
tion depiction in a real comic through motion lines is by permission.
given in Fig. 1, and using both motion lines and multi-
ple contours is shown in Fig. 3. 3 EXPERIMENTAL RESULTS
Motion lines: also known as action lines or speed
Example results of motion depiction using multiple
lines, is the most common style of representing mo-
contours, motion lines and their combination are given
tion in comic books. Depicting motion in comics has
in Figs. 5 - 9. The initial stereo image pair of the image
a meaningful purpose in story telling and strongly de-
sequence, in epipolar geometry, is presented in Fig. 5
pends on an artist’s imagination. Usually, motion for
(a) and (b). Their resulting disparity image computed
each object is presented through one path in several mo-
by employing the stereo matching algorithm presented
tion lines (see Fig. 1 and Fig. 3). In a natural scene, one
in [BG05] is given in Fig. 5 (c), along with the labeled
or more objects can be sources of motion. Represent-
disparity layers shown in Fig. 5 (e). The color segmen-
ing just the motion of the object with, for instance, the
tation result, see Fig. 5 (f), is used in order to locally
longest path would be a great limitation. As a remedy,
enhance the contrast to better convey the shape of the
we give control to the user to select a desired object and
natural scene before further processing. Fig. 5 (h) illus-
type of motion to depict by selecting one of the object’s
trates the output of our algorithm in the style of motion
tracked points. Fig. 4 (b) shows a tracked point and its
depiction using motion lines together with the drawn-
path selected from the tracking points in Fig. 4 (a). To
like representation of the natural scene from Fig. 5 (g).
preserve the usual simple motion illustration in comics,
Another example of the same motion depiction style is
we fit primitives (i.e. circle, ellipse, straight line) to the
given in Fig. 9 (c).
path of the tracked point to get the final motion path
Two sets of images, presented in Fig. 6 and Fig. 7,
and to have a possibility to extend its length. An exam-
demonstrate various styles of motion depiction through
ple of fitting a primitive, in this case a circle, is shown
multiple contours and multiple contours joined with
in Fig. 4 (c). By further selecting points on the con-
motion lines, respectively. The corresponding frames
tour in the Edge Combination image, Fig. 4 (d), close
from the original sequence are shown in Fig. 8. The
to the tracked point, the user passes the same shape of
multiple contours styles that we illustrate in these fig-
the motion path to these points, and in this way more
ures are obtained by changing the transparency of con-
motion lines can be created from a tracked path. For a
tours through frames, using a complete or a part of a
more natural, hand-drawn look, the number of points in
moving object’s contour and their combination.
the final motion path, for each motion line, is reduced
by removing a small random number of points at the
beginning of the path. 4 SUMMARY AND OUTLOOK
Multiple contours: is another, very important, style In this paper we have presented an algorithm to depict
of motion representation which uses a complete or a motion in comics-like form, with an artistic drawn-like
part of a moving object’s contour. To imitate this style representation of the scene, from a stereo image
we use Edge Combination images extracted from the sequence. We have devised several motion depiction
frames in a regular interval and draw them on the final styles using motion lines and/or multiple contours.
result. Regularity in frame selection gives a good im- We have shown how silhouettes extracted by the Edge
pression of the object’s motion speed. The variety of Combination algorithm can serve in motion depiction
styles can be even larger by changing the transparency and also to portray the impression of the change of an
of contours through frames. object’s speed.
 In future steps, we intend to experiment with a
spline representation of the motion path. This should
give more flexibility and the possibility of depicting
more complex motions, but it may also raise problems
in motion path length extension. We also plan to
experiment with other stylistic properties of comics
and explore additional techniques of motion illustration
in still images.

ACKNOWLEDGEMENTS
We wish to thank Michael Bleyer for providing the dis-
parity maps. This research has been funded by the Aus-
trian Science Fund (FWF) (project no. P15663) and by
(a)
the Austrian Federal Ministry for Education, Science,
and Culture, as well as the European Social Fund (ESF)
under grant 31.963/46-VII/9/2002.

REFERENCES
[BE01] Gabriel J. Brostow and Irfan Essa. Image-based motion
blur for stop motion animation. In Proceedings of SIG-
GRAPH ’01, pages 561–566, August 2001.
[BG05] Michael Bleyer and Margrit Gelautz. Graph-based surface
reconstruction from stereo pairs using image segmenta-
tion. In Proceedings of SPIE, volume 5665, pages 288–
299, 2005.
[CRH03] John P. Collomosse, David Rowntree, and Peter M. Hall.
Cartoon-style rendering of motion from video. In Pro-
ceedings of Video, Vision and Graphics (VVG), pages
(b)
117–124, July 2003.
[Cut02] James E. Cutting. Representing motion in a static image:
constraints and parallels in art, science, and popular cul-
ture. Perception, 31(10):1165–1193, 2002.
[Har99] Robert C. Harvey. Children of the Yellow Kid: The Evolu-
tion of the American Comic Strip. University of Washing-
ton Press, 1999.
[KE05] Byungmoon Kim and Irfan Essa. Video-based nonphoto-
realistic and expressive illustration of motion. In Proceed-
ings of Computer Graphics International (CGI ’05), pages
32–35, June 2005.
[McC00] Scott McCloud. Reinventing Comics: How Imagination
and Technology Are Revolutionizing an Art Form. Harper
Paperbacks, 2000. (c)
[MG05] Danijela Marković and Margrit Gelautz. Drawing the
real. In Proceedings of GRAPHITE’05, pages 237–243,
November 2005.
[MSG05] Danijela Marković, Efstathios Stavrakis, and Margrit
Gelautz. Parameterized sketches from stereo images. In
Proceedings of SPIE, pages 783–791, January 2005.
[MSS99] Maic Masuch, Stefan Schlechtweg, and Ronny Schulz.
Speedlines: depicting motion in motionless pictures. In
Proceedings of SIGGRAPH ’99: ACM SIGGRAPH 99
Conference abstracts and applications, page 277, August
1999.
[Sab01] Roger Sabin. Comics, Comix & Graphic Novels: A His-
tory Of Comic Art. Phaidon Press, 2001.
[ST94] Jianbo Shi and Carlo Tomasi. Good features to track.
In Proceedings of IEEE Conference on Computer Vision (d)
and Pattern Recognition (CVPR’94), pages 593–600, June Figure 4: (a) Points to track, (b) motion path of the
1994. selected point, (c) fitted circle to the motion path, (d)
selected new point on the contour in the Edge Combi-
nation image and its path.
 (a) (b)

(c) (d)

(e) (f)

(g)

(h)
Figure 5: (a) Left camera image, (b) right camera im-
age, (c) disparity image, (d) Edge Combination image,
Figure 6: Results of motion depiction through multiple
(e) layers image, (f) color segmented image, (g) drawn
contours.
image, (h) drawn image with depicted motion.
 Figure 8: Intermediate frames (from top left to bottom)
used for computation of the contours in Figs. 6 and 7.

(a) (b)

(c)
Figure 7: Results of motion depiction through multiple
Figure 9: (a) Left camera image, (b) right camera im-
contours and motion lines.
age, (c) drawn image with depicted motion.