SKETCH: An Interface for Sketching 3D Scenes

Robert C. Zeleznik
Kenneth P. Herndon John F. Hughes
fbcz,kph,[email protected]
Brown University site of
the NSF Science and Technology Center
for Computer Graphics and Scientific Visualization
PO Box 1910, Providence, RI 02912

Abstract SKETCH is designed to bridge the gap between hand sketches

Sketching communicates ideas rapidly through approximate visual and computer-based modeling programs, combining some of the
images with low overhead (pencil and paper), no need for precision features of pencil-and-paper sketching and some of the features
or specialized knowledge, and ease of low-level correction and of CAD systems to provide a lightweight, gesture-based interface
revision. In contrast, most 3D computer modeling systems are to “approximate” 3D polyhedral modeling. Conceptually, our ap-
good at generating arbitrary views of precise 3D models and support proach is very similar to Landay and Myers’ use of sketching to
high-level editing and revision. The SKETCH application described support the early stages of conventional 2D interface design [16].
in this paper attempts to combine the advantages of each in order SKETCH uses a gestural mode of input in which all operations are
to create an environment for rapidly conceptualizing and editing available directly in the 3D scene through a three-button mouse. The
approximate 3D scenes. To achieve this, SKETCH uses simple user sketches the salient features of any of a variety of 3D primitives
non-photorealistic rendering and a purely gestural interface based and, following four simple placement rules, SKETCH instantiates
on simplified line drawings of primitives that allows all operations the corresponding 3D primitive in the 3D scene. SKETCH allows
to be specified within the 3D world. both geometry and the camera to be gesturally manipulated, and
uses an automatic grouping mechanism, similar to that described
Keywords: Interaction Techniques, 3D Modeling, Gestural Inter-
by Bukowski and Sequin [6], to make it easier to transform aggre-
face, Direct Manipulation, Sketching, Nonphotorealistic Rendering
gates of geometry. Since the set of geometric primitives is more
CR Categories: I.3.8. [Computer Graphics]: Applications; I.3.6. restricted than those in most CAD systems, the user approximates
[Computer Graphics]: Methodology and Techniques — Interaction complex shapes with aggregates of simpler primitives. Since we
Techniques know these conceptual models are approximations (often to only
partially formed mental images) SKETCH renders them with non-
1 Introduction photorealistic rendering techniques designed to help viewers see
what they want to see.
SKETCH targets the exploration and communication of 3D geo- We also imagine that SKETCH might be used as part of a story-
metric ideas. Traditionally, people have attacked conceptual design boarding system, for generating a series of scenes and camera views
with paper and pencil, not with computers, even though computer in planning a 3D animation.
models offer numerous advantages. The reasons for this include The accompanying videotape1 shows the features of SKETCH
the low overhead of a single-tool interface (pencil), the lack of spe- and indicates the utility of its simple approach in creating and editing
cial knowledge needed to draw, the ease with which many kinds of 3D models.
changes can be made, and the fact that precision is not required to
express an idea. Consider Ann sketching a table with an oval top 2 Related work
for Joe. Joe gets an immediate sense of the object, without Ann
having to indicate the precise locations of the legs, nor the exact A variety of efforts have been made to simplify the process of
shape of the top. By scribbling over what she has sketched, Ann can generating 3D models, including the “idea sketching” described by
make the top round or square or freeform without affecting Joe’s Akeo et al. [1]. Akeo allows users to scan real sketches into the
perception that the legs are attached to the top. (Imagine doing this computer where they are “marked-up” with perspective vanishing
in a typical CAD or drawing program.) Nevertheless, pencil and lines and 3D cross sections. The scanned data is then projected onto
paper are still imperfect. After many changes, the paper can become the 3D mark-up to complete the process.
cluttered. Drastic alterations such as showing the model from dif- Nearly all CAD applications employ some form of 2D sketching,
ferent viewpoints require new drawings, and collections of drawn although sketching is rarely used in 3D views. A notable exception
objects cannot be transformed as a unit. While computer models do is Artifice’s Design Workshop [2], which allows cubes, walls, and
not have these disadvantages, they are typically considerably more constructive solid geometry (CSG) operations to be constructed
difficult to create. directly in the 3D view. However, the overall style of interaction is
still menu-oriented and the set of primitives is small.
The considerable work done in the area of drawing interpretation,
surveyed by Wang and Grinstein [28], focuses solely on interpreting
an entire line drawing at once. In contrast, we attempt to provide
a complete interface for progressively conceptualizing 3D scenes
using aspects of drawing interpretation to recognize primitives from
1
The videotape can be obtained upon request from the authors.
a gesture stream. Viking [20] uses a constraint based approach to of strokes shown in Table 1.
derive 3D geometry from 2D sketches. In Viking, the user draws Each axis-aligned stroke is aligned with the projection of one
line segments, and the system automatically generates a number of the three principal axes of the world. We have also tried aligning
constraints which then must be satisfied in order to re-create a 3D strokes with the three principal axes of the surface over which
shape. The difficulty with these approaches is that even though they the gesture is drawn. In general, this latter approach seems more
are generally restricted to polygonal objects, they are often slow and effective, although it also presents some difficulties, especially for
difficult to implement. In addition, they are often intolerant of noisy curved surfaces and for gestures which span over different surfaces.
input and may either be unable to find a reasonable 3D solution, or Since we have not yet adequately handled these concerns in our
may find an unexpected solution. Branco et al. [5] combine drawing implementation, we will assume for the rest of the paper that all
interpretation with more traditional 3D modeling tools, like CSG lines are aligned with the world’s principal axes except those that
operators in order to simplify the interpretation process; however, are drawn with the “tearing” or freehand strokes.
their system is limited by a menu-oriented interaction style and does mouse action stroke
not consider constructing and editing full 3D scenes.
Deering [10], Sachs et al. [22], Galyean and Hughes [11], and click and release dot
Butterworth et al. [7] take a very different approach to constructing click and drag with- axis-aligned line: line follows axis
3D models that requires 3D input devices as the primary input mech- out delaying whose screen projection is most nearly
anism. A variety of systems have incorporated gesture recognition parallel to dragged-out segment
into their user interfaces, including Rubine [21], who uses gesture click and drag, then non-axis-aligned line
recognition in a 2D drawing program, but we know of no systems “tearing” motion to
that have extended the use of gesture recognition for 3D modeling. “rip” line from axis
We also use a variety of direct-manipulation interaction tech- click, pause, draw freehand curve
niques for transforming 3D objects that are related to the work of click with Shift key freehand curve drawn on surface of ob-
Snibbe et al. [25], and Strauss and Cary [27]. In addition, we also ex- pressed, draw jects in scene
ploit some very simple flexible constrained manipulation techniques
that are similar to those described by Bukowski and Sequin [6]. The Table 1: The five stroke classes.
latter automatically generates motion constraints for an object di- Interactors, the second type of gestural element, are made with
rectly from that object’s semantics. Therefore, for example, when the second mouse button. The two classes of interactors, a “click
a picture frame is dragged around a room, the frame’s back always and drag” and a “click,” have no visual representation.
remains flush with some wall in the room to avoid unnatural situa- In addition to gestural elements, SKETCH supports direct-
tions in which the picture frame might float in mid-air. Also, when manipulation of camera parameters with the third mouse button, as
a table is manipulated, all of the objects that are on top of the table outlined in Table 2. Third-button manipulations are not discussed
are automatically moved as well. further in this paper.
In our system, since we have less semantic information than mouse action camera manipulation
Bukowski, we have less opportunity to automatically generate ap-
propriate constraints, and therefore we occasionally require the user click and drag pan: point on film plane beneath mouse
to explicitly sketch constraints in addition to geometry. Our con- remains beneath mouse
straint techniques are fast, flexible and almost trivial to implement, click, pause, drag zoom/vertical pan: dragging horizon-
but they are not as powerful as the constrained manipulation de- tally zoom in/out towards clicked-on
scribed by Gleicher [12] or Sistare [24]. Although Gleicher exploits point, dragging vertically pan up/down
the fact that constraints always start off satisfied, thereby reducing click near window rotate: performs continuous XY con-
constraint satisfaction to constraint maintenance, he still must solve boundary, drag troller rotation about center of screen [8]
systems of equations during each manipulation which are often slow click on object “focus”: camera moves so that object is
and subject to numerical instability. Other approaches like Bier’s in center of view [18]
snap-dragging [4] are also related to our constrained manipulation, shift-click change rendering: cycles through avail-
although we never present the user with a set of constraint choices able rendering styles (see Section 5)
from which to select.
Lansdown and Schofield [17] and Salisbury et al. [23] provide Table 2: Gestures for camera manipulation.
interesting techniques for non-photorealistic rendering, although
none of these systems specifically targets interactive rendering. 4 The implementation
3 The interface SKETCH processes sequences of strokes and interactors to per-
All interaction with SKETCH is via a three-button mouse2 with oc- form various modeling functions with a finite-state machine. The
casional use of one modifier key on the keyboard, and a single ortho- mapping between gestural input and modeling functions is easy to
graphic window onto the 3D scene. The mouse is used to generate remember and gives the user a clear and direct correspondence.
gestures rather than to select operations from menus. Choosing an However, one of the principal difficulties in developing a good
operation like object creation, transformation or grouping is seam- gesture-based interface is managing the delicate tradeoff among
lessly integrated with the natural expression of intent. SKETCH gestures that are natural, gestures that are effective, and gestures that
infers intended tools by recognizing gestures — sequences of two are effective within a system that may already use similar gestures
types of gestural elements — in its input stream. for other functions. For superior gestures to evolve, this tradeoff
Strokes, the first type of gestural element, are pixel-tracks on the should continue to be explored especially with user studies.
film plane3, made with the first mouse button. There are five classes
4.1 Creating geometry
2
We think that a tablet/pen and an active LCD screen implemen- We believe gestures can be a natural interface for the fundamentally
tation might be even better. See Section 6. visual task of creating 3D geometry. The difficulty is choosing
3 the “right” gesture for each geometric primitive. In SKETCH, we
A plane perpendicular to the view direction and close enough
to the eyepoint not to interfere with the objects in the scene. define “primary” gestures for instantiating primitives as sequences
of strokes that correspond to important visual features — generally salient” vertex (the trivalent vertex for a cuboid, for example, or the
edges — in partial drawings of the primitives. (see Figure 2 for first vertex of the two parallel strokes that indicate a cylinder); a
an overview of all such gestures.) For instance, a drawing of three ray is traced through this vertex to hit a surface at some point in the
non-collinear line segments which meet at a point imply a corner, scene. The object is then instantiated so that the salient vertex is
based on our visual understanding of drawings [19]; consequently, placed at the intersected surface point (Figure 1).4
we interpret similar gestures composed of three line strokes as a The third placement rule exploits invariants of vertex junctions
cuboid construction operation. in line drawings, as described by Clowes [9]. However, our use of T
We also provide alternate construction gestures using non-edge junctions is related to the treatment given by Lamb and Bandopad-
strokes. For example, an object of revolution is sketched via its hay [15]. In particular, T-shaped junctions arise in line drawings
profile and axis, and cuboids can be created by sketching a single when a line indicating the edge of one surface, Estem, ends some-
edge and two “dimensioning segments” (perpendicular to the edge) where along a line segment indicating the edge of another surface,
that meet at a vertex lying anywhere along this edge. These alter- Ebar. These junctions generally signify that the surface associated
native gestures take their structure from the notions of generative with Estem is occluded by the surface associated with Ebar, al-
modeling [26]. though it does not necessarily indicate that the two surfaces meet.
SKETCH’s other primitives — cones, cylinders, spheres, objects In SKETCH, a similar notion exists when a straight line segment
of revolution, prisms, extrusions, ducts and superquadrics — have (except for connected polyline segments) of a gesture ends along
their own gestures. For most, SKETCH forces some aspect of the an edge of an object already in the scene (Figure 3). To uphold the
shapes to be axis-aligned, so that the gestures are easier to both intuition that such T junctions indicate the occlusion of one surface
draw and recognize. For example, to create a duct, the user strokes by another, SKETCH first places the gesture line into the 3D scene
a closed freehand curve for its cross section, and another freehand according to the previous two placement rules. Then, SKETCH
curve for its path of extrusion. However, an arbitrary 3D curve is sends a ray out along the gesture line (toward the T junction). If this
not uniquely determined by a single 2D projection, so SKETCH’s ray intersects the object that defined the bar of the T junction and
ducts must have extrusion paths that lie on an axis-aligned plane, the normal at the point of intersection is pointed in approximately
specified by a third gesture — an axis-aligned line stroke normal to the opposite direction of the ray, then the gesture edge is extended
the plane on which the path of extrusion should be projected. so that it will meet that surface.
The small number of primitive objects sometimes requires the If the ray does not intersect the surface, then the object defined by
user to build up geometry from simpler pieces, and precludes some the surface is translated along the viewing vector toward the viewer
complex objects — freeform surfaces and true 3D ducts,for example until its edge exactly meets the end of the gesture edge. If the end
— from being made at all. But in exchange for this, we believe that of the gesture edge is never met (because it was farther away from
our small set of primitives minimizes cognitive load on the user and the viewer), then neither the gesture, nor the existing objects are
makes gesture recognition and disambiguation easier. Future work, modified. We never translate objects away from the viewer as a
including user studies, should explore this tradeoff. result of T junctions; tests of this behavior on a variety of users
indicated that it was both unintuitive and undesirable.
4.2 Placing geometry The final rule determines whether the new geometry should be
Object creation requires placement of the object in the scene. We CSG-subtracted from the scene when added to it. If one or more
of the gesture strokes are drawn into an existing surface (i.e., the
base object placement on four rules: first, geometry is placed so that
its salient features project onto their corresponding gesture strokes in dot product of a stroke and the normal to the existing surface on
which it is drawn is negative), then the new piece of geometry is
the film plane; second,new objects are instantiated in contact with an
existing object when possible; third, certain invariants of junctions placed in the scene and subtracted from the existing object (Figure
2). CSG subtraction is recomputed each time the new geometry is
in line drawings [9] that indicate the placement or dimension of
geometry are exploited; and fourth, CSG subtraction is inferred manipulated. If the new geometry is moved out of the surface from
which it was subtracted, CSG subtraction is no longer recomputed.
automatically from the direction of gesture strokes (Figure 2).
This makes possible such constructions as the desk drawer in the
These easy-to-understand rules often generate good placement
Editing-Grouping-Copying section of the videotape.
choices; when they do not, users can edit the results. Furthermore,
the few users that the system has had so far have rapidly learned
to use the simple rules to their advantage, “tricking” the algorithm 4.3 Editing
into doing what they want. (This may be a consequence of their SKETCH supports multiple techniques for editing geometry. Some
programming background.) exploit paper and pencil editing techniques by recognizing editing
gestures composed of strokes (e.g.,oversketching and drawing shad-
ows). Others use gestures that contain an interactor to transform
shapes as a whole by translation or rotation.
Resizing. A common way to “resize” a surface with pencil and
paper is to sketch back and forth over its bounding lines until they
are of the right size. SKETCH recognizes a similar “oversketching”
gesture to reshape objects. If two approximately coincident lines
are drawn in opposite directions nearly parallel to an existing edge,
SKETCH infers a resizing operation (Figure 2). This sketching
Figure 1: A series of strokes is drawn in the film plane in red operation works for all primitives constructed of straight line seg-
(left). The salient vertex is projected into the scene thus defining ments, including cubes, cylinders and extrusions. Additionally, the
the placement of new geometry (green). Though this figure suggests two endpoints of an extrusion path can be attached to two objects
a perspective camera, we use a parallel projection in our application. 4
If the ray intersects no surface (possible because we use a finite
ground rectangle instead of an infinite ground plane), the object is
The first rule determines object placement except for translation placed in the plane perpendicular to the view direction that passes
along the view direction. This ambiguity is generally resolved by through the origin; this turns out in practice to be a reasonable
the second rule, implemented as follows: each gesture has a “most compromise.
in the scene; whenever either object moves, the extrusion will re- however, since this stroke is followed by an interactor, a translation
size to maintain the span. However, general reshaping of objects gesture is recognized and no geometry is created. Similarly, if the
defined by freehand curves is more difficult and not yet fully imple- user drags perpendicular to the constraint axis instead of parallel to
mented. We are currently adapting Baudel’s mark-based interaction it, the gesture is interpreted as a single axis rotation. (This gesture
paradigm [3] for use in reshaping 3D objects. roughly corresponds to the motion one would use in the real world
Sketching shadows. Shadows are an important cue for determin- to rotate an object about an axis.)
ing the depth of an object in a scene [29]. In SKETCH, we exploit To translate in one of the three axis-aligned planes, two perpen-
this relationship by allowing users to edit an object’s position by dicular lines must be stroked on an object. The directions of these
drawing its shadow. The gesture for this is first to stroke a dot over two lines determine the plane in which the object is constrained to
an object, and then to stroke its approximate shadow — a set of translate. If the two perpendicular lines are drawn over a different
impressionistic line strokes — on another surface using the Shift object from the one manipulated, they are interpreted as a contact
modifier key.5 The dot indicates which object is being shadowed, constraint (although non-intuitive, this gesture is effective in prac-
and the displacement of the shadow from the object determines the tice). This forces the manipulated object to move so that it is always
new position for the object (as if there were a directional light source in contact with some surface in the scene (but not necessarily the
directed opposite to the normal of the surface on which the shadow object over which the gesture was drawn) while tracking the cursor.
is drawn). The resulting shadow is also “interactive” and can be Finally, a dot stroke drawn on an object before using an interactor is
manipulated as described by Herndon et al. [13]. interpreted as the viewing vector; the object will be constrained to
Transforming. Objects can be transformed as a unit by using translate along this vector. This constraint is particularly useful for
a “click-and-drag” interactor (with the second mouse button): the fine-tuning the placement of an object if SKETCH has placed it at
click determines the object to manipulate, and the drag determines the “wrong” depth; however, since we use an orthographic view that
the amount and direction of the manipulation. By default, objects does not automatically generate shadows, feedback for this motion
are constrained to translate, while following the cursor, along the is limited to seeing the object penetrate other objects in the scene.
locally planar surface on which they were created. However, this We believe that a rendering mode in which shadows were automat-
motion can be further constrained or can be converted to a rotational ically generated for all objects would be beneficial, although we
motion. have not implemented such a mode because of the expected com-
It is important to keep in mind that our interaction constraints are putational overhead. We did, however, mock up a rendering mode
all very simple. In SKETCH, instead of using a constraint solver in which just the manipulated object automatically cast its shadow
capable of handling a wide variety of constraints, we associate on the rest of the scene. People in our group generally found the
an interaction handler with each geometric object. This handler shadow helpful, but were slightly disturbed that none of the other
contains constraint information including which plane or axis an objects cast shadows.
object is constrained to translate along, or which axis an object In each case, the manipulation constraint, once established, is
is constrained to rotate about. Then when the user manipulates maintained during subsequent interactions until a new constraint is
an object, all of the mouse motion data is channeled through that drawn for that object. The only exception is that single axis rotation
object’s handler which converts the 2D mouse data into constrained and single axis translation constraints can both be active at the same
3D transformations. To define which of our simple constraints is time; depending on how the user gestures — either mostly parallel
active, we require that the user explicitly specify the constraint with to the translation axis or mostly perpendicular to the rotation axis
a gesture. Whenever a new constraint is specified for an object it — a translation or rotation operation, respectively, is chosen.
will persist until another constraint is specified for that object. Each Finally, objects are removed from the scene by clicking on them
new constraint for an object overwrites any previous constraint on with an interactor gesture. In early versions of SKETCH we used an
the object. apparently more natural gesture to remove objects: the user “tossed”
The advantages of such a simple constraint system are that it is them away by translating them with a quick throwing motion, as
robust, fast, and easy to understand. A more sophisticated constraint one might brush crumbs from a table. We found, however, that this
engine would allow a greater variety of constrained manipulation, gesture presented a complication: it was too easy to toss out the
but it would also require that the user be aware of which constraints wrong object, especially if its screen size were small.
were active and how each constraint worked. It would also re-
quire additional gestures so that the user could specify these other 4.4 Grouping and copying
constraints.
Systems such as Kurlander and Feiner’s [14] attempt to infer con- By default, objects are automatically unidirectionally grouped with
straints from multiple drawings, but this approach has the drawback the surface on which they were created, an idea borrowed from
that multiple valid configurations of the system need to be made Bukowski and Sequin [6], generally resulting in hierarchical scenes.
in order to define a constraint. Such approaches may also infer Each geometric object in SKETCH contains a list of objects that
constraints that the user never intended, or may be too limited to be are grouped to it. Whenever an object is transformed, that object
able to infer constraints that a user wants. will also apply the same transformation to all other objects that are
Constrained transformation. The gestures for constraining grouped to it; each grouped object will in turn transform all objects
object transformations to single-axis translation or rotation, or to grouped to itself. Cycles can occur and are handled by allowing
plane-aligned translation are composed of a series of strokes that each object to move only once for each mouse motion event.
define the constraint, followed by a “click-and-drag” interactor to This kind of hierarchical scene is generally easier to manipulate
perform the actual manipulation (Figure 2). To constrain the motion than a scene without groupings since the grouping behavior typi-
of an object to an axis-aligned axis, the user first strokes a con- cally corresponds to both expected and useful relationships among
straint axis, then translates the object with an interactor by clicking objects. For example, objects drawn on top of a table move when-
and dragging parallel to the constraint axis. The constraint axis is ever the table is manipulated, but when the objects are manipulated,
stroked just as if a new piece of geometry were being constructed; the table does not follow. Grouping also applies to non-vertical
relationships, so a picture frame drawn on a wall is grouped with
5
Recall from Table 1 that Shift-modified strokes normally the wall.
produce lines drawn on the surface of objects without special In some cases, grouping is bidirectional. The choice of bi-
interpretation. directional and uni-directional grouping is guided by what we be-
lieve is an inherent difference in the way people interpret relation- “Sketchy” rendering styles are essential because they often enable
ships between certain horizontal versus vertical drawing elements. users to focus on the essence of a problem rather than unimportant
When an object is drawn that extends horizontally between two details. Non-photorealistic rendering draws a user’s attention away
surfaces, like a rung on a ladder, the two surfaces that are spanned from imperfections in the approximate scenes she creates while
are grouped bidirectionally, so that if one rail of the ladder moves also increasing the scene’s apparent complexity and ambiguity. By
so does the other. Although the rung moves whenever either rail is making scenes more ambiguous, users can get beyond SKETCH’s
manipulated, the rails do not move when the rung is manipulated. approximate polygonal models to see what they want to see. This is
The grouping relationship for objects that span vertically, however, an important concept: we do not believe that sketchy rendering adds
establishes only one-way relationships: the topmost object is uni- noise to a signal; rather we believe that it conveys the very wide
directionally grouped to the bottommost object and the spanning tolerance in the user’s initial estimates of shape. The user is saying
object is similarly grouped to the topmost object. Thus, a table “I want a box about this long by about that high and about that
leg that spans between a floor and a table top causes the top to be deep.” Showing a picture of a box with exactly those dimensions
grouped to the floor and the leg to be grouped to the top, but the floor is misleading, because it hides the important information that the
is not grouped to the top. We only exploit the difference between dimensions are not yet completely determined.
horizontal and vertical elements to distinguish these two grouping A line drawing effect is achieved by rendering all polygonal ob-
relationships. However we believe it is important to study with user jects completely white, and then rendering the outlines and promi-
tests how effective this automatic grouping approach actually is, and nent edges of the scene geometry with multiple deliberately jittered
perhaps to determine as well if there are other ways that we might lines; the z -buffer therefore handles hidden-line removal. A char-
be able to exploit the differences between vertical and horizontal coal effect is created by mapping colors to grayscale and increasing
elements. the ambient light in the scene; a watercolor effect that washes out
Unlike Bukowski, object grouping is not automatically recom- colors is created by increasing the scene’s ambient light. There are a
puted as objects are moved around the scene. Therefore, if an object number of other techniques that we would like to explore, including
is moved away from a surface, it will still be grouped with the sur- pen and ink style textures, and drawing hidden edges with dashed
face. Grouping relationships are recomputed only when objects lines.
are moved using the contact constraint mentioned in Section 4.3 Objects are assigned a default random color when they are created
— the moved object is grouped to the surface it now contacts and to help differentiate them from the scenery. We can also copy colors
ungrouped from any surface it no longer contacts. We have found from one object to another. By just placing the cursor on top of
this approach to automatic grouping to be simple and effective, al- one object and pressing the Shift modifier, we can “pick up” that
though in some environments, Bukowski’s approach may be more object’s color. Then, we can “drop” this color on another object
appropriate. by placing the cursor over it and releasing the modifier. We can
Lassoing groups. SKETCH also allows the user to explicitly also explicitly specify colors or textures for objects. In our present
establish groups by stroking a lasso around them (Figure 2).6 De- implementation, we do this by placing the cursor over the object
ciding which objects are considered inside the lasso is based on the and typing the name of the color or texture. Although this interface
heuristic that the geometric center and all of the visible corners of requires the keyboard, it is consistent with SKETCH’s interface
the object must be inside the lasso; shapes like spheres must be com- philosophy of not making users search through a 2D interface for
pletely contained in the lasso. SKETCH currently approximates this tools to create particular effects. In the future, we expect that voice
heuristic by first projecting an object’s geometric center and all of recognition, perhaps in conjunction with gesturing, will be a more
its crease vertices (where there is a discontinuity in the derivative of effective way to establish surface properties for objects (and perhaps
the surface) and silhouette vertices into the film plane, then testing other operations as well).
whether all these projected points are contained within the lasso.
Currently, no test is made for whether objects are occluded or not; 6 Future Work
future work should address the ambiguities that arise in using the
lasso operation. We regard SKETCH as a proof-of-concept application, but it has
All lassoed objects are copied if they are manipulated while the many flaws. Many of the gestures were based on an ad hoc trial
Shift modifier is used. Lassoed objects can be scaled by dragging and error approach, and some of the gestures still do not satisfy
the lasso itself. us. For example, the pause in the freehand curve gesture rapidly
Repeating gestures. A different form of copying is used when becomes annoying in practice — the user wants to do something,
a user wants to repeat the last geometry-creation operation. After and is forced to wait. Possible solutions of course include using
SKETCH recognizes gesture strokes corresponding to a geometric more modifier keys, although we would rather find a solution that
construction, it creates and places the new geometry in the scene, preserves the simplicity of the interface.
but does not erase the gesture strokes. Thus, the user can click on SKETCH is based on an interface that is stretched to its limits.
any of these strokes (using button 1) in order to “drag and drop” We expect that adding just a few more gestures will make the system
(re-execute) them elsewhere. Gesture strokes are erased when a hard to learn and hard to use. We’d like to perform user studies on
new gesture is started or when any object is manipulated. These ease of use, ease of learning, and expressive power for novice users
techniques are shown in the videotape. as a function of the number of gestures. We’re also interested in try-
ing to determine to what extent artistic and spatial abilities influence
users’ preference for sketching over other modeling interfaces.
5 Rendering We have begun to implement a tablet-based version of SKETCH.
SKETCH renders orthographic views of 3D scenes using a con- The current generation of tablet pens include pressure sensitivity
ventional z -buffer. Color Plates I-VI show some of the rendering in addition to a single finger-controlled button, and one "eraser-
techniques that SKETCH supports. like" button. In order to develop an equivalent interface for the
tablet, we simply need to treat a specific pressure level as a button
6
The lasso gesture is similar to the sphere gesture. We differen- click to achieve the equivalent of three buttons. Therefore, the
tiate between them by requiring that the sphere gesture be followed button 1 drawing interactions described for the mouse are done by
by a dot stroke, whereas the lasso is simply a free-hand closed curve simply pressing hard enough with the penpoint of the tablet pen.
stroke followed by a manipulation gesture. To achieve the button 2 operations of the mouse, the user simply
presses the finger controlled button on the tablet pen. Finally, [11] T. Galyean and J. Hughes. Sculpting: An interactive volumet-
to effect camera motion, the user turns the pen over and uses its ric modeling technique. Computer Graphics (SIGGRAPH ’91
"eraser" to manipulate the camera. Our initial efforts with a tablet Proceedings), 25(4):267–274, July 1991.
based interface lead us to believe that a tablet based system could be [12] M. Gleicher. Integrating constraints and direct manipula-
far more effective than a mouse based system, especially if pressure tion. Computer Graphics (1992 Symposium on Interactive
sensitivity is cleverly exploited. 3D Graphics), 25(2):171–174, March 1992.
SKETCH is a tool for initial design — the “doodling” stage,
where things are deliberately imprecise. But initial design work [13] K.P. Herndon, R.C. Zeleznik, D.C. Robbins, D.B. Conner,
should not be cast away, and we are examining ways to export S.S. Snibbe, and A.van Dam. Interactive shadows. UIST ’92
models from SKETCH to modelers that support more precise edit- Proceedings, pages 1–6, Nov. 1992.
ing, so that the sketch can be moved towards a final design. Since [14] D. Kurlander and S. Feiner. Inferring constraints from multiple
subsequent analysis and design often requires re-thinking of some snapshots. ACM Transactions on Graphics, 12(4):277–304,
initial choices, we are also interested in the far more difficult task Oct. 1993.
of re-importing refined models into SKETCH and then re-editing
them, without losing the high-precision information in the models [15] D. Lamb and A. Bandopadhay. Interpreting a 3D object from a
except in the newly-sketched areas. rough 2D line drawing. Visualization ’90 Proceedings, pages
The scenes shown here and in the video are relatively simple. 59–66, 1990.
Will sketching still work in a complex or cluttered environments? [16] J.A. Landay and B.A. Myers. Interactive sketching for the
We do not yet have enough experience to know. Perhaps gestures early stages of user interface design. Proceedings of CHI’95,
to indicate an “area of interest,” which cause the remainder of the pages 43–50, 1995.
scene to become muted and un-touchable might help. [17] J. Lansdown and S. Schofield. Expressive rendering: A review
The tradeoffs in gesture design described in Section 4 must be of nonphotorealistic techniques. IEEE Computer Graphics &
further explored, especially with user-studies. Applications, pages 29–37, May 1995.
7 Acknowledgments [18] J.D. Mackinlay, S.K. Card, and G.G. Robertson. Rapid con-
trolled movement through a virtual 3d workspace. In Proceed-
Thanks to Dan Robbins, Tim Miller, and Lee Markosian for many ings of the 1986 Workshop on Interactive 3D Graphics, pages
helpful discussions. Thanks also to Andries van Dam and the Graph- 171–176, October 1986.
ics Group, as well as our sponsors: grants from NASA, NSF, Mi- [19] G. Magnan. Using technical art: An industry guide. John
crosoft, Sun and Taco; hardware from SGI, HP and Sun.
Wiley and Sons, Inc., 1970.
References [20] D. Pugh. Designing solid objects using interactive sketch
interpretation. Computer Graphics (1992 Symposium on In-
[1] M. Akeo, H. Hashimoto, T. Kobayashi, and T. Shibusawa. teractive 3D Graphics), 25(2):117–126, Mar. 1992.
Computer graphics system for reproducing three-dimensional
[21] D. Rubine. Specifying gestures by example. Computer Graph-
shape from idea sketch. Eurographics ’94 Proceedings,
ics (SIGGRAPH ’91 Proceedings), 25(4):329–337, July 1991.
13(3):477–488, 1994.
[22] E. Sachs, A. Roberts, and D. Stoops. 3-draw: A tool for de-
[2] Artifice, Inc. Design Workshop. Macintosh application.
signing 3D shapes. IEEE Computer Graphics & Applications,
[3] T. Baudel. A mark-based interaction paradigm for free-hand pages 18–25, Nov. 1991.
drawing. UIST ’94 Proceedings, pages 185–192, Nov. 1994.
[23] M. Salisbury, S. Anderson, R. Barzel, and D. Salesin. In-
[4] E.A. Bier. Snap-dragging in three dimensions. Computer teractive pen–and–ink illustration. Computer Graphics (SIG-
Graphics (1990 Symposium on Interactive 3D Graphics), GRAPH ’94 Proceedings), pages 101–108, July 1994.
24(2):193–204, Mar. 1990.
[24] S. Sistare. Graphical interaction techniques in constraint-
[5] V. Branco, A. Costa, and F.N. Ferriera. Sketching 3D models based geometric modeling. Proceedings of Graphics Interface
with 2D interaction devices. Eurographics ’94 Proceedings, ’91, pages 85–92, June 1991.
13(3):489–502, 1994.
[25] S.S. Snibbe, K.P. Herndon, D.C. Robbins, D.B. Conner, and
[6] R. Bukowski and C. Séquin. Object associations: A sim- A. van Dam. Using deformations to explore 3D widget de-
ple and practical approach to virtual 3D manipulation. Com- sign. Computer Graphics (SIGGRAPH ’92 Proceedings),
puter Graphics (1995 Symposium on Interactive 3D Graph- 26(2):351–352, July 1992.
ics), pages 131–138, Apr. 1995.
[26] J.M. Snyder and J.T. Kajiya. Generative modeling: A sym-
[7] J. Butterworth, A. Davidson, S. Hench, and T.M. Olano. bolic system for geometric modeling. Computer Graphics
3DM: A three dimensional modeler using a head-mounted (SIGGRAPH ’92 Proceedings), 26(2):369–378, July 1992.
display. Computer Graphics (1992 Symposium on Interactive
[27] P. Strauss and R. Carey. An object-oriented 3D graphics
3D Graphics), 25(2):135–138, Mar. 1992.
toolkit. Computer Graphics (SIGGRAPH ’92 Proceedings),
[8] M. Chen, S. Joy Mountford, and Abigail Sellen. A study in 26(2):341–349, July 1992.
interactive 3-D rotation using 2-D control devices. Computer
[28] W. Wang and G. Grinstein. A survey of 3D solid reconstruction
Graphics (SIGGRAPH ’88 Proceedings), 22(4):121–129, Au-
from 2D projection line drawings. Computer Graphics Forum,
gust 1988.
12(2):137–158, June 1993.
[9] M. Clowes. On seeing things. Artificial Intelligence, (2):79–
[29] L.R. Wanger, J.A. Ferwerda, and D.P. Greenberg. Perceiv-
116, 1971. North-Holland.
ing spatial relationships in computer-generated images. IEEE
[10] M. Deering. Holosketch: A virtual reality sketch- Computer Graphics and Applications, 12(3):44–58, May
ing/animation tool. ACM Transactions on Computer-Human 1992.
Interaction, 2(3):220–238, 1995.