Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

Spatial Interaction with Haptic Holograms

Wendy Plesniak and Ravikanth Pappu

Spatial Imaging Group, MIT Media Lab
20 Ames Street E15-416
Cambridge, MA 02139. USA.
{wjp, pappu}@media.mit.edu

Abstract niques, holography is certainly the most prominent among

We present a comprehensive report of our work in the them.
design and implementation of systems that combine holog-
raphy with haptic feedback. Two separate holo-haptic sys- In both the systems described in this paper, we employ
tems are described. First, we describe a static system holography to generate visual spatial images and combine
wherein a user can see a free-standing static holographic them with co-located force display. First, we describe a
image and inspect its shape using a force-feedback stylus. system wherein a user can see a free-standing, static holo-
In the second system, visual display is provided by a graphic image and inspect it using a force-feedback
dynamic holographic video system, and the user can mod- device. In the second system, spatial visual imagery is pro-
ify the geometry of the image in near real-time. We also vided by a dynamic holographic video system, and, using
suggest a preliminary set of guidelines for presenting the same force-feedback device, a user can inspect and
physically believable multimodal simulations in coinci- modify the geometry of the image in near real-time. Both
dent visual-haptic workspaces. systems provide coincident workspaces, wherein the hap-
tic and visual feedback originate from the same spatio-
Keywords: multimodal interaction, virtual and augmented temporal location.
reality, holography, haptics.
The structure of this paper is as follows. First, in the
1. Introduction context of coincident workspaces, we compare holo-
graphic displays to other techniques for displaying 3D
images. Then, for each of our holo-haptic systems, we
The last decade has witnessed a great deal of interest in
present its architecture, operation, and discuss system per-
building systems with multimodal input and output capa-
formance and related issues. Finally, we suggest a prelimi-
bilities. The availability of large computational bandwidth
nary set of guidelines for presenting physically believable
coupled with innovative sensing and display technologies
multimodal simulations in coincident visual-haptic work-
has inspired a varied and interesting array of spatial inter-
spaces.
active workspaces [1][2][3][4][5]. At the same time, there
is growing experimental evidence that the availability of
binocular visual depth cues, spatial audio, kinesthetic and 2.0 Why holographic displays?
tactile feedback improve performance of some spatial
tasks [6][7][8]. These studies underscore the need for mul- Why should we use holographic displays in coincident
timodal workspaces with spatial cueing, and motivate workspaces? In this section, we present a concise over-
research on bringing new sensing and display technologies view of three popular alternative techniques for 3D dis-
into the mainstream. play, and compare them to holography.

In the Spatial Imaging Group at the MIT Media Lab, we 2.1 Stereoscopic systems with beamsplitters
build experimental spatial displays. We believe that by
engaging binocular vision and motion parallax, these dis- A historically popular and frequently used technique
plays offer a powerful way to disambiguate spatial infor- employs a half-silvered mirror to combine two-view or
mation in a scene and help a viewer to better understand head-tracked stereo images with a view of the hand and
the shapes and layout of displayed objects. Although we physical objects in the manual workspace. In these sys-
investigate several different kinds of 3D imaging tech- tems, the stereo graphics (or video) can neither occlude the
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

real world, nor can they be occluded by it (unless hands or necessarily frustrate the perception of the three-dimen-
objects are tracked, and corresponding occlusion compu- sionality of images; the link between accommodation and
tations are performed.) The result is a ghost-like display convergence is plastic and many displays disrupt it with
in which simulated and real objects sometimes appear to no serious perceptual effects. However, the extent to
be transparent. In such a configuration, volumes of simu- which accommodation-convergence mismatch may be
lated and real objects can freely interpenetrate; when co- responsible for errors in perception and visually-guided
located, their appearance is strange, and their absolute movement is undetermined.
depths may be inaccurately perceived [9].
In coincident visuo-manual applications using astig-
When stereoscopic video or computer graphics are used matic displays, there is also an accommodation mismatch
to supply the display, depicted objects can be realistically between simulated and physical objects. For instance,
rendered and displayed at high resolution. 3D scenes thus when an operator moves a finger or a hand-held tool into
generated can be replete with pictorial cues to depth and zero-disparity alignment with a feature on the simulated
can be rendered with convincing shading, texture and object, conflicting depth cues may be reported by accom-
reflections. However, pixellated stereoscopic displays modation.
have an inherent depth quantization associated with them;
for a CRT with 0.25mm pixel pitch, and a viewer with an 2.2 Volumetric displays
interpupillary distance of 65mm located 500mm from the
display plane, the depth quantization is on the order of Volumetric displays work by sweeping or scanning out
2mm. Since human stereo acuity is on the order of 1mm at 3D space with points or planes of light. In these displays,
this distance, these displays cannot yet provide adequate solid objects are approximated by a spatial arrangement
depth resolution. of image slices or by many points of light spatially assem-
bled to indicate object structure. There are several types of
This kind of system also fixes a scenes depth of field volumetric displays available: slice-stacking displays, like
during rendering or scene optical capture, often without the varifocal mirror [10], rotating plane/helix displays,
considering an appropriate depth of focus for viewers such as the Texas Instruments OmniView; and systems
eyes. Resulting stereo images often have an abnormally which actually emit photons from within the display vol-
large depth of field which may account for the distracting ume.
nature of unfuseable images in stereo displays (while
diplopic vision goes virtually unnoticed during normal These displays have some advantages: and there is usu-
binocular viewing). ally no mismatch between accommodation and conver-
gence since all object points have a true spatial location
If head-tracking is employed to provide motion paral- to which eyes can freely converge and focus, and they typ-
lax, scene jitter from tracker noise and lag between fast ically have a wide field of view. However, the low density
head movement and scene update can be problems. Addi- with which many such systems display object points
tionally, some sort of viewing apparatus must be worn, causes depth quantization, and since the light emitting
such as LCD shutter glasses, to multiplex the correct ste- points in these systems are isotropic radiators, view-
reo views to left and right eyes (and often to provide head- dependent shading is not possible. Worse still, inter- and
tracking information to the system). Though these glasses intra-object occlusion relationships are not displayed
are not particularly uncomfortable, autostereoscopic (unless viewer position is tracked and corresponding
viewing is generally preferred, especially for viewers who backface cull and occlusion computations are performed
already wear glasses. on the object point database). Since occlusion is almost
always the most powerful cue to depth in a scene, images
Inherent in stereoscopic and other astigmatic displays is appear translucent without it, and the impression of three-
an accommodation-convergence mismatcha functional dimensionality can be seriously compromised.
disengagement of several systems of the visual system
which normally behave in cooperation. Stereoscopic sys- Finally, in coincident visuo-manual applications, these
tems require a viewer to accommodate to (or near) the dis- displays are mostly entirely inappropriate most cannot
play plane to see the imagery in focus. Yet, 3D imagery is physically admit the hand into their display volume.
likely designed to also appear at different depths than that
of the display surface. To fuse this imagery, a viewer must 2.3 Re-imaging systems
converge to a location different than the display surface
while keeping the image in focus. This mismatch does not Re-imaging display devices use optical systems to com-
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

bine or condition images and relay them into a viewers flexible image content, giving them an advantage over re-
space. For instance, Dimensional Medias High Definition imaging displays; 2D component images can be realisti-
Volumetric Display and SEGAs Time Traveller arcade cally rendered using computer graphics or optically cap-
game relay images of real 3D models and 2D computer tured with a scanning camera and computationally
graphics (flat or pre-distorted) to a viewer. This class of predistorted when necessary [13]. Thus, holographic ste-
systems employs optical components such as parabolic reograms can be replete with pictorial cues to depth in
mirrors, lenses, and beamsplitters to re-image already- addition to binocular and motion parallax, and full paral-
existing 3D objects and/or 2D display screens. As a result, lax holograms (in principle) allow a person to freely con-
interacting with the optical output in a way that modifies verge and accommodate to any depth in the image
the true shape of the displayed object is not possible [11]. volume. When computer graphic or other digitized com-
ponent images are used to generate the display, however,
Some of these systems, like DMAs HoloGlobe, are holographic stereograms are subject to the same depth
capable of displaying a large free-standing image with a quantization of any system using a pixellated display
wide field of view. In addition, a viewer can see the image screen.
under ambient lighting conditions and without wearing
viewing aids. Accommodation and convergence probably From a technological standpoint, holographic stereo-
behave the same as they do during normal binocular view- grams are becoming faster and less expensive to produce
ing. Additionally, since these systems re-image existing [14]. The availability of new recording materials and pro-
physical 3D models, the visual realism of the display can cessing techniques have improved diffraction efficiency
be quite striking. and signal-to-noise ratio, yielding brighter, cleaner
images. New display formats that incorporate their own
However, in a coincident visuo-manual workspace, it is illumination (edge-illuminated) are becoming compact
possible for the interacting hand to literally block image and portable [15]. And finally, in recent years, research on
projection. If the spatial image being occluded by the electronic holography has yielded systems which provide
hand is located nearer to the viewers eyes than the hand, full color, moving holographic images [16].
the available cues to depth become strongly contradic-
tious binocular cues and motion parallax report accu- In order to project a visual spatial image into the
rate depth relationships while occlusion reports the viewers manipulatory space, display flexible image con-
opposite depth ordering. The projective geometries of re- tent, and provide as many cues to depth as possible, we
imaging displays make them prone to this particular prob- chose to incorporate holographic displays into our spatial
lem. visuo-haptic systems. Our first experiment involved the
combination of static edge-illuminated holograms and
2.4 Holographic displays coincident force display.

Holographic displays permit this problem to arise as 3.0 Static haptic holography
well. If a persons hand is interposed between object
points (nearer the viewers eye) and the farther hologram The combination of haptics and holography was first
plane, image reconstruction is blocked and the hand investigated by researchers at De Montfort University for
appears to occlude a nearer object. While this cue conflict an object inspection task [17]. In this work, visual display
is disturbing, it occurs only during certain configurations was provided by a reflection transfer hologram which pre-
of the hand and object; this is less distracting than the sented an aerial image of a control valve. A Computer
ever-present hand/image composite displayed by stereo- Controlled Tactile Glove (CCTG) provided coincident
scopic systems with a half-silvered-mirror. However, haptic display of the same data. Early informal experi-
holographic displays, which are actively being researched ments in combining reflection transfer holograms with
and developed, have increasingly many advantages to force-feedback were also performed by researchers at the
offer. MIT Media Laboratorys Spatial Imaging Group. Reflec-
tion holograms require front overhead illumination for
A holographic stereogram is a discretized hologram image reconstruction; thus, in either of these holo-haptic
[12] which optically projects a series of 2D perspective efforts, the interacting hand could literally block the
views of a scene into the displays viewzone. There, a per- reflection holograms illumination and prevent image
son is provided autostereoscopic viewing of the scene, reconstruction.
and scene parallax changes (without lag) in concert with
the viewers head motion. Holographic stereograms offer This problem was addressed in our laboratory by
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

employing full-parallax edge-illuminated holograms in cesses. For any given optical printing setup, the specific
combination with the PhantomTM force-feedback inter- geometries of the hologram printer can be specified in
face for the inspection of static 3D models [18]. In this HoloBuild during the design process, recorded in the con-
work, the systems for optically recording holograms and figuration file, and used later during hologram printing.
producing the haptics simulations are separate, but both
take the same 3D geometry description as input. The 3D model data
hologram-production pipeline also requires specification

HoloBuild

HoloServe
scene lighting
of camera geometry, scene lighting information and all camera parameters
visual material properties for rendering. The haptics pipe- material properties (visual)
texture, reflection maps
line requires tactual and bulk material property specifica-
tion to accompany the geometric description. rendered preview

3.1 Implementation holo configuration

3.1.1 Static hologram modeling and printing. We have (a) Holographic stereogram design and preview
developed a computer graphics rendering server (HoloS-
erve) and client application for both visual scene design
holo configuration
(HoloBuild) and holographic stereogram printing (Holo-
Print).

hologram printer

HoloServe
HoloPrint
HoloBuild makes it possible to: view render request
printer
control successful render reply
import 3D model data,
interactively light the scene,
tailor the computer graphics view and camera to
match hologram geometry,
(b) Holographic stereogram printing pipeline
change model rendering parameters and
request that a specific perspective view be ren- Figure 1. Design and Preview, and Render-on-Demand
dered by HoloServe. hologram printing pipelines

HoloServe uses SGIs Graphics Library and takes

advantage of available rendering hardware; individual HoloServe, as well as HoloBuild, were implemented on
perspective views of complicated computer graphic mod- an SGI Onyx Reality Engine, and HoloPrint can be run on
els can be generated in a few seconds. any unix machine. These design-preview rendering and
printing processes communicate over ethernet using
All final model and rendering parameters designed remote procedure calls (rpc).
using HoloBuild are saved in a configuration file. This file
is read directly by HoloServe for rendering component 3.1.2 Haptic Modeling. The haptic simulation, (Holo-
images at hologram printing time. During the hologram Feel) uses the same geometrical description as HoloServe
printing process, the HoloPrint application requests that a to render a force image of the scene in space, which we
specific perspective view be rendered, and upon notifica- co-locate with the projected holographic image. From a
tion of rendering completion, sends appropriate exposure client scene design tool, HoloFeel receives the particular
and frame-advance control sequences to the hologram scene geometry and haptic modeling parameters that cor-
printer. In this manner, the newly rendered frame is respond to a given holographic image (Figure 2). Then, to
recorded and the printer is readied for the next exposure display the simulation computed with HoloFeel, we use
(Figure 1b). The combination of HoloServe and HoloPrint the Phantom haptic interface, a three degree-of-freedom
marked our first render-on-demand holographic printer, (d.o.f.) mechanical linkage with a three d.o.f. passive gim-
obviating the need for storing large numbers of pre-ren- bal that supports a simple thimble or stylus used by the
dered frames on disk for subsequent hologram recording. hand. Six encoders on the device provide positional infor-
mation, and three servo motors provide force display in a
Both HoloBuild and HoloServe were designed to workspace of 290 x 400 x 560mm3. HoloFeel is imple-
accommodate a wide variety of hologram viewing geome- mented on a Pentium PC and runs with an average servo
tries, and both one-step and multiple-step printing pro-
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

rate of 2KHz. haptic hemisphere with a hemi-ovoid).

computation
HoloFeel

Scene design tool

ethernet
3d model data
(tcp) Object modeling params
SGI Onyx

display (a) Rendered frame (40, 50). (b) Rendered frame (75, 50).
Figure 3. Rendered component images
Phantom Coincident
Haptic final
Display Volume holo
Device
The second and slightly more complex holo-haptic
example used an arrangement of blocks forming a maze,
Figure 2. Static holo-haptic system architecture which is oriented against a vertical back plane. The blocks
varied in size and spacing, and the channels formed
between them are narrow. The back plane was visually
3.2 Results and haptically texture mapped with a vertical grating. A
rendered component frame and the final hologram are
This suite of applications was used to produce two two- shown in Figure 3(b) and Figure 4(b).
optical-step edge-illuminated holographic stereograms
used in conjunction with haptic simulations. The simpler The full-parallax master provided a much smaller hori-
hologram displayed a hemisphere affixed to a vertically zontal range for head motion in the final hologram
oriented plane. The plane was both visually and haptically approximately 35 degrees. The master was comprised of
textured with a vertical grating; the hemisphere itself was 6700 exposures of pseudoscopically-rendered frames [13]
visually texture mapped but had no haptic surface relief. and the final hologram was produced in an additional
Haptic modeling included static and dynamic friction, and optical transfer step. As in the previous hologram, the
object damping and compliance. A rendered component mastering and transferring steps used a recording wave-
frame and the final hologram are shown in Figure 3(a) and length of 528nm.
Figure 4(a).
The total depth of the final hologram was approximately
The full-parallax master hologram (first optical step) 35 mm, and the entire model reconstructed in front of the
permitted a broad angular range of head motion of hologram plane. Image plane width and height were each
approximately 50 degrees horizontally and 30 degrees 100 mm. The final hologram was illuminated with an
vertically. The master contained 15,000 exposures of LED centered at 520 nm. The model presented contains
pseudoscopically-rendered [13] frames. The final edge- more image features (edges) than did the simple hemi-
illuminated hologram was printed in an additional optical sphere hologram; the maze hologram offered us a chance
transfer step. Mastering and transferring steps [14] used a to examine maze-tracing performance in this coincident
recording wavelength of 528nm. and an offset (using stereo graphics) visuo-haptic work-
space configuration.
The total depth of the final hologram was approxi-
mately 40 mm, all in front of the image plane. Image In addition to the robust and compact design of edge-
plane width and height were each 100 mm. The hologram illuminated displays, the principal benefit of using this
was illuminated with an LED centered at 520nm, which hologram format in concert with haptic applications lies
yields a bright image with slight spectral blurring. The in the steep-angle lighting it incorporates; in the final
multi-modal scene presented was intended to have very viewing configuration, the hand and haptic apparatus do
few formal features; this tangible hologram provided a not block the illumination source as they interact with the
simple example with which to examine perceptual toler- image. However, while haptic inspection of the holo-
ances for spatial misregistration and mismatches in curva- graphic image is possible using this hologram format, we
ture of the visual and haptic models (e.g. by replacing the still encounter some discord between the hologram and
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

visible co-located hand and haptic apparatus. This slightly visual display and the user never actually visually
more complex volume of information provided greater observes her hand. In such workspace configurations,
opportunity for us to examine some of the sensory con- slight spatial misregistrations, or changes in scale
flicts which occur in a coincident holo-haptic workspace. between the visual and haptic display can be virtually
unnoticeable. Yet too much intermodality disparity can
cause the visual and haptic cues to be perceived as
entirely separate events, and may be quite confusing or
annoying.

Tolerances are lower still when visual and haptic work-

spaces are superimposed. In our coincident workspace
format, we observed several conflicts between what is
seen and what is felt; these intra- and intersensory con-
flicts are described in turn below.

3.3.1 Spatial misregistration. When exploring a surface

with the Phantom and visually monitoring the device,
simultaneous visual and haptic cues to the surface loca-
tion are available. When we feel contact, the visible loca-
(a) haptic inspection of hemisphere hologram tion of the stylus tip is perceived to be co-located with the
haptic surface. During contact, if the holographic surface
and the haptic surface are not precisely aligned, the mis-
registration is strikingly obvious to vision. These conflict-
ing visual cues erode the impression of sensing a single
object. Instead, the impression of two separate representa-
tions is evident. This condition is shown in Figure 5(a).

3.3.2 Occlusion violations. As mentioned earlier, occlu-

sion is perhaps the most powerful cue to layout in a scene.
When we see the image of an object being blocked by the
image of another, we understand the occluded object to be
farther from our eye than the occluding one. In our holo-
haptic systems, it is possible to position the haptic appara-
tus between hologram and image and actually block its
(b) haptic inspection of block hologram reconstruction; in an observers view of the scene, occlu-
sion relationships contradict other depth cues reporting
Figure 3. Holograms combined with force image true scene layout as shown in Figure 5(d). Even in the
presence of correct depth accounting from stereopsis and
motion parallax, Perception appears to favor the depth
3.3 Sensory Conflicts
ordering reported by occlusion relationships.
As we readily observe in our everyday interactions, har-
3.3.3 Contact with multiple surface points. Obviously,
monious multisensory stimulation usually gives rise to
holograms present spatial images which cannot by them-
correct perception of objects and events. The broad body
selves exhibit a restoring force when penetrated by an
of work on multi-sensory interaction indicates that some
object. With no haptic simulation running to detect colli-
disparity between visual and haptic information can dis-
sions with model surfaces and to display contact forces,
tort the overall percept while still being tolerated. The
the haptic apparatus is free to pass through the holo-
ability of sensorimotor systems to adapt to discordant sen-
graphic image undeterred. Our haptic simulation can pre-
sory input permits us to perform well even in the presence
vent a single point on the stylus from penetrating the
of distortion, so long as sensory feedback is available.
model, but current device limitations preclude emulation
This fact is extremely useful in offset visuo-haptic work-
of the kind of multi-point contact that occurs in the physi-
space configurations, wherein the tracked hand or device
cal world.
position is represented as a graphical element on the
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

visual tip 1
surface
haptic

hologram
apparatus
2
3

haptic stylus
surface

Spatial misregistration results Restoring force is displayed due

(a) in visual disparity between force (b) to surface penetration at 1, while
and optical images. the visible surface penetration
from 2 to 3 is not haptically
reported.
Scene, stylus, & observer: Observers view:

observer
hologram

min
observer
max

Broad spectrum illumination In the true spatial layout, In the apparent depth
(c) of hologram causes depth- (d) the stylus is positioned ordering, the stylus is
dependent optical image blur. behind the holographic seen occluding (in
image of the sphere. front of) the sphere.
Figure 5. Conflicting sensory cues present in holo-haptic displays

During each haptic control loop cycle, the simulation artifact of a holograms diffractive properties is the chro-
checks for a surface collision all along the stylus probe; matic blurring that occurs with broad spectrum illumina-
even if it finds many, it can only compute and display tion. In the transmission edge-illuminated holograms used
forces for one. If a model surface has been penetrated by in this work, the holographic image plays out high and
the stylus tip, it is assumed the viewers primary attention farther from the hologram in wavelengths shorter than the
is focused there, and forces due to this collision are com-
recording wavelength, and lower and closer in longer
puted and displayed. However, if not the tip, but other
ones. If the illumination source used in hologram recon-
points along the probe have penetrated the model, then the
struction is not monochromatic, spectral blur will be evi-
collision closest to the tip is used for computation and dis-
play. dent in the final image. Image elements close to the
hologram plane will be quite clear, but those farther from
The situation permits another kind of occlusion viola- the hologram plane will exhibit blur in accordance with
tion to occur as shown in Figure 5(b). While the stylus tip source bandwidth.
is seen and felt in contact with some geometry, the stylus
may be rotated around its tip and swept through proximal Since a viewer generally expects scene elements closer
holographic image volume. Parts of the users hand may to the eye to be more keenly resolvable, the blurry image
also penetrate the image. Seeing such physical objects and elements near the viewer challenge the impression of
holographic image coexist in the same physical volume image solidity. This condition, shown in Figure 5(c), is
presents a confusing impression of depth and object solid- recognized as problematic on its own, but adding coinci-
ity in the scene. dent haptic display causes further difficulty. Usually an
objects bulk material properties (e.g. stiffness) remain
3.3.4 Optical-haptic surface property mismatch. An uniform throughout the display volume. If the haptic and
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

visual output are precisely in register, then near the holo- display where changes have occurred. From the point of
gram plane the stylus will be exactly coincident with an view of a user, who is holding the stylus and pressing it
imaged surface during contact. However, far from the into the holographic image, a single multimodal represen-
image plane, the stylus will visually penetrate the blurry tation of the simulation can be seen and felt changing in
image of the surface by a substantial distance before con- response to the applied force. The system architecture is
tact is felt. As mentioned earlier, misregistration between shown below in Figure 6.
the image surface and stylus tip during contact, especially
when close to the viewers eye, can diminish the simula- computation
tion quality. In addition, visual and haptic information holovideo module
presented is conflicting; by visual report, the surface qual- Hologram
haptics module Workspace unix Modeling
ities change substantially with depth though their haptic Resource SGI Onyx + RE
Force Modeling ethernet Manager pipe
quality remains the same. Pentium PC (WRM) SCSI
(tcp)
SGI Onyx Holovideo Output
In addition to the occurrence of these cue conflicts, the Cheops

image quality was somewhat compromised, mostly due to

non-uniform illumination. This problem is not inherent to display
the display format; better illumination can certainly be
devised. Nonetheless, these full-parallax haptic-holgrams Phantom Coincident Holographic
presented a compelling way to haptically inspect simu- Haptic Display Volume Video
Device System
lated spatial objects using a compact table-top display.
For instance, when a person taps the edge of a block, or
visually and haptically locates a trough in an objects tex-
ture with the stylus tip and follows it through the image, Figure 6. Dynamic holo-haptic system architecture
the impression of a single multimodal representation is
quite strong. The principal disadvantage to these displays
4.2 Haptic Modeling
(and no small one at that) is that they are static; they have
utility for inspecting three dimensional shapes, but do not
permit interaction with or modification of the data pre- The multimodal image displayed represents a spinning
sented. surface of revolution (initially a cylindrical stock)
which can be interactively lathed by user. The haptic
cylinder, initially and in subsequent stages of carving, is
4. Dynamic haptic holography
represented as a surface of revolution with two caps. It has
a mass of 1 gm, an algorithmically defined vertical grating
The logical extension of using the Phantom to inspect a as surface texture, static and dynamic frictional proper-
static holographic image is to allow an operator to modify ties, stiff spring bulk resistance, and rotates about its axis
the geometry of the image using the Phantom. This at one revolution per second. The cylinder model strad-
requires (ideally) a real-time, dynamic holographic dles a static haptic plane (which spatially corresponds
display. For the past several years, one of the research with the physical output plane of the holovideo optical
projects in our group has been to build a real-time system); the haptic plane is modeled with the same bulk
holographic video display (holovideo). Combining haptic and frictional properties as the cylinder. Currently, the
simulation and force feedback with holovideo permits us haptics simulation is implemented on a Pentium PC with
to render dynamic scenes in a coincident workspace. an average servo rate of 1.5KHz.

4.1 System architecture The radius profile of the surface of revolution is repre-
sented as a cubic B-spline curve with 28 control points, all
Two separate modules comprise the computation which of which are initially set to the same radius value (25mm)
feeds the displays; a haptics module that performs force to let us begin with a cylinder. The curve evaluated
modeling, and the holovideo module which pre-computes between the middle 21 points defines the profile of the
holograms and drives rapid local holographic display cylinder body; the remaining top three and bottom four
updates based on changes to the model. The haptics and points lie beyond the actual extent of the cylinder, and
hologram modules are organized by the Workspace serve to lock the shape at its top and bottom, respec-
Resource Manager (WRM) which is notified of geometry tively. Control points are modified as force is exerted on
changes imparted to the haptic model by the user, and the shape at height h, between control points Pi and Pi+1.
requests hologram updates to local regions of the visual
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

A new radius for the surface of revolution at this height framebuffer. Because holovideo has a non-standard
can be computed by evaluating the nonuniform rational display format, an image-processing system developed at
B-spline formulation. the MIT Media Lab, Cheops, was extended to support it.
Cheops has three different module types: processor, input/
The cylinder can be felt spinning beneath the users memory, and output, and an optional memory module
touch, and when pressed with enough force (i.e., when the provides up to 0.5 Gbytes local to the system. These
surface has been penetrated by some threshold distance modules are interconnected by two linear buses. One of
D) the surface deforms. A very simple method for surface these buses, the Nile bus, is capable of sustained high
deformation is used: the two control points straddling the bandwidth (>100 Mbyte/sec.) transfer of samples and the
penetration location are displace toward the central cylin- second, the Global bus, is capable of 32 Mbyte/sec.
der axis by a fraction of the penetration distance, and this transfer [21].
changes the radius profile of the surface of revolution.
4.4 Implementation
The upper control point is displaced by tkD, and the
lower by (1-t)kD, with t being the normalized distance 4.4.1 Haptics Module. The Workspace Resource Man-
between the contact point and the lower control point, ager (WRM) initializes its own model of the surface of
used in the B-spline formulation. The closer control point revolution, which starts as a cylinder of desired height and
is displaced by a greater distance. If contact occurs
radius. It then initiates the haptic simulation by making
directly on a control point, then that point alone is dis-
placed by kD. Thus, control point displacement modifies client calls to the haptics module on the Pentium PC.
the circumference of the cylinder at height h, as force is These calls request creation of a haptic cylinder of the
interactively applied. same height and radius at a desired location. The haptics
module commences physical simulation of this spinning
The parameters k and D can be adjusted to make carv- cylinder, and computes collisions of the Phantom tip with
ing the rotating cylinder require more or less force. A the computational model. Based on these collisions,
minimum radius of 15mm is enforced, so that once the forces are computed and displayed to the operators hand,
surface has deformed this much, the control points update and any resulting shape modifications are reflected in the
no further. The control point density, 4.17 points/cm, was model update.
experimentally determined to be high enough to accom-
modate local model changes, yet sparse enough to avoid Changes in the cylinders underlying B-spline
unstable deep notching of the haptic surface. representation are automatically communicated from the
haptics module to the WRM approximately 30 times per
4.3 Holographic Video Modeling second. The information sent contains the location where
change begins on the curve (the number of the bottom-
We employ the second generation of holovideo in this most control point), and values of the six affected control
work [16]. This system is capable of displaying points, ordered from bottom to top. It is assumed that
monochromatic, horizontal-parallax-only (HPO) images model changes occur reasonably slowly, so that no more
in a volume of 150 x 57.5 x 150 mm3, and the viewing than six control points are updated within 33 ms. Since
angle is 30. The 3D image produced by holovideo computing a deformation means updating at most two
supports the most important depth cues: stereopsis, control points surrounding the point of contact, our
motion parallax, occlusion, and many pictorial and communication rate means that we can only guarantee
physiological cues to depth. reporting accurate model changes from contact in a region
6.9 mm high within an update interval. Though this
For the present purpose, we may consider holovideo to assumption usually puts us within the realm of normal
be a black box which accepts two inputs: a computer- interaction speed, eventually, communicating a variable
generated hologram (CGH) and light [19][20]. The output number of control points to reflect the precise region of
of the black box is a 3D holographic image whose visual change would be more robust, and future work will
and geometrical characteristics depend on how the CGH implement this change.
was computed. Each CGH contains an enormous amount
of data 36 megasamples (at 1 byte per sample) 4.4.2 Workspace Resource Manager. Once the WRM
apportioned into 144 hololines of 256 kilosamples each. receives the message, the changed control points are used
The CGH is made available to the display via a to update its own representation of the radius profile. The
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

Interaction WRM computes Radius values are Quantized radii

modifies new radius quantized to fit used to determine
haptic values for each pre-computed set new hololines
model hololine of holograms for display

(a) computational flow

14 R[36] changes hololine 36

...
13 New radius
R[42] values are {

updated control points

12 }

...
curve region affected
R[48] computed and
quantized

...
11 for the
hololines
10 between
9 each pair
8 of control
points on
7 the curve.

...
6 R[84] changes hololine 84

(b) process details

Figure 7. Method of propagating haptic model changes to holographic display

WRM determines which lines of the holovideo display from each of these five separate holograms.
will be affected by the updated region of the curve. Since
the final holographic image will span 120 lines of the First we must determine how many and which lines we
display, we maintain a state vector, R, with 120 elements should change on the holovideo display. The number of
whose values represent the exact radii of the surface of display lines that require updating will vary, depending on
exactly which model control points are displaced. In
revolution at corresponding display lines. A set of six
regions near the top or bottom of the carved shape, a
holovideo display lines correspond to the space between
smaller region of the curve contributes to the visible
any two adjacent control points in the WRMs model. extent of the shape, so fewer display lines will require
change. The new radius values in R corresponding to
If as many as six control points have changed, it is changed display lines are quantized to match one of the
necessary to recompute radii for the 48 display lines set of five holographic cylinder radii, and each is assigned
spanning eight control points, between which the curve a radius code based on its quantized value as shown
will have been affected (Figure 7). These new radius below:
values are reflected in the state vector R. In the current
radius (mm) 25.0 22.5 20.0 17.5 15.0
implementation, the WRMs model can also be rendered
code 5 4 4 2 1
to a graphics display using SGIs Graphics Library for
debugging purposes, and to provide a means for remotely A message, which contains the number of the hololine
monitoring a users performance. marking the start of the update region, the number of lines
that need to be updated, and the radius codes of each new
Because it is not yet possible to compute 36 Mbyte line, is sent to the holovideo output module on Cheops. In
holograms in real time [19], we decided to pre-compute order to minimize the display update time, we are
five cylinder holograms for use in updating the display, as currently updating a maximum of 32 hololines per cycle,
explained shortly. Each hologram displays a cylinder with representing only the display lines between the original
a different radius, the initial cylinder, and four six control points sent by the haptics module.
progressively smaller ones, rcyl (mm) = {25.0, 22.5, 20.0,
17.5, 15.0}, ending with the minimum-radius cylinder. 4.4.3 Holovideo Indexing. Upon receiving the update
All holographic cylinders are 47.9 mm high. These message, the holovideo output module must instruct
holograms, from largest to smallest radius, are loaded Cheops to collect the appropriate hololines and dispatch
sequentially into the Cheops memory module. At system them to the display. This is accomplished by indexing into
start-up, the cylinder with the largest radius is displayed.
the memory module with the radius code to determine the
As the initial haptic cylinder is carved, a visual
correct cylinder to display, and then writing the
approximation to the resulting surface of revolution is
assembled on the display by loading the appropriate lines corresponding hololine to the output card (Figure 8). The
final holographic image is assembled using hololines
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

line 12
holo5 holo4 holo3 holo2 holo1

line 132
r = 25mm. r = 22.5mm. r = 20mm. r = 17.5mm. r = 15mm.

final

Figure 8. Method of assembling final holographic image from pre-computed hologram set

from the five individual holograms. It must be noted that apparently subtracted away. Higher bandwidth spatial
this method of hologram assembly is valid only for HPO light modulators, efficient data compression techniques,
holograms; for full-parallax holograms, the entire improvements in computation speed, and higher
hologram would have to be recomputed. In the absence of bandwidth data pipelines will all help to alleviate this
the computation and communication bandwidth necessary problem in future generations of the holovideo system.
to update fully-computed holograms in real-time, pre-
computed hologram indexing enables rapid, local
updating.

4.5 Results

When an operator carves the holographic surface of

revolution with the Phantom, the hologram image changes
due to force apparently applied by the tip of the stylus.
The resulting shape can be explored by moving the stylus
tip around the surface without exerting too much force.
Physical objects in the workspace may also be explored,
so that both physical and simulated forces can be dis-
played to the operator alternatively in the same work-
space. When the operator maintains the correct viewing
distance for holovideo, the perception of a single multi-
modal stimulus is quite convincing. Images of an operator
interacting with the image are shown in Figure 9. Differ-
ent stages of carving are also shown in the lower part of
the figure.

4.5.1 System Lag. A compelling multimodal

representation depends heavily on minimizing, to
imperceptible levels, the time lag between the operator
Figure 9: An operator carving
effecting changes in the haptic model and the result of that the image on holovideo. Physical
change appearing on the visual display [22]. A reasonable hardcopy produced by the 3D
visual update rate (20+ frames per second) is not currently printer is shown at right.
possible on holovideo, principally due to the speed at
which we can communicate with and update the display.
The effect of the resulting system lag, on the order of 0.5
sec., is that an operator can see the stylus tip penetrating
into the holographic surface before the surface is
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

Since the visual display is holographic, the full range of misregistration of the visual and haptic output, the visual
horizontal parallax is always available in the viewzone; no discrepancy is striking. Due to the lag present in the
lag is encountered with motion of the operators head. holovideo pipeline, our simulation is vulnerable to this
Additionally, no special eyewear is necessary to perceive problem when the operator is actively carving the surface.
the stereo information. Like static haptic holograms, the display does afford bin-
ocular disparity, motion parallax and pictorial cues to
4.5.2 Differences in Visual and Haptic Renderings. Our depth and layout. Unlike stereoscopic, half-silvered mir-
haptic simulation models a spinning surface of revolution, ror displays, the hand and haptic apparatus are able to
but the visual representation does not spin. In order to occlude the holographic image. Thus, unless hologram
represent a spinning holographic image, we need to be reconstruction is blocked, occlusion relationships are cor-
able to update all the hololines spanned by the image at a rect in the visual scene.
reasonable rate. As mentioned above, our system
currently suffers a low frame rate with the update of only 5. Discussion
32 lines; thus we chose to forgo animating the spinning of
the holographic surface. When visual update can be more The dynamic and the static systems described in this
rapid, this visual animation should be included. paper offer interaction with a holographic images on the
table-top; this marks a long-held goal in the field of
When the stylus tip is touched to a detail on the holography. In both of these systems, holographic images
holographic image, touch, stereopsis and horizontal in the manipulatory space are accompanied by real
motion parallax reinforce the perception that the stylus objects as well (at very least the hand and haptic appara-
and the holographic surface detail are spatially co-located. tus). In the resulting mixed-reality setting, visual, haptic
However, as is the case for all HPO holograms, the lack of and physical behavior differences between the holo-
vertical parallax causes a slight vertical shift that graphic image and juxtaposed physical objects can be
increases with image depth to accompany vertical head quite striking.
motion.
Even if we have done our best to render the holographic
4.5.3 Differences Between Simulation and Real Task. images with a solid, three-dimensional appearance, dis-
Differences between the haptic feedback in our simulation crepancies between spatial images and real objects call
and the feeling of carving on an actual lathe are important attention to the boundary between simulation and reality.
to note. Among them are that the simple material Noticeable distinction between real and synthetic objects
properties we currently simulate are quite different from may not necessarily impact performance in this space, but
those of wood or metal moving against a cutting tool. to the extent that we want to render a physically believ-
Additionally, since a cut applied at an instantaneous able scene, we need to consider the underlying issues
position on the cylinder surface results in a surface more carefully.
modification that extends around the entire shape
circumference, the user does not experience the feeling of Based on observations in our laboratory and discussions
continuously removing material as the shape spins under with users of our systems, we have compiled a prelimi-
the stylus. Of course, one obvious departure from reality nary set of guidelines for generating physically believable
is the 90 change in orientation of the lathe axis. visual-haptic displays in mixed-reality settings. We sug-
gest that physical believability depends on how well the
4.5.4 Sensory affordances and conflicts. The sensory stimuli representing a simulated object would correspond
conflicts in this display include all those found in the to stimuli generated by an actual physical instantiation of
static holo-haptic displays. Additionally, since holovideo that object. Rendering methods and display characteristics
is an HPO (and therefore astigmatic) display, are obviously important factors. Additionally, all sensory
accommodation and convergence are disjoined to some modalities employed in a spatial display should act in
extent (the authors are not currently aware of any concert to model some basic rules that, based on our expe-
systematic study to determine the behavior of rience, physical objects usually obey. To begin qualifying
accommodation when a person is viewing HPO this topic, we group these guidelines into display, render-
holographic stereograms). ing, and modeling factors, for presenting physically
believable multimodal simulations in coincident work-
At the moment when an operator feels that the stylus tip spaces:
is in contact with the surface, if the tip is seen either
penetrating the surface or not making contact at all due to
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

Display factors developing algorithms for computing smooth-shaded and

Simulated and real objects should appear with the visually textured holographic images. Also, to improve
same luminance, contrast, spatial resolution, our visual display update rate, we are modifying our
color balance, and clarity. pipeline to write hologram lines directly to the memory
Visual and force images of objects should have sta- module.
ble spatial and temporal properties (no percepti-
ble temporal intermittence, spatial drift, or Obviously, the performance of our dynamic holo-haptic
wavering). system is limited by the available computation and
No time lag should be detectable between a users communication bandwidth. This issue raises its own set of
action and the multi-modal response or effect of problems, some of which will no doubt be eliminated as
that action in the workspace. bandwidth becomes abundant. There are other issues,
A viewers awareness of display technology should such as computed holographic image quality, which merit
be minimized. further investigation and will continue to occupy the
minds of researchers in the field of holographic imaging.
Rendering factors
Computer graphic rendering or optical capture 7. Conclusion
geometry should match the system viewing
geometry. We have described two interactive multimodal spatial
Illumination used in simulated scenes should match imaging systems in this paper. The first was a static sys-
that in the real scene (simulated shadows and tem which allowed an operator to inspect a free-standing
specular reflections should not behave differ- static holographic image with a force feedback stylus. In
ently. the second system, the operator was allowed to modify
Optical and haptic material properties, as repre- the holographic image by carving. The component haptic
sented, should be compatible (a surface that and holographic subsystems were described and the
looks rough shouldnt feel soft and spongy). implementation of the whole system was detailed in each
case. The sensory conflicts encountered in a coincident
Modeling factors display format were discussed. A preliminary set of
The volumes of simulated objects should not inter- guidelines was suggested for presenting physically believ-
penetrate with real or other simulated objects. able visual-haptic simulations in coincident workspaces.
Occlusion, stereopsis, and motion parallax cues
should report the same depth relationships. 8. Acknowledgments
Convergence and accommodation should provide
compatible reports of absolute depth. We would like to thank the Honda R&D Company,
Accommodation should be permitted to operate NEC, IBM, the Digital Life Consortium at the MIT Media
freely throughout the volume of a simulated Lab, the Office of Naval Research (Grant N0014-96-
scene. 11200), Mitsubishi Electric Research Laboratories, and
The range of fusion and diplopia should be the same the Interval Research Corporation for their support of this
for simulated and real scene. research. We would also like to acknowledge valuable
All multisensory stimuli should appear to arise from conversations with members of the Spatial Imaging
a single source, and should be in precise spatial Group and the community at the MIT Media Laboratory,
register. especially Professor Stephen Benton, Professor Hiroshi
Ishii, Carlton Sparrell, Michael Halle, and John
Undoubtedly, more issues remain to be added to this Underkoffler. We thank Adam Kropp and Benjie Chen for
list; the factors noted above already prescribe high extending and improving the holographic design, preview
technological hurdles for visual and haptic display and printing software. We also thank Michael Klug for his
designers. assistance in the holography lab and Yael Maguire for his
3D printing effort.
6. Ongoing and future work
9. References
To improve the feel of our holo-haptic systems, we
are developing new materials simulation and modeling a [1] Wiegand, T.E.v., The Virtual Workbench & the Elec-
more realistic haptic representation of carving. To tronics Training Task. Internal memo, 1994. VETT
improve rendering quality on holovideo, we are
Published in Proceedings of the IEEE International Conference on Multimedia Computing and Systems, June 1999

project at MIT/RLE, 77 Massachusetts Ave. MIT, Cam- [13] Halle, M., Kropp, A., (1997). Fast Computer Graph-
bridge MA 02139-4307, http://mimsy.mit.edu/. ics Rendering for Full-Parallax Spatial Displays. in: S.A.
Benton, ed., SPIE Proc. Practical Holography XI.
[2] Yokokohji, Y., Hollis, R.L., Kanade, T., (1996).
Vision-based Visual/Haptic Registration for WYSIWYF [14] Klug, M., et al. (1997), Optics for full-parallax holo-
Display. International Conference on Intelligent Robots graphic stereograms, in S.A. Benton, ed., Proceedings of
and Systems, pp. 1386-1393. the IS&T/SPIEs Symposium on Electronic Imaging,
Practical Holography XI.
[3] Deering, M., (1992). High Resolution Virtual Reality.
Proceedings SIGGRAPH92, Computer Graphics, Vol. [15] W.J. Farmer, et al., (1991). Application of the edge-
26, No.2, pp. 195-202. July. lit format to holographic stereograms, in: S.A.Benton, ed.,
SPIE Vol. 1461, Practical Holography V pp. 215-226
[4] Ishii, H. and Ullmer, B., (1997). Tangible Bits:
Towards Seamless Interfaces between People, Bits and [16] St. Hilaire, P., (1994). Scalable Optical Architec-
Atoms, (CHI 97), ACM, Atlanta, pp. 234-241, March. tures for Electronic Holography, Ph.D. Thesis, MIT Pro-
gram in Media Arts and Sciences, Massachusetts Institute
[5] Wellner, P., Mackay, W., and Gold, R., (1993). Com- of Technology.
puter Augmented Environments: Back to the Real World,
CACM, Vol. 36, No. 7, July. [17] Jones, M.R.E., (1994). The Haptic Hologram, Pro-
ceedings of SPIE, Fifth International Symposium on Dis-
[6] Servos, P., Goodale, M.A., Jakobson, L.S., (1992). play Holography, Vol. 2333, pp. 444-447.
The role of binocular vision in prehension: a kinematic
analysis. Vision Research. Vol 32, No. 8, pp. 1513-1521.
[18] Plesniak, W., Klug, M., (1997). Tangible holography:
[7] Marotta, J.J., Goodale, M.A. (1997). Elevation in the adding synthetic touch to 3D display, in S.A. Benton, ed.,
visual scene: calibrating a monocularly guided reach. Proceedings of the IS&T/SPIEs Symposium on Electronic
Invest. Opthal Vis Sci 38:988. Imaging, Practical Holography XI.

[8] Marotta, J.J., Kruyer, A., Goodale, M.A., (1998). The [19] Pappu, R., et al., (1997). A generalized pipeline for
role of head movements in the control of manual prehen- preview and rendering of synthetic holograms, in S.A.
sion. Exp Brain Res 120:134-138. Benton, ed., Proceedings of the IS&T/SPIEs Symposium
on Electronic Imaging, Practical Holography XI.
[9] Ellis, S.R., Bucher, U.J., (1994). Distance Perception
of Stereoscopically Presented Virtual Objects Optically
Superimposed on Physical Objects by a Head Mounted [20] Underkoffler, J., (1991). Toward Accurate Compu-
See-Through Display. Proceedings of the 38th Annual tation of Optically Reconstructed Holograms, S.M. The-
meeting of the Human Factors and Ergonomics Society. sis, Media Arts and Sciences Section, Massachusetts
Nashville, TN. Institute of Technology.

[10] Traub, A.C., (1967). Stereoscopic Display Using [21] Watlington, J., et al., (1995). A hardware architecture
Varifocal Mirror Oscillations. Applied Optics, 6,6, pp. for rapid generation of electro-holographic fringe pat-
1085-1087, June. terns, in S.A. Benton, ed., Proceedings of the IS&T/
SPIEs Symposium on Electronic Imaging, Practical
[11] Halle, Michael, (1997). Autostereoscopic Displays Holography IX.
and Computer Graphics Computer Graphics (A publica-
tion of ACM SIGGRAPH) Volume 31, Number 2, May. [22] Mark, W.R., et al., (1996). Adding Force Feedback
to Graphics Systems: Issues and Solutions. Computer
[12] Halle, M.H., (1994).Holographic stereograms as Graphics Proceedings, Annual Conference Series, ACM
discrete imaging systems, in: S.A. Benton, ed., SPIE SIGGRAPH, pp.447-452.
Proc. Vol. #2176: Practical Holography VIII, pp. 73-84.