Teleimmersion System

Dr. Gregorij Kurillo

Contents
n n n n n n n

Introduction Teleimmersion @ UCB Current problems Multi-stereo camera system Networking Rendering Conclusion

Teleimmersion
n

Tele-Immersion is aimed to enable users in geographically distributed sites to collaborate in real time in a shared simulated environment as if they were in the same physical room Combines computer vision, graphics and network communications

What is Teleimmersion?
n

Tele-immersion connects remote places through a shared virtual environment

Internet 2 3D scene Interactive 3D display at remote location

Internet 2 3D scene

Interactive 3D display at remote location

Internet 2 3D scene

Interactive 3D display at remote location

Bidirectional 3D data transmission

Meeting in Virtual Space

site 1

site 3 site 2

Applications
n

Collaborative work

3D CAD design Ergonomics Entertainment (games) Remote learning and training Coordination of activities (dancing, rehabilitation)

Measurement tool
3D motion capture of body segments Medicine & rehabilitation

Teleimmersion @ UCB
n n n n n n

CITRIS lab in Hearst Memorial Mining Building 360 degree stereo capturing Full-body 3D reconstruction Real-time data recording Real-time rendering Collaboration through internet II with UIUC

Teleimmersion @ UCB

TI-System
n n n n n

48 cameras 12 stereo clusters 8 IR pattern projectors 4 microphones and 4 speakers 13 PC computers (2 or 4 CPUs, Windows XP) 2 projectors for passive stereo projection (using circular polarization) Real time rendering

3D Reconstruction
Image capture
(640x480)

Resize (320x240)

30 ms

Rectification

Background subtraction

Edge extraction Region-based correlation

15~20 ms

15~20 ms

Reconstruction
(Triangulation) Total: 130~180 ms

Image capture & Resize

Background subtraction

Edge extraction & RB Correlation

Reconstruction Depth Map

Real World, Real Problems

Holes & noise

Calibration

Missing data

Poor blending of pixels

Headaches
n n n n n

Cameras producing a lot of noise (i.e. outliers) Poor reconstruction (e.g. holes, flickering) Small range of capturing Some cameras do not work properly Delays in data transfer

Cameras
n

48 Dragonfly CCD Cameras (Point Grey)

12 color and 36 black & white cameras Firewire connection, IEEE 1394 640x480 resolution, 15 FPS Synchronization through hardware triggering

CCD Sensor
n n n

1/3 Sony CCD with square pixels 60dB signal/noise ratio CCD noise has three major components*:
photon noise read noise fixed pattern noise

n n

100 dark images recorded with delay of 20ms Stability of background noise examined on 100 images recorded with delay of 1s
*G. Kamberova, R. Bajcsy, Sensor errors and the uncertainties in stereo reconstruction, in: K.W. Bowyer, P.J. Phillips, eds., Empirical Evaluation Techniques in Computer Vision, pp. 96-116. IEEE Computer Society Press, 1998.

Average intensity of 100 dark images (Clust #1, Cam #1)

5.9 5.89 5.88 Avg. intensity [0..255] 5.87
Intensity [0..255] 5.9 5.89 5.88 5.87 5.86 5.85 5.84 5.83 5.82 5.81

Mean intensity of a set of dark images (Clust #1, Cam #1)

5.86 5.85 5.84 5.83 5.82 5.81 5.8 0 100 200 Rows 300 400 500

20

40 frames

60

80

100

Noise Analysis (Dark Images)

Camera 1
Cluster

Camera 2

Camera 3

1 2 3 4 5 6 7 8 9 10 11 12

min max mean std min max mean std min max mean std 4 8 5.8753 0.0392 4 8 16 5.8852 0.0395 5.8830 0.0409 4 0 3 0.0003 0.0017 0 1 6 0.0000 0.0000 0 0.0001 0.0013 0 4 0.0046 0.0069 0 1 4 0.0001 0.0012 0 0.0004 0.0020 0 1 0.0003 0.0016 0 1 3 0.0000 0.0005 0 0.0004 0.0023 0 1 0.0037 0.0061 0 3 3 0.0033 0.0058 0 0.0109 0.0104 0 3 0.0000 0.0002 0 5 1 0.0000 0.0001 0 0.0000 0.0000 0 1 0.0011 0.0032 0 2 1 0.0034 0.0058 0 0.0028 0.0053 0 1 0.0000 0.0004 0 1 1 0.0001 0.0013 0 0.0002 0.0007 0 3 0.0022 0.0048 0 1 5 0.0019 0.0043 0 0.0006 0.0026 0 2 0.0027 0.0051 0 1 1 0.0032 0.0056 0 0.0050 0.0070 0 1 0.0000 0.0005 0 1 1 0.0004 0.0019 0 0.0003 0.0016 0 2 0.0130 0.0112 0 2 2 0.0133 0.0123 0 0.0120 0.0108

min ... minimal intensity value in the average of 100 dark images max ... maximal intensity value in the average of 100 dark images mean ... average intensity of 100 dark images std ... average per-pixel standard deviation of intensity of 100 dark images

Stereo Clusters
n n n n

12 camera clusters Trinocular stereo system + color camera on top Problem of point correspondence Optimization of viewing volume
P

x1

x0

x2

o1

o0

o2

x1

x0

x2

o1

o0

o2

Rotation of the side cameras for 0 and 3 degrees

Rotation of top camera for 0 and 2 degrees

10

New Stereo Cluster

n n n

Optimized position of the cameras More robust structure Protection of camera circuit boards

Optimization of Viewing Volume

n n n n n n

Configuration of stereo clusters Volume coverage analysis Camera configuration analysis Re-arrange the cameras to increase coverage Solve existing problems of the ti-system Wide coverage of volume (e.g. for dancing)

11

Volume Coverage Simulator

12

Camera Simulator

Problems with the old configuration

13

Camera Arrangement
Before After

Lenses
n

6mm lens with the volume coverage of about 45Hx34V

4mm lens with the volume coverage of about 64Hx49V

14

6mm lens

4mm lens

n n

Resized volume = decreased resolution of captured objects Use of wide angle lenses for lower body parts

Illumination
n

Photometric properties of objects:

Lambertian (e.g. matte surfaces) Diffuse (diffuser) Reflective (e.g. mirror) Specular (e.g. metal objects)

Specular light creates problem with 3D reconstruction (correspondence of points!) Diffusive light needed to prevent shadows

15

Illumination
n

Camera hardware parameters:

Lens aperture

Camera software parameters:

Exposure Brightness Shutter Gain

Lights

Too bright

Too dark

- Lens aperture - Shutter - Gain

16

Lighting
n n n n

Diffused lights Daylight illumination brightline SeriesONE 1.1 Eight 55W 5600K lamps 2 ceiling fixtures 4 top-down fixtures 2 floor fixtures

What Should I Wear???

n n n

Texture helps correlation Use of IR patterns to increase texture Colors that work well:
Skin colors Soft colors (e.g. yellow, pink, off-white)

Black or dark colors tend to blend with the background + less features extracted = poor 3D reconstruction

17

Camera Calibration
n n n

Tsai* camera calibration method Pinhole perspective projection model Two step calibration:
Intrinsic parameters (grid collection) Extrinsic parameters (LED point collection)

n n

Nonlinear optimization Camera Calibration Toolbox for Matlab (ETHZ)

* R.Y. Tsai,An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 364-374, 1986.

18

Camera Model
n

Model for image formation:

x' fs x y ' = 0 1 0 fs fs y 0 X0 ox 1 0 0 0 Y R T 0 1 0 0 0 oy 0 1 Z 0 1 0 0 1 0 1

x ' = K 0 X = K 0 gX 0
n

Radial distortion model:

x = xd (1 + a1r 2 + a2 r 4 ) y = yd (1 + a1r 2 + a2 r 4 )
2 2 r 2 = xd + yd

Intrinsic Parameters
n n

Automatic grid detection Parameters (K):

Focal length Principal point Skew coefficient Radial distortion

Intrinsic Parameters (after 20 iterations): Focal Length: fc = [ 817.30967 817.05456 ] [ 0.29904 0.29325 ] Principal point: cc = [ 354.95341 280.30902 ] [ 0.00000 0.00000 ] Skew: alpha_c = [ 0.00000 ] [ 0.00000 ] Distortion: kc = [ -0.14879 0.00000 -0.00064 0.00582 0.00000 ] Pixel error: err = [ 0.11482 0.11512 ]

19

Extrinsic Parameters
n n

LED position capture Parameters:

Rotations (Rc) Positions (Tc)

Extrinsic Parameters (after 20 iterations): R0 = [-0.63760 -35.83037 1.04434] T0 = [949.62098 18.49745 645.78758] Rn = [-0.63783 -35.83675 1.04257] Tn = [949.80910 18.47836 645.91826]

20

Camera Calibration Errors

n n

Depth error due to lens distortion Depth error due to non-parallel CCD to the lens plane:

1 2%, 2 7%, 3 14% (at 2000mm)

Depth error due to the composition of calibration errors (position and rotation between cameras)
Relative roll: 1 0.05%, 2 0.1%, 3 0.70% Relative pitch: 1 0.05%, 2 0.2%, 3 0.75% Relative yaw: 1 1%, 2 30%, 3 45%

* W. Zhao, N. Nandhakumar, Effects of camera alignment errors on stereoscopic depth estimates, Pattern Recognition, vol. 29, pp. 2115-2126, 1996

21

Improved Calibration
n n

Described calibration is very time-consuming Accuracy of the entire system depends on calibration of one camera Multi-camera self-calibration using only LED points (T. Svoboda, 2005*) Internal calibration of cameras inside clusters + calibration of clusters by known geometry (IR)

*T. Svoboda, D. Martinec, and T. Pajdla. A convenient multi-camera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments, pp 407-422, 14(4), August 2005.

Networking
n

n n n

Ability to send captured data through network in real time with minimum delays Provide reliable transfer of 3D data Transfer of 3D points not possible Compression and other transfer schemes used

22

Data Capturing
n n

n n

640x480 b&w and color images Raw data saved in run-lengthencoding (RLE) format Z-lib lossless compression Composed of packets whose size depends on the image coverage

Header [10b] Header:

Depth [2b]

B(lue) [1b] [4b]

G(reen) [1b]

R(ed) [1b]

Frame num [2b]

Compress. size [4b]

Network Configuration
UCB RENDERER
Cam #1

UIUC RENDERER
Cam #1

Cam #2

Cam #2

Internet (TCP/IP)

Cam #12

100Mbit / 1 GBit

Cam #6 TRIGGER

TRIGGER

23

Analysis of Network Traffic

# DATE ACTION PRO Local Address:Port Remote Address:Port Status Bytes 56295 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3011 169.229.144.111:2365 SUCCESS 10 56296 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3011 169.229.144.111:2365 SUCCESS 10 56297 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3011 169.229.144.111:2365 SUCCESS 17786 56298 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3010 169.229.144.110:1680 SUCCESS 10 56299 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3010 169.229.144.110:1680 SUCCESS 10 56300 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 10 56301 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 10 56302 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 13140 56303 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 13140 56304 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 14600 56305 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3010 169.229.144.110:1680 SUCCESS 46839 56306 28/09/2006 13:37:28 RECEIVE TCP 169.229.144.34:3003 169.229.144.41:1586 SUCCESS 6969 56307 28/09/2006 13:37:29 RECEIVE TCP 169.229.144.34:3002 169.229.144.40:1644 SUCCESS 10 56308 28/09/2006 13:37:29 RECEIVE TCP 169.229.144.34:3002 169.229.144.40:1644 SUCCESS 10

One person, one minute, average of three trials One 3D stream for 320x240 (pixels/frame): 70 to 200 KB/s One TI site (10 3D Streams): 1,360 KB/s Most data comes through in one TCP/IP package
Average transfer speed during 1 minute trial (n=3)
250

Transfer speed KB/s

200 150 Std 100 50 0 1 2 3 4 5 6 7 8 9 10 11 12 Clus te r # Avg

24

Networking Problems
n

Current problems:
Each camera has to send data to multiple renderers = more traffic Reliability of connection Reliability of transfer speed

New Network Configuration

RENDERER 1 RENDERER 2

Internet (TCP/IP)
UCB GATEWAY 100Mbit / 1 GBit
Cam #1 Cam #1

UIUC GATEWAY

Cam #2

Cam #2

Cam #12

Cam #6 TRIGGER

TRIGGER

25

Rendering
n n n n n

3D graphic representation of captured data OpenGL-based application Renderer is a server, cameras are clients Real-time (10FPS) rendering of 75,000 3D points Different types of rendering:
Quads Splatting

Rendering

26

Volume Rendering
n

Quad rendering

Direct rendering of points Fast (10 FPS)

Splatting

Pre-processing Sensitive to outliers Slow (4 FPS)

* M. Zwicker, H. Pfister, J. van Baar, M. Gross, Surface Splatting, Proceedings of SIGGRAPH 2001

Quad Rendering vs. Splatting

27

TI-Renderer
n

Future improvements:

Faster rendering Import common 3D data files (e.g. OBJ, 3DS) Projection of the scene close to the user Use of hand as a user interface Gaze/face orientation tracking Scene-graph support for collaborative work

Tele-Immersive Dance

28

Tai-Chi

Whys Part I
n

Dark images, color flickering

White balance, color calibration Optimization of aperture, gain etc.

Holes in reconstruction
Improper aperture of lenses Poor illumination Not enough texture Too dark materials

29

Whys Part II
n

Noise (outlier points)

Poor calibration of stereo cameras Noisy background subtraction Incorrect correlation of points

Hiccups in rendering

Delayed triggering due to slow reconstruction on some computers

30