Session 1

Introduction
Michael Bleyer
LVA Stereo Vision
VU Stereo Vision (3.0 ECTS/2.0 WS)
Anrechenbarkeit:
• Wahlfach im Masterstudium “Computergraphik & Digitale
Bildverarbeitung”
• Wahlfach im Masterstudium “Medieninformatik”
Webseite der LVA:
• http://www.ims.tuwien.ac.at/teaching_detail.php?ims_id=188.HQK
VU Stereo Vision (3.0 ECTS/2.0 WS)
Vorlesungstermine (9 Einheiten):
• Fr 5. März (10.00-11.30)
• Fr 12. März (10.00-11.30)
• Fr 19. März (10.00-11.30)
• Fr 26. März (10.00-11.30)
• Fr 16. April (10.00-11.30)
• Fr 23. April (10.00-11.30)
• Mi 28. April (10.00-11.30)
• Fr 7. Mai (10.00-11.30)
• Mi 12. Mai (10.00-11.30)
• Mündliche Prüfung nach Vereinbarung
Ort:
• Seminarraum 188/2
Topics Covered in the Lecture (Preliminary List)
Basics:
• 3D Perception, Stereo Matching Problem, Applications
• Stereo Pipeline, Challenges in Stereo Matching, Middlebury benchmark
Local Stereo Methods:
• Adaptive Window Methods
Global Stereo Matching:
• Global optimization:
− Dynamic Programming, Belief Propagation, Move-Making Algorithms
• Stereo Models:
− Smoothness Priors, Occlusion Handling
The Data Term
• Sampling insensitive measures, Role of color, Illumination-invariant measures.
Segmentation-Based Stereo
Recent work at IMS
Homework
1. Step:
• You will implement block matching for stereo.
2. Step:
• You will make the algorithm of step 1 computationally fast (sliding
window technique)
• Reflects the first branch of stereo research: real-time matching
3. Step:
• You will improve the algorithm of step 1 to deliver high-quality
results.
• Competition:
− What is your ranking in the Middlebury benchmark?
• It is up to you what tricks of this lecture you are going to
use.
• Reflects the second branch of stereo research: high-quality
(but slow) matching
3D Perception
Michael Bleyer
LVA Stereo Vision
3D Perception
Human-Eye Brain
Separation(~6.5cm)
Left 2D Image Right 2D Image 3D View

3D Perception
Human-Eye Brain
If we ensure that the left eye sees a
Separation(~6.5cm)
2D image and the right eye sees
another one, our brain will try to
overlay the images to generate a 3D
impression.
How can we use this for watching
3D movies?
Anaglyphs
Two images of complementary
color are overlaid to generate
one image.
Glasses required (e.g.
red/green)
Red filter cancels out red
image component, green filter
cancels out green component (Anaglyph Image)
Each eye gets one image =>
3D impression
Current 3D cinemas use this
principle. However,
polarization filters are used (Red/Green Glasses)
instead of color filters.
Shutter Glasses
Display flickers between left
and right image (i.e. each
even frame shows left image,
each odd frame shows right
image)
(Shutter Glasses and 120 Hz Display)
When left frame is shown,
shutter glasses close right
eye and vice versa.
Requires new displays of
high frame rate (120Hz).
Currently pushed by Nvidea
to address gaming market.
(Nvidea Artwork)
Autostereoscopic Displays
No glasses required!
Matrix of many transparent lenses
put on the display.
Lenses distort pixels so that left eye
gets a left image and right eye gets
a right image (if you are standing in
a sweet spot) => 3D impression
Novel viewpoint capability:
• You can walk in front of the display and
get a perceptively correct depth
impression depending on your current
viewpoint. (Philips Wowvx Display)
You will get a demo soon
Free Viewing (No glasses required, but some practice)
The way how you usually look at the display (no 3D):
Parallel Viewing:
Left Image Right Image
Cross Eye Viewing:
Right Image Left Image
Most likely the simpler method.

Learning Cross Eye Viewing
Right 2D Image Left 2D Image

Take a pencil and hold it in the middle of your eyes.
Look at the pencil and slowly change its distance to your eyes
If you found the right distance, you see a third image inbetween
left and right images.
This third image is in 3D
Practise, it is worth the effort.
3D on YouTube
Computational
Stereo
Michael Bleyer
LVA Stereo Vision
Computational Stereo
Brain
Replace human eyes with a pair of slightly
displaced cameras.

Displacement Brain
(Stereo Baseline)
Replace human eyes with a pair of slightly
displaced cameras.

Displacement Brain
(Stereo Baseline)

Displacement Computer
(Stereo Baseline)

(Stereo Baseline)

(Stereo Baseline)
How can we accomplish a fully
automatic 2D to 3D conversion?

What is Disparity?
The amount to which a single pixel is displaced in the

two images is called disparity.
A pixel’s disparity is inversely proportional to its depth in
the scene.
What is Disparity?
Background
Disparity
(Small)

the scene.
What is Disparity?
Foreground
Disparity (Large)

the scene.
Disparity Encoding
The disparity of each pixel is encoded by a grey value.

High grey values represent high disparities (and low
gray values small disparities).
The resulting image is called disparity map.
Disparity and Depth
The disparity map contains sufficient information for

generating a 3D model.
Disparity and Depth
The challenging part is to compute
the disparity map.
This task is known as the stereo
matching problem.
Stereo matching will be the topic of
this lecture!!!
The disparity map contains sufficient information for
generating a 3D model.
Applications
(just a few examples)
3D Reconstruction from aerial images
Stereo cameras are mounted on an airplane to obtain a

terrain map.
Images taken from http://www.robotic.de/Heiko.Hirschmueller/
3D Reconstruction of Cities
City of Dubrovnik reconstructed from images taken from

Flickr in a fully automatic way.
[S. Agarwal, N. Snavely, I. Simon, S. Seitz and R. Szeliski “Building Rome in a Day”,
ICCV, 2009]
Driver Assistance / Autonomous driving cars
For example, use stereo to measure distance to other

cars.
DARPA Grand Challenge
Image taken from http://www.cs.auckland.ac.nz/~rklette/talks/08_AI.pdf
The Mars Rover
Reconstruct the surface of Mars using stereo vision

Human Motion Capture
Fit a 3D model of the human body to the computed

point cloud.
[R. Plänkers and P. Fua, “Articulated Soft Objects for Multiview Shape and Motion
Capture”, PAMI, 2003]
Bilayer Segmentation – Z-Keying
Goal: Divide image into a foreground and a background region.

Simple background subtraction will fail if there is motion in the background.
Solution:
• Compute depth map
• If the depth of a pixel is larger than a predefined threshold, pixels belongs to the foreground
[A. Criminisi, G. Cross, A. Blake and V. Kolmogorov, “Bilayer Segmentation
of Live Video”, CVPR, 2006]
Novel View Generation
Left View Virtual Interpolated Right View

(recorded) View (not recorded) (recorded)
Given a 3D model of the scene, one can use a virtual camera to

record new views from arbitrary viewpoints.
For example: Freeze frame effect from the movie Matrix.
[L. Zitnick, S. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, "High-quality video
view interpolation using a layered representation", SIGGRAPH, 2004]
Understanding Human Vision
If we can teach the computer to see in 3D, we can also
learn more about the way how the human perceives
depth.
Summary
3D Perception
• Principle of human 3D vision
• Ways for watching movies in 3D
• Stereo Matching Problem
Applications

Session 1

Uploaded by

Copyright:

Available Formats

Session 1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Session 1

Uploaded by

Copyright:

Available Formats

Introduction

Left 2D Image Right 2D Image 3D View

 Most likely the simpler method.

Right 2D Image Left 2D Image

Left 2D Image Right 2D Image 3D View

Left 2D Image Right 2D Image 3D View

Left 2D Image Right 2D Image 3D View

Left 2D Image Right 2D Image 3D View

Left 2D Image Right 2D Image 3D View

Left 2D Image Right 2D Image 3D View

 The amount to which a single pixel is displaced in the

 The amount to which a single pixel is displaced in the

 The amount to which a single pixel is displaced in the

 The disparity of each pixel is encoded by a grey value.

 The disparity map contains sufficient information for

 Stereo cameras are mounted on an airplane to obtain a

 City of Dubrovnik reconstructed from images taken from

 For example, use stereo to measure distance to other

 Reconstruct the surface of Mars using stereo vision

 Fit a 3D model of the human body to the computed

 Goal: Divide image into a foreground and a background region.

Left View Virtual Interpolated Right View

 Given a 3D model of the scene, one can use a virtual camera to

You might also like

Most likely the simpler method.

The amount to which a single pixel is displaced in the

The amount to which a single pixel is displaced in the

The amount to which a single pixel is displaced in the

The disparity of each pixel is encoded by a grey value.

The disparity map contains sufficient information for

Stereo cameras are mounted on an airplane to obtain a

City of Dubrovnik reconstructed from images taken from

For example, use stereo to measure distance to other

Reconstruct the surface of Mars using stereo vision

Fit a 3D model of the human body to the computed

Goal: Divide image into a foreground and a background region.

Given a 3D model of the scene, one can use a virtual camera to