Scribd Logo
Upload

Welcome to Scribd!

  • 49 views

    Image and Depth From A Conventional Camera With A Coded Aperture

    Uploaded by

    NA Karthik
    Using a single conventional camera with a coded aperture, this document describes a method to obtain both a depth map and an all-focused image from a single input image. A coded aperture pattern is placed inside the camera lens to make defocus patterns easier to discriminate at different depths compared to a conventional aperture. An image prior that favors sparse image gradients is used to regularize the deconvolution process. Experimental results demonstrate depth estimation and the ability to digitally refocus an image after the fact. While depth accuracy is limited, this technique provides a low-cost way to obtain depth information from a standard camera without sacrificing image resolution.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd

    Image and Depth From A Conventional Camera With A Coded Aperture

    Uploaded by

    NA Karthik
    49 views61 pages
    Using a single conventional camera with a coded aperture, this document describes a method to obtain both a depth map and an all-focused image from a single input image. A coded aperture pattern is placed inside the camera lens to make defocus patterns easier to discriminate at different depths compared to a conventional aperture. An image prior that favors sparse image gradients is used to regularize the deconvolution process. Experimental results demonstrate depth estimation and the ability to digitally refocus an image after the fact. While depth accuracy is limited, this technique provides a low-cost way to obtain depth information from a standard camera without sacrificing image resolution.

    Original Title

    CodedAperture-LevinEtAl-SIGGRAPH07.ppt

    Copyright

    © © All Rights Reserved

    Available Formats

    PPT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Using a single conventional camera with a coded aperture, this document describes a method to obtain both a depth map and an all-focused image from a single input image. A coded aperture pattern is placed inside the camera lens to make defocus patterns easier to discriminate at different depths compared to a conventional aperture. An image prior that favors sparse image gradients is used to regularize the deconvolution process. Experimental results demonstrate depth estimation and the ability to digitally refocus an image after the fact. While depth accuracy is limited, this technique provides a low-cost way to obtain depth information from a standard camera without sacrificing image resolution.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd
    Download as ppt, pdf, or txt
    49 views61 pages

    Image and Depth From A Conventional Camera With A Coded Aperture

    Uploaded by

    NA Karthik
    Using a single conventional camera with a coded aperture, this document describes a method to obtain both a depth map and an all-focused image from a single input image. A coded aperture pattern is placed inside the camera lens to make defocus patterns easier to discriminate at different depths compared to a conventional aperture. An image prior that favors sparse image gradients is used to regularize the deconvolution process. Experimental results demonstrate depth estimation and the ability to digitally refocus an image after the fact. While depth accuracy is limited, this technique provides a low-cost way to obtain depth information from a standard camera without sacrificing image resolution.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd
    Download as ppt, pdf, or txt
    You are on page 1of 61

    Image and Depth from a

    Conventional Camera with a


    Coded Aperture
    Anat Levin, Rob Fergus,
    Frdo Durand, William Freeman
    MIT CSAIL

    Output #1: Depth map

    Single input image:

    Output #1: Depth map

    Single input image:

    Output #2: All-focused image

    Lens and defocus


    Image of a point
    light source

    Lens aperture

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Lens and defocus


    Image of a
    defocused point
    light source

    Lens aperture

    Object

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Lens and defocus


    Image of a
    defocused point
    light source

    Lens aperture

    Object

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Lens and defocus


    Image of a
    defocused point
    light source

    Lens aperture

    Object

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Lens and defocus


    Image of a
    defocused point
    light source

    Lens aperture

    Object

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Depth and defocus


    Out of focus

    ill
    po
    s
    In focus

    Depth from defocus:

    ed

    Infer depth by analyzing


    local scale of defocus blur

    Challenges
    Hard to discriminate a smooth scene from defocus blur

    Out of focus
    Hard to undo defocus blur

    Input

    Ringing with conventional


    deblurring algorithm

    Key contributions
    Exploit prior on natural images
    - Improve deconvolution
    - Improve depth discrimination

    Natural

    Coded

    aperture (mask inside lens)

    - make defocus patterns different from


    natural images and easier to discriminate

    Unnatural

    Related Work
    Depth from (de)focus
    e.g. Pentland, Chaudhuri, Favaro et al.

    Plenoptic/ light field cameras


    e.g. Adelson and Wang, Ng et al.

    Wave front coding


    e.g. Cathey & Dowski

    Coded apertures for light gathering:


    e.g. Fenimore and Cannon

    Blind Deconvolution
    e.g. Kundur and Hatzinakos , Fergus et al, Levin

    Never recover both depth AND full resolution image from a single image
    Except: Veeraraghavan, Raskar, Agrawal, Mohan, Tumblin SIGGRAPH07
    optimize debluring while we optimize depth discrimination

    Defocus as local convolution


    Calibrated blur kernels at
    different depths
    Input defocused
    image

    Defocus as local convolution

    yy f k x
    Input defocused
    image

    Local
    sub-window

    Calibrated
    blur kernels
    at depth k

    Sharp
    sub-window

    Depth k=1:

    y fk x

    Depth k=2:

    y fk x

    Depth k=3:

    y fk x

    Overview
    Try deconvolving local input windows with different scaled filters:




    Somehow: select best scale.

    ?
    ?
    ?

    Larger scale

    Correct scale

    Smaller scale

    Challenges
    Hard to deconvolve even
    when kernel is known
    Input

    Hard to identify
    correct scale:

    ?
    ?
    ?

    Ringing with the traditional


    Richardson-Lucy deconvolution
    algorithm

    Larger scale

    Correct scale

    Smaller scale

    Deconvolution is ill posed

    f x y

    Deconvolution is ill posed

    f x y
    Solution 1:

    Solution 2:

    Idea 1: Natural images prior


    What makes images special?
    Natural
    Image

    gradient

    Natural images have sparse gradients


    put a penalty on gradients

    Unnatural

    Deconvolution with prior

    | f x y | i (xi )

    x arg min

    Convolution error

    Derivatives prior

    +
    Low

    Equal convolution error

    +
    High

    Comparing deconvolution algorithms


    (Non blind) deconvolution code available online:
    http://groups.csail.mit.edu/graphics/CodedAperture/

    Input

    Richardson-Lucy

    (x) x

    (x) x

    0.8

    spread gradients

    localizes gradients

    Gaussian prior

    Sparse prior

    Comparing deconvolution algorithms


    (Non blind) deconvolution code available online:
    http://groups.csail.mit.edu/graphics/CodedAperture/

    Input

    Richardson-Lucy

    (x) x

    (x) x

    0.8

    spread gradients

    localizes gradients

    Gaussian prior

    Sparse prior

    Recall: Overview
    Try deconvolving local input windows with different scaled filters:

    Larger scale

    Correct scale

    Smaller scale

    ?
    ?
    ?

    Somehow: select best scale.


    Challenge: smaller scale not so different than correct

    Idea 2: Coded Aperture


    Mask (code) in aperture plane
    - make defocus patterns different from
    natural images and easier to discriminate

    Conventional
    aperture

    Our coded
    aperture

    Solution: lens with occluder

    Object

    Lens

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Solution: lens with occluder


    Image of a
    defocused point
    light source

    Aperture pattern

    Object

    Lens with coded


    aperture

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Solution: lens with occluder


    Image of a
    defocused point
    light source

    Aperture pattern

    Object

    Lens with coded


    aperture

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Solution: lens with occluder


    Image of a
    defocused point
    light source

    Aperture pattern

    Object

    Lens with coded


    aperture

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Solution: lens with occluder


    Image of a
    defocused point
    light source

    Aperture pattern

    Object

    Lens with coded


    aperture

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Solution: lens with occluder


    Image of a
    defocused point
    light source

    Aperture pattern

    Object

    Lens with coded


    aperture

    Camera
    sensor

    Point
    spread
    function
    Focal plane

    Why coded?
    Coded aperture- reduce uncertainty in scale identification
    Conventional
    Larger scale

    Correct scale

    Smaller scale

    Coded

    spectrum

    Sharp Image

    Filter, 2nd scale

    2nd observed image

    Frequency

    Frequency

    Frequency

    Frequency

    Frequency

    Spatial convolution

    spectrum

    spectrum

    Filter, 1st scale

    Frequency

    1st observed image

    spectrum

    spectrum

    Sharp Image

    spectrum

    Convolution- frequency domain representation

    Output spectrum has zeros


    frequency multiplication
    where filter spectrum has zeros

    spectrum

    Estimated image

    spectrum

    Filter, correct scale

    spectrum

    Estimated image

    spectrum

    Coded aperture: Scale estimation and division by zero

    Filter, wrong scale

    Frequency

    Observed image

    Frequency

    spectrum

    Division by zero

    spatial ringing

    Frequency

    Frequency

    Frequency

    spectrum

    Estimated image

    spectrum

    Filter, correct scale

    spectrum

    Estimated image

    spectrum

    Division by zero with a conventional aperture?

    Filter, wrong scale

    Frequency

    Observed image

    Frequency

    Frequency

    division of tiny value by zero


    no spatial ringing

    ?
    Frequency

    spectrum

    Frequency

    Filter Design
    Analytically search for a pattern maximizing discrimination
    between images at different defocus scales (KL-divergence)
    Account for image prior and physical constraints

    See paper for


    details

    Score
    More discrimination
    between scales

    Less discrimination
    between scales

    Sampled aperture patterns

    Conventional
    aperture

    Zero frequencies- pros and cons


    Previous talk:

    Our solution:

    Include zero frequencies:

    No zero frequencies:

    +
    -

    Filter can be easily inverted


    Weaker depth discrimination

    +
    +

    Zeros improve depth discrimination


    Inversion difficult
    Inversion made possible
    with image priors

    Depth results

    Regularizing depth estimation


    Try deblurring with 10 different aperture scales

    x arg min | f x y |2
    Convolution error

    i (xi )
    Derivatives prior

    Keep minimal error scale in each local window + regularization

    Input

    Local depth estimation

    Regularized depth

    Regularizing depth estimation

    Local depth estimation

    Input

    Regularized depth

    Sometimes, manual intervention

    Input

    Local depth estimation

    Regularized depth

    After user corrections

    All focused results

    Input

    All-focused
    (deconvolved)

    Close-up

    Original image

    All-focus image

    Input

    All-focused
    (deconvolved)

    Close-up

    Original image

    All-focus image

    Nave sharpening

    Comparison- conventional aperture result


    Ringing due to wrong scale estimation

    Comparison- coded aperture result

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Application: Digital refocusing from a single image

    Coded aperture: pros and cons


    AND depth at a single shot
    + Image
    + No loss of image resolution
    + Simple modification to lens
    - Depth is coarse

    unable to get depth at untextured areas,


    might need manual corrections.

    + But depth is a pure bonus


    - Loss some light
    + But deconvolution increases depth of field

    Deconvolution code available


    http://groups.csail.mit.edu/graphics/CodedAperture/

    50mm f/1.8:

    $79.95

    Cardboard:

    $1

    Tape:

    $1

    Depth acquisition:

    priceless

    You might also like