Novel View Synthesis

In this work, we proposed a novel depth adaptive tone mapping scheme for stereo HDR imaging and 3D display. We are interested in the case where different exposures are taken from different viewpoints. The scheme employed a new depth-adaptive cross-trilateral filter (DA-CTF) for recovering High Dynamic Range (HDR) images from multiple Low Dynamic Range (LDR) images captured at different exposure levels. Explicitly leveraging additional depth information in the tone mapping operation correctly identify global contrast change and detail visibility change by preserving the edges and reducing halo artifacts in the synthesized 3D views by depth-image-based rendering (DIBR) procedure. The experiments show that the proposed DA-CTF and DIBR scheme outper-forms state-of-the-art operators in the enhanced depiction of tone mapped HDR stereo images on LDR displays.

This paper presents a new hybrid Kinect-variety based synthesis scheme that renders artifacts-free multiple views for autostereoscopic/ auto-multiscopic displays. The proposed approach does not explicitly require dense scene depth information for synthesizing novel views from arbitrary viewpoints. Instead, the integrated framework first constructs a consistent minimal image-space parameterization of the underlying 3D scene. The compact representation of scene structure is formed using only implicit sparse depth information of few reference scene points extracted from raw RGB-D data. The views from arbitrary positions can be inferred by moving the novel camera in parameterized space by enforcing Euclidean constraints on reference scene images under a full-perspective projection model. Unlike state-of-the-art DIBR methods, where input depth map accuracy is crucial for high quality output, our proposed algorithm does not depend on precise per-pixel geometry information. Therefore, it simply sidesteps to recover and refine the incomplete or noisy depth estimates with advanced filling or upscaling techniques. Our approach performs fairly well in unconstrained indoor/outdoor environments, where the performance of range sensors or dense depth-based algorithms could be seriously affected due to scene complex geometric conditions. We demonstrate that the proposed hybrid scheme provides guarantees on the completeness, optimality with respect to the inter-view consistency of the algorithm. In the experimental validation, we performed a quantitative evaluation as well as subjective assessment on scene with complex geometric or surface properties. Comparison with latest representative DIBR methods is additionally performed to demonstrate the superior performance of the proposed scheme.

This paper presents a novel low-cost hybrid Kinect-variety based content generation scheme for 3DTV displays. The integrated framework constructs an efficient consistent image-space parameterization of 3D scene structure using only sparse depth information of few reference scene points. Under full-perspective camera model, the enforced Euclidean constraints simplify the synthesis of high quality novel multiview content for distinct camera motions. The algorithm does not rely on complete precise scene geometry information, and are unaffected by scene complex geometric properties, unconstrained environmental variations and illumination conditions. It, therefore, performs fairly well under a wider set of operation condition where the 3D range sensors fail or reliability of depth-based algorithms are suspect. The robust integration of vision algorithm and visual sensing scheme complement each other's shortcomings. It opens new opportunities for envisioning vision-sensing applications in uncontrolled environments. We demonstrate that proposed robust integration provides guarantees on the completeness and consistency of the algorithm. This leads to improved reliability on an extensive set of experimental results.

The demand for 3D TV systems is going high and technology is rapidly improving. High quality 3D content production is crucial for working on novel ways to show glasses-free 3D. For creating a truly immersive experience, it is essential to support advanced functionalities like free-viewpoint viewing of natural video. Other multi-media features which increase user interactivity with television content, like editing or mixing of scene components, virtual panning, tilting or zoom-in, a video featuring visual 3D effects as frozen movement, etc., must also be realized. It is also desirable for users to enjoy 3D vision with an increased field of view.

Supporting these requirements is one of the key issues, using only a limited number of
real cameras. Capturing arbitrary cameras using huge multi-camera rigs is tedious (and costly). It causes in flexibility in the acquisition of the natural environment. Although standardised Multi-view video plus depth (MVD) data format, allows to create virtual views using depth-image-based rendering (DIBR) in calibrated setting. But synthesizing a large number of intermediate views, for high-quality free-viewpoint video, is challenging. DIBR is known to suffer from inherent visibility and resampling problems. This resulted in appearance of artifacts like holes, cracks, corona, etc., in synthesized images. Existing techniques reduces rendering artifacts only from nearby viewpoints, but are inadequate to deal with arbitrary virtual camera movements. Besides, DIBR fails to provide multimedia functionalities such as graphical realism.

This thesis proposed novel architectures based on uncalibrated cameras for multi-view 3D TV systems that supports the above mentioned features. We rst proposed a signal representation that improves the content generation pipeline and the interactivity of DIBR. It is based on MVD which is being standardized by MPEG. We introduced novel DIBR algorithms based on a segmentation cue that address the rendering problems encountered in creating special effects and wide baseline extensions. We also proposed a 3D warping scheme for reduction of computation cost of DIBR rendering.

We designed another image-based system for multi-view 3D TV based on a full-perspective parameterized variety model. An ecient algebraic scheme is proposed that addresses the problem of characterizing the set of images of a 3D scene. The system is flexible to take the input from uncalibrated handheld cameras. Image-based parameterization of scene space allows to render high-quality virtual views from arbitrary viewpoints without 3D model and using few sample images. The proposed signal representation and free-viewpoint rendering method overcome major shortcomings of geometry-based methods, where performance is seriously affected due to scene complex conditions.

Further, we proposed a new high-quality multi-view rendering scheme for glasses-free
3D TV by integrating Kinect with the parameterized variety model. This scheme does not need dense depth, any hardware modification or active-passive depth imaging modalities to recover missing depth. In fact, it presents a new way to deal with the challenges inherent with DIBR and noisy depth estimates.

Towards the end, we proposed a new approach to 3D images that promises glasses-free multiple-perspective 3D. We answered an important question \How to represent the space of a broad class of perspective and non-perspective stereo varieties within a single, unified framework ?". We presented a representation that combines the geometric space of multiple uncalibrated perspective views with the appearance space of manifolds in a globally optimized way. Our approach works for uncalibrated static/dynamic sequences and render high-quality content for multi-perspective 3D TV. Finally, we addressed the challenges in creating good quality composite 3D for postproduction visual-effects. Our proposed DIBR compositing technique is regularized to handle large warps, vertical disparities, and stereo
baseline changes. Overall, all proposed schemes are backwards-compatible with
state-of-the-art MPEG tools and existing image-based rendering systems.

This paper presents a novel parameterized variety based view
synthesis scheme for 3DTV and multi-view systems. We have generalized the parameterized image variety approach to image based rendering proposed in [1] to handle full perspective cameras. An algebraic geometry framework is proposed for the parameterization of the variety associated with full perspective images, by image positions of three reference scene points. A complete parameterization of the 3D scene is constructed. This
allows to generate realistic novel views from arbitrary viewpoints without explicit 3D reconstruction, taking few multi-view images as input from uncalibrated cameras.

Another contribution of this paper is to provide a generalised and flexible architecture based on this variety model for multi-view 3DTV. The novelty of the architecture lies in merging this variety based approach with standard depth image based view synthesis pipeline, without explicitly obtaining sparse or dense 3D points. This integrated framework subsequently overcomes the problems associated with existing depth based representations. The key aspects of this joint framework are: 1) Synthesis of artifacts free novel views from arbitrary camera positions for wide angle viewing. 2) Generation of signal representation compatible with
standard multi-view systems. 3) Extraction of reliable view dependent depth maps from arbitrary virtual viewpoints without recovering exact 3D points. 4) Intuitive interface for virtual view specification based on scene content. Experimental results on standard multi-view sequences are presented to demonstrate the effectiveness of the proposed scheme.

This paper presents a novel image fusion scheme for view
synthesis based on a layered depth profile of the scene and
scale periodic transform. To create a layered depth profile of
the scene, we utilize the unique properties of scale transform
considering the problem of depth map computation from reference
images as a certain shift-variant problem. The problem
of depth computation is solved without deterministic stereo
correspondences or rather than representing image signals in
terms of shifts. Instead, we pose the problem of image signals
being representable as scale periodic function, and compute
appropriate depth estimates determining the scalings of a basis
function. The rendering process is formulated as a novel
image fusion in which the textures of all probable matching
points are adaptively determined, leveraging implicitly the geometric
information. The results demonstrate superiority of
the proposed approach in suppressing geometric, blurring or
flicker artifacts in rendered wide-baseline virtual videos.

This paper presents a new depth-image-based rendering algorithm
for free-viewpoint 3DTV applications. The cracks,
holes, ghost countors caused by visibility, disocclusion, resampling
problems associated with 3D warping lead to serious
rendering artifacts in synthesized virtual views. This challenging
problem of hole filling is formulated as an algebraic
matrix completion problem on a higher dimensional space of
monomial features described by a novel variety model. The
high-level idea of this work is to exploit the linear or nonlinear
structures of the data and interpolate missing values by solving
algebraic varieties associated with Hankel matrices as a
member of Krylov subspace. The proposed model effectively
handles artifacts appear in wide-baseline spatial view interpolation
and arbitrary camera movements. Our model has a
low runtime and results excel with state-of-the-art methods in
quantitative and qualitative evaluation.