Light Fields

The depth of field (DoF) effect is a useful tool in photography and cinematography because of its aesthetic value. However, capturing and displaying dynamic DoF effect were until recently a quality unique to expensive and bulky movie cameras. A computational approach to generate realistic DoF effects for mobile devices such as tablets is proposed. We first calibrate the rear-facing stereo cameras and rectify the stereo image pairs through FCam API, then generate a low-res disparity map using graph cuts stereo matching and subsequently upsample it via joint bilateral upsampling. Next, we generate a synthetic light field by warping the raw color image to nearby viewpoints, according to the corresponding values in the upsampled high-resolution disparity map. Finally, we render dynamic DoF effect on the tablet screen with light field rendering. The user can easily capture and generate desired DoF effects with arbitrary aperture sizes or focal depths using the tablet only, with no additional hardware or software required. The system has been examined in a variety of environments with satisfactory results, according to the subjective evaluation tests. 1 Introduction Dynamic depth of field (DoF) effect is a useful tool in photography and cinematography because of its aesthetic value. Capturing and displaying dynamic DoF effect were until recently a quality unique to expensive and bulky movie cameras. Problems such as radial distortion may also arise if the lens system is not setup properly. Recent advances in computational photography enable the user to refocus an image at any desired depth after it has been taken. The hand-held plenoptic camera 1 places a micro-lens array behind the main lens, so that each microlens image captures the scene from a slightly different viewpoint. By fusing these images together, one can generate photographs focusing at different depths. However, due to the spatial-angular tradeoff 2 of the light field camera, the resolution of the final rendered image is greatly reduced. To overcome this problem, Georgiev and Lumsdaine 3 introduced the focused plenoptic camera and significantly increased spatial resolution near the main lens focal plane. However, angular resolution is reduced and may introduce aliasing effects to the rendered image. Despite recent advances in computational light field imaging , the costs of plenoptic cameras are still high due to the complicated lens structures. Also, this complicated structure makes it difficult and expensive to integrate light field cameras into small hand-held devices like smartphones or tablets. Moreover, the huge amount of data generated by the plenop-tic camera prohibits it from performing light field rendering on video streams. To address this problem, we develop a light field rendering algorithm on mobile platforms. Because our algorithm works on regular stereo camera systems, it can be directly

With the improvement of both acquisition techniques, and computational and storage capabilities, we are witnessing an increasing
presence of multidimensional scene representations. Two-dimensional, conventional images are gradually losing their hegemony,
leaving room for novel formats. Among these, light fields are gaining importance, further propelled by the recent reappearance of
virtual reality. Content generation is one of the stumbling blocks in this realm, and light fields are one of the main input sources
of content. As their use becomes more common, a key challenge is the ability to edit or modify the appearance of the objects in
the light field. This paper presents a method for manipulating the appearance of gloss in light fields. In particular, we propose a
multidimensional filtering approach in which the specular highlights are filtered in the spatial and angular domains to target a
desired increase of the material roughness. The filtering kernel is computed based on surface normals and view direction. Our
technique generates angularly-coherent plausible edits in both synthetic and captured light fields.

: We retarget media by considering them as viscous objects whose local stiffness is related to local visual saliency. Object particles are modeled as mass-points connected by springs. Stiffer springs correspond to high-content regions which are preserved, while deformations are spread across pliable ones. Time consistency for videos is assured using strong connections between correspondent nodes in consequent frames planes. The whole method is implemented as an ODE system implemented on a GPU-based lattice providing real-time performances.

Global illumination in VR applications remains an elusive goal. While it potentially has a positive impact on presence, the significant real-time computation and integration complexities involved have been stumbling blocks. In this paper we present recent and ongoing work in the Virtual Light Field paradigm for global illumination as a solution to this problem. We discuss its suitability for real-time VR applications and detail recent work in integrating it with the XVR system for real-time GPU-based rendering in a CAVE TM . This rendering method achieves real-time rendering of L(S|D) * E solutions in time independent of illumination complexity and largely independent of geometric complexity.

Light field data records the amount of light at multiple points in space, captured e.g. by an array of cameras or by a light-field camera that uses microlenses. Since the storage and transmission requirements for such data are tremendous, compression techniques for light fields are gaining momentum in recent years. Although plenty of efficient compression formats do exist for still and moving images, only a little research on the impact of these methods on light field imagery is performed. In this paper, we evaluate the impact of state-of-the-art image and video compression methods on quality of images rendered from light field data. The methods include recent video compression standards, especially AV1 and XVC finalised in 2018. To fully exploit the potential of common image compression methods on four-dimensional light field imagery, we have extended these methods into three and four dimensions. In this paper, we show that the four-dimensional light field data can be compressed much more than independent still images while maintaining the same visual quality of a perceived picture. We gradually compare the compression performance of all image and video compression methods, and eventually answer the question, "What is the best compression method for light field data?".

We propose a novel approach that jointly removes reflection or translucent layer from a scene and estimates scene depth. The input data are captured via light field imaging. The problem is couched as minimizing the rank of the transmitted scene layer via Robust Principle Component Analysis (RPCA). We also impose regularization based on piece-wise smoothness, gradient sparsity, and layer independence to simultaneously recover 3D geometry of the transmitted layer. Experimental results on synthetic and real data show that our technique is robust and reliable, and can handle a broad range of challenging layer separation problems.

: The techniques in this paper employ two computer-assisted optical effects: synthetic aperture photography and synthetic aperture illumination. On the left, we aim a camera at an array of planar mirrors, yielding 22 different views of a statuette partially obscured by a plant. By rectifying, shifting, and adding these views together, we simulate a camera with a wide aperture and a shallow depth of field. Using appropriate shifts, we can position the focal plane of this synthetic camera astride the statuette, blurring out the plant. On the right we replace the camera with a video projector. By shifting, keystoning, and projecting multiple copies of a binary pattern, we produce a real image with a similarly shallow depth of field. Using appropriate shifts, we can position this image astride the statuette. On this plane the image is well focused; elsewhere, it is blurry.

Most image sensors are planar, opaque, and inflexible. We present a novel image sensor that is based on a luminescent concentrator (LC) film which absorbs light from a specific portion of the spectrum. The absorbed light is re-emitted at a lower frequency and transported to the edges of the LC by total internal reflection. The light transport is measured at the border of the film by line scan cameras. With these measurements, images that are focused onto the LC surface can be reconstructed. Thus, our image sensor is fully transparent, flexible, scalable and, due to its low cost, potentially disposable.

Human observers group local shading patterns into global super-patterns that appear to be illuminated in some unitary fashion. Many years ago, this was noticed for the case of uniform, unidirectional illumination. Recently, we found that it also applies to convergent and divergent illumination flows, but that human observers are blind to rotational light flow patterns (in the sense of being unable to group the local shading patterns). We now report that human observers are also blind to deformation patterns. This is perhaps interesting because convergent, divergent, rotational, and deformation patterns all occur in natural light fields. This is an idiosyncrasy of the human visual system, on par with the fact that visual awareness fails to present the observer with saddle shapes.

This paper presents a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. The proposed scheme learns stacked multiplicative layers from subsets of light field views determined from different scanning orders. The multiplicative layers are optimized using a fast data-driven convolutional neural network (CNN). The essential factor for multiplicative layers representation, which has not been considered in previous compression approaches, is the origin of redundancy, i.e., the low-rank structure of light field data. The spatial correlation in layer patterns is exploited with varying low ranks through factorization derived from singular value decomposition on a Krylov subspace. Further, encoding with HEVC efficiently removes intra-view and inter-view correlation in low-rank approximated layers. The initial subset of approximated decoded views from multiplicative representation is used to construct Fourier disparity layer (FDL) representation. The FDL model synthesizes the second subset of views identified by a pre-defined hierarchical prediction order. The correlations between the prediction residue of synthesized views are further eliminated by encoding the residual signal. The set of views obtained from decoding the residual is employed to refine the FDL model and predict the next view subsets with improved accuracy. This hierarchical procedure is repeated until all light field views are encoded. The critical advantage of the proposed hybrid layered representation and coding scheme is that it utilizes not just spatial and temporal redundancies but efficiently exploits the strong intrinsic similarities among neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. Besides, the scheme is flexible to realize a range of multiple bitrates at the decoder within a single integrated system. The compression performance analyzed with real light field shows substantial bitrate savings, maintaining good reconstruction quality.

We describe a unified representation of occluders in light transport and photography using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. Our key theoretical result is that shield field can be used to decouple the effects of occluders and incident illumination. We first describe the properties of shield fields in the frequency-domain and briefly analyze the "forward" problem of efficiently computing cast shadows. Afterwards, we apply the shield field signal-processing framework to make several new observations regarding the "inverse" problem of reconstructing 3D occluders from cast shadows-extending previous work on shape-from-silhouette and visual hull methods. From this analysis we develop the first singlecamera, single-shot approach to capture visual hulls without requiring moving or programmable illumination. We analyze several competing camera designs, ultimately leading to the development of a new large-format, mask-based light field camera that exploits optimal tiled-broadband codes for light-efficient shield field capture. We conclude by presenting a detailed experimental analysis of shield field capture and 3D occluder reconstruction.

Light field techniques allow the rendering of objects in time complexity unrelated to their geometric complexity. The technique discretely samples the space of light rays exiting the boundary around an object and then reconstructs a requested view from these data. In order to generate high quality images a dense sampling of the space is required which leads to large data sets. These data sets exhibit a high degree of coherence and should be compressed in order to make their size manageable.