Multi View Stereo

: A sequence of poses captured from eight video recordings of a capoeira turn kick. Our algorithm delivers spatio-temporally coherent geometry of the moving performer that captures both the time-varying surface detail as well as details in his motion very faithfully.

City environments often lack textured areas, contain repetitive structures, strong lighting changes and therefore are very difficult for standard 3D modeling pipelines. We present a novel unified framework for creating 3D city models which overcomes these difficulties by exploiting image segmentation cues as well as presence of dominant scene orientations and piecewise planar structures. Given panoramic street view sequences, we first demonstrate how to robustly estimate camera poses without a need of bundle adjustment and propose a multi-view stereo method which operates directly on panoramas, while enforcing the piecewise planarity constraints in the sweeping stage. At last, we propose a new depth fusion method which exploits the constraints of urban environments and combines advantages of volumetric and viewpoint based fusion methods. Our technique avoids expensive voxelization of space, operates directly on 3D reconstructed points through effective kd-tree representation, and obtains a final surface by tessellation of backprojections of those points into the reference image.

Digital Surface Model (DSM) can be generated from stereo pairs of satellite or aerial images. Among the most state-of-the-art matching algorithms, Semi-Global Matching (SGM) has widely been used for generating DSM from both satellite and aerial images. This paper presents an approach to improve the accuracy of DSM generated by SGM from multi-view satellite images using a novel technique including several filters. The filters are used for deleting mismatches between very tall buildings in urban areas and removing the sea regions. The technique, in contrast to the recent multi-view matching approaches, considers some of the points generated with only a pair of images in the final DSM. The approach is implemented on five sequential high resolution images acquired by the Worldview-2 satellite. The results are locally evaluated in shape and quantitative terms in comparison with commercial software to reveal the capability of the approach to generate a reliable and dense point cloud. Experiments show that the proposed method can achieve below half-pixel accuracy.

Automatic semantic segmentation of images is becoming a very prominent research field with many promising and reliable solutions already available. Labelled images as input for the photogrammetric pipeline have enormous potential to improve the 3D reconstruction results. To support this argument,  in this work we discuss the contribution of image semantic labelling towards image-based  3D reconstruction in photogrammetry. We experiment semantic information in various steps starting from feature matching to dense 3Dreconstruction. Labelling in 2D is considered as an easier task in terms of data availability and algorithm maturity. However, since semantic labelling of all the images involved in the reconstruction may be a costly, laborious and time-consuming task, we propose to use a deep learning architecture to automatically generate semantically segmented images. To this end, we have trained a Convolutional Neural Network (CNN) on historic building façade images that will be further enriched in the future. The first results of this study are promising, with improved performance on the quality of the 3D reconstruction and the possibility to transfer the labelling results from 2D to 3D.

Patch-based  stereo  is  nowadays  a  commonly  used image-based technique  for  dense  3D  reconstruction  in  large  scale  multi-view applications. The typical steps of such a pipeline can be summarized in stereo pair selection, depth map computation, depth map refinement and, finally, fusion in order to generate a complete and accurate representation of the scene in 3D. In this study, we aim to support the standard dense 3D reconstruction of scenes as implemented in the open source library OpenMVS by using semantic priors. To this end, during the depth map fusion step, along with the depth consistency check between depth maps of neighbouring views referring to the same part of the 3D scene, we impose extra semantic constraints in order to remove possible errors and selectively obtain segmented point clouds per label, boosting  automation towards this direction. I n order to reassure semantic coherence between neighbouring views, additional semantic criterions can be considered, aiming to elim inate mismatches of pixels belonging in different classes.

3D digitisation techniques, especially image-based three-dimensional (3D) surface recording is a cost-efficient and intuitive technique that quickly gained ground in archaeological documentation during the 21st century, mainly because its execution became much more straightforward during the past decade. As a result, many archaeological 3D documentation projects suddenly saw daylight. The majority of the resulting 3D models serves heritage conservation and restoration purposes or is used to extract metrical information or morphological characteristics of the object under study. Since one could also obtain most of these outcomes using more traditional approaches, the value of 3D surface documentation in terms of archaeological analysis and interpretation is still not entirely clear. This dissertation questions, therefore, the added value of digital 3D models for archaeological knowledge gaining. It explores if these 3D surfaces can open up new analytical pathways that would otherwise stay closed.

Since various archaeological sources usually require specific research methodologies, this dissertation solely focuses on vernacular dry-stone architecture found in the Slovene Kras region. Using twenty-two image-based 3D surface models of shepherd shelters dating to the Early Modern Era, the author tries to answer questions about the functionality and biography (i.e. construction and use) of these shelters, thereby demonstrating the (in)effectiveness of 3D documentation as a starting point for this type of archaeological analysis.

To unravel which properties best served the shelters’ function of protection against environmental factors and research how their use was affected by this suitability to protect, several analytical approaches were taken. All possible inner and outer dimensions of the 3D shelter models were measured, their volume of construction material and interior space calculated, various cross sections extracted and their surfaces checked for possible traces of usage wear. Besides, the influence of the environment (i.e. sunshine and rain) on these dry-stone constructions was virtually simulated, and 3D shepherd figures were digitally placed inside them. The newly obtained data made it clear that all shelters were performing their function virtually equally-well.

The level of usage of the shelters could only be inferred from the detailed digitised 3D surfaces. Not only was it possible to observe traces of wear on the surface of the digital shelters, but the latter also enabled the quantification of the erosion and compaction of the soil inside every shelter due to the repeated walking on its floor.

The orientation of the entrance was the only characteristic related to the shelters’ usage. Frequently used shelters feature a southwest entrance, which provides the best protection against the north-eastern burja wind, while additionally appearing favourable in the case of the incoming sunlight.

The 3D models also proved essential to estimate each shelter’s stone mass and compute their centre of gravity. Both pieces of derived data served as the foundation for determining the shelters’ temporal and metabolic building costs. With an estimated caloric expenditure of 660 kcal per building day, the construction of most shelters was not very demanding nor did it necessarily involve a change in the dietary intake of the shepherds. The exception to this are larger shelters, for which the construction took many person-days. In these cases, one can assume a slight increase in food intake or less working hours per day. However, the construction could also be carried out over a more extended period (i.e. during many non-consecutive days) or just by multiple builders.

Finally, the analytical potential of these digital shelter models has also been exhibited by populating them with 3D models of sitting or standing adult and child shepherds. In this way, the maximum number of shepherds that fit per shelter while not positioned with their back to each other was determined, thereby also simulating possible ways of communication.

Besides the potential to extract new data and deduce original insights from archaeological 3D surface models, their analytical usage also brings a whole set of specific problems and considerations along. Some of those can be tackled if one knows how the 3D surface model was generated (which means that the photo acquisition, model generation and processing stages should be carefully documented). It is, for instance, impossible to directly compute if these dry-stone shelters could keep their inhabitants dry, simply because image-based modelling is unable to digitise all the cavities between the stones. However, simulating surface water run-off to detect concave regions that would collect water – and thus make it more likely for the structure not to be waterproof – is still possible.

Moreover, data generated by a computer does not give these data an explicitly or inherently greater importance than their analogue counterparts, nor does the (potential for) high geometrical accuracy of 3D surface models render them ‘more objective’. Creating and analysing digital 3D surface models is still subject to many technical and interpretive decisions, which influence both the end-result and the newly derived insights. For example, calculating the volume of stone material is impossible without estimating the shelter’s boundaries. However, delineating shelters from their natural environment or shared dry-stone constructions (such as a wall) is highly interpretative.

3D documentation as a data generation tool for archaeological analysis and interpretation of vernacular architecture thus proved to be challenging and promising at the same time. Only time will tell if the analytical potential of such 3D surface data will be embraced and exploited by archaeologists to yield insights that are not obtainable using other means. The primary challenge still faced by many field archaeologists that deal with digital 3D surface data lies in intensive collaboration with geomatics professionals. Such collaborations would enable a better understanding of the data recording particularities and analytical possibilities offered by the latest 3D technologies. Moreover, the geomatics industry could develop the software tools that enable the analyses envisioned by the heritage professionals. At the same time, the geomatics professionals often resort to cultural heritage examples to merely showcase their innovations in 3D surface recording. Although this yields an enormous quantity of digitised geometry, such 3D surface records do not inherently possess any added value for archaeological research, nor do they help to solve other significant issues like data accessibility, metadata standardisation and photo acquisition protocols.

The problem that arises here is thus twofold. Archaeologists and other cultural heritage specialists often use digital 3D data in unimaginative or even inappropriate ways due to lack of technical understanding and interdisciplinary separation, while pertinent archaeological questions and heritage-specific problems usually get excluded from the 3D digitisation wave preached and practised by the geomatics industry. Breaking this vicious circle will take much patience and can only become a reality upon intensified cooperation (and not just communication!) between the two professions.

The “Convento de Cristo”, located in the city of Tomar, represents the best that has been produced in the history of Portuguese architecture. This monumental site, built between the 12th and 17th century, includes a Templar castle and a convent of the Order of Christ, and is classified as World Heritage by UNESCO since 1983.
In this paper we intend to discuss the potential and usability of the methods and tools of digital photogrammetry and terrestrial laser scanning (TLS) to the recording and production of base documentation as support to heritage interventions. A set of documentary case studies accomplished in this monumental site will be presented. In these case studies documentation for stratigraphical analysis prior to Rehabilitation and Restoration planning design, was produced regarding the knowledge of the historical and cultural evolution of the site. The archaeological recording was done at the level of deposits and walls. In the same context, documentation as support to Conservation interventions was also produced.
The following approaches were used: i) rectified photography as a low-cost and very practical method, ii) automatic three dimensional reconstruction from multiple images with the state-of-the-art structure-from-motion (SFM) software VSFM (Visual SFM), followed by a dense reconstruction with the multi-view-stereo software CMVS+PMVS, in a approach that minimized the processing time, maintaining quality of output, iii) TLS as a tool for geometric and radiometric survey, iv) image interpretation and processing of TLS reflectance images as a tool for the differentiation of materials and constructive systems and assessment of the conservation state. Three dimensional data enabled spatial comparison between the phases of excavation. The complimentary potential between the techniques of TLS and automatic photogrammetry was demonstrated by using low altitude aerial images (taken with helium balloon with remotely controlled digital camera) allowing a more comprehensive documentation, in particular in those situations where TLS presents limitations (higher parts of buildings). Interpretation and processing of TLS reflectance images was demonstrated to be a valid tool for the inquiry of the architectural object that allows one to extract relevant information both for Archaeology and Conservation. It was possible to distinguish between several types of joints (even to perceive early interventions done with cement mortars), to detect several constructive patterns through the analysis of the stone cuts, to calculate a vegetation index, and to differentiate between organic and inorganic materials.

This article presents a novel method for acquiring high-quality solid models of complex 3D shapes from multiple calibrated photographs. After the purely geometric constraints associated with the silhouettes found in each image have been used to construct a coarse surface approximation in the form of a visual hull, photoconsistency constraints are enforced in three consecutive steps: (1) the rims where the surface grazes the visual hull are first identified through dynamic programming; (2) with the rims now fixed, the visual hull is carved using graph cuts to globally optimize the photoconsistency of the surface and recover its main features; (3) an iterative (local) refinement step is finally used to recover fine surface details. The proposed approach has been implemented, and experiments with seven real data sets are presented, along with qualitative and quantitative comparisons with several state-of-the-art image-based-modeling algorithms.

Acquiring 3d shape from images is a classic problem in Computer Vision occupying researchers for at least 20 years. Only recently however have these ideas matured enough to provide highly accurate results. We present a complete algorithm to reconstruct 3d objects from images using the stereo correspondence cue. The technique can be described as a pipeline of four basic building blocks: camera calibration, image segmentation, photo-consistency estimation from images, and surface extraction from photo-consistency. In this chapter we will put more emphasis on the latter two: namely how to extract geometric information from a set of photographs without explicit camera visibility, and how to combine different geometry estimates in an optimal way. recovering the geometry just from photographs is not only a challenging problem but also, in the general case, an ill-posed problem. It is challenging because lighting and material properties play a very important role in the image formation model. The same geometry with a different material or different lighting conditions can give extremely different photographs. It is also an ill-posed problem because, in the general case, different combinations of geometry, lighting and material can produce exactly the same photographs, making it impossible to recover a single scene geometry. The main recipe to make the problem well-posed is to use priors on the types of surface that one expects. Traditionally the most common type of prior is the smooth surface prior. However when dealing with special classes of objects such as human faces or man-made objects, more evolved priors have been successfully used, e.g. human faces [54], buildings or planes .

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialize a multiview photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: First, we describe a robust technique to estimate light directions and intensities and, second, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and, hence, allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how, even in the case of highly textured objects, this technique can greatly improve on correspondence-based multiview stereo results.

The surface of most heritage objects holds important clues about their creation. To answer specific research questions about a 16 thcentury mural painting located in the Bischofstor of Vienna's St. Stephen's Cathedral, the three-dimensional (3D) geometry of the entire painted surface was digitised in minuscule detail using thousands of overlapping photographs. Although this article provides image acquisition and processing specifics, it aims to assess which image-based modelling workflow can achieve the most detailed, noise-free, two-and-a-half dimensional (2.5D) raster surface of this mural painting. Other than their full 3D counterparts and in contrast to the focus of most academic research, 2.5D raster surfaces are ideally suited for visualising and analysing sizeable, detailed surfaces. They are, therefore, still the preferred surface encoding of many heritage projects that want to leverage digital surface approximations to further heritage insights (and not just use them as mere eyecatchers). In the end, only a combination of different 2.5D rasters was able to accurately represent the variable surface of this mural painting with the right amount of spatial detail.

IET indexes its books and journal in SCOPUS and IEEE Xplore.
Computer Vision (CV) and Sensors play a decisive role in the operation of Unmanned Aerial Vehicle (UAV), but there exists a void when it comes to analysing the extent of their impact on the entire UAV system. In general, the fact that a UAV is a Cyber-Physical System (CPS) is not taken into account. In this proposal, we propose to expand on earlier books covering the use of CV and sensing in UAVs. Among other things, an entirely autonomous UAV can help to (i) obtain information about the environment, (ii) work for an extended period of time without human interference, (iii) move either all or part of itself all over its operating location devoid of human help and (iv) stay away from dangerous situations for people and their possessions. A Vision System (VS) entails the way CV data will be utilized, the appropriate architecture for total avionics integration, the control interfaces, and the UAV operation. Since the VS core is its sensors and cameras, multi-sensor fusion, navigation, hazard detection, and ground correlation in real time are important operational aspects that can benefit from CV knowledge and technology. This book will aim to collect and shed some light on the existing information on CV software and hardware for UAVs as well as pinpoint aspects that need additional thinking. It will list standards and a set of prerequisites (or lack of them thereof) when it comes to CV deployment in UAVs. The issue of data fusion takes a centre place when the book explores ways to deal with sensor data and images as well as their integration and display. The best practices to fuse image and sensor information to enhance UAV performance by means of CV can greatly improve all aspects of the corresponding CPS. The CPS viewpoint can improve the way UAVs interact with the Internet of Things (IoT), use cloud computing, meet communications requirements, implement hardware/software paradigms necessary to handle video streaming, incorporate satellite data, and combine CV with Virtual/Augmented Realities.

VOLUME 2-DEPLOYMENT AND APPLICATIONS: This tome introduces procedures, standards, and prerequisites for the deployment of Computer Vision (CV) in UAVs from their application point of view. It discusses existing/desirable open source software tools, image banks, benchmarks, Quality of Experience (QoE), Quality of Service (QoS) and how CV can benefit from a Robot Operating System (ROS) in surveillance, remote sensing, inspection, maintenance and repair among other usages, while offering an assessment of current bottlenecks and trends. It will pave the road towards better studies on the necessity and viability of implementing collaborative environments for visualization, knowledge management and teleoperation of UAVs. This is planned to be the companion volume of Estrela, Hemanth, Saotome (Eds) / Imaging and Sensing for Unmanned Aerial Vehicles: Volume 1-Control and Performance.

Editor(s):
Dr. Vania V. Estrela, https://www.linkedin.com/in/vania-v-estrela-96b9bb29/
Universidade Federal Fluminense (UFF), RJ, Brazil
[email protected]

Dr. Jude Hemanth,https://www.karunya.edu/ece/drjude.html
Karunya University, Coimbatore, India
[email protected]

Dr. Osamu Saotome, https://www.linkedin.com/in/osamu-saotome-83935818
Instituto Tecnológico de Aeronáutica, CTA-ITA-IEEA, São José dos Campos, SP, Brazil
[email protected]

CONTENTS:
1. Image Acquisition and Restoration in UAVs
2. Image Fusion in UAVs
3. Super-Resolution Imaging in UAVs
4. 2D/3D/4D Imaging in UAVs
5. Multi-view Image and ToF Sensor Fusion in UAVs
6. Range Imaging in UAVs
7. Multispectral and Hyperspectral Imaging in UAVs
8. Imaging Standards and UAVs
9. Virtual/Augmented Reality in UAVs
10. Collaborative Environments in UAVs
11. Archiving, Storage, and Compression in UAVs
12. Analysis, Indexing, Retrieval in UAVs
13. Multicast/Broadcast/Streaming in UAVs
14. Modelling, Simulation and UAVs
15.  Image-Oriented Estimation and Identification in UAVs
16. Open Source Software in UAVs
17. Image Banks and Benchmarks in UAVs
18. Quality of Experience (QoE) and Quality of Service (QoS) in UAVs
19. Robot Operating System (ROS) in UAVs
20. Cloud Computing in UAVs

Specification and Schedule

July 1st, 2017: Call for Chapter Abstracts
September 1, 2017: One-Page Chapter Abstract (up to 1000 words) Submission Deadline. Free style.
          A proposal must outline one of topics fromthe list above (mention its number, for instance 1. and reference PBCE120B).
November 30, 2017: Last Day for Notification of Acceptance
Jan 30, 2018: Full Chapter Submissions
March 30 2018: Review Chapter Submissions and send comments to authors
May 31, 2018: Receive revised Chapter Submissions
June 30, 2018: Notification of Final Acceptance
July 31, 2018: Gather all material, figure files and copyrights permission forms
Aug 30, 2018: Book editors to finalize introduction and conclusion chapters
Sept 15th, 2018, Delivery of full manuscript to the IET
Scheduled publication: Feb/March 2019

Readership: Graduate students and Researchers in the fields of Electrical and Computer Engineering, Computer Science, Mechanical Engineering, Civil Engineering, Humanitarian Engineering, Control Systems, Geoscience and Remote Sensing, Instrumentation and Measurement, Intelligent Transportation Systems, Oceanic Engineering, Safety Engineering, Reliability, Robotics and Automation, Signal Processing, Technology and Engineering Management, Environmental Engineering, Public Health Management, Non-Invasive Testing/Monitoring and Vehicular Technology.

Additional Information: Dr. Vania V. Estrela,    [email protected]
                                          Dr. Jude Hemanth,      [email protected]

We consider the problem of recovering 3D surface displacements using both shading and multi-view stereo cues. In contrast to traditional disparity or depth map representations, the object centered displacement map representation enables the recovery of complete 3D objects while also ensuring the reconstruction is not biased towards a particular image. Although displacement mapping requires a base surface, this base mesh is easily obtained using traditional computer vision techniques (e.g., shape-from-silhouette or structure-from-motion). Our method exploits shading variation due to object rotation relative to the light source, allowing the recovery of displacements in both textured and textureless regions in a common framework. In particular, shading cues are integrated into a multi-view stereo photoconsistency function through the surface normals that are implied by the displacement map. The analytic gradient of this photo-consistency function is used to drive a multiresolution conjugate gradient optimization. We demonstrate the geometric quality of the reconstructed displacements on several example objects including a human face.

The surface of most heritage objects holds important clues about their creation. To answer specific research questions about the creation of a mural painting located in the Bishop's Gate porch of St. Stephen's Cathedral in Vienna, the three-dimensional geometry of the entire painted surface was digitised in minuscule detail using thousands of overlapping photographs. After a brief introduction to the basics of image-based surface digitisation, this paper describes in detail how the photographs were taken and how they were processed to create a digital three-dimensional surface. To appreciate the subtle surface variations encoded in this digital model, various relief visualisation techniques have been applied. The paper ends with a discussion of the new insights into the creation of this painting that are enabled by these visualisations.

This paper presents an approach to render novel views from input photographs, a task which is commonly referred to as image based rendering. We first compute dense view dependent depthmaps using consistent segmentation. This method jointly computes multiview stereo and segments input photographs while accounting for mixed pixels (matting). We take the images with depth as our input and then propose two rendering algorithms to render novel views using the segmentation, both realtime and off-line. We demonstrate the results of our approach on a wide variety of scenes.

Skin Prestretch Growth Multi-view stereo Isogeometric analysis a b s t r a c t Skin is our interface with the outside world. In its natural environment, it displays unique mechanical characteristics, such as prestretch and growth. While there is a general agreement on the physiological importance of these features, they remain poorly characterized, mainly because they are difficult to access with standard laboratory techniques. Here we present a new, inexpensive technique to characterize living skin using multi-view stereo and isogeometric analysis. Based on easy-to-create hand-held camera images, we quantify prestretch, deformation and growth in a controlled porcine model of chronic skin expansion. Over a period of 5 weeks, we gradually inflate an implanted tissue expander, take weekly photographs of the experimental scene, reconstruct the geometry from a tattooed surface grid and create parametric representations of the skin surface. After 5 weeks of expansion, our method reveals an average area prestretch of 1.44, an average area stretch of 1.87 and an average area growth of 2.25. Area prestretch is maximal in the ventral region with a value of 2.37, whereas area stretch and area growth are maximal above the center of the expander, with values of 4.05 and 4.81, respectively. Our study has immediate impact on understanding living skin to optimize treatment planning and decision making in plastic and reconstructive surgery. Beyond these direct implications, our experimental design has broad applications in clinical research and basic sciences: it serves as a simple, robust, low cost, easy-to-use tool to reconstruct living membranes, which are difficult to characterize in a conventional laboratory setup.

We present a system for rendering novel viewpoints from a set of calibrated and silhouette-segmented images using the visual hull together with multi-view stereo. The visual hull predicted from the object silhouettes is used to restrict the search range of the multi-view stereo. This reduces redundant computation and the possibility of incorrect matches. Unlike previous visual hull approaches, we do

Gesture recognition has been implemented in many systems for different purposes. The way it is used can be different from system to system. Few used them as touch objects using sensors while few use them as pointing object where the camera is in front of the hand, but the location is being pointed by human can be detected only if the camera is behind that person. This paper presents an automatic system to detect the location on the wall where the user is currently pointing by his hand. This is done with the detection of head and both hands. The shoulders and elbows' locations are identified using geometry analysis to understand the current gesture of the human and finally the direction where user hand is pointing would be detected. The face was detected using Haar classifier and shoulders were estimated using the rectangle method near the head. This method does not work in poor illumination conditions and is made to detect only one person at a time. Also user is supposed to wear half sleeve or sleeveless shirt for the better segmentation. the input domain, the direct sensing approach requires capturing and interpreting the motion of head, eye gaze, face, hand, arms or even the whole body. This paper aims at finding the location of palms, shoulders and elbows. By doing this we will be aim to classify what is being pointed at.

Through examples in Convento de Cristo, it is discussed the contribution that graphical documentation produced through automatic photogrammetry, under SFM/MVS (structure-from-motion / multiview-stereo) methodology, and laser scanning, can represent to the enhancement of the recording processes of architectural heritage, originating a base documentation more reliable, and therefore potentiating later analysis processes that have to be done by conservation agents.
Através de aplicações ao Convento de Cristo, discute-se o contributo que a documentação gráfica produzida através da fotogrametria automática, sob a metodologia SFM/MVS (structure-from-motion / multi-view-stereo), e do varrimento laser 3D, pode dar para a melhoria dos processos de registo do património arquitectónico, dando origem a uma documentação base mais fiel ao objecto, e por isso potenciadora dos processos de interpretação e análise a efectuar pelos agentes da Conservação.

Abstract. In this paper an approach is presented that obtains virtual models from sequences of images. The system can deal with uncalibrated image sequences acquired with a hand held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through auto-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense ...

Multi-view stereo, novel view synthesis and high dynamic range (HDR) imaging are three pertinent areas of concern for high quality 3D view generation. This paper presents a novel parameterized variety based model that integrates these different domains into one common framework with an envisioned goal, to accommodate multi-view stereo for multiple exposure input views and to render photo-realistic HDR images from arbitrary virtual viewpoints for high quality 3D reconstruction. We extend the parameterized variety approach for rendering presented earlier by Genc and Ponce [1] to handle full perspective cameras. An efficient algebraic framework is proposed to construct an explicit parameterization of the space of all multi-view multi-exposed images. This characterization of differently exposed views allow to simultaneously recover artifacts free HDR images, and reliable depth maps from arbitrary camera viewpoints. High quality, HDR textured 3D model of the scene is obtained using these images and recovered geometry information.

The advent of high-resolution digital cameras and sophisticated multi-view stereo algorithms offers the promises of unprecedented geometric fidelity in image-based modeling tasks, but it also puts unprecedented demands on camera calibration to fulfill these promises. This paper presents a novel approach to camera calibration where top-down information from rough camera parameter estimates and the output of a publicly available multiview-stereo system [6] on scaled-down input images are used to effectively guide the search for additional image correspondences and significantly improve camera calibration parameters using a standard bundle adjustment algorithm [14]. The proposed method has been tested on several real datasets-including objects without salient features for which image correspondences cannot be found in a purely bottom-up fashion, and image-based modeling tasks-including the construction of visual hulls where thin structures are lost without our calibration procedure.

Estimation of plant canopy using low-altitude imagery can help monitor the normal growth status of crops and is highly beneficial for various digital farming applications such as precision crop protection. However, extracting 3D canopy information from raw images requires studying the effect of sensor viewing angle by taking into accounts the limitations of the mobile platform routes inside the field. The main objective of this research was to estimate wheat (Triticum aestivum L.) leaf parameters, including leaf length and width, from the 3D model representation of the plants. For this purpose, experiments with different camera viewing angles were conducted to find the optimum setup of a mono-camera system that would result in the best 3D point clouds. The angle-control analytical study was conducted on a four-row wheat plot with a row spacing of 0.17 m and with two seeding densities and growth stages as factors. Nadir and six oblique view image datasets were acquired from the plot with 88% overlapping and were then reconstructed to point clouds using Structure from Motion (SfM) and Multi-View Stereo (MVS) methods. Point clouds were first categorized into three classes as wheat canopy, soil background, and experimental plot. The wheat canopy class was then used to extract leaf parameters, which were then compared with those values from manual measurements. The comparison between results showed that (i) multiple-view dataset provided the best estimation for leaf length and leaf width, (ii) among the single-view dataset, canopy, and leaf parameters were best modeled with angles vertically at −45° and horizontally at 0° (VA −45, HA 0), while (iii) in nadir view, fewer underlying 3D points were obtained with a missing leaf rate of 70%. It was concluded that oblique imagery is a promising approach to effectively estimate wheat canopy 3D representation with SfM-MVS using a single camera platform for crop monitoring. This study contributes to the improvement of the proximal sensing platform for crop health assessment.

This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in general indoor or outdoor 3D scenes with potential static obstacles, using multiple calibrated video streams. Most shape-fromsilhouette techniques use a two-classification of space occupancy and silhouettes, based on image regions that match or disagree with a static background appearance model. Binary silhouette information becomes insufficient to unambiguously carve 3D space regions as the number and density of dynamic objects increases. In such difficult scenes, multi-view stereo methods suffer from visibility problems, and rely on color calibration procedures tedious to achieve outdoors. We propose a new algorithm to automatically detect and reconstruct scenes with a variable number of dynamic objects. Our formulation distinguishes between m different shapes in the scene by using automatically learnt view-specific appearance models, eliminating the color calibration requirement. Bayesian reasoning is then applied to solve the m-shape occupancy problem, with m updated as objects enter or leave the scene. Results show that this method yields multiple silhouette-based estimates that drastically improve scene reconstructions over traditional twolabel silhouette scene analysis. This enables the method to also efficiently deal with multi-person tracking problems.

We present a new formulation to multi-view stereo that treats the problem as probabilistic 3D segmentation. Previous work has used the stereo photo-consistency criterion as a detector of the boundary between the 3D scene and the surrounding empty space. Here we show how the same criterion can also provide a foreground/background model that can predict if a 3D location is inside or outside the scene. This model replaces the commonly used naive foreground model based on ballooning which is known to perform poorly in concavities. We demonstrate how the probabilistic visibility is linked to previous work on depth-map fusion and we present a multi-resolution graph-cut implementation using the new ballooning term that is very efficient both in terms of computation time and memory requirements.

: A sequence of poses captured from eight video recordings of a capoeira turn kick. Our algorithm delivers spatio-temporally coherent geometry of the moving performer that captures both the time-varying surface detail as well as details in his motion very faithfully.

In this paper we describe an experience undertaken in the Faculty of Architecture of Technical University of Lisbon, concerning the introduction of a 3D Digitization course in the frame of the PhD doctorate program in Architecture and in the frame of the Master’s programs in Architecture, Urbanism and Design.We start by describing the theoretical and instrumental frameworks proposed. Then we describe and discuss a set of two exercises developed during one semester, giving examples of the work produced by the students. Finally we end with some considerations to be taken into account in future editions of the course.

This paper addresses the problem of obtaining 3d detailed reconstructions of human faces in real-time and with inexpensive hardware. We present an algorithm based on a monocular multi-spectral photometric-stereo setup. This system is known to capture high-detailed deforming 3d surfaces at high frame rates and without having to use any expensive hardware or synchronized light stage. However, the main challenge of such a setup is the calibration stage, which depends on the lights setup and how they interact with the specific material being captured, in this case, human faces. For this purpose we develop a self-calibration technique where the person being captured is asked to perform a rigid motion in front of the camera, maintaining a neutral expression. Rigidity constrains are then used to compute the head's motion with a structure-from-motion algorithm. Once the motion is obtained, a multi-view stereo algorithm reconstructs a coarse 3d model of the face. This coarse model is then used to estimate the lighting parameters with a stratified approach: In the first step we use a RANSAC search to identify purely diffuse points on the face and to simultaneously estimate this diffuse reflectance model. In the second step we apply non-linear optimization to fit a non-Lambertian reflectance model to the outliers of the previous step. The calibration procedure is validated with synthetic and real data.

This paper presents a novel formulation for the multiview scene reconstruction problem. While this formulation benefits from a volumetric scene representation, it is amenable to a computationally tractable global optimisation using Graph-cuts. The algorithm proposed uses the visual hull of the scene to infer occlusions and as a constraint on the topology of the scene. A photo consistencybased surface cost functional is defined and discretised with a weighted graph. The optimal surface under this discretised functional is obtained as the minimum cut solution of the weighted graph. Our method provides a viewpoint independent surface regularisation, approximate handling of occlusions and a tractable optimisation scheme. Promising experimental results on real scenes as well as a quantitative evaluation on a synthetic scene are presented.

Boosted by the Middlebury challenge, the precision of dense multi-view stereovision methods has increased drastically in the past few years. Yet, most methods, although they perform well on this benchmark, are still inapplicable to large-scale data sets taken under uncontrolled conditions. In this paper, we propose a multi-view stereo pipeline able to deal at the same time with very large scenes while still producing highly detailed reconstructions within very reasonable time. The keys to these benefits are twofold: (i) a minimum s-t cut based global optimization that transforms a dense point cloud into a visibility consistent mesh, followed by (ii) a mesh-based variational refinement that captures small details, smartly handling photo-consistency, regularization and adaptive resolution. Our method has been tested on numerous large-scale outdoor scenes. The accuracy of our reconstructions is also measured on the recent dense multi-view benchmark proposed by Strecha et al., showing our results to compare more than favorably with the current state-of-the-art.

Occlusion is usually modelled in two images symmetrically in previous stereo algorithms which cannot work for multi-view stereo efficiently. In this paper, we present a novel formulation that handles occlusion using only one depth map in an asymmetrical way. Consequently, multiview information is efficiently accumulated to achieve high accuracy. The resulting energy function is complex and approximate graph cut based solutions are proposed. Our approach complements the theory and extends the applicability of using graph cut in stereo. The experiments demonstrate that the approach is comparable with the state of the art and potentially more efficient for multi-view stereo.

Figure 1: A sequence of poses captured from eight video recordings of a capoeira turn kick. Our algorithm delivers spatio-temporally coherent geometry of the moving performer that captures both the time-varying surface detail as well as details in his motion very faithfully.

This paper presents a real-time multi-view stereo algorithm, which is based on local winner-take-all optimization. When computing the disparity maps for a given view, the algorithm performs 3 steps: cost volume generation, cost volume merging, and disparity selection. The main focus of this paper is on the second step and a new cost volume merging method is proposed, which combines the adaptive weight and the Parzen window approaches. The proposed method can deal with noise and visibility problems effectively, even with poorly generated cost volumes as input. The experimental results demonstrate the validity of our presented approach.