A quasi-dense approach to surface reconstruction from uncalibrated images

LNMVSNet: A Low-Noise Multi-View Stereo Depth Inference Method for 3D Reconstruction

With the widespread adoption of modern RGB cameras, an abundance of RGB images is available everywhere. Therefore, multi-view stereo (MVS) 3D reconstruction has been extensively applied across various fields because of its cost-effectiveness and accessibility, which involves multi-view depth estimation and stereo matching algorithms. However, MVS tasks face noise challenges because of natural multiplicative noise and negative gain in algorithms, which reduce the quality and accuracy of the generated models and depth maps. Traditional MVS methods often struggle with noise, relying on assumptions that do not always hold true under real-world conditions, while deep learning-based MVS approaches tend to suffer from high noise sensitivity. To overcome these challenges, we introduce LNMVSNet, a deep learning network designed to enhance local feature attention and fuse features across different scales, aiming for low-noise, high-precision MVS 3D reconstruction. Through extensive evaluation of multiple benchmark datasets, LNMVSNet has demonstrated its superior performance, showcasing its ability to improve reconstruction accuracy and completeness, especially in the recovery of fine details and clear feature delineation. This advancement brings hope for the widespread application of MVS, ranging from precise industrial part inspection to the creation of immersive virtual environments.

A Light Multi-View Stereo Method with Patch-Uncertainty Awareness

Multi-view stereo methods utilize image sequences from different views to generate a 3D point cloud model of the scene. However, existing approaches often overlook coarse-stage features, impacting the final reconstruction accuracy. Moreover, using a fixed range for all the pixels during inverse depth sampling can adversely affect depth estimation. To address these challenges, we present a novel learning-based multi-view stereo method incorporating attention mechanisms and an adaptive depth sampling strategy. Firstly, we propose a lightweight, coarse-feature-enhanced feature pyramid network in the feature extraction stage, augmented by a coarse-feature-enhanced module. This module integrates features with channel and spatial attention, enriching the contextual features that are crucial for the initial depth estimation. Secondly, we introduce a novel patch-uncertainty-based depth sampling strategy for depth refinement, dynamically configuring depth sampling ranges within the GRU-based optimization process. Furthermore, we incorporate an edge detection operator to extract edge features from the reference image's feature map. These edge features are additionally integrated into the iterative cost volume construction, enhancing the reconstruction accuracy. Lastly, our method is rigorously evaluated on the DTU and Tanks and Temples benchmark datasets, revealing its low GPU memory consumption and competitive reconstruction quality compared to other learning-based MVS methods.

Multi-Scale Geometric Consistency Guided and Planar Prior Assisted Multi-View Stereo

In this paper, we propose some efficient multi-view stereo methods for accurate and complete depth map estimation. We first present our basic methods with Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection (ACMH & ACMH+). Based on our basic models, we develop two frameworks to deal with the depth estimation of ambiguous regions (especially low-textured areas) from two different perspectives: multi-scale information fusion and planar geometric clue assistance. For the former one, we propose a multi-scale geometric consistency guidance framework (ACMM) to obtain the reliable depth estimates for low-textured areas at coarser scales and guarantee that they can be propagated to finer scales. For the latter one, we propose a planar prior assisted framework (ACMP). We utilize a probabilistic graphical model to contribute a novel multi-view aggregated matching cost. At last, by taking advantage of the above frameworks, we further design a multi-scale geometric consistency guided and planar prior assisted multi-view stereo (ACMMP). This greatly enhances the discrimination of ambiguous regions and helps their depth sensing. Experiments on extensive datasets show our methods achieve state-of-the-art performance, recovering the depth estimation not only in low-textured areas but also in details. Related codes are available at https://github.com/GhiXu.

Miper-MVS: Multi-scale iterative probability estimation with refinement for efficient multi-view stereo

Multi-view stereo reconstruction aims to construct 3D scenes from multiple 2D images. In recent years, learning-based multi-view stereo methods have achieved significant results in depth estimation for multi-view stereo reconstruction. However, the current popular multi-stage processing method cannot solve the low-efficiency problem satisfactorily owing to the use of 3D convolution and still involves significant amounts of calculation. Therefore, to further balance the efficiency and generalization performance, this study proposed a multi-scale iterative probability estimation with refinement, which is a highly efficient method for multi-view stereo reconstruction. It comprises three main modules: 1) a high-precision probability estimator, dilated-LSTM that encodes the pixel probability distribution of depth in the hidden state, 2) an efficient interactive multi-scale update module that fully integrates multi-scale information and improves parallelism by interacting information between adjacent scales, and 3) a Pi-error Refinement module that converts the depth error between views into a grayscale error map and refines the edges of objects in the depth map. Simultaneously, we introduced a large amount of high-frequency information to ensure the accuracy of the refined edges. Among the most efficient methods (e.g., runtime and memory), the proposed method achieved the best generalization on the Tanks & Temples benchmarks. Additionally, the performance of the Miper-MVS was highly competitive in DTU benchmark. Our code is available at https://github.com/zhz120/Miper-MVS.

A Pose-Only Solution to Visual Reconstruction and Navigation

Visual navigation and three-dimensional (3D) scene reconstruction are essential for robotics to interact with the surrounding environment. Large-scale scenarios and computational robustness are great challenges facing the research community to achieve this goal. This paper raises a pose-only imaging geometry representation and algorithms that might help solve these challenges. The pose-only representation, equivalent to the classical multiple-view geometry, is discovered to be linearly related to camera global translations, which allows for efficient and robust camera motion estimation. As a result, the spatial feature coordinates can be analytically reconstructed and do not require nonlinear optimization. Comprehensive experiments demonstrate that the computational efficiency of recovering the scene and associated camera poses is significantly improved by 2-4 orders of magnitude.

SurRF: Unsupervised Multi-View Stereopsis by Learning Surface Radiance Field

The recent success in supervised multi-view stereopsis (MVS) relies on the onerously collected real-world 3D data. While the latest differentiable rendering techniques enable unsupervised MVS, they are restricted to discretized (e.g., point cloud) or implicit geometric representation, suffering from either low integrity for a textureless region or less geometric details for complex scenes. In this paper, we propose SurRF, an unsupervised MVS pipeline by learning Surface Radiance Field, i.e., a radiance field defined on a continuous and explicit 2D surface. Our key insight is that, in a local region, the explicit surface can be gradually deformed from a continuous initialization along view-dependent camera rays by differentiable rendering. That enables us to define the radiance field only on a 2D deformable surface rather than in a dense volume of 3D space, leading to compact representation while maintaining complete shape and realistic texture for large-scale complex scenes. We experimentally demonstrate that the proposed SurRF produces competitive results over the-state-of-the-art on various real-world challenging scenes, without any 3D supervision. Moreover, SurRF shows great potential in owning the joint advantages of mesh (scene manipulation), continuous surface (high geometric resolution), and radiance field (realistic rendering).

GigaMVS: A Benchmark for Ultra-Large-Scale Gigapixel-Level 3D Reconstruction

Multiview stereopsis (MVS) methods, which can reconstruct both the 3D geometry and texture from multiple images, have been rapidly developed and extensively investigated from the feature engineering methods to the data-driven ones. However, there is no dataset containing both the 3D geometry of large-scale scenes and high-resolution observations of small details to benchmark the algorithms. To this end, we present GigaMVS, the first gigapixel-image-based 3D reconstruction benchmark for ultra-large-scale scenes. The gigapixel images, with both wide field-of-view and high-resolution details, can clearly observe both the Palace-scale scene structure and Relievo-scale local details. The ground-truth geometry is captured by the laser scanner, which covers ultra-large-scale scenes with an average area of 8667 m ² and a maximum area of 32007 m ². Owing to the extremely large scale, complex occlusion, and gigapixel-level images, GigaMVS exposes problems that emerge from the poor scalability and efficiency of the existing MVS algorithms. We thoroughly investigate the state-of-the-art methods in terms of geometric and textural measurements, which point to the weakness of the existing methods and promising opportunities for future works. We believe that GigaMVS can benefit the community of 3D reconstruction and support the development of novel algorithms balancing robustness, scalability and accuracy.

PDE-Based 3D Surface Reconstruction from Multi-View 2D Images

Partial differential equation (PDE) based surfaces own a lot of advantages, compared to other types of 3D representation. For instance, fewer variables are required to represent the same 3D shape; the position, tangent, and even curvature continuity between PDE surface patches can be naturally maintained when certain conditions are satisfied, and the physics-based nature is also kept. Although some works applied implicit PDEs to 3D surface reconstruction from images, there is little work on exploiting the explicit solutions of PDE to this topic, which is more efficient and accurate. In this paper, we propose a new method to apply the explicit solutions of a fourth-order partial differential equation to surface reconstruction from multi-view images. The method includes two stages: point clouds data are extracted from multi-view images in the first stage, which is followed by PDE-based surface reconstruction from the obtained point clouds data. Our computational experiments show that the reconstructed PDE surfaces exhibit good quality and can recover the ground truth with high accuracy. A comparison between various solutions with different complexity to the fourth-order PDE is also made to demonstrate the power and flexibility of our proposed explicit PDE for surface reconstruction from images.

SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-View Stereopsis

Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the 'incompleteness' and the 'inaccuracy' problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting.

Visibility-Aware Point-Based Multi-View Stereo Network

We introduce VA-Point-MVSNet, a novel visibility-aware point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Furthermore, our visibility-aware multi-view feature aggregation allows the network to aggregate multi-view appearance cues while taking into account visibility. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. The code of VA-Point-MVSNet proposed in this work will be released at https://github.com/callmeray/PointMVSNet.

How to Build a 2D and 3D Aerial Multispectral Map?—All Steps Deeply Explained

The increased development of camera resolution, processing power, and aerial platforms helped to create more cost-efficient approaches to capture and generate point clouds to assist in scientific fields. The continuous development of methods to produce three-dimensional models based on two-dimensional images such as Structure from Motion (SfM) and Multi-View Stereopsis (MVS) allowed to improve the resolution of the produced models by a significant amount. By taking inspiration from the free and accessible workflow made available by OpenDroneMap, a detailed analysis of the processes is displayed in this paper. As of the writing of this paper, no literature was found that described in detail the necessary steps and processes that would allow the creation of digital models in two or three dimensions based on aerial images. With this, and based on the workflow of OpenDroneMap, a detailed study was performed. The digital model reconstruction process takes the initial aerial images obtained from the field survey and passes them through a series of stages. From each stage, a product is acquired and used for the following stage, for example, at the end of the initial stage a sparse reconstruction is produced, obtained by extracting features of the images and matching them, which is used in the following step, to increase its resolution. Additionally, from the analysis of the workflow, adaptations were made to the standard workflow in order to increase the compatibility of the developed system to different types of image sets. Particularly, adaptations focused on thermal imagery were made. Due to the low presence of strong features and therefore difficulty to match features across thermal images, a modification was implemented, so thermal models could be produced alongside the already implemented processes for multispectral and RGB image sets.

Parallel Structure from Motion for Sparse Point Cloud Generation in Large-Scale Scenes

Scene reconstruction uses images or videos as input to reconstruct a 3D model of a real scene and has important applications in smart cities, surveying and mapping, military, and other fields. Structure from motion (SFM) is a key step in scene reconstruction, which recovers sparse point clouds from image sequences. However, large-scale scenes cannot be reconstructed using a single compute node. Image matching and geometric filtering take up a lot of time in the traditional SFM problem. In this paper, we propose a novel divide-and-conquer framework to solve the distributed SFM problem. First, we use the global navigation satellite system (GNSS) information from images to calculate the GNSS neighborhood. The number of images matched is greatly reduced by matching each image to only valid GNSS neighbors. This way, a robust matching relationship can be obtained. Second, the calculated matching relationship is used as the initial camera graph, which is divided into multiple subgraphs by the clustering algorithm. The local SFM is executed on several computing nodes to register the local cameras. Finally, all of the local camera poses are integrated and optimized to complete the global camera registration. Experiments show that our system can accurately and efficiently solve the structure from motion problem in large-scale scenes.

Distributed Very Large Scale Bundle Adjustment by Global Camera Consensus

The increasing scale of Structure-from-Motion is fundamentally limited by the conventional optimization framework for the all-in-one global bundle adjustment. In this paper, we propose a distributed approach to coping with this global bundle adjustment for very large scale Structure-from-Motion computation. First, we derive the distributed formulation from the classical optimization algorithm ADMM, Alternating Direction Method of Multipliers, based on the global camera consensus. Then, we analyze the conditions under which the convergence of this distributed optimization would be guaranteed. In particular, we adopt over-relaxation and self-adaption schemes to improve the convergence rate. After that, we propose to split the large scale camera-point visibility graph in order to reduce the communication overheads of the distributed computing. The experiments on both public large scale SfM data-sets and our very large scale aerial photo sets demonstrate that the proposed distributed method clearly outperforms the state-of-the-art method in efficiency and accuracy.

Capturing the Geometry of Object Categories from Video Supervision

We propose an unsupervised method to learn the 3D geometry of object categories by looking around them. Differently from traditional approaches, this method does not require CAD models or manual supervision. Instead, using only video sequences showing object instances from a moving viewpoint, the method learns a deep neural network that can predict several aspects of the 3D geometry of such objects from single images. The network has three components. The first is a Siamese viewpoint factorization network that robustly aligns the input videos and learns to predict the absolute viewpoint of the object from a single image. The second is a depth estimation network that performs monocular depth prediction. The third is a shape completion network that predicts the full 3D shape of the object from the output of the monocular depth prediction module. While the three modules solve very different task, we show that they all benefit significantly from allowing networks to perform probabilistic predictions. This results in a self-assessment mechanism which is crucial for obtaining high quality predictions. Our network achieves state-of-the-art results on viewpoint prediction, depth estimation, and 3D point cloud estimation on public benchmarks.

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials

We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.

Object Segmentation Ensuring Consistency Across Multi-Viewpoint Images

We present a hybrid approach that segments an object by using both color and depth information obtained from views captured from a low-cost RGBD camera and sparsely-located color cameras. Our system begins with generating dense depth information of each target image by using Structure from Motion and Joint Bilateral Upsampling. We formulate the multi-view object segmentation as the Markov Random Field energy optimization on the graph constructed from the superpixels. To ensure inter-view consistency of the segmentation results between color images that have too few color features, our local mapping method generates dense inter-view geometric correspondences by using the dense depth images. Finally, the pixel-based optimization step refines the boundaries of the results obtained from the superpixel-based binary segmentation. We evaluate the validity of our method under various capture conditions such as numbers of views, rotations, and distances between cameras. We compared our method with the state-of-the-art methods that use the standard multi-view datasets. The comparison verified that the proposed method works very efficiently especially in a sparse wide-baseline capture environment.

CODE: Coherence Based Decision Boundaries for Feature Correspondence

A key challenge in feature correspondence is the difficulty in differentiating true and false matches at a local descriptor level. This forces adoption of strict similarity thresholds that discard many true matches. However, if analyzed at a global level, false matches are usually randomly scattered while true matches tend to be coherent (clustered around a few dominant motions), thus creating a coherence based separability constraint. This paper proposes a non-linear regression technique that can discover such a coherence based separability constraint from highly noisy matches and embed it into a correspondence likelihood model. Once computed, the model can filter the entire set of nearest neighbor matches (which typically contains over 90 percent false matches) for true matches. We integrate our technique into a full feature correspondence system which reliably generates large numbers of good quality correspondences over wide baselines where previous techniques provide few or no matches.

3D Human Model Reconstruction from Sparse Uncalibrated Views

Using a two-stage algorithm, the proposed technique can tackle the challenges of reconstructing high-quality 3D models of humans wearing regular clothes from sparse uncalibrated cameras. The proposed algorithm based on nonrigid dense correspondences (NRDC) requires fewer images than previous methods because it does not require an initial sparse matching. The authors validated the proposed algorithm using images from an existing dataset and images captured by a cell phone camera.

Improved Feature Matching for Mobile Devices with IMU

Thanks to the recent diffusion of low-cost high-resolution digital cameras and to the development of mostly automated procedures for image-based 3D reconstruction, the popularity of photogrammetry for environment surveys is constantly increasing in the last years. Automatic feature matching is an important step in order to successfully complete the photogrammetric 3D reconstruction: this step is the fundamental basis for the subsequent estimation of the geometry of the scene. This paper reconsiders the feature matching problem when dealing with smart mobile devices (e.g., when using the standard camera embedded in a smartphone as imaging sensor). More specifically, this paper aims at exploiting the information on camera movements provided by the inertial navigation system (INS) in order to make the feature matching step more robust and, possibly, computationally more efficient. First, a revised version of the affine scale-invariant feature transform (ASIFT) is considered: this version reduces the computational complexity of the original ASIFT, while still ensuring an increase of correct feature matches with respect to the SIFT. Furthermore, a new two-step procedure for the estimation of the essential matrix E (and the camera pose) is proposed in order to increase its estimation robustness and computational efficiency.

Performance analysis of different surface reconstruction algorithms for 3D reconstruction of outdoor objects from their digital images

3D reconstruction of geo-objects from their digital images is a time-efficient and convenient way of studying the structural features of the object being modelled. This paper presents a 3D reconstruction methodology which can be used to generate photo-realistic 3D watertight surface of different irregular shaped objects, from digital image sequences of the objects. The 3D reconstruction approach described here is robust, simplistic and can be readily used in reconstructing watertight 3D surface of any object from its digital image sequence. Here, digital images of different objects are used to build sparse, followed by dense 3D point clouds of the objects. These image-obtained point clouds are then used for generation of photo-realistic 3D surfaces, using different surface reconstruction algorithms such as Poisson reconstruction and Ball-pivoting algorithm. Different control parameters of these algorithms are identified, which affect the quality and computation time of the reconstructed 3D surface. The effects of these control parameters in generation of 3D surface from point clouds of different density are studied. It is shown that the reconstructed surface quality of Poisson reconstruction depends on Samples per node (SN) significantly, greater SN values resulting in better quality surfaces. Also, the quality of the 3D surface generated using Ball-Pivoting algorithm is found to be highly depend upon Clustering radius and Angle threshold values. The results obtained from this study give the readers of the article a valuable insight into the effects of different control parameters on determining the reconstructed surface quality.

Robust Point Sets Matching by Fusing Feature and Spatial Information Using Nonuniform Gaussian Mixture Models

Most of the traditional methods that handle the point sets matching between two images are based on local feature descriptors and the succedent mismatch eliminating strategies, which usually suffers from the sparsity of the initial match set because some correct ambiguous associations are easily filtered out by the ratio test of SIFT matching due to their second ranking in feature similarity. In this paper, we propose a nonuniform Gaussian mixture model (NGMM) for point sets matching between a pair of images which combines feature with position information of the local feature points extracted from the image pair to achieve point sets matching in a GMM framework. The proposed point set matching using an NGMM is able to change the correspondence assignments throughout the matching process and has the potential to match up even ambiguous matches correctly. The proposed NGMM framework can be either used to directly find matches between two point sets obtained from two images or applied to remove outliers in a match set. When finding matches, NGMM tries to learn a nonrigid transformation between the two point sets and provide a probability for every found match to measure the reliability of the match. Then, a probability threshold can be used to get the final robust match set. When removing outliers, NGMM requires that the vector field formed by the correct matches to be coherent and the matches contradicting the coherent vector field will be regarded as mismatches to be removed. A number of comparison and evaluation experiments reveal the good performance of the proposed NGMM framework in both finding matches and discarding mismatches.

Sparse Multi-View Consistency for Object Segmentation

Multiple view segmentation consists in segmenting objects simultaneously in several views. A key issue in that respect and compared to monocular settings is to ensure propagation of segmentation information between views while minimizing complexity and computational cost. In this work, we first investigate the idea that examining measurements at the projections of a sparse set of 3D points is sufficient to achieve this goal. The proposed algorithm softly assigns each of these 3D samples to the scene background if it projects on the background region in at least one view, or to the foreground if it projects on foreground region in all views. Second, we show how other modalities such as depth may be seamlessly integrated in the model and benefit the segmentation. The paper exposes a detailed set of experiments used to validate the algorithm, showing results comparable with the state of art, with reduced computational complexity. We also discuss the use of different modalities for specific situations, such as dealing with a low number of viewpoints or a scene with color ambiguities between foreground and background.

A quasi-dense matching approach and its calibration application with Internet photos

This paper proposes a quasi-dense matching approach to the automatic acquisition of camera parameters, which is required for recovering 3-D information from 2-D images. An affine transformation-based optimization model and a new matching cost function are used to acquire quasi-dense correspondences with high accuracy in each pair of views. These correspondences can be effectively detected and tracked at the sub-pixel level in multiviews with our neighboring view selection strategy. A two-layer iteration algorithm is proposed to optimize 3-D quasi-dense points and camera parameters. In the inner layer, different optimization strategies based on local photometric consistency and a global objective function are employed to optimize the 3-D quasi-dense points and camera parameters, respectively. In the outer layer, quasi-dense correspondences are resampled to guide a new estimation and optimization process of the camera parameters. We demonstrate the effectiveness of our algorithm with several experiments.

Characterization of living skin using multi-view stereo and isogeometric analysis

Skin is our interface with the outside world. In its natural environment, it displays unique mechanical characteristics, such as prestretch and growth. While there is a general agreement on the physiological importance of these features, they remain poorly characterized, mainly because they are difficult to access with standard laboratory techniques. Here we present a new, inexpensive technique to characterize living skin using multi-view stereo and isogeometric analysis. Based on easy-to-create hand-held camera images, we quantify prestretch, deformation and growth in a controlled porcine model of chronic skin expansion. Over a period of 5 weeks, we gradually inflate an implanted tissue expander, take weekly photographs of the experimental scene, reconstruct the geometry from a tattooed surface grid and create parametric representations of the skin surface. After 5 weeks of expansion, our method reveals an average area prestretch of 1.44, an average area stretch of 1.87 and an average area growth of 2.25. Area prestretch is maximal in the ventral region with a value of 2.37, whereas area stretch and area growth are maximal above the center of the expander, with values of 4.05 and 4.81, respectively. Our study has immediate impact on understanding living skin to optimize treatment planning and decision making in plastic and reconstructive surgery. Beyond these direct implications, our experimental design has broad applications in clinical research and basic sciences: it serves as a simple, robust, low cost, easy-to-use tool to reconstruct living membranes, which are difficult to characterize in a conventional laboratory setup.

Confronting passive and active sensors with non-Gaussian statistics

This paper has two motivations: firstly, to compare the Digital Surface Models (DSM) derived by passive (digital camera) and by active (terrestrial laser scanner) remote sensing systems when applied to specific architectural objects, and secondly, to test how well the Gaussian classic statistics, with its Least Squares principle, adapts to data sets where asymmetrical gross errors may appear and whether this approach should be changed for a non-parametric one. The field of geomatic technology automation is immersed in a high demanding competition in which any innovation by one of the contenders immediately challenges the opponents to propose a better improvement. Nowadays, we seem to be witnessing an improvement of terrestrial photogrammetry and its integration with computer vision to overcome the performance limitations of laser scanning methods. Through this contribution some of the issues of this “technological race” are examined from the point of view of photogrammetry. A new software is introduced and an experimental test is designed, performed and assessed to try to cast some light on this thrilling match. For the case considered in this study, the results show good agreement between both sensors, despite considerable asymmetry. This asymmetry suggests that the standard Normal parameters are not adequate to assess this type of data, especially when accuracy is of importance. In this case, standard deviation fails to provide a good estimation of the results, whereas the results obtained for the Median Absolute Deviation and for the Biweight Midvariance are more appropriate measures.

How to select good neighboring images in depth-map merging based 3D modeling

Depth-map merging based 3D modeling is an effective approach for reconstructing large-scale scenes from multiple images. In addition to generate high quality depth maps at each image, how to select suitable neighboring images for each image is also an important step in the reconstruction pipeline, unfortunately to which little attention has been paid in the literature until now. This paper is intended to tackle this issue for large scale scene reconstruction where many unordered images are captured and used with substantial varying scale and view-angle changes. We formulate the neighboring image selection as a combinatorial optimization problem and use the quantum-inspired evolutionary algorithm to seek its optimal solution. Experimental results on the ground truth data set show that our approach can significantly improve the quality of the depth-maps as well as final 3D reconstruction results with high computational efficiency.

Image-based modeling of unwrappable façades

In this paper, we propose an unwrappable representation for image-based façade modeling from multiple registered images. An unwrappable façade is represented by the mutually orthogonal baseline and profile. We first reconstruct semidense 3D points from images, then the baseline and profile are extracted from the point cloud to construct the base shape and compose the textures of the building from the images. Through our unwrapping process, the reconstructed 3D points and composed textures are further mapped to an unwrapped space that is parameterized by the baseline and profile. In doing so, the unwrapped space becomes equivalent to the planar space in which planar façade modeling techniques can be used to reconstruct the details of the buildings. Finally, the augmented details can be wrapped back to the original 3D space to generate the final model. This newly introduced unwrappable representation extends the state-of-the-art modeling for planar façades to a more general class of façades. We demonstrate the power of the unwrappable representation with a few examples in which the façade is not planar.

Global propagation of affine invariant features for robust matching

Local invariant features have been successfully used in image matching to cope with viewpoint change, partial occlusion, and clutters. However, when these factors become too strong, there will be a lot of mismatches due to the limited repeatability and discriminative power of features. In this paper, we present an efficient approach to remove the false matches and propagate the correct ones for the affine invariant features which represent the state-of-the-art local invariance. First, a pair-wise affine consistency measure is proposed to evaluate the consensus of the matches of affine invariant regions. The measure takes into account both the keypoint location and the region shape, size, and orientation. Based on this measure, a geometric filter is then presented which can efficiently remove the outliers from the initial matches, and is robust to severe clutters and non-rigid deformation. To increase the correct matches, we propose a global match refinement and propagation method that simultaneously finds a optimal group of local affine transforms to relate the features in two images. The global method is capable of producing a quasi-dense set of matches even for the weakly textured surfaces that suffer strong rigid transformation or non-rigid deformation. The strong capability of the proposed method in dealing with significant viewpoint change, non-rigid deformation, and low-texture objects is demonstrated in experiments of image matching, object recognition, and image based rendering.

Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes

In this paper, we propose a depth-map merging based multiple view stereo method for large-scale scenes which takes both accuracy and efficiency into account. In the proposed method, an efficient patch-based stereo matching process is used to generate depth-map at each image with acceptable errors, followed by a depth-map refinement process to enforce consistency over neighboring views. Compared to state-of-the-art methods, the proposed method can reconstruct quite accurate and dense point clouds with high computational efficiency. Besides, the proposed method could be easily parallelized at image level, i.e., each depth-map is computed individually, which makes it suitable for large-scale scene reconstruction with high resolution images. The accuracy and efficiency of the proposed method are evaluated quantitatively on benchmark data and qualitatively on large data sets.

A quantitative evaluation of confidence measures for stereo vision

We present an extensive evaluation of 17 confidence measures for stereo matching that compares the most widely used measures as well as several novel techniques proposed here. We begin by categorizing these methods according to which aspects of stereo cost estimation they take into account and then assess their strengths and weaknesses. The evaluation is conducted using a winner-take-all framework on binocular and multibaseline datasets with ground truth. It measures the capability of each confidence method to rank depth estimates according to their likelihood for being correct, to detect occluded pixels, and to generate low-error depth maps by selecting among multiple hypotheses for each pixel. Our work was motivated by the observation that such an evaluation is missing from the rapidly maturing stereo literature and that our findings would be helpful to researchers in binocular and multiview stereo.

High accuracy and visibility-consistent dense multiview stereo

Since the initial comparison of Seitz et al., the accuracy of dense multiview stereovision methods has been increasing steadily. A number of limitations, however, make most of these methods not suitable to outdoor scenes taken under uncontrolled imaging conditions. The present work consists of a complete dense multiview stereo pipeline which circumvents these limitations, being able to handle large-scale scenes without sacrificing accuracy. Highly detailed reconstructions are produced within very reasonable time thanks to two key stages in our pipeline: a minimum s-t cut optimization over an adaptive domain that robustly and efficiently filters a quasidense point cloud from outliers and reconstructs an initial surface by integrating visibility constraints, followed by a mesh-based variational refinement that captures small details, smartly handling photo-consistency, regularization, and adaptive resolution. The pipeline has been tested over a wide range of scenes: from classic compact objects taken in a laboratory setting, to outdoor architectural scenes, landscapes, and cultural heritage sites. The accuracy of its reconstructions has also been measured on the dense multiview benchmark proposed by Strecha et al., showing the results to compare more than favorably with the current state-of-the-art methods.

AN INTELLIGENT LEARNING APPROACH TO L-GRAMMAR EXTRACTION FROM IMAGE SEQUENCES OF REAL PLANTS

In this paper, we propose an automatic analyzing and transforming approach to L-system grammar extraction from real plants. Instead of using manually designed rules and cumbersome parameters, our method establishes the relationship between L-system grammars and the iterative trend of botanical entities, which reflect the endogenous factors that caused the plant branching process. To realize this goal, we use a digital camera to take multiple images of unfoliaged (leafless) plants and capture the topological and geometrical data of plant entities using image processing methods. The data then stored into specific data structures. A Hidden Markov based statistical model is then employed to reveal the hidden relations of plant entities which have been classified into categories based on their statistical properties extracted by a classic EM algorithm, the hidden relations have been integrated into the target L-system as grammars. Results show that our method is capable of automatically generating L-grammars for a given unfoliaged plant no matter what branching type it is belongs to.

GENERALIZATION OF THE DESARGUES THEOREM FOR SPARSE 3D RECONSTRUCTION

Visual perception for walking machines needs to handle more degrees of freedom than for wheeled robots. For humanoid, four- or six-legged robots, camera motion is 6D instead of 3D or planar motion. Classical 3D reconstruction methods cannot be applied directly, because explicit sensor motion is needed. In this paper, we propose an algorithm for 3D reconstruction of an unstructured environment using motion-free uncalibrated single camera. Computer vision techniques are employed to obtain an incremental geometrical reconstruction of the environment, therefore using vision as a sensor for robot control tasks like navigation, obstacle avoidance, manipulation, tracking, etc. and 3D model acquisition. The main contribution is that the offline 3D reconstruction problem is considered as a point trajectory search through the video stream. The algorithm takes into account the temporal aspect of the sequence of images in order to have an analytical expression of the geometrical locus of the point trajectories through the sequence of images. The approach is a generalization of the Desargues theorem applied to multiple views taken from nearby viewpoints. Experiments on both synthetic and real image sequences show the simplicity and efficiency of the proposed method. This method provides an alternative technical solution easy to use, flexible in the context of robotic applications and can significantly improve the 3D estimation accuracy.

Accurate, dense, and robust multiview stereopsis

This paper proposes a novel algorithm for multiview stereopsis that outputs a dense set of small rectangular patches covering the surfaces visible in the images. Stereopsis is implemented as a match, expand, and filter procedure, starting from a sparse set of matched keypoints, and repeatedly expanding these before using visibility constraints to filter away false matches. The keys to the performance of the proposed algorithm are effective techniques for enforcing local photometric consistency and global visibility constraints. Simple but effective methods are also proposed to turn the resulting patch model into a mesh which can be further refined by an algorithm that enforces both photometric consistency and regularization constraints. The proposed approach automatically detects and discards outliers and obstacles and does not require any initialization in the form of a visual hull, a bounding box, or valid depth ranges. We have tested our algorithm on various data sets including objects with fine surface details, deep concavities, and thin structures, outdoor scenes observed from a restricted set of viewpoints, and "crowded" scenes where moving obstacles appear in front of a static structure of interest. A quantitative evaluation on the Middlebury benchmark shows that the proposed method outperforms all others submitted so far for four out of the six data sets.

Location registration and recognition (LRR) for serial analysis of nodules in lung CT scans

In the clinical workflow for lung cancer management, the comparison of nodules between CT scans from subsequent visits by a patient is necessary for timely classification of pulmonary nodules into benign and malignant and for analyzing nodule growth and response to therapy. The algorithm described in this paper takes (a) two temporally-separated CT scans, I(1) and I(2), and (b) a series of nodule locations in I(1), and for each location it produces an affine transformation that maps the locations and their immediate neighborhoods from I(1) to I(2). It does this without deformable registration and without initialization by global affine registration. Requiring the nodule locations to be specified in only one volume provides the clinician more flexibility in investigating the condition of the lung. The algorithm uses a combination of feature extraction, indexing, refinement, and decision processes. Together, these processes essentially "recognize" the neighborhoods. We show on lung CT scans that our technique works at near interactive speed and that the median alignment error of 134 nodules is 1.70mm compared to the error 2.14mm of the Diffeomorphic Demons algorithm, and to the error 3.57mm of the global nodule registration with local refinement. We demonstrate on the alignment of 250 nodules, that the algorithm is robust to changes caused by cancer progression and differences in breathing states, scanning procedures, and patient positioning. Our algorithm may be used both for diagnosis and treatment monitoring of lung cancer. Because of the generic design of the algorithm, it might also be used in other applications that require fast and accurate mapping of regions.