High accuracy and visibility-consistent dense multiview stereo

A Comprehensive Review of Vision-Based 3D Reconstruction Methods

With the rapid development of 3D reconstruction, especially the emergence of algorithms such as NeRF and 3DGS, 3D reconstruction has become a popular research topic in recent years. 3D reconstruction technology provides crucial support for training extensive computer vision models and advancing the development of general artificial intelligence. With the development of deep learning and GPU technology, the demand for high-precision and high-efficiency 3D reconstruction information is increasing, especially in the fields of unmanned systems, human-computer interaction, virtual reality, and medicine. The rapid development of 3D reconstruction is becoming inevitable. This survey categorizes the various methods and technologies used in 3D reconstruction. It explores and classifies them based on three aspects: traditional static, dynamic, and machine learning. Furthermore, it compares and discusses these methods. At the end of the survey, which includes a detailed analysis of the trends and challenges in 3D reconstruction development, we aim to provide a comprehensive introduction for individuals who are currently engaged in or planning to conduct research on 3D reconstruction. Our goal is to help them gain a comprehensive understanding of the relevant knowledge related to 3D reconstruction.

A Parallel Method for Open Hole Filling in Large-Scale 3D Automatic Modeling Based on Oblique Photography

Common methods of filling open holes first reaggregate them into closed holes and then use a closed hole filling method to repair them. These methods have problems such as long calculation times, high memory consumption, and difficulties in filling large-area open holes. Hence, this paper proposes a parallel method for open hole filling in large-scale 3D automatic modeling. First, open holes are automatically identified and divided into two categories (internal and external). Second, the hierarchical relationships between the open holes are calculated in accordance with the adjacency relationships between partitioning cells, and the open holes are filled through propagation from the outer level to the inner level with topological closure and height projection transformation. Finally, the common boundaries between adjacent open holes are smoothed based on the Laplacian algorithm to achieve natural transitions between partitioning cells. Oblique photography data from an area of 28 km2 in Dongying, Shandong, were used for validation. The experimental results reveal the following: (i) Compared to the Han method, the proposed approach has a 12.4% higher filling success rate for internal open holes and increases the filling success rate for external open holes from 0% to 100%. (ii) Concerning filling efficiency, the Han method can achieve hole filling only in a small area, whereas with the proposed method, the size of the reconstruction area is not restricted. The time and memory consumption are improved by factors of approximately 4–5 and 7–21, respectively. (iii) In terms of filling accuracy, the two methods are basically the same.

Surface Reconstruction Assessment in Photogrammetric Applications

The image-based 3D reconstruction pipeline aims to generate complete digital representations of the recorded scene, often in the form of 3D surfaces. These surfaces or mesh models are required to be highly detailed as well as accurate enough, especially for metric applications. Surface generation can be considered as a problem integrated in the complete 3D reconstruction workflow and thus visibility information (pixel similarity and image orientation) is leveraged in the meshing procedure contributing to an optimal photo-consistent mesh. Other methods tackle the problem as an independent and subsequent step, generating a mesh model starting from a dense 3D point cloud or even using depth maps, discarding input image information. Out of the vast number of approaches for 3D surface generation, in this study, we considered three state of the art methods. Experiments were performed on benchmark and proprietary datasets of varying nature, scale, shape, image resolution and network designs. Several evaluation metrics were introduced and considered to present qualitative and quantitative assessment of the results.

Coral Reef Monitoring by Scuba Divers Using Underwater Photogrammetry and Geodetic Surveying

Underwater photogrammetry is increasingly being used by marine ecologists because of its ability to produce accurate, spatially detailed, non-destructive measurements of benthic communities, coupled with affordability and ease of use. However, independent quality control, rigorous imaging system set-up, optimal geometry design and a strict modeling of the imaging process are essential to achieving a high degree of measurable accuracy and resolution. If a proper photogrammetric approach that enables the formal description of the propagation of measurement error and modeling uncertainties is not undertaken, statements regarding the statistical significance of the results are limited. In this paper, we tackle these critical topics, based on the experience gained in the Moorea Island Digital Ecosystem Avatar (IDEA) project, where we have developed a rigorous underwater photogrammetric pipeline for coral reef monitoring and change detection. Here, we discuss the need for a permanent, underwater geodetic network, which serves to define a temporally stable reference datum and a check for the time series of photogrammetrically derived three-dimensional (3D) models of the reef structure. We present a methodology to evaluate the suitability of several underwater camera systems for photogrammetric and multi-temporal monitoring purposes and stress the importance of camera network geometry to minimize the deformations of photogrammetrically derived 3D reef models. Finally, we incorporate the measurement and modeling uncertainties of the full photogrammetric process into a simple and flexible framework for detecting statistically significant changes among a time series of models.

Using ALS Data to Improve Co-Registration of Photogrammetry-Based Point Cloud Data in Urban Areas

Globally, urban areas are rapidly expanding and high-quality remote sensing products are essential to help guide such development towards efficient and sustainable pathways. Here, we present an algorithm to address a common problem in digital aerial photogrammetry (DAP)-based image point clouds: vertical mis-registration. The algorithm uses the ground as inferred from airborne laser scanning (ALS) data as a reference surface and re-aligns individual point clouds to this surface. We demonstrate the effectiveness of the proposed method for the city of Kuopio, in central Finland. Here, we use the standard deviation of the vertical coordinate values as a measure of the mis-registration. We show that such standard deviation decreased substantially (more than 1.0 m) for a large proportion (23.2%) of the study area. Moreover, it was shown that the method performed better in urban and suburban areas, compared to vegetated areas (parks, forested areas, and so on). Hence, we demonstrate that the proposed algorithm is a simple and effective method to improve the quality and usability of DAP-based point clouds in urban areas.

High-Fidelity Monocular Face Reconstruction Based on an Unsupervised Model-Based Face Autoencoder

In this work, we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that serves as decoder. The core innovation is the differentiable parametric decoder that encapsulates image formation analytically based on a generative model. Our decoder takes as input a code vector with exactly defined semantic meaning that encodes detailed face pose, shape, expression, skin reflectance, and scene illumination. Due to this new way of combining CNN-based with model-based face reconstruction, the CNN-based encoder learns to extract semantically meaningful parameters from a single monocular input image. For the first time, a CNN encoder and an expert-designed generative model can be trained end-to-end in an unsupervised manner, which renders training on very large (unlabeled) real world datasets feasible. The obtained reconstructions compare favorably to current state-of-the-art approaches in terms of quality and richness of representation. This work is an extended version of [1] , where we additionally present a stochastic vertex sampling technique for faster training of our networks, and moreover, we propose and evaluate analysis-by-synthesis and shape-from-shading refinement approaches to achieve a high-fidelity reconstruction.

Multi-Variant Accuracy Evaluation of UAV Imaging Surveys: A Case Study on Investment Area

The main focus of the presented study is a multi-variant accuracy assessment of a photogrammetric 2D and 3D data collection, whose accuracy meets the appropriate technical requirements, based on the block of 858 digital images (4.6 cm ground sample distance) acquired by Trimble^® UX5 unmanned aircraft system equipped with Sony NEX-5T compact system camera. All 1418 well-defined ground control and check points were a posteriori measured applying Global Navigation Satellite Systems (GNSS) using the real-time network method. High accuracy of photogrammetric products was obtained by the computations performed according to the proposed methodology, which assumes multi-variant images processing and extended error analysis. The detection of blurred images was preprocessed applying Laplacian operator and Fourier transform implemented in Python using the Open Source Computer Vision library. The data collection was performed in Pix4Dmapper suite supported by additional software: in the bundle block adjustment (results verified using RealityCapure and PhotoScan applications), on the digital surface model (CloudCompare), and georeferenced orthomosaic in GeoTIFF format (AutoCAD Civil 3D). The study proved the high accuracy and significant statistical reliability of unmanned aerial vehicle (UAV) imaging 2D and 3D surveys. The accuracy fulfills Polish and US technical requirements of planimetric and vertical accuracy (root mean square error less than or equal to 0.10 m and 0.05 m).

Scalable point cloud meshing for image-based large-scale 3D modeling

Image-based 3D modeling is an effective method for reconstructing large-scale scenes, especially city-level scenarios. In the image-based modeling pipeline, obtaining a watertight mesh model from a noisy multi-view stereo point cloud is a key step toward ensuring model quality. However, some state-of-the-art methods rely on the global Delaunay-based optimization formed by all the points and cameras; thus, they encounter scaling problems when dealing with large scenes. To circumvent these limitations, this study proposes a scalable point-cloud meshing approach to aid the reconstruction of city-scale scenes with minimal time consumption and memory usage. Firstly, the entire scene is divided along the x and y axes into several overlapping chunks so that each chunk can satisfy the memory limit. Then, the Delaunay-based optimization is performed to extract meshes for each chunk in parallel. Finally, the local meshes are merged together by resolving local inconsistencies in the overlapping areas between the chunks. We test the proposed method on three city-scale scenes with hundreds of millions of points and thousands of images, and demonstrate its scalability, accuracy, and completeness, compared with the state-of-the-art methods.

Non-Rigid Vehicle-Borne LiDAR-Assisted Aerotriangulation

VLS (Vehicle-borne Laser Scanning) can easily scan the road surface in the close range with high density. UAV (Unmanned Aerial Vehicle) can capture a wider range of ground images. Due to the complementary features of platforms of VLS and UAV, combining the two methods becomes a more effective method of data acquisition. In this paper, a non-rigid method for the aerotriangulation of UAV images assisted by a vehicle-borne light detection and ranging (LiDAR) point cloud is proposed, which greatly reduces the number of control points and improves the automation. We convert the LiDAR point cloud-assisted aerotriangulation into a registration problem between two point clouds, which does not require complicated feature extraction and match between point cloud and images. Compared with the iterative closest point (ICP) algorithm, this method can address the non-rigid image distortion with a more rigorous adjustment model and a higher accuracy of aerotriangulation. The experimental results show that the constraint of the LiDAR point cloud ensures the high accuracy of the aerotriangulation, even in the absence of control points. The root-mean-square error (RMSE) of the checkpoints on the x, y, and z axes are 0.118 m, 0.163 m, and 0.084m, respectively, which verifies the reliability of the proposed method. As a necessary condition for joint mapping, the research based on VLS and UAV images in uncontrolled circumstances will greatly improve the efficiency of joint mapping and reduce its cost.

Detail Preserved Surface Reconstruction from Point Cloud

In this paper, we put forward a new method for surface reconstruction from image-based point clouds. In particular, we introduce a new visibility model for each line of sight to preserve scene details without decreasing the noise filtering ability. To make the proposed method suitable for point clouds with heavy noise, we introduce a new likelihood energy term to the total energy of the binary labeling problem of Delaunay tetrahedra, and we give its s-t graph implementation. Besides, we further improve the performance of the proposed method with the dense visibility technique, which helps to keep the object edge sharp. The experimental result shows that the proposed method rivalled the state-of-the-art methods in terms of accuracy and completeness, and performed better with reference to detail preservation.

Improving Details of Building Façades in Open LiDAR Data Using Ground Images

Recent open data initiatives allow free access to a vast amount of light detection and ranging (LiDAR) data in many cities. However, most open LiDAR data of cities are acquired by airborne scanning, where points on building façades are sparse or even completely missing due to occlusions in the urban environment, leading to the absence of façade details. This paper presents an approach for improving the LiDAR data coverage on building façades by using point cloud generated from ground images. A coarse-to-fine strategy is proposed to fuse these two-point clouds of different sources with very limited overlaps. First, the façade point cloud generated from ground images is leveled by adjusting the facade normal to perpendicular to the upright direction. Then leveling façade point cloud is geolocated by alignment between images GPS data and their structure from motion (SfM) coordinates. Next, a modified coherent point drift algorithm with (surface) normal consistency is proposed to accurately align the façade point cloud to the LiDAR data. The significance of this work resides in the use of 2D overlapping points on the building outlines instead of the limited 3D overlap between the two-point clouds. This way we can still achieve reliable and precise registration under incomplete coverage and ambiguous correspondence. Experiments show that the proposed approach can significantly improve the façade details in open LiDAR data, and achieve 2 to 10 times higher registration accuracy, when compared to classic registration methods.

Hierarchical Regularization of Building Boundaries in Noisy Aerial Laser Scanning and Photogrammetric Point Clouds

Aerial laser scanning or photogrammetric point clouds are often noisy at building boundaries. In order to produce regularized polygons from such noisy point clouds, this study proposes a hierarchical regularization method for the boundary points. Beginning with detected planar structures from raw point clouds, two stages of regularization are employed. In the first stage, the boundary points of an individual plane are consolidated locally by shifting them along their refined normal vector to resist noise, and then grouped into piecewise smooth segments. In the second stage, global regularities among different segments from different planes are softly enforced through a labeling process, in which the same label represents parallel or orthogonal segments. This is formulated as a Markov random field and solved efficiently via graph cut. The performance of the proposed method is evaluated for extracting 2D footprints and 3D polygons of buildings in metropolitan area. The results reveal that the proposed method is superior to the state-of-art methods both qualitatively and quantitatively in compactness. The simplified polygons could fit the original boundary points with an average residuals of 0.2 m, and in the meantime reduce up to 90% complexities of the edges. The satisfactory performances of the proposed method show a promising potential for 3D reconstruction of polygonal models from noisy point clouds.

Reconstruction of Cylindrical Surfaces Using Digital Image Correlation

A technique for the reconstruction of cylindrical surfaces using optical images with an extension of least squares matching is presented. This technique is based on stereo-image acquisition of a cylindrical object, and it involves displacing the camera following the object length. The basic concept behind this technique is that variations in the camera viewpoint over a cylindrical object produce perspective effects similar to a conic section in an image sequence. Such parallax changes are continuous and can be modelled by a second-order function, which is combined with an adaptive least squares matching (ALSM) for the 3D object reconstruction. Using this concept, a photogrammetric intersection with only two image patches can be used to model a cylindrical object with high accuracy. Experiments were conducted with a cylinder on a panel with coded targets to assess the 3D reconstruction accuracy. The accuracy assessment was based on a comparison between the estimated diameter and the diameter directly measured over the cylinder. The difference between the diameters indicated an accuracy of 1/10 mm, and the cylindrical surface was entirely reconstructed.

Fusion of Infrared Thermal Image and Visible Image for 3D Thermal Model Reconstruction Using Smartphone Sensors

Thermal infrared imagery provides temperature information on target objects, and has been widely applied in non-destructive testing. However, thermal infrared imagery is not always able to display detailed textures of inspected objects, which hampers the understanding of geometric entities consisting of temperature information. Although some commercial software has been developed for 3D thermal model displays, the software requires the use of expensive specific thermal infrared sensors. This study proposes a cost-effective method for 3D thermal model reconstruction based on image-based modeling. Two smart phones and a low-cost thermal infrared camera are employed to acquire visible images and thermal images, respectively, that are fused for 3D thermal model reconstruction. The experiment results demonstrate that the proposed method is able to effectively reconstruct a 3D thermal model which extremely approximates its corresponding entity. The total computational time for the 3D thermal model reconstruction is intensive while generating dense points required for the creation of a geometric entity. Future work will improve the efficiency of the proposed method in order to expand its potential applications to in-time monitoring.

An automated device for the digitization and 3D modelling of insects, combining extended-depth-of-field and all-side multi-view imaging

Digitization of natural history collections is a major challenge in archiving biodiversity. In recent years, several approaches have emerged, allowing either automated digitization, extended depth of field (EDOF) or multi-view imaging of insects. Here, we present DISC3D: a new digitization device for pinned insects and other small objects that combines all these aspects. A PC and a microcontroller board control the device. It features a sample holder on a motorized two-axis gimbal, allowing the specimens to be imaged from virtually any view. Ambient, mostly reflection-free illumination is ascertained by two LED-stripes circularly installed in two hemispherical white-coated domes (front-light and back-light). The device is equipped with an industrial camera and a compact macro lens, mounted on a motorized macro rail. EDOF images are calculated from an image stack using a novel calibrated scaling algorithm that meets the requirements of the pinhole camera model (a unique central perspective). The images can be used to generate a calibrated and real color texturized 3Dmodel by ‘structure from motion’ with a visibility consistent mesh generation. Such models are ideal for obtaining morphometric measurement data in 1D, 2D and 3D, thereby opening new opportunities for trait-based research in taxonomy, phylogeny, eco-physiology, and functional ecology.

Robust, Efficient Depth Reconstruction With Hierarchical Confidence-Based Matching

In recent years, taking photos and capturing videos with mobile devices have become increasingly popular. Emerging applications based on the depth reconstruction technique have been developed, such as Google lens blur. However, depth reconstruction is difficult due to occlusions, non-diffuse surfaces, repetitive patterns, and textureless surfaces, and it has become more difficult due to the unstable image quality and uncontrolled scene condition in the mobile setting. In this paper, we present a novel hierarchical framework with multi-view confidence-based matching for robust, efficient depth reconstruction in uncontrolled scenes. Particularly, the proposed framework combines local cost aggregation with global cost optimization in a complementary manner that increases efficiency and accuracy. A depth map is efficiently obtained in a coarse-to-fine manner by using an image pyramid. Moreover, confidence maps are computed to robustly fuse multi-view matching cues, and to constrain the stereo matching on a finer scale. The proposed framework has been evaluated with challenging indoor and outdoor scenes, and has achieved robust and efficient depth reconstruction.

Geometry Guided Multi-Scale Depth Map Fusion via Graph Optimization

In depth discontinuous and untextured regions, depth maps created by multiple view stereopsis are with heavy noises, but existing depth map fusion methods cannot handle it explicitly. To tackle the problem, two novel strategies are proposed: 1) a more discriminative fusion method, which is based on geometry consistency, measuring the consistency, and stability of surface geometry computed on both partial and global surfaces, different from traditional methods only using visibility consistency; 2) a graph optimization method which fuses pyramids of depth maps as mutual complementary information is available in different scales, and differs from existing multi-scale fusion methods. The method considers both sampling scale of a point and relations among points, and is proven to be solvable by graph cuts. Experimental results verify the superior performance of the proposed method to the traditional visibility consistency-based methods, and the proposed method is also compared favorably with a number of state-of-the-art methods. Moreover, the proposed method achieves the highest completeness among all the methods compared.

Three-dimensional measurement of small inner surface profiles using feature-based 3-D panoramic registration

Rapid development in the performance of sophisticated optical components, digital image sensors, and computer abilities along with decreasing costs has enabled three-dimensional (3-D) optical measurement to replace more traditional methods in manufacturing and quality control. The advantages of 3-D optical measurement, such as noncontact, high accuracy, rapid operation, and the ability for automation, are extremely valuable for inline manufacturing. However, most of the current optical approaches are eligible for exterior instead of internal surfaces of machined parts. A 3-D optical measurement approach is proposed based on machine vision for the 3-D profile measurement of tiny complex internal surfaces, such as internally threaded holes. To capture the full topographic extent (peak to valley) of threads, a side-view commercial rigid scope is used to collect images at known camera positions and orientations. A 3-D point cloud is generated with multiview stereo vision using linear motion of the test piece, which is repeated by a rotation to form additional point clouds. Registration of these point clouds into a complete reconstruction uses a proposed automated feature-based 3-D registration algorithm. The resulting 3-D reconstruction is compared with x-ray computed tomography to validate the feasibility of our proposed method for future robotically driven industrial 3-D inspection.

Estimating 3D Leaf and Stem Shape of Nursery Paprika Plants by a Novel Multi-Camera Photography System

For plant breeding and growth monitoring, accurate measurements of plant structure parameters are very crucial. We have, therefore, developed a high efficiency Multi-Camera Photography (MCP) system combining Multi-View Stereovision (MVS) with the Structure from Motion (SfM) algorithm. In this paper, we measured six variables of nursery paprika plants and investigated the accuracy of 3D models reconstructed from photos taken by four lens types at four different positions. The results demonstrated that error between the estimated and measured values was small, and the root-mean-square errors (RMSE) for leaf width/length and stem height/diameter were 1.65 mm (R² = 0.98) and 0.57 mm (R² = 0.99), respectively. The accuracies of the 3D model reconstruction of leaf and stem by a 28-mm lens at the first and third camera positions were the highest, and the number of reconstructed fine-scale 3D model shape surfaces of leaf and stem is the most. The results confirmed the practicability of our new method for the reconstruction of fine-scale plant model and accurate estimation of the plant parameters. They also displayed that our system is a good system for capturing high-resolution 3D images of nursery plants with high efficiency.

Image-based modeling of unwrappable façades

In this paper, we propose an unwrappable representation for image-based façade modeling from multiple registered images. An unwrappable façade is represented by the mutually orthogonal baseline and profile. We first reconstruct semidense 3D points from images, then the baseline and profile are extracted from the point cloud to construct the base shape and compose the textures of the building from the images. Through our unwrapping process, the reconstructed 3D points and composed textures are further mapped to an unwrapped space that is parameterized by the baseline and profile. In doing so, the unwrapped space becomes equivalent to the planar space in which planar façade modeling techniques can be used to reconstruct the details of the buildings. Finally, the augmented details can be wrapped back to the original 3D space to generate the final model. This newly introduced unwrappable representation extends the state-of-the-art modeling for planar façades to a more general class of façades. We demonstrate the power of the unwrappable representation with a few examples in which the façade is not planar.