High-speed three-dimensional shape measurements of objects with laser speckles and acousto-optical deflection

DSCNet: lightweight and efficient self-supervised network via depthwise separable cross convolution blocks for speckle image matching

Speckle structured light has become a research hotspot due to its ability to acquire target three-dimensional information with single image projection in recent years. To address the challenges of a low number of extracted speckle feature points, high mismatch rate and poor real-time performance in traditional algorithms, as well as the obstacle of requiring expensive annotation data in deep learning-based methods, a lightweight and efficient self-supervised convolutional neural network (CNN) is proposed to achieve high-precision and rapid matching of speckle images. First, to efficiently utilize the speckle projection information, a feature extraction backbone based on the depthwise separable cross convolution blocks is proposed. Second, in the feature detection module, a softargmax detection head is designed to refine the coordinates of speckle feature points to sub-pixel accuracy. In the feature description module, a coarse-to-fine module is presented to further refine matching accuracy. Third, we adopt strategies of transfer learning and self-supervised learning to improve the generalization and feature representation capabilities of the model. Data augmentation and real-time training techniques are used to improve the robustness of the model. The experimental results show that the proposed method achieves a mean matching accuracy of 91.62% for speckle feature points on the pilot's helmet, with mere 0.95% mismatch rate. The full model runs at 42ms for a speckle image pair on an RTX 3060.

Distortion spot correction and center location base on deep neural network and MBAS in measuring large curvature aspheric optical element

Large curvature aspheric optical elements are widely used in visual system. But its morphological detection is very difficult because its accuracy requirement is very high. When we use the self-developed multi-beam angle sensor (MBAS) to detect large curvature aspheric optical elements, the accuracy will be reduced due to spot distortion. Therefore, we propose a scheme combining distorted spot correction neural network (DSCNet) and gaussian fitting method to improve the detection accuracy of distorted spot center. We develop a spot discrimination method to determine spot region in multi-spot images. The spot discrimination threshold is obtained by the quantitative distribution of pixels in the connected domain. We design a DSCNet, which corrects the distorted spot to Gaussian spot, to extract the central information of distorted spot images by multiple pooling. The experimental results demonstrate that the DSCNet can effectively correct the distorted spot, and the spot center can be extracted to sub-pixel level, which improves the measurement accuracy of the MBAS. The standard deviations of plano-convex lenses with curvature radii of 500 mm, 700 mm and 1000 mm measured with the proposed method are respectively 0.0112 um, 0.0086 um and 0.0074 um.

High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking

Close-range 3D sensors based on the structured light principle have a constrained measuring range due to their depth of field (DOF). Focus stacking is a method to extend the DOF. The additional time to change the focus is a drawback in high-speed measurements. In our research, the method of chromatic focus stacking was applied to a high-speed 3D sensor with 180 fps frame rate. The extended DOF was evaluated by the distance-dependent 3D resolution derived from the 3D-MTF of a tilted edge. The conventional DOF of 14 mm was extended to 21 mm by stacking two foci at 455 and 520 nm wavelength. The 3D sensor allowed shape measurements with extended DOF within 44 ms.

Advances and Prospects of Vision-Based 3D Shape Measurement Methods

Vision-based three-dimensional (3D) shape measurement techniques have been widely applied over the past decades in numerous applications due to their characteristics of high precision, high efficiency and non-contact. Recently, great advances in computing devices and artificial intelligence have facilitated the development of vision-based measurement technology. This paper mainly focuses on state-of-the-art vision-based methods that can perform 3D shape measurement with high precision and high resolution. Specifically, the basic principles and typical techniques of triangulation-based measurement methods as well as their advantages and limitations are elaborated, and the learning-based techniques used for 3D vision measurement are enumerated. Finally, the advances of, and the prospects for, further improvement of vision-based 3D shape measurement techniques are proposed.

3D face imaging with the spatial-temporal correlation method using a rotary speckle projector

In this paper, a compact, cost-effective, and fast rotary speckle projector (RSP) is designed and manufactured for high-precision three-dimensional (3D) face data acquisition. Compared with the common speckle projectors, RSP uses a simple speckle pattern design method and has a good performance in high-speed projection and compact structure, which allows a flexible balance between measurement accuracy and time cost in a real acquisition task. Using a carefully designed rotation angle of the speckle mask, temporally and spatially non-correlative speckle patterns in the measurement volume can be generated. The rotation angle of the speckle mask is carefully checked and optimally selected via detailed theoretical analysis, simulation, and experiments to ensure 3D reconstruction accuracy across the reconstruction area. Subsequently, a binocular 3D face imaging system composed of the RSP and two cameras is constructed. With captured stereo speckle image pairs, we adopted our previously well-established spatial-temporal correlation method to determine the disparity. The accuracy of the 3D face imaging system was verified by using a real face mask, which is standardized by a certified, high-precision industrial 3D scanner. The real face data collection under various expressions has demonstrated that the proposed system also has a good performance for 3D face imaging in dynamic scenes.

Single-shot 3D shape measurement using an end-to-end stereo matching network for speckle projection profilometry

Speckle projection profilometry (SPP), which establishes the global correspondences between stereo images by projecting only a single speckle pattern, has the advantage of single-shot 3D reconstruction. Nevertheless, SPP suffers from the low matching accuracy of traditional stereo matching algorithms, which fundamentally limits its 3D measurement accuracy. In this work, we propose a single-shot 3D shape measurement method using an end-to-end stereo matching network for SPP. To build a high-quality SPP dataset for training the network, by combining phase-shifting profilometry (PSP) and temporal phase unwrapping techniques, high-precision absolute phase maps can be obtained to generate accurate and dense disparity maps with high completeness as the ground truth by phase matching. For the architecture of the network, a multi-scale residual subnetwork is first leveraged to synchronously extract compact feature tensors with 1/4 resolution from speckle images for constructing the 4D cost volume. Considering that the cost filtering based on 3D convolution is computationally costly, a lightweight 3D U-net network is proposed to implement efficient 4D cost aggregation. In addition, because the disparity maps in the SPP dataset should have valid values only in the foreground, a simple and fast saliency detection network is integrated to avoid predicting the invalid pixels in the occlusions and background regions, thereby implicitly enhancing the matching accuracy for valid pixels. Experiment results demonstrated that the proposed method improves the matching accuracy by about 50% significantly compared with traditional stereo matching methods. Consequently, our method achieves fast and absolute 3D shape measurement with an accuracy of about 100µm through a single speckle pattern.

3-D face registration solution with speckle encoding based spatial-temporal logical correlation algorithm

3-D information acquisition (registration) of whole face plays a significant role in 3-D human face recognition application. In this paper, we develop a prototype of 3-D system consisting of two binocular measurement units that allows a full 3-D reconstruction by utilizing the advantages of a novel correlation algorithm. In this system, we use optical modulation to produce temporally and spatially varying high-density binary speckle patterns to encode the tested face, then propose a spatial-temporal logical correlation (STLC) stereo matching algorithm to fast determine the accurate disparity with a coarse and refined strategy. Finally the 3-D information of whole face from left- and right ear (~180°) can be obtainable by fusing the data from two measurement units. Comparative researches are performed to test a plastic model and a real human face by simulating real application situations. The results verify the feasibility and good performances of our computational frameworks and experimental configuration in terms of accuracy and time cost, which show a good application prospect in our future 3-D human face recognition research.

GOBO projection for 3D measurements at highest frame rates: a performance analysis

Aperiodic sinusoidal patterns that are cast by a GOBO (GOes Before Optics) projector are a powerful tool for optically measuring the surface topography of moving or deforming objects with very high speed and accuracy. We optimised the first experimental setup that we were able to measure inflating car airbags at frame rates of more than 50 kHz while achieving a 3D point standard deviation of ~500 µm. Here, we theoretically investigate the method of GOBO projection of aperiodic sinusoidal fringes. In a simulation-based performance analysis, we examine the parameters that influence the accuracy of the measurement result and identify an optimal pattern design that yields the highest measurement accuracy. We compare the results with those that were obtained via GOBO projection of phase-shifted sinusoidal fringes. Finally, we experimentally verify the theoretical findings. We show that the proposed technique has several advantages over conventional fringe projection techniques, as the easy-to-build and cost-effective GOBO projector can provide a high radiant flux, allows high frame rates, and can be used over a wide spectral range.

Calibration of an array projector used for high-speed three-dimensional shape measurements using a single camera

Geometric calibration of digital light processing projectors in single-camera, fringe-projecting 3D measurement systems have been studied assuming the projector is inverse pinhole modeled. Conversely, a high-speed multi-aperture array projector (MAAP) projecting aperiodic fringes is not dependent on a digital mirror device and cannot be pinhole modeled. With MAAP projection, a stereo camera setup is required. This paper presents a model-less method to calibrate a MAAP by direct measurement of its illumination field and re-enables 3D measurements with a single camera even with surface discontinuities present. Experimental proof of principle and preliminary measurement performance are shown.

Optical 3-D surface reconstruction with color binary speckle pattern encoding

This paper proposes a novel 3-D surface profile measurement scheme by only a single-shot color binary speckle pattern (CBSP) and a temporal-spatial correlation matching algorithm, which can be applied to measurements of dynamic and static objects. R/G/B channels of CBSP are coupled with three carefully designed black and white binary speckle patterns (BWBSPs), whose physical features are associated with the system configuration parameters. We mathematically deduce the concrete details of how to design such a pattern and its relationship with the system parameters selected in the experiment. During 3-D reconstruction, we develop an extended temporal-spatial correlation framework to determine the correspondence between two stereo images sequences that are composed of R/G/B images separated from a captured color stereo image pair. Comparative experiments and analysis are implemented to assess the measurement accuracy using standard workpieces (dumbbell and optical flat). The results indicate that the proposed approach enjoys better performance than the conventional BWBSP-based method in terms of spatial resolution, accuracy, and efficient reconstructed points. An experiment of applying CBSP to measuring a moving A4 paper is also presented, demonstrating the success of our computational framework. Finally discussions concerning the limitations of this method are implemented.

Snapshot phase shift fringe projection 3D surface measurement

We propose a novel snapshot phase shift fringe projection three-dimensional (3D) surface measurement method using polarization-coded light illumination and polarization camera. The light from the light source is split into two beams, one is left circularly polarized and the other is right circularly polarized, to illuminate the object simultaneously. A four-channel division of focal plane (DoFP) polarization camera is employed to capture the light reflected from the object surface. Four images with a phase shift of π/2 are extracted from the snapshot image and then analyzed to reconstruct a 3D object surface. The proposed method is the first snapshot phase shift fringe projection approach for 3D surface imaging. It is insensitive to motion and has the potential for ultrafast 3D surface imaging.

3D shape measurement of static and moving objects with adaptive spatiotemporal correlation

Common correlation-based photogrammetric 3D shape measurement techniques evaluate either temporal or spatial features. Temporal approaches achieve high accuracies but are limited to the measurement of static objects. Spatial techniques can deal with moving objects but provide relatively inaccurate results. Our goal is to combine these methods in order to measure dynamic scenes that contain static and moving objects. Therefore, we present a spatiotemporal correlation that adapts its temporal and spatial support locally to the motion of the measured objects. In addition, our technique compensates motion by warping the correlated image regions temporally. Our approach is based on structured illumination of random patterns, which are well suited for dynamic scenes due to high possible frame rates. The proposed technique is tested with simulated data and real measurements.

Optimized stereo matching in binocular three-dimensional measurement system using structured light

In this paper, we develop an optimized stereo-matching method used in an active binocular three-dimensional measurement system. A traditional dense stereo-matching algorithm is time consuming due to a long search range and the high complexity of a similarity evaluation. We project a binary fringe pattern in combination with a series of N binary band limited patterns. In order to prune the search range, we execute an initial matching before exhaustive matching and evaluate a similarity measure using logical comparison instead of a complicated floating-point operation. Finally, an accurate point cloud can be obtained by triangulation methods and subpixel interpolation. The experiment results verify the computational efficiency and matching accuracy of the method.

Coherent two-beam interference fringe projection for highspeed three-dimensional shape measurements

Two-beam interference is a fundamental and well-understood approach to create Fizeau's interference fringes. With a Mach-Zehnder interferometer, we utilize these two-beam interference Fizeau fringes for three-dimensional (3D) shape measurements. By introducing an acousto-optical deflector the phase of the interference fringes can be shifted with a rate of up to 200,000 Hz. When used in conjunction with highspeed cameras, this stereo-photogrammetric approach performs well for highspeed applications in comparison with the commonly used digital light processing projectors for stripe projection. Maximum speed and the achievable accuracy are discussed. Experiments and media substantiate the suitability, accuracy, and speed of this technique for very fast 3D shape measurements.

Generalized phase evaluation for stereophotogrammetric correspondence assignment

Phase-shifting fringe projection is the primary structured illumination method for high-accuracy, three-dimensional (3D) shape measurements in the fields of profilometry and stereophotogrammetry. Many different schemes for the phase evaluation and the phase-shifted fringe pattern design exist. Here we focus on the role of the phase evaluation in the context of stereophotogrammetry, where the nominal phase value itself is merely used as an image feature that can be exploited to establish a correspondence between the two camera views. Starting from the classical phase evaluation function, we will discuss its essential properties for a highly accurate correspondence mapping. Based on the findings, we generalize the classical function to derive a generalized phase value for a sequence of stereo images. An experimental comparison between a correspondence assignment using the classical phase evaluation function and a specifically chosen general phase evaluation function is given.