Composite deep learning framework for absolute 3D shape measurement based on single fringe phase retrieval and speckle correlation

Fourier Ptychographic Neural Network Combined with Zernike Aberration Recovery and Wirtinger Flow Optimization

Fourier ptychographic microscopy, as a computational imaging method, can reconstruct high-resolution images but suffers optical aberration, which affects its imaging quality. For this reason, this paper proposes a network model for simulating the forward imaging process in the Tensorflow framework using samples and coherent transfer functions as the input. The proposed model improves the introduced Wirtinger flow algorithm, retains the central idea, simplifies the calculation process, and optimizes the update through back propagation. In addition, Zernike polynomials are used to accurately estimate aberration. The simulation and experimental results show that this method can effectively improve the accuracy of aberration correction, maintain good correction performance under complex scenes, and reduce the influence of optical aberration on imaging quality.

Single-shot 3D measurement of highly reflective objects with deep learning

Three-dimensional (3D) measurement methods based on fringe projection profilometry (FPP) have been widely applied in industrial manufacturing. Most FPP methods adopt phase-shifting techniques and require multiple fringe images, thus having limited application in dynamic scenes. Moreover, industrial parts often have highly reflective areas leading to overexposure. In this work, a single-shot high dynamic range 3D measurement method combining FPP with deep learning is proposed. The proposed deep learning model includes two convolutional neural networks: exposure selection network (ExSNet) and fringe analysis network (FrANet). The ExSNet utilizes self-attention mechanism for enhancement of highly reflective areas leading to overexposure problem to achieve high dynamic range in single-shot 3D measurement. The FrANet consists of three modules to predict wrapped phase maps and absolute phase maps. A training strategy directly opting for best measurement accuracy is proposed. Experiments on a FPP system showed that the proposed method predicted accurate optimal exposure time under single-shot condition. A pair of moving standard spheres with overexposure was measured for quantitative evaluation. The proposed method reconstructed standard spheres over a large range of exposure level, where prediction errors for diameter were 73 µm (left) and 64 µm (right) and prediction error for center distance was 49 µm. Ablation study and comparison with other high dynamic range methods were also conducted.

High-speed structured light based 3D scanning using an event camera

For a structured light system, scan speed and reconstruction accuracy are usually compromised for limited sensor bandwidth. The bio-inspired camera, also known as the event camera, has high temporal resolution and redundancy-suppressing properties, showing potential to be utilized in a high-speed structured light system. In this paper, we present an event-based structured light system for high-speed 3D scanning, which is composed of an event camera (CeleX-V) and a high-speed digital light projector (TI-DLP6500). The events are triggered by blinking a single pseudo-random pattern by controlling the projector. A simple yet effective algorithm is proposed to generate the event frames from the event stream, and a digital image correlation method is then performed to calculate the displacements, deriving the 3D surfaces of the target objects. A prototype of our proposed system is built with off-the-shelf devices and tested in both static and dynamic scenes. Experiments verify that the proposed system successfully achieves up to a 1000 fps scan rate with an accuracy of 0.27 mm at a distance of 90 cm.

Spatial phase-shifting profilometry by use of polarization for measuring 3D shapes of metal objects

In this paper, we present a polarization spatial phase-shifting method for fringe projection profilometry. It enables us to measure the three-dimensional shape of a metal object in a fast way requiring only a single-shot implementation. With this method, a couple of projectors are equipped, in front of their lens, with linear polarization filters having orthogonal polarization directions, so that they can simultaneously cast two sinusoidal fringe patterns having different phase shifts onto the measured metal surfaces without mixture. To register the two projected patterns, we suggest a fringe alignment method based on the epipolar geometry between the projectors. By taking advantage of the property of metal surfaces in maintaining polarization state of incident light, the deformed fringe patterns on the measured surfaces are captured by using two coaxially-arranged polarization cameras. As a result, the fringe phases are calculated by using a two-step phase-shifting algorithm and further the 3D shapes of the measured surfaces are reconstructed. Experimental results demonstrate the proposed method to be valid and efficient in measuring metal objects.

Single-shot 3D shape measurement using an end-to-end stereo matching network for speckle projection profilometry

Speckle projection profilometry (SPP), which establishes the global correspondences between stereo images by projecting only a single speckle pattern, has the advantage of single-shot 3D reconstruction. Nevertheless, SPP suffers from the low matching accuracy of traditional stereo matching algorithms, which fundamentally limits its 3D measurement accuracy. In this work, we propose a single-shot 3D shape measurement method using an end-to-end stereo matching network for SPP. To build a high-quality SPP dataset for training the network, by combining phase-shifting profilometry (PSP) and temporal phase unwrapping techniques, high-precision absolute phase maps can be obtained to generate accurate and dense disparity maps with high completeness as the ground truth by phase matching. For the architecture of the network, a multi-scale residual subnetwork is first leveraged to synchronously extract compact feature tensors with 1/4 resolution from speckle images for constructing the 4D cost volume. Considering that the cost filtering based on 3D convolution is computationally costly, a lightweight 3D U-net network is proposed to implement efficient 4D cost aggregation. In addition, because the disparity maps in the SPP dataset should have valid values only in the foreground, a simple and fast saliency detection network is integrated to avoid predicting the invalid pixels in the occlusions and background regions, thereby implicitly enhancing the matching accuracy for valid pixels. Experiment results demonstrated that the proposed method improves the matching accuracy by about 50% significantly compared with traditional stereo matching methods. Consequently, our method achieves fast and absolute 3D shape measurement with an accuracy of about 100µm through a single speckle pattern.