Real-time motion-induced-error compensation in 3D surface-shape measurement

Temporal phase unwrapping based on unequal phase-shifting code

In fringe projection profilometry (FPP) based on temporal phase unwrapping (TPU), reducing the number of projecting patterns has become one of the most important works in recent years. To remove the 2π ambiguity independently, this paper proposes a TPU method based on unequal phase-shifting code. Wrapped phase is still calculated from N-step conventional phase-shifting patterns with equal phase-shifting amount to guarantee the measuring accuracy. Particularly, a series of different phase-shifting amounts relative to the first phase-shifting pattern are set as codewords, and encoded to different periods to generate one coded pattern. When decoding, Fringe order with a large number can be determined from the conventional and coded wrapped phases. In addition, we develop a self-correction method to eliminate the deviation between the edge of fringe order and the 2π discontinuity. Thus, the proposed method can achieve TPU but need to only project one additional coded pattern (e. g. 3+1), which can significantly benefit dynamic 3D shape reconstruction. The theoretical and experimental analysis verify that the proposed method performs high robustness on the reflectivity of the isolated object while ensuring the measuring speed.

Nonlinear error full-field compensation method for phase measuring profilometry

Phase measuring profilometry (PMP) has the highest measuring accuracy among structured light projection-based three-dimensional (3D) sensing methods. Due to their low-cost and high-resolution features, commercial projectors are extensively used in PMP, but they are all designed with a gamma effect purpose that considers the characteristics of human vision. Affected by the gamma effect, a set of phase-shifting sinusoidal deformed patterns captured in PMP may contain high-order harmonics which lead to nonlinear phase errors. Then, a novel nonlinear error full-field compensation method is proposed. First, the unwrapped phases modulated by the reference plane are measured several times, and their average phase is taken as the measured phase modulated by the reference plane to eliminate random errors as much as possible. Second, an expected phase plane is fitted from this average phase with the least-squares method. Third, the nonlinear phase error can be detected by subtracting the fitted expected phase from this average phase. Finally, the full-field look-up table (LUT) can be established between the nonlinear phase error and the measured phase. When an object is measured, the unwrapped phase modulated by the object is taken as the measured phase of the LUT, so the corresponding nonlinear phase error can be directly searched in the LUT. In this way, the full-field nonlinear phase error can be efficiently compensated. Experimental results show the feasibility and validity of the proposed method. The mean absolute error (MAE) can be improved from 0.48 mm to 0.06 mm, and the root mean square error (RMSE) can be improved from 0.55 mm to 0.07 mm.

Deep learning-based method for non-uniform motion-induced error reduction in dynamic microscopic 3D shape measurement

The non-uniform motion-induced error reduction in dynamic fringe projection profilometry is complex and challenging. Recently, deep learning (DL) has been successfully applied to many complex optical problems with strong nonlinearity and exhibits excellent performance. Inspired by this, a deep learning-based method is developed for non-uniform motion-induced error reduction by taking advantage of the powerful ability of nonlinear fitting. First, a specially designed dataset of motion-induced error reduction is generated for network training by incorporating complex nonlinearity. Then, the corresponding DL-based architecture is proposed and it contains two parts: in the first part, a fringe compensation module is developed as network pre-processing to reduce the phase error caused by fringe discontinuity; in the second part, a deep neural network is employed to extract the high-level features of error distribution and establish a pixel-wise hidden nonlinear mapping between the phase with motion-induced error and the ideal one. Both simulations and real experiments demonstrate the feasibility of the proposed method in dynamic macroscopic measurement.

Quasi-pixelwise motion compensation for 4-step phase-shifting profilometry based on a phase error estimation

Phase-shifting profilometry (PSP) is widely used in 3D shape measurement due to its high accuracy. However, in dynamic scenarios, the motion of objects will introduce phase-shifting errors and result in measurement errors. In this paper, a novel compensation method based on 4-step phase-shifting profilometry is proposed to reduce motion-induced errors when objects undergo uniform or uniformly accelerated motion. We utilize the periodic characteristic of fringe patterns to estimate the phase errors from only four phase-shifting patterns and realize a pixel-wise error compensation. This method can also be applied to non-rigid deforming objects and help restore high-quality texture. Both simulation and experiments demonstrate that the proposed method can effectively improve the measurement accuracy and reduce surface ripples introduced by motion for a standard monocular structured-light system.

Advances and Prospects of Vision-Based 3D Shape Measurement Methods

Vision-based three-dimensional (3D) shape measurement techniques have been widely applied over the past decades in numerous applications due to their characteristics of high precision, high efficiency and non-contact. Recently, great advances in computing devices and artificial intelligence have facilitated the development of vision-based measurement technology. This paper mainly focuses on state-of-the-art vision-based methods that can perform 3D shape measurement with high precision and high resolution. Specifically, the basic principles and typical techniques of triangulation-based measurement methods as well as their advantages and limitations are elaborated, and the learning-based techniques used for 3D vision measurement are enumerated. Finally, the advances of, and the prospects for, further improvement of vision-based 3D shape measurement techniques are proposed.

Real-time motion-induced error compensation for 4-step phase-shifting profilometry

Phase-shifting profilometry has been widely used in high-accuracy three-dimensional (3D) shape measurement. However, for dynamic scenarios, the object motion will lead to extra phase shift and then motion-induced error. Convenient and efficient motion-induced error compensation is still challenging. Therefore, we proposed a real-time motion-induced error compensation method for 4-step phase-shifting profilometry. The four phase-shifting images are divided into two groups to calculate two corresponding wrapped phases, one from the first three fringes and the other from the last three fringes. As the motion-induced error doubles the frequency of the projected fringes, the average phase can effectively compensate the motion-induced error because there is a π/2 phase shift between the adjacent frames. Furthermore, we designed a time sequence by recycling the projection fringes in a proper order, and the efficiency of 3D reconstruction could be effectively improved. This method performs pixel-wise error compensation, based on which we realized 50 fps real-time 3D measurement by GPU acceleration. Experimental results demonstrate that the proposed method can effectively reduce the motion-induced error.

Spatial-temporal phase unwrapping algorithm for fringe projection profilometry

In this paper, a generalized spatial-temporal phase unwrapping algorithm (STPUA) is proposed for extracting the absolute phase of the isolated objects with intricate surfaces. This proposed algorithm can eliminate thoroughly the order jumps of various temporal phase unwrapping algorithms (TPUAs), while inheriting the high measuring accuracy of quality-guided phase unwrapping algorithms (QGPUAs). Differing from the traditional phase unwrapping algorithms, wrapped phase is first divided into several regional wrapped phases, which can be extracted successively according to its areas and unwrapped individually by QGPUAs. Meanwhile, a series of reliable points from the fringe order map obtained from the code deformed patterns are selected to map the corresponding regional unwrapped phases into an absolute phase. The radii of selecting reliable points can provide the high measuring robustness compared with the classical point-to-point TPUAs for the complex surfaces and the motion blur, while keeping the same number of patterns. Therefore, the proposed STPUA combining SPUAs and TPUAs also can be employed in real-time three-dimensional (3D) reconstruction. Theoretical analysis and experimental results are performed to verify the effectiveness and capability of the proposed algorithm.

High-speed 3D shape measurement using a rotary mechanical projector

In this paper, a fast rotary mechanical projector (RMP) is designed and manufactured for high-speed 3D shape measurement. Compared with the common high-speed projectors, RMP has a good performance in high-speed projection, which can obtain high quality projected fringes with shorter camera exposure time by using the error diffusion binary coding method and chrome plating technology. The magnitude, acceptability of systemic projection error is analyzed and quantified in detail. For the quantified error, the probability distribution function (PDF) algorithm is introduced to correct the error. Corrected projection error is reduced to more than one third of the original error. Subsequently, a monocular measurement system composed of the RMP and a single camera is constructed. The combination of the RMP device and PDF algorithm ensure the accuracy of a corresponding 3D shape measurement system. Experiments have demonstrated that the proposed solution has a good performance for the 3D measurement of high-speed scenes.

Automated reconstruction of multiple objects with individual movement based on PSP

Many methods have been proposed to reconstruct the moving object based on phase shifting profilometry. Quality reconstruction results can be achieved when a single moving object or multiple objects with same movement are measured. However, errors will be introduced when multiple objects with individual movements are reconstructed. This paper proposes an automated method to track and reconstruct the multiple objects with individual movement. First, the objects are identified automatically and their bounding boxes are obtained. Second, with the identified objects' images before movement, the objects are tracked by the KCF algorithm in the successive fringe pattern after movement. Third, the SIFT method is applied on the tracked object images and the objects' movement is described individually by the rotation matrix and translation vector. Finally, the multiple objects are reconstructed based on the different movement information. Experiments are presented to verify the effectiveness.

Real-time 3D shape measurement with dual-frequency composite grating and motion-induced error reduction

Phase-shifting profilometry has been increasingly sought and applied in dynamic three-dimensional (3D) shape measurement. However, the object motion will lead to extra phase shift error and thus measurement error. In this paper, a real-time 3D shape measurement method based on dual-frequency composite phase-shifting grating and motion-induced error reduction is proposed for a complex scene containing dynamic and static objects. The proposed method detects the motion region of a complex scene through the phase relations of the dual-frequency composite grating and reduces the motion-induced error with the combination of the phase calculated by a phase-shifting algorithm and the phase extracted by Fourier fringe analysis. It can correctly reconstruct the 3D shape of a complex dynamic scene and ensure high measurement accuracy of its static object as well. With the aid of the phase-shifting image ordering approach, the dynamic 3D shape of complex scenes can be reconstructed and the motion-induced error can also be suppressed in real time. Experimental results well proved that the proposed method is effective and practical.

High-frequency color-encoded fringe-projection profilometry based on geometry constraint for large depth range

In multi-view fringe projection profilometry (FPP), a limitation of geometry-constraint based approaches is the reduced measurement depth range often used to reduce the number of candidate points and increase the corresponding point selection reliability, when high-frequency fringe patterns are used. To extend the depth range, a new method of high-frequency fringe projection profilometry was developed by color encoding the projected fringe patterns to allow reliable candidate point selection even when six candidate points are in the measurement volume. The wrapped phase is directly retrieved using the intensity component of the hue-saturation-intensity (HSI) color space and complementary-hue is introduced to identify color codes for correct corresponding point selection. Mathematical analyses of the effect of color crosstalk on phase calculation and color code identification show that the phase calculation is independent of color crosstalk and that color crosstalk has little effect on color code identification. Experiments demonstrated that the new method can achieve high accuracy in 3D measurement over a large depth range and for isolated objects, using only two high-frequency color-encoded fringe patterns.

Dynamic 3-D measurement based on fringe-to-fringe transformation using deep learning

Fringe projection profilometry (FPP) has become increasingly important in dynamic 3-D shape measurement. In FPP, it is necessary to retrieve the phase of the measured object before shape profiling. However, traditional phase retrieval techniques often require a large number of fringes, which may generate motion-induced error for dynamic objects. In this paper, a novel phase retrieval technique based on deep learning is proposed, which uses an end-to-end deep convolution neural network to transform a single or two fringes into the phase retrieval required fringes. When the object's surface is located in a restricted depth, the presented network only requires a single fringe as the input, which otherwise requires two fringes in an unrestricted depth. The proposed phase retrieval technique is first theoretically analyzed, and then numerically and experimentally verified on its applicability for dynamic 3-D measurement.