Phase-shifting profilometry combined with Gray-code patterns projection: unwrapping error removal by an adaptive median filter

BimodalPS: Causes and Corrections for Bimodal Multi-Path in Phase-Shifting Structured Light Scanners

Structured light illumination is an active 3D scanning technique based on projecting and capturing a set of striped patterns and measuring the warping of the patterns as they reflect off a target object's surface. As designed, each pixel in the camera sees exactly one pixel from the projector; however, there are multi-path situations where a camera pixel sees light from multiple projector positions. In the case of bimodal multi-path, the camera pixel receives light from exactly two positions, which occurs along a step edge where the edge slices through a pixel which, therefore, sees both a foreground and background surface. In this paper, we present a general mathematical model to address this bimodal multi-path issue in a phase-shifting or so-called phase-measuring-profilometry scanner to measure the constructive and destructive interference between the two light paths, and by taking advantage of this interference, separate the paths and make two decoupled depth measurements. We validate our algorithm with both simulations and a number of challenging real-world scenarios, significantly outperforming the state-of-the-art methods.

Orthogonal Spatial Binary Coding Method for High-Speed 3D Measurement

Temporal phase unwrapping based on single auxiliary binary coded pattern has been proven to be effective for high-speed 3D measurement. However, in traditional spatial binary coding, it often leads to an imbalance between the number of periodic divisions and codewords. To meet this challenge, a large codewords orthogonal spatial binary coding method is proposed in this paper. By expanding spatial multiplexing from 1D to 2D orthogonal direction, it goes beyond the traditional 8 codewords to 27 codewords at three-level periodic division. In addition, a novel full-period connected domain segmentation technique based on local localization is proposed to avoid the time-consuming global iterative erosion and complex anomaly detection in traditional methods. For the decoding process, a purely spatial codewords recognition and a spatial-temporal hybrid codewords recognition methods are established to better suppress the percentage offset caused by static defocusing and dynamic motion, respectively. Obviating the need for intricate symbol recognition, the decoding process in our proposed method encompasses a straightforward analysis of statistical distribution. Building upon the development of special spatial binary coding, we have achieved a well-balance between low periodic division and large codewords for the first time. The experimental results verify the feasibility and validity of our proposed whole image processing method in both static and dynamic measurements.

Phase correction strategy based on structured light fringe projection profilometry

Fringe projection profilometry based on structured light has been widely used in 3-D vision due to its advantages of simple structure, good robustness, and high speed. The principle of this technique is to project multiple orders of stripes on the object, and the camera captures the deformed stripe map. Phase unwrapping and depth map calculation are important steps. Still, in actual situations, phase ambiguity is prone to occur at the edges of the object. In this paper, an adaptive phase segmentation and correction (APSC) method after phase unwrapping is proposed. In order to effectively distinguish the stable area and unstable area of the phase, a boundary identification method is proposed to obtain the structural mask of the phase. A phase compensation method is proposed to improve the phase accuracy. Finally, we obtain the 3-D reconstruction result based on the corrected phase. Specific experimental results verify the feasibility and effectiveness of this method.

Scene-adaptive pattern coding-based fringe projection profilometry: diffuse surfaces identification and 3-D reconstruction in cluttered scenes

Fringe projection profilometry (FPP) is one of the most widely used optical three-dimensional (3-D) perceiving techniques. However, when applied to cluttered scenes, acquiring accurate 3-D shapes is difficult because of the influences of indirect light caused by non-diffuse surfaces. In this paper, we first theoretically analyze and model the influences of indirect light in FPP, and then propose a scene-adaptive pattern coding-based method, which can design projection patterns based on the reflective properties of the scene's surfaces, to achieve accurate 3-D perceiving in cluttered scenes. Specifically, the scene confidence analysis method is first proposed to identify the reflective properties of various surfaces and localize the camera pixels of the diffuse surface. The illumination status (i.e., "0" or "1") of each projector pixel can be determined according to the camera-projection coordinate mapping and spatial pattern coding, where only diffuse surfaces can be illuminated, thus fundamentally preventing the influences of indirect light from the point of view of the light source. The 3-D shapes of diffuse surfaces can be accurately reconstructed in cluttered scenes. Different from traditional reflective properties change or light separation solutions, the proposed method can achieve accurate 3-D perceiving of cluttered scenes without additional hardware or expensive calculation. Extensive experiments verify that the proposed method outperforms the traditional methods in terms of accuracy and robustness.

FPP-SLAM: indoor simultaneous localization and mapping based on fringe projection profilometry

Simultaneous localization and mapping (SLAM) plays an important role in autonomous driving, indoor robotics and AR/VR. Outdoor SLAM has been widely used with the assistance of LiDAR and Global Navigation Satellite System (GNSS). However, for indoor applications, the commonly used LiDAR sensor does not satisfy the accuracy requirement and the GNSS signals are blocked. Thus, an accurate and reliable 3D sensor and suited SLAM algorithms are required for indoor SLAM. One of the most promising 3D perceiving techniques, fringe projection profilometry (FPP), shows great potential but does not prevail in indoor SLAM. In this paper, we first introduce FPP to indoor SLAM, and accordingly propose suited SLAM algorithms, thus enabling a new FPP-SLAM. The proposed FPP-SLAM can achieve millimeter-level and real-time mapping and localization without any expensive equipment assistance. The performance is evaluated in both simulated controlled and real room-sized scenes. The experimental results demonstrate that our method outperforms other state-of-the-art methods in terms of efficiency and accuracy. We believe this method paves the way for FPP in indoor SLAM applications.

High-Accuracy Three-Dimensional Deformation Measurement System Based on Fringe Projection and Speckle Correlation

Fringe projection profilometry (FPP) and digital image correlation (DIC) are widely applied in three-dimensional (3D) measurements. The combination of DIC and FPP can effectively overcome their respective shortcomings. However, the speckle on the surface of an object seriously affects the quality and modulation of fringe images captured by cameras, which will lead to non-negligible errors in the measurement results. In this paper, we propose a fringe image extraction method based on deep learning technology, which transforms speckle-embedded fringe images into speckle-free fringe images. The principle of the proposed method, 3D coordinate calculation, and deformation measurements are introduced. Compared with the traditional 3D-DIC method, the experimental results show that this method is effective and precise.

Three-dimensional shape measurement method based on composite cyclic phase coding

Phase coding is widely used in 3D measurement due to its good anti-interference and robustness. However, the measurement accuracy is affected by the limitation of the number of codewords. To solve this problem, we propose a 3D shape measurement method based on composite cyclic phase coding. The traditional phase coding is quantized cyclic without adding extra patterns, further adopting composite coding, using the composite cyclic phase coding grayscale values to distinguish the same cyclic codewords, and finally integrating them into a new fringe order sequentially for phase unwrapping to achieve effective expansion of codewords. The related experimental results show that the proposed method stably achieves high accuracy 3D reconstruction, which overcomes the misjudgment of codewords caused by traditional phase coding under high-frequency fringes due to system nonlinearity and noise. Meanwhile, compared with the improved phase coding method of temporal domain combined with spatial domain information, such as the method of quantized phase coding and connected region labeling, it can effectively avoid the phenomenon of error propagation, with high robustness and low algorithm complexity.

Wavelet denoising approach in long-distance optical communications

For free-space optical communication links, the light spot collected by the photodetector at the receiving terminal is not an ideal light spot that is affected by atmospheric turbulence. The light spot collected by the photodetector will also be accompanied by various noises. More importantly, the presence of all noise will bring errors to acquire the light spot's center. As a result, the tracking error can affect the stability of the optoelectronic tracking system. Therefore, it is necessary to remove noise from the collected images. The method of removing noise needs to be effective, but it cannot bring a large amount of calculation to affect the real-time performance. The calculation amount of wavelet transform is small, and the effect of noise removal is better, which can focus on local details with arbitrary expansion coefficients. An improved wavelet denoising method is proposed. The long-distance verification experiment (11.16 km) verified the effectiveness of this approach, compared with the traditional method. Furthermore, to the best of our knowledge, this new approach would be beneficial for the design of optical communication systems.

Half-Period Gray-Level Coding Strategy for Absolute Phase Retrieval

N-ary gray-level (nGL) coding strategy is an effective method for absolute phase retrieval in the fringe projection technique. However, the conventional nGL method contains many unwrapping errors at the boundaries of codewords. In addition, the number of codewords is limited in only one pattern. Consequently, this paper proposes a new gray-level coding method based on half-period coding, which can improve both these two deficiencies. Specifically, we embed every period with a 2-bit codeword, instead of a 1-bit codeword. Then, special correction and decoding methods are proposed to correct the codewords and calculate the fringe orders, respectively. The proposed method can generate n2 codewords with n gray levels in one pattern. Moreover, this method is insensitive to moderate image blurring. Various experiments demonstrate the robustness and effectiveness of the proposed strategy.

Three-dimensional measurement method based on a three-step phase-shifting fringe and a binary fringe

Gray-code plus phase-shifting is currently a commonly used method for structured light three-dimensional (3D) measurement that is able to measure complex surfaces. However, the Gray-code fringe patterns tend to be complicated, making the measurement process time-consuming. To solve this problem and to obtain faster speed without sacrificing accuracy, a 3D measurement method based on three-step phase-shifting and a binary fringe is proposed; the method contains three phase-shifting fringe patterns and an additional binary fringe pattern. The period of the binary fringe is designed to be the same as the three-step phase-shifting fringe. Because of the specific pattern design strategy, the three-step phase-shifting algorithm is used to obtain the wrapped phase, and the connected region labeling theorem is used to calculate the fringe order. A theoretical analysis, simulation, and experiments validate the efficiency and robustness of the proposed method. It can achieve high-precision 3D measurement, which performs almost the same as the Gray-code plus phase-shifting method. Since only one additional binary fringe pattern is required, it has the potential to achieve higher measurement speed.

Accurate 3D Reconstruction of Dynamic Objects by Spatial-Temporal Multiplexing and Motion-Induced Error Elimination

Three-dimensional (3D) reconstruction of dynamic objects has broad applications, including object recognition and robotic manipulation. However, achieving high-accuracy reconstruction and robustness to motion simultaneously is a challenging task. In this paper, we present a novel method for 3D reconstruction of dynamic objectS, whose main features are as follows. Firstly, a structured-light multiplexing method is developed that only requires 3 patterns to achieve high-accuracy encoding. Fewer projected patterns require shorter image acquisition time, thus, the object motion is reduced in each reconstruction cycle. The three patterns, i.e. spatial-temporally encoded patterns, are generated by embedding a specifically designed spatial-coded texture map into the temporal-encoded three-step phase-shifting fringes. A temporal codeword and three spatial codewords are extracted from the composite patterns using a proposed extraction algorithm. The two types of codewords are utilized separately in stereo matching: the temporal codeword ensures the high accuracy, while the spatial codewords are responsible for removing phase ambiguity. Secondly, we aim to eliminate the reconstruction error induced by motion between frames abbreviated as motion induced error (MiE). Instead of assuming the object to be static when acquiring the 3 images, we derive the motion of projection pixels among frames. Using the extracted spatial codewords, correspondences between different frames are found, i.e. pixels with the same codewords are traceable in the image sequences. Therefore, we can obtain the phase map at each image-acquisition moment without being affected by the object motion. Then the object surfaces corresponding to all the images can be recovered. Experimental results validate the high reconstruction accuracy and precision of the proposed method for dynamic objects with different motion speeds. Comparative experiments show that the presented method demonstrates superior performance with various types of motion, including translation in different directions and deformation.

Real-Time Φ-OTDR Vibration Event Recognition Based on Image Target Detection

Accurate and fast identification of vibration signals detected based on the phase-sensitive optical time-domain reflectometer (Φ-OTDR) is crucial in reducing the false-alarm rate of the long-distance distributed vibration warning system. This study proposes a computer vision-based Φ-OTDR multi-vibration events detection method in real-time, which can effectively detect perimeter intrusion events and reduce personnel patrol costs. Pulse accumulation, pulse cancellers, median filter, and pseudo-color processing are employed for vibration signal feature enhancement to generate vibration spatio-temporal images and form a customized dataset. This dataset is used to train and evaluate an improved YOLO-A30 based on the YOLO target detection meta-architecture to improve system performance. Experiments show that using this method to process 8069 vibration data images generated from 5 abnormal vibration activities for two types of fiber optic laying scenarios, buried underground or hung on razor barbed wire at the perimeter of high-speed rail, the system mAP@.5 is 99.5%, 555 frames per second (FPS), and can detect a theoretical maximum distance of 135.1 km per second. It can quickly and effectively identify abnormal vibration activities, reduce the false-alarm rate of the system for long-distance multi-vibration along high-speed rail lines, and significantly reduce the computational cost while maintaining accuracy.

Unsupervised deep learning for 3D reconstruction with dual-frequency fringe projection profilometry

The fringe projection profilometry (FPP) technique has been widely applied in three-dimensional (3D) reconstruction in industry for its high speed and high accuracy. Recently, deep learning has been successfully applied in FPP to achieve high-accuracy and robust 3D reconstructions in an efficient way. However, the network training needs to generate and label numerous ground truth 3D data, which can be time-consuming and labor-intensive. In this paper, we propose to design an unsupervised convolutional neural network (CNN) model based on dual-frequency fringe images to fix the problem. The fringe reprojection model is created to transform the output height map to the corresponding fringe image to realize the unsupervised training of the CNN. Our network takes two fringe images with different frequencies and outputs the corresponding height map. Unlike most of the previous works, our proposed network avoids numerous data annotations and can be trained without ground truth 3D data for unsupervised learning. Experimental results verify that our proposed unsupervised model (1) can get competitive-accuracy reconstruction results compared with previous supervised methods, (2) has excellent anti-noise and generalization performance and (3) saves time for dataset generation and labeling (3.2 hours, one-sixth of the supervised method) and computer space for dataset storage (1.27 GB, one-tenth of the supervised method).

Generalized phase unwrapping method that avoids jump errors for fringe projection profilometry

Jump errors easily occur on the discontinuity of the wrapped phase because of the misalignment between wrapped phase and fringe order in fringe projection profilometry (FPP). In this paper, a phase unwrapping method that avoids jump errors is proposed for FPP. By building two other staggered wrapped phases from the original wrapped phase and dividing each period of fringe order into three parts, the proposed generalized tripartite phase unwrapping (Tri-PU) method can be used to avoid rather than compensatorily correct jump errors. It is suitable for the phase unwrapping method assisted by fringe order with a basic wrapped phase and fringe order, no matter which method is used to recover them. The experimental results demonstrate the effectiveness and generality of the proposed method, which is simple to implement and superior to measure complex objects with sharp edges.

Phase Demodulation Method for Fringe Projection Measurement Based on Improved Variable-Frequency Coded Patterns

The phase-to-height imaging model, as a three-dimensional (3D) measurement technology, has been commonly applied in fringe projection to assist surface profile measurement, where the efficient and accurate calculation of phase plays a critical role in precise imaging. To deal with multiple extra coded patterns and 2π jump error caused to the existing absolute phase demodulation methods, a novel method of phase demodulation is proposed based on dual variable-frequency (VF) coded patterns. In this paper, the frequency of coded fringe is defined as the number of coded fringes within a single sinusoidal fringe period. First, the effective wrapped phase (EWP) as calculated using the four-step phase shifting method was split into the wrapped phase region with complete period and the wrapped phase region without complete period. Second, the fringe orders in wrapped phase region with complete period were decoded according to the frequency of the VF coded fringes and the continuous characteristic of the fringe order. Notably, the sampling frequency of fast Fourier transform (FFT) was determined by the length of the decoding interval and can be adjusted automatically with the variation in height of the object. Third, the fringe orders in wrapped phase region without complete period were decoded depending on the consistency of fringe orders in the connected region of wrapped phase. Last, phase demodulation was performed. The experimental results were obtained to confirm the effectiveness of the proposed method in the phase demodulation of both discontinuous objects and highly abrupt objects.

3-D shape reconstruction of non-uniform reflectance surface based on pixel intensity, pixel color and camera exposure time adaptive adjustment

High dynamic range 3-D shape measurement is a challenge. In this work, we propose a novel method to solve the 3-D shape reconstruction of high-reflection and colored surfaces. First, we propose a method to establish a fast pixel-level mapping between the projected image and the captured image. Secondly, we propose a color texture extraction method using a black-and-white (B/W) camera and a pixel-level projection color adjustment method. Third, we give an optimal projection fringe modulation/background intensity ratio. Fourth, we propose a method for estimating the reflectivity of the object surface and ambient light interference, and a method for adjusting the projection intensity at the pixel level and a method for estimating the optimal exposure time. Experiments show that, compared with the existing methods, the proposed method not only can obtain high-quality captured images, but also has higher measurement efficiency and wider application range.

Absolute phase retrieval for colored objects based on three phase-shifting amount codes

We propose an absolute phase retrieval method based on three phase-shifting amount codes (3-PSA-codes) to measure the colored object with one additional pattern. 3-PSA-codes adopt the coding concept of 3-digit-codes, in which the code elements of three consecutive periods are treated as a unique code word for one period. However, to measure the colored object more effectively in the proposed method, each code element is embedded into the PSA domain and retrieved from the phase difference. Fringe patterns for the wrapped phase are artfully employed in the code element retrieval. Hence, for the first time, to the best of our knowledge, the code element related to the phase can be determined by one additional pattern. It breaks the constraint that temporal methods require multiple additional patterns to overcome the adverse effect of the surface color of objects on absolute phase retrieval. Experimental results demonstrate that the proposed 3-PSA-codes have strong robustness in the measurement of the colored object.

Efficient intensity-based fringe projection profilometry method resistant to global illumination

Intensity-based fringe projection profilometry (IBFPP) is used widely because of its simple structure, high robustness, and noise resilience. Most IBFPP methods assume that any scene point is illuminated by direct illumination only, but global illumination effects introduce strong biases in the reconstruction result for many real-world scenes. To solve this problem, this paper describes an efficient IBFPP method for reconstructing three-dimensional geometry in the presence of global illumination. First, the average intensity of two sinusoidal patterns is used as a pixel-wise threshold to binarize the codeword patterns. The binarized template pattern is then used to convert other binarized fringe patterns into traditional Gray-code patterns. A proprietary compensation algorithm is then applied to eliminate fringe errors caused by environmental noise and lens defocusing. Finally, simple, efficient, and robust phase unwrapping can be achieved despite the effects of subsurface scattering and interreflection. Experimental results obtained in different environments show that the proposed method can obtain three-dimensional information reliably when influenced by global illumination.

Encoding technology of an asymmetric combined structured light for 3D measurement

Sinusoidal phase-shifting symmetrically combined with cyclic code is one of the most important encoding methods in the field of 3D measurement. Due to the modulation of the object surface and the influence of the noise of the image acquisition system, the periods of the cyclic code and the sinusoidal phase-shifting in the intensity image do not coincide completely, and they lead to large absolute phase decoding errors near the cycle boundaries, which are called cycle dislocation errors. In order to eliminate these errors in principle, the concept and method of region encoding for four-step sinusoidal phase-shifting are proposed, and the sinusoidal phase-shifting is combined with cyclic code asymmetrically. Under the premise that the cyclic code and the region code change at different times, the cycle dislocation error is reduced from one cycle of cyclic code to one pixel by the dual constraint of cyclic code and region code. The simulation measurement results of 3 ds max and the physical measurement results show that the asymmetric combination encoding method effectively eliminates the cycle dislocation errors; the maximum measurement error is reduced by an order of magnitude, and the root mean square measurement error is reduced by 70%.

Coding line structured light based on a line-scan camera and its calibration

In a conventional three-dimensional (3D) measurement technique of a line-scan camera, the projection system based on surface structured light is a compromise of traditional projection technology, which suffers from complex calibration, complex structure and low accuracy. To this end, the coding line structured light based on the coded line laser projection system is proposed to address the 3D measurement of a line-scan camera. The single-line projection and codeable characteristics of coded line laser projection system (constructed by a point laser and a micro-electro-mechanical system (MEMS) scanning galvanometer and modeled as the line projection model) are fully matched with the imaging mode of the line-scan camera. The 3D measurement model based on the height information, lateral information and absolute phase of the coding line structured light is derived. The multi-position flat display calibration method is proposed to calibrate the system parameters. In addition, in order to obtain the accurate absolute phase from the phase shift combined binary code, the periodic error correction method based on expansion-corrosion is proposed to correct the phase error. Contrary to conventional structured light methods based on a line-scan camera, the proposed method has the advantages of high measurement accuracy, high efficiency, more compactness and low cost. The experiments affirm that the coding line structured light is valid and the proposed calibration method is feasible. Experimental results also indicate that the proposed method performs well for both diffuse reflective surfaces and reflective surfaces that are difficult to measure with conventional structured light methods based on a line-scan camera.

Deep learning-based fringe modulation-enhancing method for accurate fringe projection profilometry

Fringe projection profilometry (i.e., FPP) has been one of the most popular 3-D measurement techniques. The phase error due to system random noise becomes non-ignorable when fringes captured by a camera have a low fringe modulation, which are inevitable for objects' surface with un-uniform reflectivity. The phase calculated from these low-modulation fringes may have a non-ignorable phase error and generate 3-D measurement error. Traditional methods reduce the phase error with losing details of 3-D shapes or sacrificing the measurement speed. In this paper, a deep learning-based fringe modulation-enhancing method (i.e., FMEM) is proposed, that transforms two low-modulation fringes with different phase shifts into a set of three phase-shifted high-modulation fringes. FMEM enables to calculate the desired phase from the transformed set of high-modulation fringes, and result in accurate 3-D FPP without sacrificing the speed. Experimental analysis verifies its effectiveness and accurateness.

Spatial pattern-shifting method for complete two-wavelength fringe projection profilometry

Two-wavelength fringe projection profilometry (FPP) unwraps a phase with the unambiguous phase range (UPR) of the least common multiple (LCM) of the two wavelengths. It is accurate, convenient, and robust, and thus plays an important role in shape measurement. However, when two non-coprime wavelengths are used, only a small UPR can be generated, and the unwrapping performance is compromised. In this Letter, a spatial pattern-shifting method (SPSM) is proposed to generate the maximum UPR (i.e., the product of the two wavelengths) from two non-coprime wavelengths. For the first time, to the best of our knowledge, the SPSM breaks the constraint of wavelength selection and enables a complete (i.e., either coprime or non-coprime) two-wavelength FPP. The SPSM, on the other hand, only requires spatially shift of the low-frequency pattern with the designed amounts and accordingly adjusting the fringe order determination, which is extremely convenient in implementation. Both numerical and experimental analyses verify its flexibility and correctness.

A Monitoring System for the Segmentation and Grading of Broccoli Head Based on Deep Learning and Neural Networks

Achieving the non-contact and non-destructive observation of broccoli head is the key step to realize the acquisition of high-throughput phenotyping information of broccoli. However, the rapid segmentation and grading of broccoli head remains difficult in many parts of the world due to low equipment development level. In this paper, we combined an advanced computer vision technique with a deep learning architecture to allow the acquisition of real-time and accurate information about broccoli head. By constructing a private image dataset with 100s of broccoli-head images (acquired using a self-developed imaging system) under controlled conditions, a deep convolutional neural network named "Improved ResNet" was trained to extract the broccoli pixels from the background. Then, a yield estimation model was built based on the number of extracted pixels and the corresponding pixel weight value. Additionally, the Particle Swarm Optimization Algorithm (PSOA) and the Otsu method were applied to grade the quality of each broccoli head according to our new standard. The trained model achieved an Accuracy of 0.896 on the test set for broccoli head segmentation, demonstrating the feasibility of this approach. When testing the model on a set of images with different light intensities or with some noise, the model still achieved satisfactory results. Overall, our approach of training a deep learning model using low-cost imaging devices represents a means to improve broccoli breeding and vegetable trade.

Dynamic 3-D measurement based on fringe-to-fringe transformation using deep learning

Fringe projection profilometry (FPP) has become increasingly important in dynamic 3-D shape measurement. In FPP, it is necessary to retrieve the phase of the measured object before shape profiling. However, traditional phase retrieval techniques often require a large number of fringes, which may generate motion-induced error for dynamic objects. In this paper, a novel phase retrieval technique based on deep learning is proposed, which uses an end-to-end deep convolution neural network to transform a single or two fringes into the phase retrieval required fringes. When the object's surface is located in a restricted depth, the presented network only requires a single fringe as the input, which otherwise requires two fringes in an unrestricted depth. The proposed phase retrieval technique is first theoretically analyzed, and then numerically and experimentally verified on its applicability for dynamic 3-D measurement.

3D Measurement of Human Chest and Abdomen Surface Based on 3D Fourier Transform and Time Phase Unwrapping

Monitoring respiratory movements is an effective way to improve radiotherapy treatments of thoracic and abdominal tumors, but the current approach is limited to measuring specific points in the chest and abdomen. In this paper, a dynamic three-dimensional (3D) measurement approach of the human chest and abdomen surface is proposed, which can infer tumor movement more accurately, so the radiotherapy damage to the human body can be reduced. Firstly, color stripe patterns in the RGB color model are projected, then after color correction, the collected stripe image sequences are separated into the three RGB primary color stripe image sequences. Secondly, a fringe projection approach is used to extract the folded phase combined 3D Fourier transform with 3D Gaussian filtering. By the relationship between adjacent fringe images in the time sequence, Gaussian filter parameters with individual characteristics are designed and optimized to improve the accuracy of wrapped phase extraction. In addition, based on the difference between the fractional parts of the folded phase error, one remainder equation can be determined, which is used for time-phase unwrapping. The simulation model and human experiments show that the proposed approach can obtain the 3D image sequences of the chest and abdomen surface in respiratory motion effectively and accurately with strong anti-interference ability.

Temporal phase unwrapping using deep learning

The multi-frequency temporal phase unwrapping (MF-TPU) method, as a classical phase unwrapping algorithm for fringe projection techniques, has the ability to eliminate the phase ambiguities even while measuring spatially isolated scenes or the objects with discontinuous surfaces. For the simplest and most efficient case in MF-TPU, two groups of phase-shifting fringe patterns with different frequencies are used: the high-frequency one is applied for 3D reconstruction of the tested object and the unit-frequency one is used to assist phase unwrapping for the wrapped phase with high frequency. The final measurement precision or sensitivity is determined by the number of fringes used within the high-frequency pattern, under the precondition that its absolute phase can be successfully recovered without any fringe order errors. However, due to the non-negligible noises and other error sources in actual measurement, the frequency of the high-frequency fringes is generally restricted to about 16, resulting in limited measurement accuracy. On the other hand, using additional intermediate sets of fringe patterns can unwrap the phase with higher frequency, but at the expense of a prolonged pattern sequence. With recent developments and advancements of machine learning for computer vision and computational imaging, it can be demonstrated in this work that deep learning techniques can automatically realize TPU through supervised learning, as called deep learning-based temporal phase unwrapping (DL-TPU), which can substantially improve the unwrapping reliability compared with MF-TPU even under different types of error sources, e.g., intensity noise, low fringe modulation, projector nonlinearity, and motion artifacts. Furthermore, as far as we know, our method was demonstrated experimentally that the high-frequency phase with 64 periods can be directly and reliably unwrapped from one unit-frequency phase using DL-TPU. These results highlight that challenging issues in optical metrology can be potentially overcome through machine learning, opening new avenues to design powerful and extremely accurate high-speed 3D imaging systems ubiquitous in nowadays science, industry, and multimedia.

Optimal processing scheme for restoration of phase data corrupted by strong decorrelation noise and dislocations

The presence of speckle noise and dislocations makes phase restoration potentially difficult in quantitative phase imaging and metrology. Unfortunately, there is no appropriate approach to deal with phase data corrupted by high speckle noise and phase dislocations. Usually, processing schemes may deal with low-pass phase filtering, phase unwrapping, or phase inpainting. This paper discusses the efficient processing to deal with noisy phase maps corrupted with phase dislocations. Six processing schemes, combining four operations, are evaluated. The investigation is carried out by realistic numerical simulations in which strong decorrelation phase noise and phase dislocations are generated. As a result, most robust and faster processing is established. The applicability of the optimal scheme is demonstrated through deformation measurement in dental materials.

Improved unwrapped phase retrieval method of a fringe projection profilometry system based on fewer phase-coding patterns

In this paper, based on two additional phase-coding patterns, an improved phase demodulation method is proposed. First, six equally spaced coding phases in the interval [$ - \pi $-π, $\pi $π] are embedded in different periods of the coded fringes following a certain sequence. Subsequently, since a group of phase orders can be uniquely determined by the four adjacent coding phases, the phase-order map of the object can be generated. To ensure the accuracy of decoding results, the interference coding numbers should be corrected in advance. In the meantime, the connected regions exhibiting the same orders are classified and then labeled for simplifying the decoding process. The simulation results verify the feasibility of the proposed method. By two groups of 3D imaging experiments, the applicability of this method to multiple objects and discontinuous objects is confirmed.

Enhanced phase-coding method for three-dimensional shape measurement with half-period codeword

The phase-coding method has been widely used for 3D shape measurement, which uses sinusoidal phase-shifting patterns to recover the wrapped phase and the stair phase-coding patterns to determine the fringe order. However, due to random noises and image blurring, the fringe order is always misaligned with the wrapped phase, which will lead to fringe order errors. This paper presents an enhanced phase-coding method to address this misalignment problem by using half-period codewords, in which each codeword is aligned to the half-period of the sinusoidal patterns. Then, two complementary fringe orders with half-period dislocation can be calculated, which can effectively eliminate the fringe order errors. To extend the coding range of stair phase, this paper further develops a computational scheme based on the geometric constraint method. Simulations and experiments have been carried out, and their results confirm that the enhanced method can reliably recover the 3D shape of the measured objects.

High-speed three-dimensional shape measurement based on shifting Gray-code light

The measuring technique combining a phase-shifting algorithm and Gray-code light has been widely used in three-dimensional (3D) shape measurement for static scenes because of its high robustness and anti-noise ability. However, in the high-speed measurement, phase unwrapping errors occur easily on the boundaries of adjacent Gray-code words because of the defocus of the projector, the motion of the objects and the non-uniform reflectivity of the surface. To overcome this challenge, a high-speed 3D shape measurement method based on shifting Gray-code light has been proposed in this paper. Firstly, the average intensity of three captured phase-shifting fringe images are used as a pixel-wise threshold to binarize the Gray codes and to eliminate most phase unwrapping errors caused by the non-uniform reflectivity, ambient light variations, and the defocus of projector. Then, the shifting Gray-code (SGC) coding strategy is proposed to avoid the remaining errors of phase unwrapping on the edge of the code words. In this strategy, no additional patterns are projected, and two sets of decoding words with staggered boundaries are built in the temporal sequences for one wrapped phase. Finally, the simple, efficient, and robust phase unwrapping can be achieved in the high-speed dynamic measurement. This proposed method has been applied to reconstruct 3D shape of randomly collapsing objects in a large depth range, and the experimental results demonstrate that it can reliably obtain high-quality shape and texture information at 310 frames per second.

Robust structured-light depth mapping via recursive decomposition of binary codes

Structured-light depth cameras rely on projecting and resolving coded patterns on a three-dimensional scene with high contrast. The front-end optics of such depth cameras impose a fundamental restriction on the depth-sensing range and accuracy: the patterns only remain sharp within the depth of field jointly determined by the camera and projector. We present here a robust method to improve the depth-sensing range and accuracy for a structured-light depth camera without changing the underlying optical design. Moreover, it shows the unique advantage in macrophotography of highly light-scattering objects. We analyze the proposed method theoretically and validate it in experiments.

Binocular vision profilometry for large-sized rough optical elements using binarized band-limited pseudo-random patterns

In this paper, a non-contact binocular vision profilometry method is proposed to measure a rough lens with aperture of around 300mm. A series of binarized band-limited pseudo-random patterns (BBPPs) are projected onto the rough lens, we utilize the temporal encoding method so that each pixel in the captured images has its specific code word. Homologous points could be matched via stereo matching procedure, then the surface of the rough lens will be reconstructed based on triangulation method according to the previous calibration data. Compared with the three coordinate measuring machine (CMM), this method achieves a fast and cheap measurement of the large-sized rough lens, which might be highly interesting for fast and overall measurement of metre-sized rough elements in the future.

A simple and practical jump error removal method for fringe projection profilometry based on self-alignment technique

The code-based method is one of the frequently adopted fringe projection profilometry techniques because of its robustness and high speed. However, the abnormal jump errors caused by the misalignment between the wrapped phase and the fringe order impact the phase unwrapping quality and are more serious in binary defocusing measurement with significant defocusing. This paper proposes a self-alignment technique (SAT) with high speed, no additional patterns, and no accuracy loss to eliminate such jump errors. After analyzing the relation between the grating changes and the misalignment, we designed an assist-code that can perfectly align with the wrapped phase even in significant defocusing. After that, this assist-code is used to adjust the misalignment and eliminate the jump errors. The comparison between the median filter and the proposed SAT method in simulations and experiments demonstrates that the proposed SAT method has the ability to completely remove jump errors for complex textures and step-height objects, even in significant defocusing, indicating its potential to be applied to other phase unwrapping fields, e.g., color-based methods with color coupling problems.

Reference-plane-based fast pixel-by-pixel absolute phase retrieval for height measurement

Absolute phase retrieval is essential for height measurement in digital fringe projection. However, projections of additional structured patterns that are normally required for phase unwrapping increase the measurement complexity. In this paper, we propose two reference-plane-based pixel-by-pixel absolute phase retrieval techniques with as few projections as possible, suitable for different object depth ranges. The wrapped phase on the object is absolutely unwrapped by referring just to the absolute phase map on the reference plane. Single-frequency absolute phase retrieval with one-reference-plane-based calibration is first proposed for objects within a height limit that equals a calibrated system constant. To extend the measurement depth range, dual-frequency absolute phase retrieval with two parallel reference planes is further proposed. The additional low frequency is used to choose the unwrapping reference from the two reference plane phases for unwrapping the high-frequency phase. Moreover, the proposed techniques are capable of high-frequency absolute phase unwrapping for objects with step-height surface discontinuities. Experiments have been conducted to demonstrate the efficiency of the proposed two techniques.

Suppression of the nonlinear phase error in phase shifting profilometry: considering non-smooth reflectivity and fractional period

Hilbert transform (HT) has been employed to compensate phase error arising from the nonlinear effect in phase shifting profilometry (PSP). However, in most common situations, pure HT may lead to a significant system error, which has a negative impact on subsequent phase error compensation. In this paper, system error from HT of non-stationary and non-continuous fringe is analyzed, and then a novel phase error suppression approach is presented. The cosine fringe without direct current (DC) component is reconstructed to eliminate the influence of non-smooth reflectivity, and the fractional periods at both ends of the reconstructed fringe are extended to generate fringe with integer number of periods. And then the HT is applied to the reconstructed and extended fringe. Finally, a revised phase-shifting algorithm is employed to calculate the phase with the fringe after HT. The proposed approach is suitable for PSP of the surface with non-smooth reflectivity (e.g. texture of complex colors), which is demonstrated in a series of experiments.

Dual-sensitivity profilometry with defocused projection of binary fringes

A dual-sensitivity profilometry technique based on defocused projection of binary fringes is presented. Here, two sets of fringe patterns with a sinusoidal profile are produced by applying the same analog low-pass filter (projector defocusing) to binary fringes with a high- and low-frequency spatial carrier. The high-frequency fringes have a binary square-wave profile, while the low-frequency binary fringes are produced with error-diffusion dithering. The binary nature of the binary fringes removes the need for calibration of the projector's nonlinear gamma. Working with high-frequency carrier fringes, we obtain a high-quality wrapped phase. On the other hand, working with low-frequency carrier fringes we found a lower-quality, nonwrapped phase map. The nonwrapped estimation is used as stepping stone for dual-sensitivity temporal phase unwrapping, extending the applicability of the technique to discontinuous (piecewise continuous) surfaces. We are proposing a single defocusing level for faster high- and low-frequency fringe data acquisition. The proposed technique is validated with experimental results.

Ternary Gray code-based phase unwrapping for 3D measurement using binary patterns with projector defocusing

The three-dimensional measurement technique using binary pattern projection with projector defocusing has become increasingly important due to its high speed and high accuracy. To obtain even faster speed without sacrificing accuracy, a ternary Gray code-based phase-unwrapping method is proposed by using even fewer binary patterns, which makes it possible to efficiently and accurately unwrap the phase. Theoretical analysis, simulations, and experiments are presented to validate the proposed method's efficiency and robustness.