Temporal phase unwrapping using deep learning

SE-FSCNet: full-scale connection network for single-shot phase demodulation

The accuracy of phase demodulation has significant impact on the accuracy of fringe projection 3D measurement. Currently, researches based on deep learning methods for extracting wrapped phase mostly use U-Net as the subject of network. The connection method between its hierarchies has certain shortcomings in global information transmission, which hinders the improvement of wrapped phase prediction accuracy. We propose a single-shot phase demodulation method for fringe projection based on a novel full-scale connection network SE-FSCNet. The encoder and decoder of the SE-FSCNet have the same number of hierarchies but are not completely symmetrical. At the decoder a full-scale connection method and feature fusion module are designed so that SE-FSCNet has better abilities of feature transmission and utilization compared with U-Net. A channel attention module based on squeeze and excitation is also introduced to assign appropriate weights to features with different scales, which has been proved by the ablation study. The experiments conducted on the test set have demonstrated that the SE-FSCNet can achieve higher precision than the traditional Fourier transform method and the U-Net in phase demodulation.

Efficient and robust phase unwrapping method based on SFNet

Phase unwrapping is a crucial step in obtaining the final physical information in the field of optical metrology. Although good at dealing with phase with discontinuity and noise, most deep learning-based spatial phase unwrapping methods suffer from the complex model and unsatisfactory performance, partially due to simple noise type for training datasets and limited interpretability. This paper proposes a highly efficient and robust spatial phase unwrapping method based on an improved SegFormer network, SFNet. The SFNet structure uses a hierarchical encoder without positional encoding and a decoder based on a lightweight fully connected multilayer perceptron. The proposed method utilizes the self-attention mechanism of the Transformer to better capture the global relationship of phase changes and reduce errors in the phase unwrapping process. It has a lower parameter count, speeding up the phase unwrapping. The network is trained on a simulated dataset containing various types of noise and phase discontinuity. This paper compares the proposed method with several state-of-the-art deep learning-based and traditional methods in terms of important evaluation indices, such as RMSE and PFS, highlighting its structural stability, robustness to noise, and generalization.

Phase unwrapping for phase imaging using the plug-and-play proximal algorithm

Phase unwrapping (PU) is essential for various scientific optical applications. This process aims to estimate continuous phase values from acquired wrapped values, which are limited to the interval (-π,π]. However, the PU process can be challenging due to factors such as insufficient sampling, measurement errors, and inadequate equipment calibration, which can introduce excessive noise and unexpected phase discontinuities. This paper presents a robust iterative method based on the plug-and-play (PnP) proximal algorithm to unwrap two-dimensional phase values while simultaneously removing noise at each iteration. Using a least-squares formulation based on local phase differences and reformulating it as a partially differentiable equation, it is possible to employ the fast cosine transform to obtain a closed-form solution for one of the subproblems within the PnP framework. As a result, reliable phase reconstruction can be achieved even in scenarios with extremely high noise levels.

Triple-output phase unwrapping network with a physical prior in fringe projection profilometry

Deep learning has been attracting more and more attention in the phase unwrapping of fringe projection profilometry (FPP) in recent years. In order to improve the accuracy of the deep-learning-based unwrapped phase methods from a single fringe pattern, this paper proposes a single-input triple-output neural network structure with a physical prior. In the proposed network, a single-input triple-output network structure is developed to convert the input fringe pattern into three intermediate outputs: the wrapped phase, the fringe order, the coarse unwrapped phase, and the final output high-precision unwrapped phase from the three outputs. Moreover, a new, to the best of our knowledge, loss function is designed to improve the performance of the model using a physical prior about these three outputs in FPP. Numerous experiments demonstrated that the proposed network is able to improve the accuracy of the unwrapped phase, which can also be extended to other deep learning phase unwrapping models.

Deep learning assisted non-contact defect identification method using diffraction phase microscopy

Reliable detection of defects from optical fringe patterns is a crucial problem in non-destructive optical interferometric metrology. In this work, we propose a deep-learning-based method for fringe pattern defect identification. By attributing the defect information to the fringe pattern's phase gradient, we compute the spatial phase derivatives using the deep learning model and apply the gradient map to localize the defect. The robustness of the proposed method is illustrated on multiple numerically synthesized fringe pattern defects at various noise levels. Further, the practical utility of the proposed method is substantiated for experimental defect identification in diffraction phase microscopy.

Phase unwrapping using deep learning in holographic tomography

Holographic tomography (HT) is a measurement technique that generates phase images, often containing high noise levels and irregularities. Due to the nature of phase retrieval algorithms within the HT data processing, the phase has to be unwrapped before tomographic reconstruction. Conventional algorithms lack noise robustness, reliability, speed, and possible automation. In order to address these problems, this work proposes a convolutional neural network based pipeline consisting of two steps: denoising and unwrapping. Both steps are carried out under the umbrella of a U-Net architecture; however, unwrapping is aided by introducing Attention Gates (AG) and Residual Blocks (RB) to the architecture. Through the experiments, the proposed pipeline makes possible the phase unwrapping of highly irregular, noisy, and complex experimental phase images captured in HT. This work proposes phase unwrapping carried out by segmentation with a U-Net network, that is aided by a pre-processing denoising step. It also discusses the implementation of the AGs and RBs in an ablation study. What is more, this is the first deep learning based solution that is trained solely on real images acquired with HT.

High-precision dynamic three-dimensional shape measurement of specular surfaces based on deep learning

In order to solve the difficulty of traditional phase measuring deflectometry (PMD) in considering precision and speed, an orthogonal encoding PMD method based on deep learning is presented in this paper. We demonstrate for, what we believe to be, the first time that deep learning techniques can be combined with dynamic-PMD and can be used to reconstruct high-precision 3D shapes of specular surfaces from single-frame distorted orthogonal fringe patterns, enabling high-quality dynamic measurement of specular objects. The experimental results prove that the phase and shape information measured by the proposed method has high accuracy, almost reaching the results obtained by the ten-step phase-shifting method. And the proposed method also has excellent performance in dynamic experiments, which is of great significance to the development of optical measurement and fabrication areas.

Unifying temporal phase unwrapping framework using deep learning

Temporal phase unwrapping (TPU) is significant for recovering an unambiguous phase of discontinuous surfaces or spatially isolated objects in fringe projection profilometry. Generally, temporal phase unwrapping algorithms can be classified into three groups: the multi-frequency (hierarchical) approach, the multi-wavelength (heterodyne) approach, and the number-theoretic approach. For all of them, extra fringe patterns of different spatial frequencies are required for retrieving the absolute phase. Due to the influence of image noise, people have to use many auxiliary patterns for high-accuracy phase unwrapping. Consequently, image noise limits the efficiency and the measurement speed greatly. Further, these three groups of TPU algorithms have their own theories and are usually applied in different ways. In this work, for the first time to our knowledge, we show that a generalized framework using deep learning can be developed to perform the TPU task for different groups of TPU algorithms. Experimental results show that benefiting from the assistance of deep learning the proposed framework can mitigate the impact of noise effectively and enhance the phase unwrapping reliability significantly without increasing the number of auxiliary patterns for different TPU approaches. We believe that the proposed method demonstrates great potential for developing powerful and reliable phase retrieval techniques.

Single-shot 3D measurement of highly reflective objects with deep learning

Three-dimensional (3D) measurement methods based on fringe projection profilometry (FPP) have been widely applied in industrial manufacturing. Most FPP methods adopt phase-shifting techniques and require multiple fringe images, thus having limited application in dynamic scenes. Moreover, industrial parts often have highly reflective areas leading to overexposure. In this work, a single-shot high dynamic range 3D measurement method combining FPP with deep learning is proposed. The proposed deep learning model includes two convolutional neural networks: exposure selection network (ExSNet) and fringe analysis network (FrANet). The ExSNet utilizes self-attention mechanism for enhancement of highly reflective areas leading to overexposure problem to achieve high dynamic range in single-shot 3D measurement. The FrANet consists of three modules to predict wrapped phase maps and absolute phase maps. A training strategy directly opting for best measurement accuracy is proposed. Experiments on a FPP system showed that the proposed method predicted accurate optimal exposure time under single-shot condition. A pair of moving standard spheres with overexposure was measured for quantitative evaluation. The proposed method reconstructed standard spheres over a large range of exposure level, where prediction errors for diameter were 73 µm (left) and 64 µm (right) and prediction error for center distance was 49 µm. Ablation study and comparison with other high dynamic range methods were also conducted.

Generalized Fringe-to-Phase Framework for Single-Shot 3D Reconstruction Integrating Structured Light with Deep Learning

Three-dimensional (3D) shape acquisition of objects from a single-shot image has been highly demanded by numerous applications in many fields, such as medical imaging, robotic navigation, virtual reality, and product in-line inspection. This paper presents a robust 3D shape reconstruction approach integrating a structured-light technique with a deep learning-based artificial neural network. The proposed approach employs a single-input dual-output network capable of transforming a single structured-light image into two intermediate outputs of multiple phase-shifted fringe patterns and a coarse phase map, through which the unwrapped true phase distributions containing the depth information of the imaging target can be accurately determined for subsequent 3D reconstruction process. A conventional fringe projection technique is employed to prepare the ground-truth training labels, and part of its classic algorithm is adopted to preserve the accuracy of the 3D reconstruction. Numerous experiments have been conducted to assess the proposed technique, and its robustness makes it a promising and much-needed tool for scientific research and engineering applications.

Deep learning-enabled anti-ambient light approach for fringe projection profilometry

Achieving high-quality surface profiles under strong ambient light is challenging in fringe projection profilometry (FPP) since ambient light inhibits functional illumination from exhibiting sinusoidal stripes with high quantization levels. Conventionally, large-step phase shifting approaches are presented to enhance the anti-interference capability of FPP, but the image acquisition process in these approaches is highly time-consuming. Inspired by the promising performance of deep learning in optical metrology, we propose a deep learning-enabled anti-ambient light (DLAL) approach that can help FPP extract phase distributions from a single fringe image exposed to unbalanced lighting. In this work, the interference imposed by ambient light on FPP is creatively modeled as ambient light-induced phase error (ALPE). Guided by the ALPE model, we generate the dataset by precisely adjusting the stripe contrast before performing active projection, overcoming the challenge of collecting a large sample of fringe images with various illumination conditions. Driven by the novel dataset, the generated deep learning model can effectively suppress outliers among surface profiles in the presence of strong ambient light, thereby implementing high-quality 3D surface imaging. Experimentally, we verify the effectiveness and adaptability of the proposed DLAL approach in both indoor and outdoor scenarios with strong irradiation.

Different structured-light patterns in single-shot 2D-to-3D image conversion using deep learning

Single-shot 3D shape reconstruction integrating structured light and deep learning has drawn considerable attention and achieved significant progress in recent years due to its wide-ranging applications in various fields. The prevailing deep-learning-based 3D reconstruction using structured light generally transforms a single fringe pattern to its corresponding depth map by an end-to-end artificial neural network. At present, it remains unclear which kind of structured-light patterns should be employed to obtain the best accuracy performance. To answer this fundamental and much-asked question, we conduct an experimental investigation of six representative structured-light patterns adopted for single-shot 2D-to-3D image conversion. The assessment results provide a valuable guideline for structured-light pattern selection in practice.

High-quality binary fringe generation by multi-scale optimization on intensity and phase

The optimization-based binary fringe generation method can greatly improve the quality of projected fringes for structured light measurement. However, the existing phase-based and intensity-based methods are either sensitive to the projector defocus levels or cannot reduce the phase error efficiently, and these two methods only optimize the phase or intensity at a single defocus scale. A new method is proposed to optimize the intensity and phase of binary fringe on multiple defocus scales, to the best of our knowledge, which can effectively reduce the phase error and ensure its robustness. The algorithm is accelerated by block optimization. We also proposed a strategy based on objective index evaluation to simplify the process of selecting the best binary patch. Simulation and experiment results show that the binary fringe generated by the proposed method is of high-phasing quality while keeping robust to different defocus levels of the projector.

ContransGAN: Convolutional Neural Network Coupling Global Swin-Transformer Network for High-Resolution Quantitative Phase Imaging with Unpaired Data

Optical quantitative phase imaging (QPI) is a frequently used technique to recover biological cells with high contrast in biology and life science for cell detection and analysis. However, the quantitative phase information is difficult to directly obtain with traditional optical microscopy. In addition, there are trade-offs between the parameters of traditional optical microscopes. Generally, a higher resolution results in a smaller field of view (FOV) and narrower depth of field (DOF). To overcome these drawbacks, we report a novel semi-supervised deep learning-based hybrid network framework, termed ContransGAN, which can be used in traditional optical microscopes with different magnifications to obtain high-quality quantitative phase images. This network framework uses a combination of convolutional operation and multiheaded self-attention mechanism to improve feature extraction, and only needs a few unpaired microscopic images to train. The ContransGAN retains the ability of the convolutional neural network (CNN) to extract local features and borrows the ability of the Swin-Transformer network to extract global features. The trained network can output the quantitative phase images, which are similar to those restored by the transport of intensity equation (TIE) under high-power microscopes, according to the amplitude images obtained by low-power microscopes. Biological and abiotic specimens were tested. The experiments show that the proposed deep learning algorithm is suitable for microscopic images with different resolutions and FOVs. Accurate and quick reconstruction of the corresponding high-resolution (HR) phase images from low-resolution (LR) bright-field microscopic intensity images was realized, which were obtained under traditional optical microscopes with different magnifications.

Single-exposure height-recovery structured illumination microscopy based on deep learning

Modulation-based structured illumination microscopy (SIM) is performed to reconstruct three-dimensional (3D) surface topography. Generally speaking, modulation decoding algorithms mainly include a phase-shift (PS) method and frequency analysis technique. The PS method requires at least three images with fixed PSs, which leads to low efficiency. Frequency methods could decode modulation from a single image, but the loss of high-frequency information is inevitable. In addition, these methods all need to calculate the mapping relationship between modulation and height to recover the 3D shape. In this paper, we propose a deep learning enabled single-exposure surface measurement method. With only one fringe image, this method can directly restore the height information of the object. Processes such as denoising, modulation calculation, and height mapping are all included in the neural network. Compared with traditional Fourier methods, our method has higher accuracy and efficiency. Experimental results demonstrate that the proposed method can provide accurate and fast surface measurement for different structures.

Deep learning-based Phase Measuring Deflectometry for single-shot 3D shape measurement and defect detection of specular objects

Phase Measuring Deflectometry (PMD) and Structured-Light Modulation Analysis Technique (SMAT) perform effectively in shape and defect measurements of specular objects, but the difficulty of giving consideration to accuracy and speed has also restricted the further development and application of them. Inspired by recent successes of deep learning techniques for computational imaging, we demonstrate for the first time that deep learning techniques can be used to recover high-precision modulation distributions of specular surfaces from a single-frame fringe pattern under SMAT, enabling fast and high-quality defect detection of specular surfaces. This method can also be applied to recover higher-precision phase distributions of specular surfaces from a single-frame fringe pattern under PMD, so as to realize the 3D shape measurement. In this paper, we combine depthwise separable convolution, residual structure and U-Net to build an improved U-Net network. The experimental results prove that the method has excellent performance in the phase and modulation retrieval of specular surfaces, which almost reach the accuracy of the results obtained by ten-step phase-shifting method.

SCDeep: Single-Channel Depth Encoding for 3D-Range Geometry Compression Utilizing Deep-Learning Techniques

Recent advances in optics and computing technologies have encouraged many applications to adopt the use of three-dimensional (3D) data for the measurement and visualization of the world around us. Modern 3D-range scanning systems have become much faster than real-time and are able to capture data with incredible precision. However, increasingly fast acquisition speeds and high fidelity data come with increased storage and transmission costs. In order to enable applications that wish to utilize these technologies, efforts must be made to compress the raw data into more manageable formats. One common approach to compressing 3D-range geometry is to encode its depth information within the three color channels of a traditional 24-bit RGB image. To further reduce file sizes, this paper evaluates two novel approaches to the recovery of floating-point 3D range data from only a single-channel 8-bit image using machine learning techniques. Specifically, the recovery of depth data from a single channel is enabled through the use of both semantic image segmentation and end-to-end depth synthesis. These two distinct approaches show that machine learning techniques can be utilized to enable significant file size reduction while maintaining reconstruction accuracy suitable for many applications. For example, a complex set of depth data encoded using the proposed method, stored in the JPG 20 format, and recovered using semantic segmentation techniques was able to achieve an average RMS reconstruction accuracy of 99.18% while achieving an average compression ratio of 106:1 when compared to the raw floating-point data. When end-to-end synthesis techniques were applied to the same encoded dataset, an average reconstruction accuracy of 99.59% was experimentally demonstrated for the same average compression ratio.

Time-overlapping structured-light projection: high performance on 3D shape measurement for complex dynamic scenes

High-speed three-dimensional (3D) shape measurement has been continuously researched due to the demand for analyzing dynamic behavior in transient scenes. In this work, a time-overlapping structured-light 3D shape measuring technique is proposed to realize high-speed and high-performance measurement on complex dynamic scenes. Time-overlapping structured-light projection is presented to maximumly reduce the information redundancy in temporal sequences and improve the measuring efficiency; generalized tripartite phase unwrapping (Tri-PU) is used to ensure the measuring robustness; fringe period extension is achieved by improving overlapping rate to further double the encoding fringe periods for higher measuring accuracy. Based on the proposed measuring technique, one new pixel-to-pixel and unambiguous 3D reconstruction result can be updated with three newly required patterns at a reconstruction rate of 3174 fps. Three transient scenes including collapsing wood blocks struck by a flying arrow, free-falling foam snowflakes and flying water balloon towards metal grids were measured to verify the high performance of the proposed method in various complex dynamic scenes.

Optical metrology embraces deep learning: keeping an open mind

Optical metrology practitioners ought to embrace deep learning with an open mind, while devote continuing efforts to look for its theoretical groundwork and maintain an awareness of its limits.

Deep learning in optical metrology: a review

With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional "physics-based" approach, deep-learning-enabled optical metrology is a kind of "data-driven" approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

Advances and Prospects of Vision-Based 3D Shape Measurement Methods

Vision-based three-dimensional (3D) shape measurement techniques have been widely applied over the past decades in numerous applications due to their characteristics of high precision, high efficiency and non-contact. Recently, great advances in computing devices and artificial intelligence have facilitated the development of vision-based measurement technology. This paper mainly focuses on state-of-the-art vision-based methods that can perform 3D shape measurement with high precision and high resolution. Specifically, the basic principles and typical techniques of triangulation-based measurement methods as well as their advantages and limitations are elaborated, and the learning-based techniques used for 3D vision measurement are enumerated. Finally, the advances of, and the prospects for, further improvement of vision-based 3D shape measurement techniques are proposed.

Composite fringe projection deep learning profilometry for single-shot absolute 3D shape measurement

Single-shot fringe projection profilometry (FPP) is essential for retrieving the absolute depth information of the objects in high-speed dynamic scenes. High-precision 3D reconstruction using only one single pattern has become the ultimate goal in FPP. The frequency-multiplexing (FM) method is a promising strategy for realizing single-shot absolute 3D measurement by compounding multi-frequency fringe information for phase unwrapping. In order to solve the problem of serious spectrum aliasing caused by multiplexing schemes that cannot be removed by traditional spectrum analysis algorithms, we apply deep learning to frequency multiplexing composite fringe projection and propose a composite fringe projection deep learning profilometry (CDLP). By combining physical model and data-driven approaches, we demonstrate that the model generated by training an improved deep convolutional neural network can directly perform high-precision and unambiguous phase retrieval on a single-shot spatial frequency multiplexing composite fringe image. Experiments on both static and dynamic scenes demonstrate that our method can retrieve robust and unambiguous phases information while avoiding spectrum aliasing and reconstruct high-quality absolute 3D surfaces of objects only by projecting a single composite fringe image.

Direct left-ventricular global longitudinal strain (GLS) computation with a fully convolutional network

This study's purpose was to develop a direct MRI-based, deep-learning semantic segmentation approach for computing global longitudinal strain (GLS), a known metric for detecting left-ventricular (LV) cardiotoxicity in breast cancer. Displacement Encoding with Stimulated Echoes cardiac image phases acquired from 30 breast cancer patients and 30 healthy females were unwrapped via a DeepLabV3 + fully convolutional network (FCN). Myocardial strains were directly computed from the unwrapped phases with the Radial Point Interpolation Method. FCN-unwrapped phases of a phantom's rotating gel were validated against quality-guided phase-unwrapping (QGPU) and robust transport of intensity equation (RTIE) phase-unwrapping. FCN performance on unwrapping human LV data was measured with F1 and Dice scores versus QGPU ground-truth. The reliability of FCN-based strains was assessed against RTIE-based strains with Cronbach's alpha (C-α) intraclass correlation coefficient. Mean squared error (MSE) of unwrapping the phantom experiment data at 0 dB signal-to-noise ratio were 1.6, 2.7 and 6.1 with FCN, QGPU and RTIE techniques. Human data classification accuracies were F1 = 0.95 (Dice = 0.96) with FCN and F1 = 0.94 (Dice = 0.95) with RTIE. GLS results from FCN and RTIE were -16 ± 3% vs. -16 ± 3% (C-α = 0.9) for patients and -20 ± 3% vs. -20 ± 3% (C-α = 0.9) for healthy subjects. The low MSE from the phantom validation demonstrates accuracy of phase-unwrapping with the FCN and comparable human subject results versus RTIE demonstrate GLS analysis accuracy. A deep-learning methodology for phase-unwrapping in medical images and GLS computation was developed and validated in a heterogeneous cohort.

Deep learning for digital holography: a review

Recent years have witnessed the unprecedented progress of deep learning applications in digital holography (DH). Nevertheless, there remain huge potentials in how deep learning can further improve performance and enable new functionalities for DH. Here, we survey recent developments in various DH applications powered by deep learning algorithms. This article starts with a brief introduction to digital holographic imaging, then summarizes the most relevant deep learning techniques for DH, with discussions on their benefits and challenges. We then present case studies covering a wide range of problems and applications in order to highlight research achievements to date. We provide an outlook of several promising directions to widen the use of deep learning in various DH applications.

Unsupervised deep learning for 3D reconstruction with dual-frequency fringe projection profilometry

The fringe projection profilometry (FPP) technique has been widely applied in three-dimensional (3D) reconstruction in industry for its high speed and high accuracy. Recently, deep learning has been successfully applied in FPP to achieve high-accuracy and robust 3D reconstructions in an efficient way. However, the network training needs to generate and label numerous ground truth 3D data, which can be time-consuming and labor-intensive. In this paper, we propose to design an unsupervised convolutional neural network (CNN) model based on dual-frequency fringe images to fix the problem. The fringe reprojection model is created to transform the output height map to the corresponding fringe image to realize the unsupervised training of the CNN. Our network takes two fringe images with different frequencies and outputs the corresponding height map. Unlike most of the previous works, our proposed network avoids numerous data annotations and can be trained without ground truth 3D data for unsupervised learning. Experimental results verify that our proposed unsupervised model (1) can get competitive-accuracy reconstruction results compared with previous supervised methods, (2) has excellent anti-noise and generalization performance and (3) saves time for dataset generation and labeling (3.2 hours, one-sixth of the supervised method) and computer space for dataset storage (1.27 GB, one-tenth of the supervised method).

Three-Dimensional Morphology and Size Measurement of High-Temperature Metal Components Based on Machine Vision Technology: A Review

The three-dimensional (3D) size and morphology of high-temperature metal components need to be measured in real time during manufacturing processes, such as forging and rolling. Since the surface temperature of a metal component is very high during the forming and manufacturing process, manually measuring the size of a metal component at a close distance is difficult; hence, a non-contact measurement technology is required to complete the measurement. Recently, machine vision technology has been developed, which is a non-contact measurement technology that only needs to capture multiple images of a measured object to obtain the 3D size and morphology information, and this technology can be used in some extreme conditions. Machine vision technology has been widely used in industrial, agricultural, military and other fields, especially fields involving various high-temperature metal components. This paper provides a comprehensive review of the application of machine vision technology in measuring the 3D size and morphology of high-temperature metal components. Furthermore, according to the principle and method of measuring equipment structures, this review highlights two aspects in detail: laser scanning measurement and multi-view stereo vision technology. Special attention is paid to each method through comparisons and analyses to provide essential technical references for subsequent researchers.

Single-shot 3D shape measurement using an end-to-end stereo matching network for speckle projection profilometry

Speckle projection profilometry (SPP), which establishes the global correspondences between stereo images by projecting only a single speckle pattern, has the advantage of single-shot 3D reconstruction. Nevertheless, SPP suffers from the low matching accuracy of traditional stereo matching algorithms, which fundamentally limits its 3D measurement accuracy. In this work, we propose a single-shot 3D shape measurement method using an end-to-end stereo matching network for SPP. To build a high-quality SPP dataset for training the network, by combining phase-shifting profilometry (PSP) and temporal phase unwrapping techniques, high-precision absolute phase maps can be obtained to generate accurate and dense disparity maps with high completeness as the ground truth by phase matching. For the architecture of the network, a multi-scale residual subnetwork is first leveraged to synchronously extract compact feature tensors with 1/4 resolution from speckle images for constructing the 4D cost volume. Considering that the cost filtering based on 3D convolution is computationally costly, a lightweight 3D U-net network is proposed to implement efficient 4D cost aggregation. In addition, because the disparity maps in the SPP dataset should have valid values only in the foreground, a simple and fast saliency detection network is integrated to avoid predicting the invalid pixels in the occlusions and background regions, thereby implicitly enhancing the matching accuracy for valid pixels. Experiment results demonstrated that the proposed method improves the matching accuracy by about 50% significantly compared with traditional stereo matching methods. Consequently, our method achieves fast and absolute 3D shape measurement with an accuracy of about 100µm through a single speckle pattern.

Fully-automated global and segmental strain analysis of DENSE cardiovascular magnetic resonance using deep learning for segmentation and phase unwrapping

BACKGROUND

Cardiovascular magnetic resonance (CMR) cine displacement encoding with stimulated echoes (DENSE) measures heart motion by encoding myocardial displacement into the signal phase, facilitating high accuracy and reproducibility of global and segmental myocardial strain and providing benefits in clinical performance. While conventional methods for strain analysis of DENSE images are faster than those for myocardial tagging, they still require manual user assistance. The present study developed and evaluated deep learning methods for fully-automatic DENSE strain analysis.

METHODS

Convolutional neural networks (CNNs) were developed and trained to (a) identify the left-ventricular (LV) epicardial and endocardial borders, (b) identify the anterior right-ventricular (RV)-LV insertion point, and (c) perform phase unwrapping. Subsequent conventional automatic steps were employed to compute strain. The networks were trained using 12,415 short-axis DENSE images from 45 healthy subjects and 19 heart disease patients and were tested using 10,510 images from 25 healthy subjects and 19 patients. Each individual CNN was evaluated, and the end-to-end fully-automatic deep learning pipeline was compared to conventional user-assisted DENSE analysis using linear correlation and Bland Altman analysis of circumferential strain.

RESULTS

LV myocardial segmentation U-Nets achieved a DICE similarity coefficient of 0.87 ± 0.04, a Hausdorff distance of 2.7 ± 1.0 pixels, and a mean surface distance of 0.41 ± 0.29 pixels in comparison with manual LV myocardial segmentation by an expert. The anterior RV-LV insertion point was detected within 1.38 ± 0.9 pixels compared to manually annotated data. The phase-unwrapping U-Net had similar or lower mean squared error vs. ground-truth data compared to the conventional path-following method for images with typical signal-to-noise ratio (SNR) or low SNR (p < 0.05), respectively. Bland-Altman analyses showed biases of 0.00 ± 0.03 and limits of agreement of - 0.04 to 0.05 or better for deep learning-based fully-automatic global and segmental end-systolic circumferential strain vs. conventional user-assisted methods.

CONCLUSIONS

Deep learning enables fully-automatic global and segmental circumferential strain analysis of DENSE CMR providing excellent agreement with conventional user-assisted methods. Deep learning-based automatic strain analysis may facilitate greater clinical use of DENSE for the quantification of global and segmental strain in patients with cardiac disease.

A Fringe Phase Extraction Method Based on Neural Network

In optical metrology, the output is usually in the form of a fringe pattern, from which a phase map can be generated and phase information can be converted into the desired parameters. This paper proposes an end-to-end method of fringe phase extraction based on the neural network. This method uses the U-net neural network to directly learn the correspondence between the gray level of a fringe pattern and the wrapped phase map, which is simpler than the exist deep learning methods. The results of simulation and experimental fringe patterns verify the accuracy and the robustness of this method. While it yields the same accuracy, the proposed method features easier operation and a simpler principle than the traditional phase-shifting method and has a faster speed than wavelet transform method.

Sparse-view imaging of a fiber internal structure in holographic diffraction tomography via a convolutional neural network

Deep learning has recently shown great potential in computational imaging. Here, we propose a deep-learning-based reconstruction method to realize the sparse-view imaging of a fiber internal structure in holographic diffraction tomography. By taking the sparse-view sinogram as the input and the cross-section image obtained by the dense-view sinogram as the ground truth, the neural network can reconstruct the cross-section image from the sparse-view sinogram. It performs better than the corresponding filtered back-projection algorithm with a sparse-view sinogram, both in the case of simulated data and real experimental data.

Fringe projection profilometry by conducting deep learning from its digital twin

High-accuracy and high-speed three-dimensional (3D) fringe projection profilometry (FPP) has been widely applied in many fields. Recently, researchers discovered that deep learning can significantly improve fringe analysis. However, deep learning requires numerous objects to be scanned for training data. In this paper, we propose to build the digital twin of an FPP system and perform virtual scanning using computer graphics, which can significantly save cost and labor. The proposed method extracts 3D geometry directly from a single-shot fringe image, and real-world experiments have demonstrated the success of the virtually trained model. Our virtual scanning method can automatically generate 7,200 fringe images and 800 corresponding 3D scenes within 1.5 hours.

Single-Shot 3D Shape Reconstruction Using Structured Light and Deep Convolutional Neural Networks

Single-shot 3D imaging and shape reconstruction has seen a surge of interest due to the ever-increasing evolution in sensing technologies. In this paper, a robust single-shot 3D shape reconstruction technique integrating the structured light technique with the deep convolutional neural networks (CNNs) is proposed. The input of the technique is a single fringe-pattern image, and the output is the corresponding depth map for 3D shape reconstruction. The essential training and validation datasets with high-quality 3D ground-truth labels are prepared by using a multi-frequency fringe projection profilometry technique. Unlike the conventional 3D shape reconstruction methods which involve complex algorithms and intensive computation to determine phase distributions or pixel disparities as well as depth map, the proposed approach uses an end-to-end network architecture to directly carry out the transformation of a 2D image to its corresponding 3D depth map without extra processing. In the approach, three CNN-based models are adopted for comparison. Furthermore, an accurate structured-light-based 3D imaging dataset used in this paper is made publicly available. Experiments have been conducted to demonstrate the validity and robustness of the proposed technique. It is capable of satisfying various 3D shape reconstruction demands in scientific research and engineering applications.

Dynamic 3-D measurement based on fringe-to-fringe transformation using deep learning

Fringe projection profilometry (FPP) has become increasingly important in dynamic 3-D shape measurement. In FPP, it is necessary to retrieve the phase of the measured object before shape profiling. However, traditional phase retrieval techniques often require a large number of fringes, which may generate motion-induced error for dynamic objects. In this paper, a novel phase retrieval technique based on deep learning is proposed, which uses an end-to-end deep convolution neural network to transform a single or two fringes into the phase retrieval required fringes. When the object's surface is located in a restricted depth, the presented network only requires a single fringe as the input, which otherwise requires two fringes in an unrestricted depth. The proposed phase retrieval technique is first theoretically analyzed, and then numerically and experimentally verified on its applicability for dynamic 3-D measurement.

PhUn-Net: ready-to-use neural network for unwrapping quantitative phase images of biological cells

We present a deep-learning approach for solving the problem of 2π phase ambiguities in two-dimensional quantitative phase maps of biological cells, using a multi-layer encoder-decoder residual convolutional neural network. We test the trained network, PhUn-Net, on various types of biological cells, captured with various interferometric setups, as well as on simulated phantoms. These tests demonstrate the robustness and generality of the network, even for cells of different morphologies or different illumination conditions than PhUn-Net has been trained on. In this paper, for the first time, we make the trained network publicly available in a global format, such that it can be easily deployed on every platform, to yield fast and robust phase unwrapping, not requiring prior knowledge or complex implementation. By this, we expect our phase unwrapping approach to be widely used, substituting conventional and more time-consuming phase unwrapping algorithms.