1
|
Evans RG, Devlieghere E, Keijzer R, Dirckx JJJ, Van der Jeught S. Deep Learning for Single-Shot Structured Light Profilometry: A Comprehensive Dataset and Performance Analysis. J Imaging 2024; 10:179. [PMID: 39194968 DOI: 10.3390/jimaging10080179] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2024] [Revised: 07/18/2024] [Accepted: 07/22/2024] [Indexed: 08/29/2024] Open
Abstract
In 3D optical metrology, single-shot deep learning-based structured light profilometry (SS-DL-SLP) has gained attention because of its measurement speed, simplicity of optical setup, and robustness to noise and motion artefacts. However, gathering a sufficiently large training dataset for these techniques remains challenging because of practical limitations. This paper presents a comprehensive DL-SLP dataset of over 10,000 physical data couples. The dataset was constructed by 3D-printing a calibration target featuring randomly varying surface profiles and storing the height profiles and the corresponding deformed fringe patterns. Our dataset aims to serve as a benchmark for evaluating and comparing different models and network architectures in DL-SLP. We performed an analysis of several established neural networks, demonstrating high accuracy in obtaining full-field height information from previously unseen fringe patterns. In addition, the network was validated on unique objects to test the overall robustness of the trained model. To facilitate further research and promote reproducibility, all code and the dataset are made publicly available. This dataset will enable researchers to explore, develop, and benchmark novel DL-based approaches for SS-DL-SLP.
Collapse
Affiliation(s)
- Rhys G Evans
- Industrial Vision Lab (InViLab), Faculty of Applied Engineering, Campus Groenenborger, University of Antwerp, Groenenborgerlaan 179, 2020 Antwerp, Belgium
| | - Ester Devlieghere
- Laboratory of Biomedical Physics (BIMEF), University of Antwerp, Groenenborgerlaan 171, 2020 Antwerp, Belgium
| | - Robrecht Keijzer
- Laboratory of Biomedical Physics (BIMEF), University of Antwerp, Groenenborgerlaan 171, 2020 Antwerp, Belgium
| | - Joris J J Dirckx
- Laboratory of Biomedical Physics (BIMEF), University of Antwerp, Groenenborgerlaan 171, 2020 Antwerp, Belgium
| | - Sam Van der Jeught
- Industrial Vision Lab (InViLab), Faculty of Applied Engineering, Campus Groenenborger, University of Antwerp, Groenenborgerlaan 179, 2020 Antwerp, Belgium
| |
Collapse
|
2
|
Nguyen AH, Wang Z. Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches. SENSORS (BASEL, SWITZERLAND) 2024; 24:3246. [PMID: 38794100 PMCID: PMC11125235 DOI: 10.3390/s24103246] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Revised: 04/22/2024] [Accepted: 05/16/2024] [Indexed: 05/26/2024]
Abstract
The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA;
| | - Zhaoyang Wang
- Department of Mechanical Engineering, School of Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
3
|
Li Y, Wu Z, Shen J, Zhang Q. Real-time 3D shape measurement of dynamic scenes using fringe projection profilometry: lightweight NAS-optimized dual frequency deep learning approach. OPTICS EXPRESS 2023; 31:40803-40823. [PMID: 38041372 DOI: 10.1364/oe.506343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 10/29/2023] [Indexed: 12/03/2023]
Abstract
Achieving real-time and high-accuracy 3D reconstruction of dynamic scenes is a fundamental challenge in many fields, including online monitoring, augmented reality, and so on. On one hand, traditional methods, such as Fourier transform profilometry (FTP) and phase-shifting profilometry (PSP), are struggling to balance measuring efficiency and accuracy. On the other hand, deep learning-based approaches, which offer the potential for improved accuracy, are hindered by large parameter amounts and complex structures less amenable to real-time requirements. To solve this problem, we proposed a network architecture search (NAS)-based method for real-time processing and 3D measurement of dynamic scenes with rate equivalent to single-shot. A NAS-optimized lightweight neural network was designed for efficient phase demodulation, while an improved dual-frequency strategy was employed coordinately for flexible absolute phase unwrapping. The experiment results demonstrate that our method can effectively perform 3D reconstruction with a reconstruction speed of 58fps, and realize high-accuracy measurement of dynamic scenes based on deep learning for what we believe to be the first time with the average RMS error of about 0.08 mm.
Collapse
|
4
|
Nguyen AH, Wang Z. Time-Distributed Framework for 3D Reconstruction Integrating Fringe Projection with Deep Learning. SENSORS (BASEL, SWITZERLAND) 2023; 23:7284. [PMID: 37631820 PMCID: PMC10458373 DOI: 10.3390/s23167284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 08/07/2023] [Accepted: 08/18/2023] [Indexed: 08/27/2023]
Abstract
In recent years, integrating structured light with deep learning has gained considerable attention in three-dimensional (3D) shape reconstruction due to its high precision and suitability for dynamic applications. While previous techniques primarily focus on processing in the spatial domain, this paper proposes a novel time-distributed approach for temporal structured-light 3D shape reconstruction using deep learning. The proposed approach utilizes an autoencoder network and time-distributed wrapper to convert multiple temporal fringe patterns into their corresponding numerators and denominators of the arctangent functions. Fringe projection profilometry (FPP), a well-known temporal structured-light technique, is employed to prepare high-quality ground truth and depict the 3D reconstruction process. Our experimental findings show that the time-distributed 3D reconstruction technique achieves comparable outcomes with the dual-frequency dataset (p = 0.014) and higher accuracy than the triple-frequency dataset (p = 1.029 × 10-9), according to non-parametric statistical tests. Moreover, the proposed approach's straightforward implementation of a single training network for multiple converters makes it more practical for scientific research and industrial applications.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA;
| | - Zhaoyang Wang
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
5
|
Wan M, Kong L. Single-shot 3D measurement of highly reflective objects with deep learning. OPTICS EXPRESS 2023; 31:14965-14985. [PMID: 37157349 DOI: 10.1364/oe.487917] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
Three-dimensional (3D) measurement methods based on fringe projection profilometry (FPP) have been widely applied in industrial manufacturing. Most FPP methods adopt phase-shifting techniques and require multiple fringe images, thus having limited application in dynamic scenes. Moreover, industrial parts often have highly reflective areas leading to overexposure. In this work, a single-shot high dynamic range 3D measurement method combining FPP with deep learning is proposed. The proposed deep learning model includes two convolutional neural networks: exposure selection network (ExSNet) and fringe analysis network (FrANet). The ExSNet utilizes self-attention mechanism for enhancement of highly reflective areas leading to overexposure problem to achieve high dynamic range in single-shot 3D measurement. The FrANet consists of three modules to predict wrapped phase maps and absolute phase maps. A training strategy directly opting for best measurement accuracy is proposed. Experiments on a FPP system showed that the proposed method predicted accurate optimal exposure time under single-shot condition. A pair of moving standard spheres with overexposure was measured for quantitative evaluation. The proposed method reconstructed standard spheres over a large range of exposure level, where prediction errors for diameter were 73 µm (left) and 64 µm (right) and prediction error for center distance was 49 µm. Ablation study and comparison with other high dynamic range methods were also conducted.
Collapse
|
6
|
Nguyen AH, Ly KL, Lam VK, Wang Z. Generalized Fringe-to-Phase Framework for Single-Shot 3D Reconstruction Integrating Structured Light with Deep Learning. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094209. [PMID: 37177413 PMCID: PMC10181406 DOI: 10.3390/s23094209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 04/16/2023] [Accepted: 04/18/2023] [Indexed: 05/15/2023]
Abstract
Three-dimensional (3D) shape acquisition of objects from a single-shot image has been highly demanded by numerous applications in many fields, such as medical imaging, robotic navigation, virtual reality, and product in-line inspection. This paper presents a robust 3D shape reconstruction approach integrating a structured-light technique with a deep learning-based artificial neural network. The proposed approach employs a single-input dual-output network capable of transforming a single structured-light image into two intermediate outputs of multiple phase-shifted fringe patterns and a coarse phase map, through which the unwrapped true phase distributions containing the depth information of the imaging target can be accurately determined for subsequent 3D reconstruction process. A conventional fringe projection technique is employed to prepare the ground-truth training labels, and part of its classic algorithm is adopted to preserve the accuracy of the 3D reconstruction. Numerous experiments have been conducted to assess the proposed technique, and its robustness makes it a promising and much-needed tool for scientific research and engineering applications.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA
| | - Khanh L Ly
- Department of Biomedical Engineering, The Catholic University of America, Washington, DC 20064, USA
| | - Van Khanh Lam
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC 20012, USA
| | - Zhaoyang Wang
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
7
|
Nguyen AH, Sun B, Li CQ, Wang Z. Different structured-light patterns in single-shot 2D-to-3D image conversion using deep learning. APPLIED OPTICS 2022; 61:10105-10115. [PMID: 36606771 DOI: 10.1364/ao.468984] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Accepted: 11/06/2022] [Indexed: 06/17/2023]
Abstract
Single-shot 3D shape reconstruction integrating structured light and deep learning has drawn considerable attention and achieved significant progress in recent years due to its wide-ranging applications in various fields. The prevailing deep-learning-based 3D reconstruction using structured light generally transforms a single fringe pattern to its corresponding depth map by an end-to-end artificial neural network. At present, it remains unclear which kind of structured-light patterns should be employed to obtain the best accuracy performance. To answer this fundamental and much-asked question, we conduct an experimental investigation of six representative structured-light patterns adopted for single-shot 2D-to-3D image conversion. The assessment results provide a valuable guideline for structured-light pattern selection in practice.
Collapse
|
8
|
Wang J, Li Y, Ji Y, Qian J, Che Y, Zuo C, Chen Q, Feng S. Deep Learning-Based 3D Measurements with Near-Infrared Fringe Projection. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22176469. [PMID: 36080928 PMCID: PMC9460471 DOI: 10.3390/s22176469] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 08/23/2022] [Accepted: 08/24/2022] [Indexed: 05/27/2023]
Abstract
Fringe projection profilometry (FPP) is widely applied to 3D measurements, owing to its advantages of high accuracy, non-contact, and full-field scanning. Compared with most FPP systems that project visible patterns, invisible fringe patterns in the spectra of near-infrared demonstrate fewer impacts on human eyes or on scenes where bright illumination may be avoided. However, the invisible patterns, which are generated by a near-infrared laser, are usually captured with severe speckle noise, resulting in 3D reconstructions of limited quality. To cope with this issue, we propose a deep learning-based framework that can remove the effect of the speckle noise and improve the precision of the 3D reconstruction. The framework consists of two deep neural networks where one learns to produce a clean fringe pattern and the other to obtain an accurate phase from the pattern. Compared with traditional denoising methods that depend on complex physical models, the proposed learning-based method is much faster. The experimental results show that the measurement accuracy can be increased effectively by the presented method.
Collapse
Affiliation(s)
- Jinglei Wang
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Yixuan Li
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Yifan Ji
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Jiaming Qian
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Yuxuan Che
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Chao Zuo
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Qian Chen
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| | - Shijie Feng
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Smart Computational Imaging Research Institute (SCIRI), Nanjing University of Science and Technology, Nanjing 210019, China
- Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing 210094, China
| |
Collapse
|
9
|
Ueda K, Ikeda K, Koyama O, Yamada M. Absolute phase retrieval of shiny objects using fringe projection and deep learning with computer-graphics-based images. APPLIED OPTICS 2022; 61:2750-2756. [PMID: 35471347 DOI: 10.1364/ao.450723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/16/2021] [Accepted: 03/07/2022] [Indexed: 06/14/2023]
Abstract
Fringe projection profilometry is a high-precision method used to measure the 3D shape of an object by projecting sinusoidal fringes onto an object. However, fringes projected onto a metallic or shiny object are distorted nonlinearly, which causes significant measurement errors. A high-precision measurement method for shiny objects that employs computer graphics (CG) and deep learning is proposed. We trained a deep neural network by projecting fringes on a shiny object in CG space. Our results show that the method can reduce the nonlinear fringe distortion caused by gloss in real space.
Collapse
|