3D shape, deformation, and vibration measurements using infrared Kinect sensors and digital image correlation

Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches

The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.

Time-Distributed Framework for 3D Reconstruction Integrating Fringe Projection with Deep Learning

In recent years, integrating structured light with deep learning has gained considerable attention in three-dimensional (3D) shape reconstruction due to its high precision and suitability for dynamic applications. While previous techniques primarily focus on processing in the spatial domain, this paper proposes a novel time-distributed approach for temporal structured-light 3D shape reconstruction using deep learning. The proposed approach utilizes an autoencoder network and time-distributed wrapper to convert multiple temporal fringe patterns into their corresponding numerators and denominators of the arctangent functions. Fringe projection profilometry (FPP), a well-known temporal structured-light technique, is employed to prepare high-quality ground truth and depict the 3D reconstruction process. Our experimental findings show that the time-distributed 3D reconstruction technique achieves comparable outcomes with the dual-frequency dataset (p = 0.014) and higher accuracy than the triple-frequency dataset (p = 1.029 × 10-9), according to non-parametric statistical tests. Moreover, the proposed approach's straightforward implementation of a single training network for multiple converters makes it more practical for scientific research and industrial applications.

Compact structured light generation based on meta-hologram PCSEL integration

Metasurfaces, a catalog of optical components, offer numerous novel functions on demand. They have been integrated with vertical cavity surface-emitting lasers (VCSELs) in previous studies. However, the performance has been limited by the features of the VCSELs such as low output power and large divergence angle. Although the solution of the module of VCSEL array could solve these issues, the practical application is limited by extra lens and large size. In this study, we experimentally demonstrate reconstruction of a holographic images using a compact integration of a photonic crystal surface-emitting laser and metasurface holograms designed for structured light generation. This research showcases the flexible design capabilities of metasurfaces, high output power (on the order of milliwatts), and the ability to produce well-uniformed images with a wide field of view without the need for a collection lens, making it suitable for 3D imaging and sensing.

Different structured-light patterns in single-shot 2D-to-3D image conversion using deep learning

Single-shot 3D shape reconstruction integrating structured light and deep learning has drawn considerable attention and achieved significant progress in recent years due to its wide-ranging applications in various fields. The prevailing deep-learning-based 3D reconstruction using structured light generally transforms a single fringe pattern to its corresponding depth map by an end-to-end artificial neural network. At present, it remains unclear which kind of structured-light patterns should be employed to obtain the best accuracy performance. To answer this fundamental and much-asked question, we conduct an experimental investigation of six representative structured-light patterns adopted for single-shot 2D-to-3D image conversion. The assessment results provide a valuable guideline for structured-light pattern selection in practice.

Real-Time Vibration Visualization Using GPU-Based High-Speed Vision

In this study, we developed a real-time vibration visualization system that can estimate and display vibration distributions at all frequencies in real time through parallel implementation of subpixel digital image correlation (DIC) computations with short-time Fourier transforms on a GPU-based high-speed vision platform. To help operators intuitively monitor high-speed motion, we introduced a two-step framework of high-speed video processing to obtain vibration distributions at hundreds of hertz and video conversion processing for the visualization of vibration distribution at dozens of hertz. The proposed system can estimate the full-field vibration displacements of 1920 × 1080 images in real time at 1000 fps and display their frequency responses in the range of 0–500 Hz on a computer at dozens of frames per second by accelerating phase-only DICs for full-field displacement measurement and video conversion. The effectiveness of this system for real-time vibration monitoring and visualization was demonstrated by conducting experiments on objects vibrating at dozens or hundreds of hertz.

Single-shot 3D shape acquisition using a learning-based structured-light technique

Learning three-dimensional (3D) shape representation of an object from a single-shot image has been a prevailing topic in computer vision and deep learning over the past few years. Despite extensive adoption in dynamic applications, the measurement accuracy of the 3D shape acquisition from a single-shot image is still unsatisfactory due to a wide range of challenges. We present an accurate 3D shape acquisition method from a single-shot two-dimensional (2D) image using the integration of a structured-light technique and a deep learning approach. Instead of a direct 2D-to-3D transformation, a pattern-to-pattern network is trained to convert a single-color structured-light image to multiple dual-frequency phase-shifted fringe patterns for succeeding 3D shape reconstructions. Fringe projection profilometry, a prominent structured-light technique, is employed to produce high-quality ground-truth labels for training the network and to accomplish the 3D shape reconstruction after predicting the fringe patterns. A series of experiments has been conducted to demonstrate the practicality and potential of the proposed technique for scientific research and industrial applications.

Hybrid-driven structural modal shape visualization using subtle variations in high-speed video

The phase-based motion magnification technique can exaggerate specific structural vibrations and obtain potential applications in visualizing and understanding modal shapes. However, the quality of motion magnification is affected by noise and clipping artifacts, especially in large amplifications. We propose a hybrid-driven motion magnification framework that combines Eulerian and Lagrangian motion processing. Since the structural global spatial vibration corresponding to different modal shapes usually accumulates energy differences in the timeline, from a Eulerian perspective, temporal intensity variations are denoised and separated according to the energy distribution to control spatial motions. Meanwhile, from a Lagrangian perspective, the motion magnification is realized by compensating spatial motion according to the magnified inter-frame motion vector field. By utilizing both Eulerian and Lagrangian motion processing, the proposed framework supports a larger amplification factor and achieves better performance in perceiving subtle vibrations in controlled modal tests.

Accurate 3D Shape Reconstruction from Single Structured-Light Image via Fringe-to-Fringe Network

Accurate three-dimensional (3D) shape reconstruction of objects from a single image is a challenging task, yet it is highly demanded by numerous applications. This paper presents a novel 3D shape reconstruction technique integrating a high-accuracy structured-light method with a deep neural network learning scheme. The proposed approach employs a convolutional neural network (CNN) to transform a color structured-light fringe image into multiple triple-frequency phase-shifted grayscale fringe images, from which the 3D shape can be accurately reconstructed. The robustness of the proposed technique is verified, and it can be a promising 3D imaging tool in future scientific and industrial applications.

Comparison of Depth Camera and Terrestrial Laser Scanner in Monitoring Structural Deflections

Modeling a structure in the virtual world using three-dimensional (3D) information enhances our understanding, while also aiding in the visualization, of how a structure reacts to any disturbance. Generally, 3D point clouds are used for determining structural behavioral changes. Light detection and ranging (LiDAR) is one of the crucial ways by which a 3D point cloud dataset can be generated. Additionally, 3D cameras are commonly used to develop a point cloud containing many points on the external surface of an object around it. The main objective of this study was to compare the performance of optical sensors, namely a depth camera (DC) and terrestrial laser scanner (TLS) in estimating structural deflection. We also utilized bilateral filtering techniques, which are commonly used in image processing, on the point cloud data for enhancing their accuracy and increasing the application prospects of these sensors in structure health monitoring. The results from these sensors were validated by comparing them with the outputs from a linear variable differential transformer sensor, which was mounted on the beam during an indoor experiment. The results showed that the datasets obtained from both the sensors were acceptable for nominal deflections of 3 mm and above because the error range was less than ±10%. However, the result obtained from the TLS were better than those obtained from the DC.

Single-Shot 3D Shape Reconstruction Using Structured Light and Deep Convolutional Neural Networks

Single-shot 3D imaging and shape reconstruction has seen a surge of interest due to the ever-increasing evolution in sensing technologies. In this paper, a robust single-shot 3D shape reconstruction technique integrating the structured light technique with the deep convolutional neural networks (CNNs) is proposed. The input of the technique is a single fringe-pattern image, and the output is the corresponding depth map for 3D shape reconstruction. The essential training and validation datasets with high-quality 3D ground-truth labels are prepared by using a multi-frequency fringe projection profilometry technique. Unlike the conventional 3D shape reconstruction methods which involve complex algorithms and intensive computation to determine phase distributions or pixel disparities as well as depth map, the proposed approach uses an end-to-end network architecture to directly carry out the transformation of a 2D image to its corresponding 3D depth map without extra processing. In the approach, three CNN-based models are adopted for comparison. Furthermore, an accurate structured-light-based 3D imaging dataset used in this paper is made publicly available. Experiments have been conducted to demonstrate the validity and robustness of the proposed technique. It is capable of satisfying various 3D shape reconstruction demands in scientific research and engineering applications.

Approach for accurate calibration of RGB-D cameras using spheres

RGB-D cameras (or color-depth cameras) play key roles in many vision applications. A typical RGB-D camera has only rough intrinsic and extrinsic calibrations that cannot provide the accuracy required in many vision applications. In this paper, we propose a novel and accurate sphere-based calibration framework for estimating the intrinsic and extrinsic parameters of color-depth sensor pair. Additionally, a method of depth error correction is suggested, and the principle of error correction is analyzed in detail. In our method, the feature extraction module can automatically and reliably detect the center and edges of the sphere projection, while excluding noise data and outliers, and the projection of the sphere center on RGB and depth images is used to obtain a closed solution of the initial parameters. Finally, all the parameters are accurately estimated within the framework of nonlinear global minimization. Compared to other state-of-the-art methods, our calibration method is easy to use and provides higher calibration accuracy. Detailed experimental analysis is performed to support our conclusions.

Complete grid pattern decoding method for a one-shot structured light system

Structured light 3D reconstruction methods using a De Bruijn sequence-based color grid pattern have an impressive advantage of fast and accurate decoding, which leads to fast 3D reconstruction. They are especially suitable for capturing moving objects. However, the drawback of these methods is their high false decoding rate while dealing with feature points at the object's boundaries, and objects can be prone to becoming deformed by the uneven structure of the dynamic scene. To solve this problem, we present an efficient opened-grid-point detector and a complete grid pattern decoding method. Specifically, a new, to the best of our knowledge, color grid pattern is designed to reduce the influence of color noise and increase the density of 3D cloud points. In addition, a LCD screen projected with the proposed pattern is utilized to calibrate the camera-projector system. The experiments, conducted in a laboratory without a light curtain, demonstrate that the proposed method can fully satisfy the requirements of real applications.

Laser-speckle-projection-based handheld anthropometric measurement system with synchronous redundancy reduction

Human body measurement is essential in modern rehabilitation medicine, which can be effectively combined with the technology of additive manufacturing. Digital image correlation based on laser speckle projection is a single-shot, accurate, and robust technique for human body measurement. In this paper, we present a handheld anthropometric measurement system based on laser speckle projection. A flexible retroreflective marker target is designed for multi-view data registration. Meanwhile, a synchronous redundancy-reduction algorithm based on a re-projected global disparity map is proposed. Experiment results validate that the proposed system is effective and accurate for different human body part measurements. Comparative experiments show that the proposed redundancy-reduction algorithm has high efficiency and can effectively preserve the features of complex shapes. The comprehensive performance of the algorithm is better than the other two tested methods.

Real-time 3D shape measurement using 3LCD projection and deep machine learning

For 3D imaging and shape measurement, simultaneously achieving real-time and high-accuracy performance remains a challenging task in practice. In this paper, a fringe-projection-based 3D imaging and shape measurement technique using a three-chip liquid-crystal-display (3LCD) projector and a deep machine learning scheme is presented. By encoding three phase-shifted fringe patterns into the red, green, and blue (RGB) channels of a color image and controlling the 3LCD projector to project the RGB channels individually, the technique can synchronize the projector and the camera to capture the required fringe images at a fast speed. In the meantime, the 3D imaging and shape measurement accuracy is dramatically improved by introducing a novel phase determination approach built on a fully connected deep neural network (DNN) learning model. The proposed system allows performing 3D imaging and shape measurement of multiple complex objects at a real-time speed of 25.6 fps with relative accuracy of 0.012%. Experiments have shown great promise for advancing scientific and engineering applications.

Accurate 3D Shape, Displacement and Deformation Measurement Using a Smartphone

The stereo-digital image correlation technique using two synchronized industrial-grade cameras has been extensively used for full-field 3D shape, displacement and deformation measurements. However, its use in resource-limited institutions and field settings is inhibited by the need for relatively expensive, bulky and complicated experimental set-ups. To mitigate this problem, we established a cost-effective and ultra-portable smartphone-based stereo-digital image correlation system, which only uses a smartphone and an optical attachment. This optical attachment is composed of four planar mirrors and a 3D-printed mirror support, and can split the incoming scene into two sub-images, simulating a stereovision system using two virtual smartphones. Although such a mirror-based system has already been used for stereo-image correlation, this is the first time it has been combined with a commercial smartphone. This publication explores the potential and limitations of such a configuration. We first verified the effectiveness and accuracy of this system in 3D shape and displacement measurement through shape measurement and in-plane and out-of-plane translation tests. Severe thermal-induced virtual strains (up to 15,000 με) were found in the measured results due to the smartphone heating. The mechanism for the generation of the temperature-dependent errors in this system was clearly and reasonably explained. After a simple preheating process, the smartphone-based system was demonstrated to be accurate in measuring the strain on the surface of a loaded composite specimen, with comparable accuracy to a strain gauge. Measurements of 3D deformation are illustrated by tracking the deformation on the surface of a deflating ball. This cost-effective and ultra-portable smartphone-based system not only greatly decreases the hardware investment in the system construction, but also increases convenience and efficiency of 3D deformation measurements, thus demonstrating a large potential in resource-limited and field settings.

Three-dimensional facial digitization using advanced digital image correlation

Presented in this paper is an effective technique to acquire the three-dimensional (3D) digital images of the human face without the use of active lighting and artificial patterns. The technique is based on binocular stereo imaging and digital image correlation, and it includes two key steps: camera calibration and image matching. The camera calibration involves a pinhole model and a bundle-adjustment approach, and the governing equations of the 3D digitization process are described. For reliable pixel-to-pixel image matching, the skin pores and freckles or lentigines on the human face serve as the required pattern features to facilitate the process. It employs feature-matching-based initial guess, multiple subsets, iterative optimization algorithm, and reliability-guided computation path to achieve fast and accurate image matching. Experiments have been conducted to demonstrate the validity of the proposed technique. The simplicity of the approach and the affordable cost of the implementation show its practicability in scientific and engineering applications.