Accurate depth estimation in structured light fields

Light Field Depth Estimation via Stitched Epipolar Plane Images

Depth estimation is a fundamental problem in light field processing. Epipolar-plane image (EPI)-based methods often encounter challenges such as low accuracy in slope computation due to discretization errors and limited angular resolution. Besides, existing methods perform well in most regions but struggle to produce sharp edges in occluded regions and resolve ambiguities in texture-less regions. To address these issues, we propose the concept of stitched-EPI (SEPI) to enhance slope computation. SEPI achieves this by shifting and concatenating lines from different EPIs that correspond to the same 3D point. Moreover, we introduce the half-SEPI algorithm, which focuses exclusively on the non-occluded portion of lines to handle occlusion. Additionally, we present a depth propagation strategy aimed at improving depth estimation in texture-less regions. This strategy involves determining the depth of such regions by progressing from the edges towards the interior, prioritizing accurate regions over coarse regions. Through extensive experimental evaluations and ablation studies, we validate the effectiveness of our proposed method. The results demonstrate its superior ability to generate more accurate and robust depth maps across all regions compared to state-of-the-art methods.

Fast 3D reconstruction via event-based structured light with spatio-temporal coding

Event-based structured light (SL) systems leverage bio-inspired event cameras, which are renowned for their low latency and high dynamics, to drive progress in high-speed structured light systems. However, existing event-based structured light methods concentrate on the independent construction of either time-domain or space-domain features for stereo matching, ignoring the spatio-temporal consistency towards depth. In this work, we build an event-based SL system that consists of a laser point projector and an event camera, and we devise a spatial-temporal coding strategy that realizes depth encoding in dual domains through a single shot. To exploit the spatio-temporal synergy, we further present STEM, a novel Spatio-Temporal Enhanced Matching approach for 3D reconstruction. STEM is comprised of two parts, the spatio-temporal enhancing (STE) algorithm and the spatio-temporal matching (STM) algorithm. Specifically, STE integrates the dual-domain information to increase the saliency of the temporal coding, providing a more robust basis for matching. STM is a stereo matching algorithm explicitly tailored to the unique characteristics of event data modality, which computes the disparity via a meticulously designed hybrid cost function. Experimental results demonstrate the superior performance of our proposed method, achieving a reconstruction rate of 16 fps and a low root mean square error of 0.56 mm at a distance of 0.72 m.

Visual Sensing and Depth Perception for Welding Robots and Their Industrial Applications

With the rapid development of vision sensing, artificial intelligence, and robotics technology, one of the challenges we face is installing more advanced vision sensors on welding robots to achieve intelligent welding manufacturing and obtain high-quality welding components. Depth perception is one of the bottlenecks in the development of welding sensors. This review provides an assessment of active and passive sensing methods for depth perception and classifies and elaborates on the depth perception mechanisms based on monocular vision, binocular vision, and multi-view vision. It explores the principles and means of using deep learning for depth perception in robotic welding processes. Further, the application of welding robot visual perception in different industrial scenarios is summarized. Finally, the problems and countermeasures of welding robot visual perception technology are analyzed, and developments for the future are proposed. This review has analyzed a total of 2662 articles and cited 152 as references. The potential future research topics are suggested to include deep learning for object detection and recognition, transfer deep learning for welding robot adaptation, developing multi-modal sensor fusion, integrating models and hardware, and performing a comprehensive requirement analysis and system evaluation in collaboration with welding experts to design a multi-modal sensor fusion architecture.

Learning to detect 3D planes from a single plenoptic image

In this paper, we propose a deep neural network to detect 3D planar surfaces from single plenoptic images captured by a Lytro Illum camera. Different from learning methods based on a single RGB image, we train a network to exploit the light distribution information both in spatial and angular dimensions from multi-sub-aperture images. The features from each sub-aperture image are extracted by using parameter-sharing convolutional layers and then fused to jointly infer the parameters of planes, depths, and segmentation masks. The experiments demonstrate that our approach outperforms the existing state-of-the-art methods with significant margins in the plane detection metrics.

Matching entropy based disparity estimation from light field data

A major challenge for matching-based disparity estimation from light field data is to prevent mismatches in occlusion and smooth regions. An effective matching window satisfying three characteristics: texture richness, disparity consistency, and anti-occlusion should be able to prevent mismatches to some extent. According to these characteristics, we propose matching entropy in the spatial domain of the light field to measure the amount of correct information in a matching window, which provides the criterion for matching window selection. Based on matching entropy regularization, we establish an optimization model for disparity estimation with a matching cost fidelity term. To find the optimum, we propose a two-step adaptive matching algorithm. First, the region type is adaptively determined to identify occluding, occluded, smooth, and textured regions. Then, the matching entropy criterion is used to adaptively select the size and shape of matching windows, as well as the visible viewpoints. The two-step process can reduce mismatches and redundant calculations by selecting effective matching windows. The experimental results on synthetic and real data show that the proposed method can effectively improve the accuracy of disparity estimation in occlusion and smooth regions and has strong robustness for different noise levels. Therefore, high-precision disparity estimation from 4D light field data is achieved.

Phase-unwrapping-free 3D reconstruction in structured light field system based on varied auxiliary point

Three-dimensional (3D) reconstruction is an essential task in structured light field (SLF) related techniques and applications. This paper presents a new method to reconstruct a 3D object point by using many auxiliary points adjacent to it. The relationship between two points in a SLF system is derived. Different from conventional "direct" methods that reconstruct 3D coordinates of the object point by using phase, slope, disparity etc., the proposed method is an "indirect" method as the 3D coordinates of auxiliary points are not needed. Based on the auxiliary point theory, the wrapped phase obtained by 4-step phase-shifting method is sufficient for 3D reconstruction, without the need for phase unwrapping. To the best of our knowledge, this is the first strategy that combines the intrinsic characteristics of structured light and light field for phase-unwrapping-free 3D reconstruction. This paper also analyzes the constraints between system architecture parameters and phase rectification, phase to depth ratio, and presents a relatively simple criterion to guide the system design. Experimental results show that, with an appropriate system architecture, the proposed method can realize accurate, unambiguous, and reliable 3D reconstruction without phase unwrapping.

Large DOF microscopic fringe projection profilometry with a coaxial light-field structure

Fringe projection profilometry (FPP) has been widely researched for three-dimensional (3D) microscopic measurement during recent decades. Nevertheless, some disadvantages arising from the limited depth of field and occlusion still exist and need to be further addressed. In this paper, light field imaging is introduced for microscopic fringe projection profilometry (MFPP) to obtain a larger depth of field. Meanwhile, this system is built with a coaxial structure to reduce occlusion, where the principle of triangulation is no longer applicable. In this situation, the depth information is estimated based on the epipolar plane image (EPI) of light field. In order to make a quantitative measurement, a metric calibration method which establishes the mapping between the slope of the line feature in EPI and the depth information is proposed for this system. Finally, a group of experiments demonstrate that the proposed LF-MFPP system can work well for depth estimation with a large DOF and reduced occlusion.

Fast depth estimation with cost minimization for structured light field

Depth estimation is a fundamental task in light field (LF) related applications. However, conventional light field suffers from the lack of features, which introduces depth ambiguity and heavy computation load to depth estimation. In this paper, we introduce phase light field (PLF), which uses sinusoidal fringes as patterns and the latent phases as the codes. With PLF and the re-formatted phase-epipolar-plane-images (phase EPIs), a global cost minimization framework is proposed to estimate the depth. In general, EPI-based depth estimation tests a set of candidate lines to find the optimal one with most similar intensities, and the slope of the optimal line is converted to disparity and depth. Based on this principle, for phase-EPI, we propose a cost with weighted phase variance in the candidate line, and we prove that the cost is a convex function. After that, the beetle antennae search (BAS) optimization algorithm is utilized to find the optimal line and thus depth can be obtained. Finally, a bilateral filter is incorporated to further improve the depth quality. Simulation and real experimental results demonstrate that, the proposed method can produce accurate depth maps, especially at boundary regions. Moreover, the proposed method achieves an acceleration of about 5.9 times over the state-of-the-art refocus method with comparable depth quality, and thus can facilitate practical applications.

Resolution Improvement of Light Field Imaging via a Nematic Liquid Crystal Microlens with Added Multi-Walled Carbon Nanotubes

A relatively simple method to improve the image resolution of light field based on a liquid crystal (LC) microlens doped with multi-walled carbon nanotubes (MWCNTs) was developed and evaluated. As the nanoparticles were doped in LC, its electro-optical features could enhance, leading to a short response time compared to the pure LC microlens. With the maximum use of the proposed LC microlens, a method combining aperiodicity extraction and weighted average algorithm was adopted to realize the high-resolution light field imaging. The aperiodicity extraction method was proposed, which could effectively improve resolution of view angle image. For synthesizing the full resolution image at 0 Vrms and the extracted view angle image of light field imaging at 2.0 Vrms, the final high-resolution light field imaging could be obtained in a short time by weighted average algorithm. In this way, the common problem of low resolution in light field imaging could be solved. This proposed method was in good agreement with our experimental results. And it was also in line with the development of the trend of the smart imaging sensor combining algorithm with hardware.

State-of-the-art active optical techniques for three-dimensional surface metrology: a review [Invited]

This paper reviews recent developments of non-contact three-dimensional (3D) surface metrology using an active structured optical probe. We focus primarily on those active non-contact 3D surface measurement techniques that could be applicable to the manufacturing industry. We discuss principles of each technology, and its advantageous characteristics as well as limitations. Towards the end, we discuss our perspectives on the current technological challenges in designing and implementing these methods in practical applications.

Light-field depth estimation considering plenoptic imaging distortion

Light-field imaging can simultaneously record spatio-angular information of light rays to carry out depth estimation via depth cues which reflect a coupling of the angular information and the scene depth. However, the unavoidable imaging distortion in a light-field imaging system has a side effect on the spatio-angular coordinate computation, leading to incorrectly estimated depth maps. Based on the previously established unfocused plenoptic metric model, this paper reports a study on the effect of the plenoptic imaging distortion on the light-field depth estimation. A method of light-field depth estimation considering the plenoptic imaging distortion is proposed. Besides, the accuracy analysis of the light-field depth estimation was performed by using standard components. Experimental results demonstrate that efficiently compensating the plenoptic imaging distortion results in a six-fold improvement in measuring accuracy and more consistency across the measuring depth range. Consequently, the proposed method is proved to be suitable for light-field depth estimation and three-dimensional measurement with high quality, enabling unfocused plenoptic cameras to be metrological tools in the potential application scenarios such as industry, biomedicine, entertainment, and many others.

Monocular depth estimation method using a focus tunable lens

We propose a method for the passive-type estimation of depth using a focus tunable lens. The proposed method utilizes a lens group and a focus tunable lens and charge coupled device. The target object is imaged by the group of lenses and measured by changing the focal length of the focus tunable lens. The method aims to measure depth information in the image domain and not in the object domain. An autofocusing algorithm finds the best focus position of the target object through the focus value calculated by the Sobel operator. We believe that the proposed method is applicable to depth measurement systems because it offers a simple configuration without any active light source and can operate in real time. The experiment, performed for comparison with theoretical calculations, confirms the feasibility of the proposed method.