Human gesture recognition using three-dimensional integral imaging

Deep-learning-based 3D object salient detection via light-field integral imaging

This Letter proposes an effective light-field 3D saliency object detection (SOD) method, which is inspired by the idea that the spatial and angular information inherent in a light-field implicitly contains the geometry and reflection characteristics of the observed scene. These characteristics can provide effective background clues and depth information for 3D saliency reconstruction, which can greatly improve the accuracy of object detection and recognition. We use convolutional neural networks (CNNs) to detect the saliency of each elemental image (EI) with different viewpoints in an elemental image array (EIA) and the salient EIA is reconstructed by using a micro-lens array, forming a 3D salient map in the reconstructed space. Experimental results show that our method can generate high-quality 3D saliency maps and can be observed simultaneously from different angles and positions.

Field-of-view enhanced integral imaging with dual prism arrays based on perspective-dependent pixel mapping

A field-of-view (FOV)-enhanced integral imaging system is proposed by the combined use of micro-lens array (MLA) and dual-prism array (DPA). The MLA coupled with a DPA virtually function as a new type of the MLA whose FOV is much more increased than that of the original MLA, which enables the capturing of perspective-expanded elemental image arrays (EIAs) of input 3-D scenes and the FOV-enhanced reconstruction of them. For its practical operation, a two-step digital process called perspective-dependent pixel-mapping (PDPM) is also presented. With this PDPM method, picked-up EIAs with a couple of MLAs and DPAs are remapped into the new forms of EIAs to be properly reconstructed in the conventional integral imaging system. Operational performances of the proposed system are ray-optically analyzed. In addition, the feasibility of the proposed system is also confirmed from the computational and optical experiments with test 3-D objects on the implemented prototype. Experimental results finally show a two-times increase of the FOV range of the proposed system when it is compared with that of the conventional system.

Roadmap on 3D integral imaging: sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.

Three-dimensional integral imaging in photon-starved environments with high-sensitivity image sensors

Imaging in poorly illuminated environments using three-dimensional (3D) imaging with passive imaging sensors that operate in the visible spectrum is a formidable task due to the low number of photons detected. 3D integral imaging, which integrates multiple two-dimensional perspectives, is expected to perform well in the presence of noise, as well as statistical fluctuation in the detected number of photons. In this paper, we present an investigation of 3D integral imaging in low-light-level conditions, where as low as a few photons and as high as several tens of photons are detected on average per pixel. In the experimental verification, we use an electron multiplying charge-coupled device (EM-CCD) and a scientific complementary metal-oxide-semiconductor (sCMOS) camera. For the EM-CCD, a theoretical model for the probability distribution of the pixel values is derived, then fitted with the experimental data to determine the camera parameters. Likewise, pixelwise calibration is performed on the sCMOS to determine the camera parameters for further analysis. Theoretical derivation of the expected signal-to-noise-ratio is provided for each image sensor and corroborated by the experimental findings. Further comparison between the cameras includes analysis of the contrast-to-noise ratio (CNR) as well as the perception-based image quality estimator (PIQE). Improvement of image quality metrics in the 3D reconstructed images is successfully confirmed compared with those of the 2D images. To the best of our knowledge, this is the first experimental report of low-light-level 3D integral imaging with as little as a few photons detected per pixel on average to improve scene visualization including occlusion removal from the scene.

Spatial-temporal human gesture recognition under degraded conditions using three-dimensional integral imaging

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.

Human detection in occluded scenes through optically inspired multi-camera image fusion

In this paper, a novel approach for foreground extraction has been proposed based on a popular three-dimensional imaging technique in optics, called integral imaging. In this approach, multiple viewpoint images captured from a three-dimensional scene are used to extract range information of the scene and effectively extract an object or a person, even in the presence of heavy occlusion. The algorithm consists of two parts: depth estimation and reconstruction of the targeted object at the estimated depth distance. Further processing of the resulting reconstructed image can lead to the detection of a face or a pedestrian in the scene, which may not otherwise be detectable due to partial occlusion in each of the views. The validity of our approach has been demonstrated by experimental results in different scenarios.