Light field endoscopy and its parametric description

3D object tracking using integral imaging with mutual information and Bayesian optimization

Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging's depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object's depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object's bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object's depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.

3D integral imaging depth estimation of partially occluded objects using mutual information and Bayesian optimization

Integral imaging (InIm) is useful for passive ranging and 3D visualization of partially-occluded objects. We consider 3D object localization within a scene and in occlusions. 2D localization can be achieved using machine learning and non-machine learning-based techniques. These techniques aim to provide a 2D bounding box around each one of the objects of interest. A recent study uses InIm for the 3D reconstruction of the scene with occlusions and utilizes mutual information (MI) between the bounding box in this 3D reconstructed scene and the corresponding bounding box in the central elemental image to achieve passive depth estimation of partially occluded objects. Here, we improve upon this InIm method by using Bayesian optimization to minimize the number of required 3D scene reconstructions. We evaluate the performance of the proposed approach by analyzing different kernel functions, acquisition functions, and parameter estimation algorithms for Bayesian optimization-based inference for simultaneous depth estimation of objects and occlusion. In our optical experiments, mutual-information-based depth estimation with Bayesian optimization achieves depth estimation with a handful of 3D reconstructions. To the best of our knowledge, this is the first report to use Bayesian optimization for mutual information-based InIm depth estimation.

Information theoretic performance evaluation of 3D integral imaging

Integral imaging (InIm) has proved useful for three-dimensional (3D) object sensing, visualization, and classification of partially occluded objects. This paper presents an information-theoretic approach for simulating and evaluating the integral imaging capture and reconstruction process. We utilize mutual information (MI) as a metric for evaluating the fidelity of the reconstructed 3D scene. Also we consider passive depth estimation using mutual information. We apply this formulation for optimal pitch estimation of integral-imaging capture and reconstruction to maximize the longitudinal resolution. The effect of partial occlusion in integral imaging 3D reconstruction using mutual information is evaluated. Computer simulation tests and experiments are presented.

The Clinical Application of Remimazolam Benzenesulfonate Combined with Esketamine Intravenous Anesthesia in Endoscopic Retrograde Cholangiopancreatography

In this project, algorithm-based image processing methods in 3D endoscopic image processing endoscopic retrograde cholangiopancreatography (ERCP) were analyzed. To enhance local information of images, an adaptive histogram equalization method with limited contrast is introduced. The influences of the algorithm on 3D endoscopic image peak signal-to-noise ratio (PSNR), image discrete information entropy (DE), and average mean brightness error (AMBE) of images before and after the optimization before were compared. A total of 92 patients receiving ERCP at Yuhuangding Hospital between December 2019 and December 2021 were selected and divided into the control group (fentanyl+propofol) and the observation group (remimazolam benzenesulfonate+esketamine). Mean arterial pressure heart rate (HR), oxygen saturation (SpO2), and respiratory rate (RR) of the patients at each time point including the entry into the operation room (T0), 2 minutes after the beginning of medication (T1), after endoscopy (T2), endoscopy withdrawal (T3), and postoperative awakening (T4) were recorded. The comparison of MAP between T1, T2, T3, and T4 and T0 among patients in the observation group and the control group showed statistical differences (P < 0.05). Besides, HR and RR at T4 in the observation group were obviously higher than those in the control group (P < 0.05). The comparison of SpO2 at T3 and T4 and that at T0 both showed statistical differences (P < 0.05). Awakening time and VAS scores in the observation group were obviously lower than those in the control group (P < 0.05). The incidence of bradycardia, nausea, vomiting, and chill in the observation group was all lower than that in the control group (P < 0.05). The results indicated that an effective endoscopic image processing method was established based on an image enhancement algorithm, and the combination of remimazolam benzenesulfonate and esketamine showed high safety and efficacy in ERCP.

Prism-based tri-aperture laparoscopic objective for multi-view acquisition

This paper presents the design and prototype of a novel tri-aperture monocular laparoscopic objective that can acquire both stereoscopic views for depth information and a wide field of view (FOV) for situational awareness. The stereoscopic views are simultaneously captured via a shared objective with two displaced apertures and a custom prism. Overlapping crosstalk between the stereoscopic views is diminished by incorporating a strategically placed vignetting aperture. Meanwhile, the wide FOV is captured via a central third aperture of the same objective and provides a 2D view of the surgical field 2x as large as the area imaged by the stereoscopic views. We also demonstrate how the wide FOV provides a reference data set for stereo calibration, which enables absolute depth mapping in our experimental prototype.

Lowlight object recognition by deep learning with passive three-dimensional integral imaging in visible and long wave infrared wavelengths

Traditionally, long wave infrared imaging has been used in photon starved conditions for object detection and classification. We investigate passive three-dimensional (3D) integral imaging (InIm) in visible spectrum for object classification using deep neural networks in photon-starved conditions and under partial occlusion. We compare the proposed passive 3D InIm operating in the visible domain with that of the long wave infrared sensing in both 2D and 3D imaging cases for object classification in degraded conditions. This comparison is based on average precision, recall, and miss rates. Our experimental results demonstrate that cold and hot object classification using 3D InIm in the visible spectrum may outperform both 2D and 3D imaging implemented in long wave infrared spectrum for photon-starved and partially occluded scenes. While these experiments are not comprehensive, they demonstrate the potential of 3D InIm in the visible spectrum for low light applications. Imaging in the visible spectrum provides higher spatial resolution, more compact optics, and lower cost hardware compared with long wave infrared imaging. In addition, higher spatial resolution obtained in the visible spectrum can improve object classification accuracy. Our experimental results provide a proof of concept for implementing visible spectrum imaging in place of the traditional LWIR spectrum imaging for certain object recognition tasks.

Three-dimensional polarimetric image restoration in low light with deep residual learning and integral imaging

Polarimetric imaging can become challenging in degraded environments such as low light illumination conditions or in partial occlusions. In this paper, we propose the denoising convolutional neural network (DnCNN) model with three-dimensional (3D) integral imaging to enhance the reconstructed image quality of polarimetric imaging in degraded environments such as low light and partial occlusions. The DnCNN is trained based on the physical model of the image capture in degraded environments to enhance the visualization of polarimetric imaging where simulated low light polarimetric images are used in the training process. The DnCNN model is experimentally tested on real polarimetric images captured in real low light environments and in partial occlusion. The performance of DnCNN model is compared with that of total variation denoising. Experimental results demonstrate that DnCNN performs better than total variation denoising for polarimetric integral imaging in terms of signal-to-noise ratio and structural similarity index measure in low light environments as well as low light environments under partial occlusions. To the best of our knowledge, this is the first report of polarimetric 3D object visualization and restoration in low light environments and occlusions using DnCNN with integral imaging. The proposed approach is also useful for 3D image restoration in conventional (non-polarimetric) integral imaging in a degraded environment.

Deep learning polarimetric three-dimensional integral imaging object recognition in adverse environmental conditions

Polarimetric imaging is useful for object recognition and material classification because of its ability to discriminate objects based on polarimetric signatures of materials. Polarimetric imaging of an object captures important physical properties such as shape and surface properties and can be effective even in low light environments. Integral imaging is a passive three-dimensional (3D) imaging approach that takes advantage of multiple 2D imaging perspectives to perform 3D reconstruction. In this paper, we propose a unified polarimetric detection and classification of objects in degraded environments such as low light and the presence of occlusion. This task is accomplished using a deep learning model for 3D polarimetric integral imaging data captured in the visible spectral domain. The neural network system is designed and trained for 3D object detection and classification using polarimetric integral images. We compare the detection and classification results between polarimetric and non-polarimetric 2D and 3D imaging. The system performance in degraded environmental conditions is evaluated using average miss rate, average precision, and F-1 score. The results indicate that for the experiments we have performed, polarimetric 3D integral imaging outperforms 2D polarimetric imaging as well as non-polarimetric 2D and 3D imaging for object recognition in adverse conditions such as low light and occlusions. To the best of our knowledge, this is the first report for polarimetric 3D object recognition in low light environments and occlusions using a deep learning-based integral imaging. The proposed approach is attractive because low light polarimetric object recognition in the visible spectral band benefits from much higher spatial resolution, more compact optics, and lower system cost compared with long wave infrared imaging which is the conventional imaging approach for low light environments.

High-Identical Numerical Aperture, Multifocal Microlens Array through Single-Step Multi-Sized Hole Patterning Photolithography

Imaging applications based on microlens arrays (MLAs) have a great potential for the depth sensor, wide field-of-view camera and the reconstructed hologram. However, the narrow depth-of-field remains the challenge for accurate, reliable depth estimation. Multifocal microlens array (Mf-MLAs) is perceived as a major breakthrough, but existing fabrication methods are still hindered by the expensive, low-throughput, and dissimilar numerical aperture (NA) of individual lenses due to the multiple steps in the photolithography process. This paper reports the fabrication method of high NA, Mf-MLAs for the extended depth-of-field using single-step photolithography assisted by chemical wet etching. The various lens parameters of Mf-MLAs are manipulated by the multi-sized hole photomask and the wet etch time. Theoretical and experimental results show that the Mf-MLAs have three types of lens with different focal lengths, while maintaining the uniform and high NA irrespective of the lens type. Additionally, we demonstrate the multi-focal plane image acquisition via Mf-MLAs integrated into a microscope.

Three-dimensional polarimetric integral imaging in photon-starved conditions: performance comparison between visible and long wave infrared imaging

Three-dimensional (3D) polarimetric integral imaging (InIm) to extract the 3D polarimetric information of objects in photon-starved conditions is investigated using a low noise visible range camera and a long wave infrared (LWIR) range camera, and the performance between the two sensors is compared. Stokes polarization parameters and degree of polarization (DoP) are calculated to extract the polarimetric information of the 3D scene while integral imaging reconstruction provides depth information and improves the performance of low-light imaging tasks. An LWIR wire grid polarizer and a linear polarizer film are used as polarimetric objects for the LWIR range and visible range cameras, respectively. To account for a limited number of photons per pixel using the visible range camera in low light conditions, we apply a mathematical restoration model at each elemental image of visible camera to enhance the signal. We show that the low noise visible range camera may outperform the LWIR camera in detection of polarimetric objects under low illumination conditions. Our experiments indicate that for 3D polarimetric measurements under photon-starved conditions, visible range sensing may produce a signal-to-noise ratio (SNR) that is not lower than the LWIR range sensing. We derive the probability density function (PDF) of the 2D and 3D degree of polarization (DoP) images and show that the theoretical model demonstrates agreement to that of the experimentally obtained results. To the best of our knowledge, this is the first report comparing the polarimetric imaging performance between visible range and infrared (IR) range sensors under photon-starved conditions and the relevant statistical models of 3D polarimetric integral imaging.

Single-shot structured-light-field three-dimensional imaging

This Letter reports an approach to single-shot three-dimensional (3D) imaging that is combining structured illumination and light-field imaging. The sinusoidal distribution of the radiance in the structured-light field can be processed and transformed to compute the angular variance of the local radiance difference. The angular variance across the depth range exhibits a single-peak distribution trend that can be used to obtain the unambiguous depth. The phase computation that generally requires the acquisition of multi-frame phase-shifting images is no longer mandatory, thus enabling single-shot structured-light-field 3D imaging. The proposed approach was experimentally demonstrated through a dynamic scene.

High resolution, programmable aperture light field laparoscope for quantitative depth mapping

Recent applications have shown that light field imaging can be useful for developing uniaxial three-dimensional (3D) endoscopes. The immediate challenges in implementation are a tradeoff in lateral resolution and acquiring enough depth information in the physically limited environment of minimally invasive surgery. Here we propose using programmable aperture light field imaging in laparoscopy to capture 3D information without sacrificing the camera sensor's native, high spatial resolution. This hybrid design utilizes a programmable aperture to preserve the conventional laparoscope's functionality and, upon demand, to compute a depth map for surgical guidance. A working prototype is demonstrated.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Reflection light-field microscope with a digitally tunable aperture by single-pixel imaging

Reflected light microscope is a tool for imaging opaque specimens. However, most of the existing reflected light microscopes can only obtain the two-dimensional image of the specimen. Here we demonstrate that with the help of single-pixel imaging, we can develop a reflection light-field microscopy for volumetric imaging. Importantly, using single-pixel imaging, we can digitally adjust the size of the aperture diaphragm of the proposed reflection light-field microscope for changing the depth of field and for achieving three-dimensional differential phase-contrast imaging in an arbitrary direction, without a hardware change. Our approach may benefit various reflective specimens with wide depth information in the semiconductor industry and material science.

Achromatic metalens array for full-colour light-field imaging

A light-field camera captures both the intensity and the direction of incoming light^1-5. This enables a user to refocus pictures and afterwards reconstruct information on the depth of field. Research on light-field imaging can be divided into two components: acquisition and rendering. Microlens arrays have been used for acquisition, but obtaining broadband achromatic images with no spherical aberration remains challenging. Here, we describe a metalens array made of gallium nitride (GaN) nanoantennas⁶ that can be used to capture light-field information and demonstrate a full-colour light-field camera devoid of chromatic aberration. The metalens array contains an array of 60 × 60 metalenses with diameters of 21.65 μm. The camera has a diffraction-limited resolution of 1.95 μm under white light illumination. The depth of every object in the scene can be reconstructed slice by slice from a series of rendered images with different depths of focus. Full-colour, achromatic light-field cameras could find applications in a variety of fields such as robotic vision, self-driving vehicles and virtual and augmented reality.

Light-field-based absolute phase unwrapping

Ambiguity caused by a wrapped phase is an intrinsic problem in fringe projection-based 3D shape measurement. Among traditional methods for avoiding phase ambiguity, spatial phase unwrapping is sensitive to sensor noise and depth discontinuity, and temporal phase unwrapping requires additional encoding information that leads to an increase of image sequence acquisition time or a reduction of fringe contrast. Here, to the best of our knowledge, we report a novel method of absolute phase unwrapping based on light field imaging. In a recorded light field under structured illumination, i.e., a structured light field, a wrapped phase-encoded field can be retrieved and resampled in diverse image planes associated with several possible fringe orders in a measurement volume. Then, by leveraging phase consistency constraint in the resampled wrapped phase-encoded field, correct fringe orders can be determined to unwrap the wrapped phase without any additional encoding information. Experimental results demonstrated that the proposed method was suitable for accurate and robust absolute phase unwrapping.

Light field 3D measurement using unfocused plenoptic cameras

This Letter reports a novel method to establish the metric relationship of depth value between object space and image space for unfocused plenoptic cameras. A three-dimensional (3D) measurement system was introduced to precisely construct benchmarks and matching features to compute the metric depths in the object space and the corresponding depth values in the image space for metric calibration. After metric calibration, precise measurement of the depth dimension was possible. Furthermore, with the aid of metric spatio-angular parameters determined via light field ray calibration, transverse dimensions were computed from the measured depth, realizing light field 3D measurement for unfocused plenoptic cameras. Finally, we experimentally performed accuracy analysis of the proposed method with depth measurement precision of 0.5 mm in a depth range of 300 mm, which illuminated potential applications of unfocused plenoptic cameras in the field of 3D measurement.

Glare-free retinal imaging using a portable light field fundus camera

We present the retinal plenoptoscope, a novel light field retinal imaging device designed to overcome many of the problems that limit the use of portable non-mydriatic fundus cameras, including image quality and lack of stereopsis. The design and prototype construction of this device is detailed and the ideal relationship between the eye pupil, system aperture stop and micro-image separation is investigated. A comparison of the theoretical entrance pupil size, multi-view baseline and depth resolution indicates that a higher degree of stereopsis is possible than with stereo fundus cameras. We also show that the effects of corneal backscatter on image quality can be removed through a novel method of glare identification and selective image rendering. This method is then extended to produce glare-free depth maps from densely estimated depth fields, creating representations of retinal topography from a single exposure. These methods are demonstrated on physical models and live human eyes using a prototype device based on a Lytro Illum consumer light field camera. The Retinal Plenoptoscope offers a viable, robust modality for non-mydriatic color and 3-D retinal imaging.

Dual-dimensional microscopy: real-time in vivo three-dimensional observation method using high-resolution light-field microscopy and light-field display

Here, we present dual-dimensional microscopy that captures both two-dimensional (2-D) and light-field images of an in-vivo sample simultaneously, synthesizes an upsampled light-field image in real time, and visualizes it with a computational light-field display system in real time. Compared with conventional light-field microscopy, the additional 2-D image greatly enhances the lateral resolution at the native object plane up to the diffraction limit and compensates for the image degradation at the native object plane. The whole process from capturing to displaying is done in real time with the parallel computation algorithm, which enables the observation of the sample's three-dimensional (3-D) movement and direct interaction with the in-vivo sample. We demonstrate a real-time 3-D interactive experiment with Caenorhabditis elegans.

FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples

In this work, Fourier integral microscope (FIMic), an ultimate design of 3D-integral microscopy, is presented. By placing a multiplexing microlens array at the aperture stop of the microscope objective of the host microscope, FIMic shows extended depth of field and enhanced lateral resolution in comparison with regular integral microscopy. As FIMic directly produces a set of orthographic views of the 3D-micrometer-sized sample, it is suitable for real-time imaging. Following regular integral-imaging reconstruction algorithms, a 2.75-fold enhanced depth of field and [Formula: see text]-time better spatial resolution in comparison with conventional integral microscopy is reported. Our claims are supported by theoretical analysis and experimental images of a resolution test target, cotton fibers, and in-vivo 3D-imaging of biological specimens.