Resolution-enhanced integral imaging microscopy that uses lens array shifting

Spatial resolution enhancement with line-scan light-field imaging

This Letter proposes a line-scan-based light-field imaging framework that records lines of a light-field image successively to improve its spatial resolution. In this new, to the best of our knowledge, light-field imaging method, a conventional square or hexagonal microlens array is replaced with a cylindrical one. As such, the spatial resolution along the cylindrical axis remains unaffected, but angular information is recorded in the direction perpendicular to the cylindrical axis. By sequentially capturing multiple rows of light-field images with the aid of a translation device, a high-resolution two-dimensional light-field image can then be constructed. As a proof of concept, a prototype line-scan light-field camera was built and tested with the 1951 USAF resolution chart and the high-precision calibration dot array. Good measurement accuracies in the x, y, and z directions are demonstrated and prove that line-scan light-field imaging can significantly improve spatial resolutions and could be an alternative for fast three-dimensional inspections in the production line.

Enhanced light-field image resolution via MLA translation

This work describes a method that effectively improves the spatial resolution of light-field images without sacrificing angular resolution. The method involves translating the microlens array (MLA) linearly in both x- and y-directions in multiple steps to achieve 4 ×, 9 ×, 16 × and 25 × spatial resolution improvements. Its effectiveness was firstly validated through simulations with synthetic light-field images, demonstrating that distinct spatial resolution increments can be achieved by shifting the MLA. An MLA-translation light-field camera was built based on an industrial light-field camera, with which detailed experimental tests were carried out on a 1951 USAF resolution chart and a calibration plate. Qualitative and quantitative results prove that MLA translations can significantly improve measurement accuracy in x- and y- directions while preserving z-direction accuracy. Finally, the MLA-translation light-field camera was used to image a MEMS chip to demonstrate that finer structures of the chip can be acquired successfully.

3D microscope image acquisition method based on zoom objective

Microscopy is being pursued to obtain richer and more accurate information, and there are many challenges in imaging depth and display dimension. In this paper, we propose a three-dimensional (3D) microscope acquisition method based on a zoom objective. It enables 3D imaging of thick microscopic specimens with continuous adjustable optical magnification. The zoom objective based on liquid lenses can quickly adjust the focal length, to expand the imaging depth and change the magnification by adjusting the voltage. Based on the zoom objective, an arc shooting mount is designed to accurately rotate the objective to obtain the parallax information of the specimen and generate parallax synthesis images for 3D display. A 3D display screen is used to verify the acquisition results. The experimental results show that the obtained parallax synthesis images can accurately and efficiently restore the 3D characteristics of the specimen. The proposed method has promising applications in industrial detection, microbial observation, medical surgery, and so on.

Depth Estimation for Integral Imaging Microscopy Using a 3D-2D CNN with a Weighted Median Filter

This study proposes a robust depth map framework based on a convolutional neural network (CNN) to calculate disparities using multi-direction epipolar plane images (EPIs). A combination of three-dimensional (3D) and two-dimensional (2D) CNN-based deep learning networks is used to extract the features from each input stream separately. The 3D convolutional blocks are adapted according to the disparity of different directions of epipolar images, and 2D-CNNs are employed to minimize data loss. Finally, the multi-stream networks are merged to restore the depth information. A fully convolutional approach is scalable, which can handle any size of input and is less prone to overfitting. However, there is some noise in the direction of the edge. A weighted median filtering (WMF) is used to acquire the boundary information and improve the accuracy of the results to overcome this issue. Experimental results indicate that the suggested deep learning network architecture outperforms other architectures in terms of depth estimation accuracy.

Iterative tomography with digital adaptive optics permits hour-long intravital observation of 3D subcellular dynamics at millisecond scale

Long-term subcellular intravital imaging in mammals is vital to study diverse intercellular behaviors and organelle functions during native physiological processes. However, optical heterogeneity, tissue opacity, and phototoxicity pose great challenges. Here, we propose a computational imaging framework, termed digital adaptive optics scanning light-field mutual iterative tomography (DAOSLIMIT), featuring high-speed, high-resolution 3D imaging, tiled wavefront correction, and low phototoxicity with a compact system. By tomographic imaging of the entire volume simultaneously, we obtained volumetric imaging across 225 × 225 × 16 μm3, with a resolution of up to 220 nm laterally and 400 nm axially, at the millisecond scale, over hundreds of thousands of time points. To establish the capabilities, we investigated large-scale cell migration and neural activities in different species and observed various subcellular dynamics in mammals during neutrophil migration and tumor cell circulation.

Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy

The integral imaging microscopy system provides a three-dimensional visualization of a microscopic object. However, it has a low-resolution problem due to the fundamental limitation of the F-number (the aperture stops) by using micro lens array (MLA) and a poor illumination environment. In this paper, a generative adversarial network (GAN)-based super-resolution algorithm is proposed to enhance the resolution where the directional view image is directly fed as input. In a GAN network, the generator regresses the high-resolution output from the low-resolution input image, whereas the discriminator distinguishes between the original and generated image. In the generator part, we use consecutive residual blocks with the content loss to retrieve the photo-realistic original image. It can restore the edges and enhance the resolution by ×2, ×4, and even ×8 times without seriously hampering the image quality. The model is tested with a variety of low-resolution microscopic sample images and successfully generates high-resolution directional view images with better illumination. The quantitative analysis shows that the proposed model performs better for microscopic images than the existing algorithms.

Wave-optics and spatial frequency analyses of integral imaging three-dimensional display systems

Wave optics is usually thought to be more rigorous than geometrical optics to analyze integral imaging (II) systems. However, most of the previous wave-optics investigations are directed to a certain subsystem or do not sufficiently consider the finite aperture of microlens arrays (MLAs). Therefore, a diffraction-limited model of the entire II system, which consists of pickup, image processing, and reconstruction subsystems, is proposed, and the effects of system parameters on spatial resolution are especially studied. With the help of paraxial scalar diffraction theory, the origin impulse response function of the entire II system is derived; the parameter matching condition with optimum resolution and the wave-optics principle are achieved. Besides, the modulation transfer function is then obtained and Fourier analysis is performed, which indicates that the features of MLA and the display play a critical role in spatial frequency transfer characteristics, greatly affecting the resolution. These studies might be useful for the further research and understanding of II systems, especially for the effective enhancement of resolution.

Bionic-compound-eye structure for realizing a compact integral imaging 3D display in a cell phone with enhanced performance

A bionic-compound-eye structure (BCES), which is a substitute of a microlens array, is proposed to enhance the performance of integral imaging (II) 3D display systems. Hexagonal ocelli without gaps and barriers are predesigned to obtain a continuous image, high-resolution, and uniform parallax. A curved substrate is designed to enhance the viewing angle. In addition, ocelli are fused with the substrate to form a relief structure, BCES. When they are placed above a normal display, continuous and full-parallax 3D images with 150 µm effective resolution and a 28° horizontal, 22° vertical viewing angle could be achieved, about twice as much as that of normal systems. The weight of the BCES is 31 g, and the thickness of the whole system is 22 mm; thus, the BCES-based II (BCES-II) is very compact. In addition, this structure can be easily integrated into a cell phone or iPad for compact quasi-2D and 3D adjustable display.

Light field reconstruction with randomly shot photographs

Light field imaging is a promising technique for recording and displaying three-dimensional (3D) scenes. Light field reconstruction using a conventional camera, instead of a lens-array-based plenoptic camera, is expected to have a higher resolution. However, in existing conventional-camera-based methods, the camera placement is restricted. In this paper, we propose a new flexible light field reconstruction method involving shooting using a conventional camera from random 3D positions and angles. The proposed method has the advantage of flexibility because it uses a conventional camera, and the shooting position and angle can be selected randomly. Such flexibility is expected to give it a wide range of applications.

Fast-switching laterally virtual-moving microlens array for enhancing spatial resolution in light-field imaging system without degradation of angular sampling resolution

We present an electrically controllable fast-switching virtual-moving microlens array (MLA) consisting of a stacked structure of two polarization-dependent microlens arrays (PDMLAs) with optical orthogonality, where the position of the two stacked PDMLAs is shifted by half the elemental pitch in the diagonal direction. By controlling the polarization of the incident light without the physical movement of the molecules comprising the virtual-moving MLA, the periodic sampling position of the MLA can be switched fast using a polarization-switching layer based on a fast-switching liquid crystal cell. Using the fast-switching virtual-moving MLA, the spatial-resolution-enhanced light-field (LF) imaging system was demonstrated without a decrease in the angular sampling resolution as compared to the conventional LF imaging system comprising a passive MLA; two sets of elemental image arrays were captured quickly owing to the short switching time of the virtual-moving MLA of 450 μs. From the two captured sets of the elemental array image, four-times resolution-enhanced reconstruction images of the directional-view and depth-slice images could be obtained.

High-resolution light-field imaging via phase space retrieval

By combining a high-resolution image from a standard camera with a low-resolution light-field image from a lenslet array, we numerically reconstruct a high-resolution light-field image. We experimentally demonstrate the method by creating a high-definition 3D image of a human cheek cell with a commercially available microscope.

FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples

In this work, Fourier integral microscope (FIMic), an ultimate design of 3D-integral microscopy, is presented. By placing a multiplexing microlens array at the aperture stop of the microscope objective of the host microscope, FIMic shows extended depth of field and enhanced lateral resolution in comparison with regular integral microscopy. As FIMic directly produces a set of orthographic views of the 3D-micrometer-sized sample, it is suitable for real-time imaging. Following regular integral-imaging reconstruction algorithms, a 2.75-fold enhanced depth of field and [Formula: see text]-time better spatial resolution in comparison with conventional integral microscopy is reported. Our claims are supported by theoretical analysis and experimental images of a resolution test target, cotton fibers, and in-vivo 3D-imaging of biological specimens.

Enhancement of the depth-of-field of integral imaging microscope by using switchable bifocal liquid-crystalline polymer micro lens array

An integral imaging microscopy (IIM) system with improved depth-of-field (DoF) using a custom-designed bifocal polarization-dependent liquid-crystalline polymer micro lens array (LCP-MLA) is proposed. The implemented MLA has improved electro-optical properties such as a small focal ratio, high fill factor, low driving voltage, and fast switching speed, utilizing a well-aligned reactive mesogen on the imprinted reverse shape of the lens and a polarization switching layer. A bifocal MLA switches its focal length according to the polarization angle and acquires different DoF information of the specimen. After two elemental image arrays are captured, the depth-slices are reconstructed and combined to provide a widened DoF. The fabricated bifocal MLA consists of two identical polarization-dependent LCP-MLAs with 1.6 mm and f/16 focal ratio. Our experimental results confirmed that the proposed system improves the DoF of IIM without the need for mechanical manipulation.

Computational integral imaging reconstruction of perspective and orthographic view images by common patches analysis

A novel method to computationally reconstruct perspective and orthographic view images with full resolution of a recording device from a single integral photograph is proposed. Firstly, a group of image slices that contain full yet redundant information to reconstruct the view image are generated, and the object surface is divided into pieces by the points that correspond to the centers of image slices. Secondly, the image slices that contribute to the pieces are extracted and redundant information embedded in them are figured out by common patches analysis. Finally, the view image is reconstructed by excluding the redundant information and resampling with maximum sampling rate. Each piece of the object surface is represented with 9 patches at most from 4 adjacent elemental images, and view images with high quality are reconstructed. Both simulations and experiments verify the validity of the method.

Enhanced depth-of-field of an integral imaging microscope using a bifocal holographic optical element-micro lens array

We propose and implement an integral imaging microscope with extended depth-of-field (DoF) using a bifocal holographic micro lens array (MLA). The properties of the two MLAs are switched via peristrophic multiplexing, where different properties of the MLA are recorded onto the single holographic optical element (HOE). The recorded MLA properties are perpendicular to each other: after the first mode is recorded, the HOE is rotated by 90° clockwise, and the second mode is recorded. The experimental results confirm that the DoF of the integral imaging microscopy system is extended successfully by using the bifocal MLA.

Resolution improvements in integral microscopy with Fourier plane recording

Integral microscopes (IMic) have been recently developed in order to capture the spatial and the angular information of 3D microscopic samples with a single exposure. Computational post-processing of this information permits to carry out a 3D reconstruction of the sample. By applying conventional algorithms, both depth and also view reconstructions are possible. However, the main drawback of IMic is that the resolution of the reconstructed images is low and axially heterogeneous. In this paper, we propose a new configuration of the IMic by placing the lens array not at the image plane, but at the pupil (or Fourier) plane of the microscope objective. With this novel system, the spatial resolution is increased by factor 1.4, and the depth of field is substantially enlarged. Our experiments show the feasibility of the proposed method.

A review of snapshot multidimensional optical imaging: measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons' spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition-also dubbed snapshot imaging-has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications.

Integral imaging microscopy with enhanced depth-of-field using a spatial multiplexing

A depth-of-field enhancement method for integral imaging microscopy system using a spatial multiplexing structure consisting of a beamsplitter with dual video channels and micro lens arrays is proposed. A computational integral imaging reconstruction algorithm generates two sets of depth-sliced images for the acquired depth information of the captured elemental image arrays and the well-focused depth-slices of both image sets are combined where each is focused on a different depth plane of the specimen. A prototype is implemented, and the experimental results demonstrate that the depth-of-field of the reconstructed images in the proposed integral imaging microscopy is significantly increased compared with conventional integral imaging microscopy systems.

Resolution enhancement in integral microscopy by physical interpolation

Integral-imaging technology has demonstrated its capability for computing depth images from the microimages recorded after a single shot. This capability has been shown in macroscopic imaging and also in microscopy. Despite the possibility of refocusing different planes from one snap-shot is crucial for the study of some biological processes, the main drawback in integral imaging is the substantial reduction of the spatial resolution. In this contribution we report a technique, which permits to increase the two-dimensional spatial resolution of the computed depth images in integral microscopy by a factor of √2. This is made by a double-shot approach, carried out by means of a rotating glass plate, which shifts the microimages in the sensor plane. We experimentally validate the resolution enhancement as well as we show the benefit of applying the technique to biological specimens.

Resolution-enhancement for an orthographic-view image display in an integral imaging microscope system

Due to the limitations of micro lens arrays and camera sensors, images on display devices through the integral imaging microscope systems have been suffering for a low-resolution. In this paper, a resolution-enhanced orthographic-view image display method for integral imaging microscopy is proposed and demonstrated. Iterative intermediate-view reconstructions are performed based on bilinear interpolation using neighborhood elemental image information, and a graphics processing unit parallel processing algorithm is applied for fast image processing. The proposed method is verified experimentally and the effective results are presented in this paper.

Superresolution of 3-D computational integral imaging based on moving least square method

In this paper, we propose an edge directive moving least square (ED-MLS) based superresolution method for computational integral imaging reconstruction(CIIR). Due to the low resolution of the elemental images and the alignment error of the microlenses, it is not easy to obtain an accurate registration result in integral imaging, which makes it difficult to apply superresolution to the CIIR application. To overcome this problem, we propose the edge directive moving least square (ED-MLS) based superresolution method which utilizes the properties of the moving least square. The proposed ED-MLS based superresolution takes the direction of the edge into account in the moving least square reconstruction to deal with the abrupt brightness changes in the edge regions, and is less sensitive to the registration error. Furthermore, we propose a framework which shows how the data have to be collected for the superresolution problem in the CIIR application. Experimental results verify that the resolution of the elemental images is enhanced, and that a high resolution reconstructed 3-D image can be obtained with the proposed method.

Light ray field capture using focal plane sweeping and its optical reconstruction using 3D displays

We propose a method to capture light ray field of three-dimensional scene using focal plane sweeping. Multiple images are captured using a usual camera at different focal distances, spanning the three-dimensional scene. The captured images are then back-projected to four-dimensional spatio-angular space to obtain the light ray field. The obtained light ray field can be visualized either using digital processing or optical reconstruction using various three-dimensional display techniques including integral imaging, layered display, and holography.

Plenoptic projection fluorescence tomography

A new method to obtain the three-dimensional localization of fluorochrome distributions in micrometric samples is presented. It uses a microlens array coupled to the image port of a standard microscope to obtain tomographic data by a filtered back-projection algorithm. Scanning of the microlens array is proposed to obtain a dense data set for reconstruction. Simulation and experimental results are shown and the implications of this approach in fast 3D imaging are discussed.

Scalar wave-optical reconstruction of plenoptic camera images

We investigate the reconstruction of plenoptic camera images in a scalar wave-optical framework. Previous publications relating to this topic numerically simulate light propagation on the basis of ray tracing. However, due to continuing miniaturization of hardware components it can be assumed that in combination with low-aperture optical systems this technique may not be generally valid. Therefore, we study the differences between ray- and wave-optical object reconstructions of true plenoptic camera images. For this purpose we present a wave-optical reconstruction algorithm, which can be run on a regular computer. Our findings show that a wave-optical treatment is capable of increasing the detail resolution of reconstructed objects.

Real-time interactive display for integral imaging microscopy

A real-time interactive orthographic-view image display of integral imaging (II) microscopy that includes the generation of intermediate-view elemental images (IVEIs) for resolution enhancement is proposed. Unlike the conventional II microscopes, parallel processing through a graphics processing unit is required for real-time display that generates the IVEIs and interactive orthographic-view images in high speed, according to the user interactive input. The real-time directional-view display for the specimen for which 3D information is acquired through II microscopy is successfully demonstrated by using resolution-enhanced elemental image arrays. A user interactive feature is also satisfied in the proposed real-time interactive display for II microscopy.

Real-time integral imaging system for light field microscopy

We propose a real-time integral imaging system for light field microscopy systems. To implement a 3D live in-vivo experimental environment for multiple experimentalists, we generate elemental images for an integral imaging system from the captured light field with a light field microscope in real-time. We apply the f-number matching method to generate an elemental image to reconstruct an undistorted 3D image. Our implemented system produces real and orthoscopic 3D images of micro objects in 16 frames per second. We verify the proposed system via experiments using Caenorhabditis elegans.

Perspective rectification of integral images produced using arrays of circular lenses

There are many different three-dimensional (3D) techniques to capture and deliver autostereoscopic 3D content. A promising technique that provides two-dimensional parallax as well as high-quality, full-color 3D content is integral imaging (InI). Misalignments between the lens arrays (LAs) and the camera charged coupled device, however, introduce geometric distortions in the acquired image that propagate through the different image processing stages and deteriorate the 3D effect. Here, we propose a method to accurately rectify the perspective distortion of integral images (InIms) generated using circular lenses. Using an edge-linking approach, we extracted elliptically shaped contours of elemental images in the perspectively distorted InIm. To calculate the rectification matrix, we used the images of the circular points. Subsequently, we applied a triangulation scheme followed by a statistical approach to accurately estimate the grid structure of the LA. Finally, we provided experimental results over a wide range of InIms to evaluate the robustness and accuracy of the proposed method using objective metrics.

Axially moving a lenslet array for high-resolution 3D images in computational integral imaging

This paper presents a new high-resolution computational integral imaging system employing a pickup with the axial movement of a lenslet array and a computation reconstruction algorithm with pixel-to-pixel mapping. In the proposed method, a lenslet array and its image sensor are moved together along the z-axis direction (or axial direction) and a series of elemental image arrays are obtained while moving. The elemental image arrays are then applied to pixel-to-pixel mapping without interpolation for the reconstruction of 3D slice images. Also, an analysis of the proposed reconstruction method is provided. To show the usefulness of the proposed method, experiments are conducted. The results indicate that the proposed method is superior to the existing method such as MALT in terms of image quality.

Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]

Three-dimensional (3D) sensing and imaging technologies have been extensively researched for many applications in the fields of entertainment, medicine, robotics, manufacturing, industrial inspection, security, surveillance, and defense due to their diverse and significant benefits. Integral imaging is a passive multiperspective imaging technique, which records multiple two-dimensional images of a scene from different perspectives. Unlike holography, it can capture a scene such as outdoor events with incoherent or ambient light. Integral imaging can display a true 3D color image with full parallax and continuous viewing angles by incoherent light; thus it does not suffer from speckle degradation. Because of its unique properties, integral imaging has been revived over the past decade or so as a promising approach for massive 3D commercialization. A series of key articles on this topic have appeared in the OSA journals, including Applied Optics. Thus, it is fitting that this Commemorative Review presents an overview of literature on physical principles and applications of integral imaging. Several data capture configurations, reconstruction, and display methods are overviewed. In addition, applications including 3D underwater imaging, 3D imaging in photon-starved environments, 3D tracking of occluded objects, 3D optical microscopy, and 3D polarimetric imaging are reviewed.

Integral imaging using a color filter pinhole array on a display panel

We propose methods of enhancing pinhole-type integral imaging ray density, resolution, and expressible depth range using a color filter pinhole array on a liquid crystal display panel with a projection scheme. A color filter structure on a liquid crystal display panel acts as pinhole array in integral imaging with separation of color channels. In conventional pinhole-type integral imaging, the resolution, viewing angle, and ray density are limited by the pinhole interval, the width and thickness of the pinhole structure, and the gap between the display panel and the pinhole array. To overcome the limitation of the pinhole interval, we use a color filter pinhole array on a display panel and a projection-type integral imaging scheme. The use of a color filter pinhole array and the projection scheme can enlarge the region of one elemental image and improve the resolution and ray density remarkably. This paper presents the experimental results of the proposed method and a comparison with conventional methods.

Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging

We propose a depth extraction method by using the correlation between an elemental image and a periodic function in computational integral imaging. Because each elemental image corresponds to a different perspective of the three-dimensional (3-D) object, an elemental image is regarded as the sum of the periodic spatial frequencies depending on the depth of a 3-D object. In this regard, we analyze the property of correlation between the same periodic functions and vice versa. To show the feasibility of the proposed method, we carried out our experiment and presented the results.

High-resolution far-field integral-imaging camera by double snapshot

In multi-view three-dimensional imaging, to capture the elemental images of distant objects, the use of a field-like lens that projects the reference plane onto the microlens array is necessary. In this case, the spatial resolution of reconstructed images is equal to the spatial density of microlenses in the array. In this paper we report a simple method, based on the realization of double snapshots, to double the 2D pixel density of reconstructed scenes. Experiments are reported to support the proposed approach.

Resolution comparison between integral-imaging-based hologram synthesis methods using rectangular and hexagonal lens arrays

We compare the resolution of the hologram reconstruction synthesis methods based on integral imaging using rectangular and hexagonal lens arrays. By using a hexagonal lens array instead of conventional rectangular lens array, the three-dimensional objects are sampled with hexagonal grids. Due to more efficient sampling of the hexagonal grid, the resolution of the reconstructed object is higher compared with the case of using rectangular lens array. We analyze the resolution enhancement of the hologram reconstruction quantitatively and verify it experimentally.

Nanophotonic three-dimensional microscope

Three-dimensional (3D) optical microscopy based on integral imaging techniques is limited mainly by diffraction effects and the pitch of the microlens array used to sample the specimen. We integrate nanotechnology to the integral imaging technique and demonstrate a nanophotonic 3D microscope, where a nanophotonic lens array is used to finely sample the specimen. The resolution limitation due to diffraction is reduced by capturing images before the diffraction effects predominate and hence overcomes the bottleneck of achieving high resolution in an integral imaging 3D microscope.

Parameter analysis of integral Fourier hologram and its resolution enhancement

We present a parameter analysis of the integral Fourier hologram that is generated from multiple orthographic view images of a three-dimensional object. The maximum view angle, the lens array pitch, and the projection angle step are analyzed to reveal their effects on the maximum size of the reconstructed object and its maximum spatial frequency. With these analyses, we propose a lens array shift method to enhance the resolution of the reconstructed object from the Fourier hologram. The principles are verified by computational and optical experiment.