Resolution improvements in integral microscopy with Fourier plane recording

EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.

Whole-brain neural substrates of behavioral variability in the larval zebrafish

Animals engaged in naturalistic behavior can exhibit a large degree of behavioral variability even under sensory invariant conditions. Such behavioral variability can include not only variations of the same behavior, but also variability across qualitatively different behaviors driven by divergent cognitive states, such as fight-or-flight decisions. However, the neural circuit mechanisms that generate such divergent behaviors across trials are not well understood. To investigate this question, here we studied the visual-evoked responses of larval zebrafish to moving objects of various sizes, which we found exhibited highly variable and divergent responses across repetitions of the same stimulus. Given that the neuronal circuits underlying such behaviors span sensory, motor, and other brain areas, we built a novel Fourier light field microscope which enables high-resolution, whole-brain imaging of larval zebrafish during behavior. This enabled us to screen for neural loci which exhibited activity patterns correlated with behavioral variability. We found that despite the highly variable activity of single neurons, visual stimuli were robustly encoded at the population level, and the visual-encoding dimensions of neural activity did not explain behavioral variability. This robustness despite apparent single neuron variability was due to the multi-dimensional geometry of the neuronal population dynamics: almost all neural dimensions that were variable across individual trials, i.e. the "noise" modes, were orthogonal to those encoding for sensory information. Investigating this neuronal variability further, we identified two sparsely-distributed, brain-wide neuronal populations whose pre-motor activity predicted whether the larva would respond to a stimulus and, if so, which direction it would turn on a single-trial level. These populations predicted single-trial behavior seconds before stimulus onset, indicating they encoded time-varying internal modulating behavior, perhaps organizing behavior over longer timescales or enabling flexible behavior routines dependent on the animal's internal state. Our results provide the first whole-brain confirmation that sensory, motor, and internal variables are encoded in a highly mixed fashion throughout the brain and demonstrate that de-mixing each of these components at the neuronal population level is critical to understanding the mechanisms underlying the brain's remarkable flexibility and robustness.

Light-field flow cytometry for high-resolution, volumetric and multiparametric 3D single-cell analysis

Imaging flow cytometry (IFC) combines flow cytometry and fluorescence microscopy to enable high-throughput, multiparametric single-cell analysis with rich spatial details. However, current IFC techniques remain limited in their ability to reveal subcellular information with a high 3D resolution, throughput, sensitivity, and instrumental simplicity. In this study, we introduce a light-field flow cytometer (LFC), an IFC system capable of high-content, single-shot, and multi-color acquisition of up to 5,750 cells per second with a near-diffraction-limited resolution of 400-600 nm in all three dimensions. The LFC system integrates optical, microfluidic, and computational strategies to facilitate the volumetric visualization of various 3D subcellular characteristics through convenient access to commonly used epi-fluorescence platforms. We demonstrate the effectiveness of LFC in assaying, analyzing, and enumerating intricate subcellular morphology, function, and heterogeneity using various phantoms and biological specimens. The advancement offered by the LFC system presents a promising methodological pathway for broad cell biological and translational discoveries, with the potential for widespread adoption in biomedical research.

3D object tracking using integral imaging with mutual information and Bayesian optimization

Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging's depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object's depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object's bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object's depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.

High-resolution 3D imaging in light-field microscopy through Stokes matrices and data fusion

The trade-off between the lateral and vertical resolution has long posed challenges to the efficient and widespread application of Fourier light-field microscopy, a highly scalable 3D imaging tool. Although existing methods for resolution enhancement can improve the measurement result to a certain extent, they come with limitations in terms of accuracy and applicable specimen types. To address these problems, this paper proposed a resolution enhancement scheme utilizing data fusion of polarization Stokes vectors and light-field information for Fourier light-field microscopy system. By introducing the surface normal vector information obtained from polarization measurement and integrating it with the light-field 3D point cloud data, 3D reconstruction results accuracy is highly improved in axial direction. Experimental results with a Fourier light-field 3D imaging microscope demonstrated a substantial enhancement of vertical resolution with a depth resolution to depth of field ratio of 0.19%. This represented approximately 44 times the improvement compared to the theoretical ratio before data fusion, enabling the system to access more detailed information with finer measurement accuracy for test samples. This work not only provides a feasible solution for breaking the limitations imposed by traditional light-field microscope hardware configurations but also offers superior 3D measurement approach in a more cost-effective and practical manner.

Quantitative real-time phase microscopy for extended depth-of-field imaging based on the 3D single-shot differential phase contrast (ssDPC) imaging method

Optical diffraction tomography (ODT) is a promising label-free imaging method capable of quantitatively measuring the three-dimensional (3D) refractive index distribution of transparent samples. In recent years, partially coherent ODT (PC-ODT) has attracted increasing attention due to its system simplicity and absence of laser speckle noise. Quantitative phase imaging (QPI) technologies represented by Fourier ptychographic microscopy (FPM), differential phase contrast (DPC) imaging and intensity diffraction tomography (IDT) need to collect several or hundreds of intensity images, which usually introduce motion artifacts when shooting fast-moving targets, leading to a decrease in image quality. Hence, a quantitative real-time phase microscopy (qRPM) for extended depth of field (DOF) imaging based on 3D single-shot differential phase contrast (ssDPC) imaging method is proposed in this research study. qRPM incorporates a microlens array (MLA) to simultaneously collect spatial information and angular information. In subsequent optical information processing, a deconvolution method is used to obtain intensity stacks under different illumination angles in a raw light field image. Importing the obtained intensity stack into the 3D DPC imaging model is able to finally obtain the 3D refractive index distribution. The captured four-dimensional light field information enables the reconstruction of 3D information in a single snapshot and extending the DOF of qRPM. The imaging capability of the proposed qRPM system is experimental verified on different samples, achieve single-exposure 3D label-free imaging with an extended DOF for 160 µm which is nearly 30 times higher than the traditional microscope system.

Optimal sparsity allows reliable system-aware restoration of fluorescence microscopy images

Fluorescence microscopy is one of the most indispensable and informative driving forces for biological research, but the extent of observable biological phenomena is essentially determined by the content and quality of the acquired images. To address the different noise sources that can degrade these images, we introduce an algorithm for multiscale image restoration through optimally sparse representation (MIRO). MIRO is a deterministic framework that models the acquisition process and uses pixelwise noise correction to improve image quality. Our study demonstrates that this approach yields a remarkable restoration of the fluorescence signal for a wide range of microscopy systems, regardless of the detector used (e.g., electron-multiplying charge-coupled device, scientific complementary metal-oxide semiconductor, or photomultiplier tube). MIRO improves current imaging capabilities, enabling fast, low-light optical microscopy, accurate image analysis, and robust machine intelligence when integrated with deep neural networks. This expands the range of biological knowledge that can be obtained from fluorescence microscopy.

Computational Integral Imaging Reconstruction via Elemental Image Blending without Normalization

This paper presents a novel computational integral imaging reconstruction (CIIR) method using elemental image blending to eliminate the normalization process in CIIR. Normalization is commonly used in CIIR to address uneven overlapping artifacts. By incorporating elemental image blending, we remove the normalization step in CIIR, leading to decreased memory consumption and computational time compared to those of existing techniques. We conducted a theoretical analysis of the impact of elemental image blending on a CIIR method using windowing techniques, and the results showed that the proposed method is superior to the standard CIIR method in terms of image quality. We also performed computer simulations and optical experiments to evaluate the proposed method. The experimental results showed that the proposed method enhances the image quality over that of the standard CIIR method, while also reducing memory usage and processing time.

Practical guide for setting up a Fourier light-field microscope

A practical guide for the easy implementation of a Fourier light-field microscope is reported. The Fourier light-field concept applied to microscopy allows the capture in real time of a series of 2D orthographic images of microscopic thick dynamic samples. Such perspective images contain spatial and angular information of the light-field emitted by the sample. A feature of this technology is the tight requirement of a double optical conjugation relationship, and also the requirement of NA matching. For these reasons, the Fourier light-field microscope being a non-complex optical system, a clear protocol on how to set up the optical elements accurately is needed. In this sense, this guide is aimed to simplify the implementation process, with an optical bench and off-the-shelf components. This will help the widespread use of this recent technology.

Optical projection tomography of fluorescent microscopic specimens using lateral translation of tube lens

Optical projection tomography (OPT) is a three-dimensional (3D) fluorescence imaging technique, in which projection images are acquired for varying orientations of a sample using a large depth of field. OPT is typically applied to a millimeter-sized specimen, because the rotation of a microscopic specimen is challenging and not compatible with live cell imaging. In this Letter, we demonstrate fluorescence optical tomography of a microscopic specimen by laterally translating the tube lens of a wide-field optical microscope, which allows for high-resolution OPT without rotating the sample. The cost is the reduction of the field of view to about halfway along the direction of the tube lens translation. Using bovine pulmonary artery endothelial cells and 0.1 µm beads, we compare the 3D imaging performance of the proposed method with that of the conventional objective-focus scan method.

Compact, Hybrid Light-Sheet and Fourier Light-Field Microscopy with a Single Objective for High-Speed Volumetric Imaging In Vivo

Volumetric imaging of biodynamics at high spatiotemporal resolutions in vivo is vital in biomedical studies, in which Fourier light field microscopy (FLFM) is a promising technique. However, the commonly used wide-field illumination strategy in FLFM introduces intense out of depth-of-field background, which not only degrades the image quality, but also introduces reconstruction artifacts. Employing light sheet illumination is an effective way to alleviate the background and reduce photobleaching in light-field microscopy. Unfortunately, the introduction of light-sheet illumination often requires an extra objective and precise alignment, which increases the system complexity. Here, we propose the compact, hybrid light-sheet and FLFM (CLS-FLFM), which uses only a single objective to achieve both light-sheet illumination and Fourier light-field imaging simultaneously. With a micromirror under the objective, we focus the light sheet, which ensures selective-volume-illumination, on the imaging plane of the FLFM to perform volumetric imaging. We demonstrate the superior performance of CLS-FLFM in inhibiting background in both structural and dynamical imaging of larval zebrafish in vivo. We envision that CLS-FLFM finds wide applications in high-speed, background-inhibited volumetric imaging of biodynamics in vivo.

High-Quality 3D Visualization System for Light-Field Microscopy with Fine-Scale Shape Measurement through Accurate 3D Surface Data

We propose a light-field microscopy display system that provides improved image quality and realistic three-dimensional (3D) measurement information. Our approach acquires both high-resolution two-dimensional (2D) and light-field images of the specimen sequentially. We put forward a matting Laplacian-based depth estimation algorithm to obtain nearly realistic 3D surface data, allowing the calculation of depth data, which is relatively close to the actual surface, and measurement information from the light-field images of specimens. High-reliability area data of the focus measure map and spatial affinity information of the matting Laplacian are used to estimate nearly realistic depths. This process represents a reference value for the light-field microscopy depth range that was not previously available. A 3D model is regenerated by combining the depth data and the high-resolution 2D image. The element image array is rendered through a simplified direction-reversal calculation method, which depends on user interaction from the 3D model and is displayed on the 3D display device. We confirm that the proposed system increases the accuracy of depth estimation and measurement and improves the quality of visualization and 3D display images.

3D super-resolution live-cell imaging with radial symmetry and Fourier light-field microscopy

Live-cell imaging reveals the phenotypes and mechanisms of cellular function and their dysfunction that underscore cell physiology, development, and pathology. Here, we report a 3D super-resolution live-cell microscopy method by integrating radiality analysis and Fourier light-field microscopy (rad-FLFM). We demonstrated the method using various live-cell specimens, including actins in Hela cells, microtubules in mammary organoid cells, and peroxisomes in COS-7 cells. Compared with conventional wide-field microscopy, rad-FLFM realizes scanning-free, volumetric 3D live-cell imaging with sub-diffraction-limited resolution of ∼150 nm (x-y) and 300 nm (z), milliseconds volume acquisition time, six-fold extended depth of focus of ∼6 µm, and low photodamage. The method provides a promising avenue to explore spatiotemporal-challenging subcellular processes in a wide range of cell biological research.

Comparing synthetic refocusing to deconvolution for the extraction of neuronal calcium transients from light fields

Significance: Light-field microscopy (LFM) enables fast, light-efficient, volumetric imaging of neuronal activity with calcium indicators. Calcium transients differ in temporal signal-to-noise ratio (tSNR) and spatial confinement when extracted from volumes reconstructed by different algorithms. Aim: We evaluated the capabilities and limitations of two light-field reconstruction algorithms for calcium fluorescence imaging. Approach: We acquired light-field image series from neurons either bulk-labeled or filled intracellularly with the red-emitting calcium dye CaSiR-1 in acute mouse brain slices. We compared the tSNR and spatial confinement of calcium signals extracted from volumes reconstructed with synthetic refocusing and Richardson-Lucy three-dimensional deconvolution with and without total variation regularization. Results: Both synthetic refocusing and Richardson-Lucy deconvolution resolved calcium signals from single cells and neuronal dendrites in three dimensions. Increasing deconvolution iteration number improved spatial confinement but reduced tSNR compared with synthetic refocusing. Volumetric light-field imaging did not decrease calcium signal tSNR compared with interleaved, widefield image series acquired in matched planes. Conclusions: LFM enables high-volume rate, volumetric imaging of calcium transients in single cell somata (bulk-labeled) and dendrites (intracellularly loaded). The trade-offs identified for tSNR, spatial confinement, and computational cost indicate which of synthetic refocusing or deconvolution can better realize the scientific requirements of future LFM calcium imaging applications.

Background inhibited and speed-loss-free volumetric imaging in vivo based on structured-illumination Fourier light field microscopy

Benefiting from its advantages in fast volumetric imaging for recording biodynamics, Fourier light field microscopy (FLFM) has a wide range of applications in biomedical research, especially in neuroscience. However, the imaging quality of the FLFM is always deteriorated by both the out-of-focus background and the strong scattering in biological samples. Here we propose a structured-illumination and interleaved-reconstruction based Fourier light field microscopy (SI-FLFM), in which we can filter out the background fluorescence in FLFM without sacrificing imaging speed. We demonstrate the superiority of our SI-FLFM in high-speed, background-inhibited volumetric imaging of various biodynamics in larval zebrafish and mice in vivo. The signal-to-background ratio (SBR) is improved by tens of times. And the volumetric imaging speed can be up to 40 Hz, avoiding artifacts caused by temporal under-sampling in conventional structured illumination microscopy. These suggest that our SI-FLFM is suitable for applications of weak fluorescence signals but high imaging speed requirements.

Snapshot volumetric imaging with engineered point-spread functions

The biological world involves intracellular and intercellular interactions that occur at high speed, at multiple scales and in three dimensions. Acquiring 3D images, however, typically requires a compromise in either spatial or temporal resolution compared to 2D imaging. Conventional 2D fluorescence imaging provides high spatial resolution but requires plane-by-plane imaging of volumes. Conversely, snapshot methods such as light-field microscopy allow video-rate imaging, but at the cost of spatial resolution. Here we introduce 3D engineered point-spread function microscopy (3D-EPM), enabling snapshot imaging of real-world 3D extended biological structures while retaining the native resolution of the microscope in space and time. Our new computational recovery strategy is the key to volumetrically reconstructing arbitrary 3D structures from the information encapsulated in 2D raw EPM images. We validate our technique on both point-like and extended samples, and demonstrate its power by imaging the intracellular motion of chloroplasts undergoing cyclosis in a sample of Egeria densa. Our technique represents a generalised computational methodology for 3D image recovery which is readily adapted to a diverse range of existing microscopy platforms and engineered point-spread functions. We therefore expect it to find broad applicability in the study of rapid biological dynamics in 3D.

Hyperspectral three-dimensional absorption imaging using snapshot optical tomography

Hyperspectral imaging (HSI) records a series of two-dimensional (2D) images for different wavelengths to provide the chemical fingerprint at each pixel. Combining HSI with a tomographic data acquisition method, we can obtain the chemical fingerprint of a sample at each point in three-dimensional (3D) space. The so-called 3D HSI typically suffers from low imaging throughput due to the requirement of scanning the wavelength and rotating the beam or sample. In this paper we present an optical system which captures the entire four-dimensional (4D), i.e., 3D structure and 1D spectrum, dataset of a sample with the same throughput of conventional HSI systems. Our system works by combining snapshot projection optical tomography (SPOT) which collects multiple projection images with a single snapshot, and Fourier-transform spectroscopy (FTS) which results in superior spectral resolution by collecting and processing a series of interferogram images. Using this hyperspectral SPOT system we imaged the volumetric absorbance of dyed polystyrene microbeads, oxygenated red blood cells (RBCs), and deoxygenated RBCs. The 4D optical system demonstrated in this paper provides a tool for high-throughput chemical imaging of complex microscopic specimens.

Volumetric Imaging of Neural Activity by Light Field Microscopy

Recording the highly diverse and dynamic activities in large populations of neurons in behaving animals is crucial for a better understanding of how the brain works. To meet this challenge, extensive efforts have been devoted to developing functional fluorescent indicators and optical imaging techniques to optically monitor neural activity. Indeed, optical imaging potentially has extremely high throughput due to its non-invasive access to large brain regions and capability to sample neurons at high density, but the readout speed, such as the scanning speed in two-photon scanning microscopy, is often limited by various practical considerations. Among different imaging methods, light field microscopy features a highly parallelized 3D fluorescence imaging scheme and therefore promises a novel and faster strategy for functional imaging of neural activity. Here, we briefly review the working principles of various types of light field microscopes and their recent developments and applications in neuroscience studies. We also discuss strategies and considerations of optimizing light field microscopy for different experimental purposes, with illustrative examples in imaging zebrafish and mouse brains.

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy

Current interest in Fourier lightfield microscopy is increasing, due to its ability to acquire 3D images of thick dynamic samples. This technique is based on simultaneously capturing, in a single shot, and with a monocular setup, a number of orthographic perspective views of 3D microscopic samples. An essential feature of Fourier lightfield microscopy is that the number of acquired views is low, due to the trade-off relationship existing between the number of views and their corresponding lateral resolution. Therefore, it is important to have a tool for the generation of a high number of synthesized view images, without compromising their lateral resolution. In this context we investigate here the use of a neural radiance field view synthesis method, originally developed for its use with macroscopic scenes acquired with a moving (or an array of static) digital camera(s), for its application to the images acquired with a Fourier lightfield microscope. The results obtained and presented in this paper are analyzed in terms of lateral resolution and of continuous and realistic parallax. We show that, in terms of these requirements, the proposed technique works efficiently in the case of the epi-illumination microscopy mode.

Fourier lightfield microscopy: a practical design guide

In this work, a practical guide for the design of a Fourier lightfield microscope is reported. The fundamentals of the Fourier lightfield are presented and condensed on a set of contour plots from which the user can select the design values of the spatial resolution, the field of view, and the depth of field, as function of the specifications of the hardware of the host microscope. This work guides the reader to select the parameters of the infinity-corrected microscope objective, the optical relay lenses, the aperture stop, the microlens array, and the digital camera. A user-friendly graphic calculator is included to ease the design, even to those who are not familiar with the lightfield technology. The guide is aimed to simplify the design process of a Fourier lightfield microscope, which sometimes could be a daunting task, and in this way, to invite the widespread use of this technology. An example of a design and experimental results on imaging different types of samples is also presented.

Improving image resolution on point-like sources in a type 1 light-field camera

A ray-trace simulation of a type 1 light-field imager is used to show that resolutions significantly better than the lenslet scale can be deterministically reached in reconstructed images of isolated point-like sources. This is enabled by computationally projecting the system pupil onto the lenslet-array plane to better estimate the lenslet-plane-crossing locations through which the rays from a point source have passed on their way to the detector array. Improving light-field type 1 image resolution from the lenslet scale to the pixel scale can significantly enhance signal-to-noise ratios on faint point-like sources such as fluorescent microbes, making the technique of interest in, e.g., in situ microbial life searches in extreme environments.

Handheld and Cost-Effective Fourier Lightfield Microscope

In this work, the design, building, and testing of the most portable, easy-to-build, robust, handheld, and cost-effective Fourier Lightfield Microscope (FLMic) to date is reported. The FLMic is built by means of a surveillance camera lens and additional off-the-shelf optical elements, resulting in a cost-effective FLMic exhibiting all the regular sought features in lightfield microscopy, such as refocusing and gathering 3D information of samples by means of a single-shot approach. The proposed FLMic features reduced dimensions and light weight, which, combined with its low cost, turn the presented FLMic into a strong candidate for in-field application where 3D imaging capabilities are pursued. The use of cost-effective optical elements has a relatively low impact on the optical performance, regarding the figures dictated by the theory, while its price can be at least 100 times lower than that of a regular FLMic. The system operability is tested in both bright-field and fluorescent modes by imaging a resolution target, a honeybee wing, and a knot of dyed cotton fibers.

Snapshot three-dimensional absorption imaging of microscopic specimens

Snapshot projection optical tomography (SPOT) uses a micro-lens array (MLA) to simultaneously capture the projection images of a three-dimensional (3D) specimen corresponding to different viewing directions. Compared to other light-field imaging techniques using an MLA, SPOT is dual telecentric and can block high-angle stray rays without sacrificing the light collection efficiency. Using SPOT, we recently demonstrated snapshot 3D fluorescence imaging. Here we demonstrate snapshot 3D absorption imaging of microscopic specimens. For the illumination, we focus the incoherent light from a light-emitting diode onto a pinhole, which is placed at a conjugate plane to the sample plane. SPOT allows us to capture the ray bundles passing through the specimen along different directions. The images recorded by an array of lenslets can be related to the projections of 3D absorption coefficient along the viewing directions of lenslets. Using a tomographic reconstruction algorithm, we obtain the 3D map of absorption coefficient. We apply the developed system to different types of samples, which demonstrates the optical sectioning capability. The transverse and axial resolutions measured with gold nanoparticles are 1.3 μm and 2.3 μm, respectively.

Hyperspectral Three-Dimensional Fluorescence Imaging Using Snapshot Optical Tomography

Hyperspectral three-dimensional (3D) imaging can provide both 3D structural and functional information of a specimen. The imaging throughput is typically very low due to the requirement of scanning mechanisms for different depths and wavelengths. Here we demonstrate hyperspectral 3D imaging using Snapshot projection optical tomography (SPOT) and Fourier-transform spectroscopy (FTS). SPOT allows us to instantaneously acquire the projection images corresponding to different viewing angles, while FTS allows us to perform hyperspectral imaging at high spectral resolution. Using fluorescent beads and sunflower pollens, we demonstrate the imaging performance of the developed system.

High-resolution Fourier light-field microscopy for volumetric multi-color live-cell imaging

Volumetric interrogation of the organization and processes of intracellular organelles and molecules in cellular systems with a high spatiotemporal resolution is essential for understanding cell physiology, development, and pathology. Here, we report high-resolution Fourier light-field microscopy (HR-FLFM) for fast and volumetric live-cell imaging. HR-FLFM transforms conventional cell microscopy and enables exploration of less accessible spatiotemporal-limiting regimes for single-cell studies. The results present a near-diffraction-limited resolution in all three dimensions, a five-fold extended focal depth to several micrometers, and a scanning-free volume acquisition time up to milliseconds. The system demonstrates instrumentation accessibility, low photo damage for continuous observation, and high compatibility with general cell assays. We anticipate HR-FLFM to offer a promising methodological pathway for investigating a wide range of intracellular processes and functions with exquisite spatiotemporal contextual details.

Light-field microscopy for fast volumetric brain imaging

Recording neural activities over large populations is critical for a better understanding of the functional mechanisms of animal brains. Traditional optical imaging technologies for in vivo neural activity recording are usually limited in throughput and cannot cover a large imaging volume at high speed. Light-field microscopy features a highly parallelized imaging collection mechanism and can simultaneously record optical signals from different depths. Therefore, it can potentially increase the imaging throughput substantially. Furthermore, its unique instantaneous volumetric imaging capability enables the capture of highly dynamic processes, such as recording whole-animal neural activities in freely moving Caenorhabditis elegans and whole-brain neural activity in freely swimming larval zebrafish during prey capture. Here, we summarize the principles of and considerations in the practical implementation of light-field microscopy as currently applied in biological imaging experiments. We also discuss the strategies that light-field microscopy can employ when imaging thick tissues in the presence of scattering and background interference. Finally, we present a few examples of applying light-field microscopy in neuroscientific studies in several important animal models.

wFLFM: enhancing the resolution of Fourier light-field microscopy using a hybrid wide-field image

We introduce wFLFM, an approach that enhances the resolution of Fourier light-field microscopy (FLFM) through a hybrid wide-field image. The system exploits the intrinsic compatibility of image formation between the on-axis FLFM elemental image and the wide-field image, allowing for minimal instrumental and computational complexity. The numerical and experimental results of wFLFM present a two- to three-fold improvement in the lateral resolution without compromising the 3D imaging capability in comparison with conventional FLFM.

Roadmap on 3D integral imaging: sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.

Three-dimensional real-time darkfield imaging through Fourier lightfield microscopy

We report a protocol that takes advantage of the Fourier lightfield microscopy concept for providing 3D darkfield images of volumetric samples in a single-shot. This microscope takes advantage of the Fourier lightfield configuration, in which a lens array is placed at the Fourier plane of the microscope objective, providing a direct multiplexing of the spatio-angular information of the sample. Using the proper illumination beam, the system collects the light scattered by the sample while the background light is blocked out. This produces a set of orthographic perspective images with shifted spatial-frequency components that can be recombined to produce a 3D darkfield image. Applying the adequate reconstruction algorithm high-contrast darkfield optical sections are calculated in real time. The presented method is applied for fast volumetric reconstructions of unstained 3D samples.

Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy

Miniature fluorescence microscopes are a standard tool in systems biology. However, widefield miniature microscopes capture only 2D information, and modifications that enable 3D capabilities increase the size and weight and have poor resolution outside a narrow depth range. Here, we achieve the 3D capability by replacing the tube lens of a conventional 2D Miniscope with an optimized multifocal phase mask at the objective's aperture stop. Placing the phase mask at the aperture stop significantly reduces the size of the device, and varying the focal lengths enables a uniform resolution across a wide depth range. The phase mask encodes the 3D fluorescence intensity into a single 2D measurement, and the 3D volume is recovered by solving a sparsity-constrained inverse problem. We provide methods for designing and fabricating the phase mask and an efficient forward model that accounts for the field-varying aberrations in miniature objectives. We demonstrate a prototype that is 17 mm tall and weighs 2.5 grams, achieving 2.76 μm lateral, and 15 μm axial resolution across most of the 900 × 700 × 390 μm3 volume at 40 volumes per second. The performance is validated experimentally on resolution targets, dynamic biological samples, and mouse brain tissue. Compared with existing miniature single-shot volume-capture implementations, our system is smaller and lighter and achieves a more than 2× better lateral and axial resolution throughout a 10× larger usable depth range. Our microscope design provides single-shot 3D imaging for applications where a compact platform matters, such as volumetric neural imaging in freely moving animals and 3D motion studies of dynamic samples in incubators and lab-on-a-chip devices.

Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser

Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plane to encode 3D information, then computationally reconstructs the volume by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths; the random positions provide a larger FOV compared to a conventional MLA, and the diverse focal lengths improve the axial depth range. To predict system performance based on diffuser parameters, we, for the first time, establish a theoretical framework and design guidelines, which are verified by numerical simulations, and then build an experimental system that achieves < 3 µm lateral and 4 µm axial resolution over a 1000 × 1000 × 280 µm³ volume. Our diffuser design outperforms the MLA used in LFM, providing more uniform resolution over a larger volume, both laterally and axially.

Imaging volumetric dynamics at high speed in mouse and zebrafish brain with confocal light field microscopy

A detailed understanding of the function of neural networks and how they are supported by a dynamic vascular system requires fast three-dimensional imaging in thick tissues. Here we present confocal light field microscopy, a method that enables fast volumetric imaging in the brain at depths of hundreds of micrometers. It uses a generalized confocal detection scheme that selectively collects fluorescent signals from the in-focus volume and provides optical sectioning capability to improve imaging resolution and sensitivity in thick tissues. We demonstrate recording of whole-brain calcium transients in freely swimming zebrafish larvae and observe behaviorally correlated activities in single neurons during prey capture. Furthermore, in the mouse brain, we detect neural activities at depths of up to 370 μm and track blood cells at 70 Hz over a volume of diameter 800 μm × thickness 150 μm and depth of up to 600 μm.

What about computational super-resolution in fluorescence Fourier light field microscopy?

Recently, Fourier light field microscopy was proposed to overcome the limitations in conventional light field microscopy by placing a micro-lens array at the aperture stop of the microscope objective instead of the image plane. In this way, a collection of orthographic views from different perspectives are directly captured. When inspecting fluorescent samples, the sensitivity and noise of the sensors are a major concern and large sensor pixels are required to cope with low-light conditions, which implies under-sampling issues. In this context, we analyze the sampling patterns in Fourier light field microscopy to understand to what extent computational super-resolution can be triggered during deconvolution in order to improve the resolution of the 3D reconstruction of the imaged data.

Snapshot projection optical tomography

We present a new plenoptic microscopy configuration for 3D snapshot imaging, which is dual telecentric and can directly record true projection images corresponding with different viewing angles. It also allows blocking high-angle stray rays without sacrificing the light collection efficiency. This configuration named as snapshot projection optical tomography (SPOT) arranges an objective lens and a microlens array (MLA) in a 4-f telecentric configuration and places an aperture stop at the back focal plane of a relay lens. We develop a forward imaging model for SPOT, which can also be applied to existing light field microscopy techniques using an MLA as tube lens. Using the developed system, we demonstrate snapshot 3D imaging of various fluorescent beads and a biological cell, which confirms the capability of SPOT for imaging specimens with an extended fluorophore distribution as well as isolated fluorochromes. The transverse and vertical resolutions are measured to be 0.8 μm and 1.6 μm, respectively.

Light field camera based on hexagonal array of flat-surface nanostructured GRIN lenses

In this paper we present a light field camera system where a flat-surface hexagonal array of nanostructured gradient index lenses was used as a lens matrix. In our approach we use an array of 469 gradient index microlenses with a diameter of 20 µm and 100% fill factor. To develop the single lens and the lenslet array we used a modified stack-and-draw technology. In this technique, variation of refractive index is achieved by using quantized gradient index profiles and rods from different types of glasses. We show experimental results of using this type of lenses for imaging in a system of two kinds of light field cameras. In the first one, the microlens array is located in the focal plane of the main lens. The image is reconstructed, in this case using a Fourier slice photography algorithm. This allowed a partial reconstruction of a 3D scene with spatial and depth resolution of 20 µm and field of view of 500×500×500 µm. In the second configuration, the microlens array is located between a sample and a microscopic objective, thus allowing for superresolution 3D reconstruction of a microscopic image. The scale-invariant feature transform method was used for image reconstruction and obtained a partial 3D reconstruction with a field of view of 150×115×80 µm and a spatial resolution of 2 µm and depth resolution of 10 µm.

Fourier light-field microscopy

Observing the various anatomical and functional information that spans many spatiotemporal scales with high resolution provides deep understandings of the fundamentals of biological systems. Light-field microscopy (LFM) has recently emerged as a scanning-free, scalable method that allows for high-speed, volumetric imaging ranging from single-cell specimens to the mammalian brain. However, the prohibitive reconstruction artifacts and severe computational cost have thus far limited broader applications of LFM. To address the challenge, in this work, we report Fourier LFM (FLFM), a system that processes the light-field information through the Fourier domain. We established a complete theoretical and algorithmic framework that describes light propagation, image formation and system characterization of FLFM. Compared with conventional LFM, FLFM fundamentally mitigates the artifacts, allowing high-resolution imaging across a two- to three-fold extended depth. In addition, the system substantially reduces the reconstruction time by roughly two orders of magnitude. FLFM was validated by high-resolution, artifact-free imaging of various caliber and biological samples. Furthermore, we proposed a generic design principle for FLFM, as a highly scalable method to meet broader imaging needs across various spatial levels. We anticipate FLFM to be a particularly powerful tool for imaging diverse phenotypic and functional information, spanning broad molecular, cellular and tissue systems.

Robust Depth Estimation for Light Field Microscopy

Light field technologies have seen a rise in recent years and microscopy is a field where such technology has had a deep impact. The possibility to provide spatial and angular information at the same time and in a single shot brings several advantages and allows for new applications. A common goal in these applications is the calculation of a depth map to reconstruct the three-dimensional geometry of the scene. Many approaches are applicable, but most of them cannot achieve high accuracy because of the nature of such images: biological samples are usually poor in features and do not exhibit sharp colors like natural scene. Due to such conditions, standard approaches result in noisy depth maps. In this work, a robust approach is proposed where accurate depth maps can be produced exploiting the information recorded in the light field, in particular, images produced with Fourier integral Microscope. The proposed approach can be divided into three main parts. Initially, it creates two cost volumes using different focal cues, namely correspondences and defocus. Secondly, it applies filtering methods that exploit multi-scale and super-pixels cost aggregation to reduce noise and enhance the accuracy. Finally, it merges the two cost volumes and extracts a depth map through multi-label optimization.

FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples

In this work, Fourier integral microscope (FIMic), an ultimate design of 3D-integral microscopy, is presented. By placing a multiplexing microlens array at the aperture stop of the microscope objective of the host microscope, FIMic shows extended depth of field and enhanced lateral resolution in comparison with regular integral microscopy. As FIMic directly produces a set of orthographic views of the 3D-micrometer-sized sample, it is suitable for real-time imaging. Following regular integral-imaging reconstruction algorithms, a 2.75-fold enhanced depth of field and [Formula: see text]-time better spatial resolution in comparison with conventional integral microscopy is reported. Our claims are supported by theoretical analysis and experimental images of a resolution test target, cotton fibers, and in-vivo 3D-imaging of biological specimens.

Fast and high-resolution light field acquisition using defocus modulation

In the conventional microlens-array-based light field imaging system, there is a trade-off between the angular and spatial resolutions. Light field reconstruction from images captured by focal plane sweeping, such as light field moment imaging (LFMI) and light field reconstruction with back projection (LFBP), can achieve high transverse resolution comparable to the modern camera sensor. However, the acquisition of a series of focal plane sweeping images along the optical axis is time consuming and requires fine alignment. Furthermore, different focal-plane-based light field reconstruction techniques require images with different characteristics. To solve these problems, we present an efficient approach for fast light field acquisition with precise focal plane sweeping capture by defocus modulation, rather than mechanical movement. Also, we verify the validity and the improvement of our system. With the controllable point spread function, we can capture images for light field reconstruction with both LFMI and LFBP. Otherwise, we quantitatively compare the two methods using images captured with the proposed systems.

Integral imaging techniques for flexible sensing through image-based reprojection

In this work, a 3D reconstruction approach for flexible sensing inspired by integral imaging techniques is proposed. This method allows the application of different integral imaging techniques, such as generating a depth map or the reconstruction of images on a certain 3D plane of the scene that were taken with a set of cameras located at unknown and arbitrary positions and orientations. By means of a photo-consistency measure proposed in this work, all-in-focus images can also be generated by projecting the points of the 3D plane into the sensor planes of the cameras and thereby capturing the associated RGB values. The proposed method obtains consistent results in real scenes with different surfaces of objects as well as changes in texture and lighting.