Fast, volumetric live-cell imaging using high-resolution light-field microscopy

Highly sensitive volumetric single-molecule imaging

Volumetric subcellular imaging has long been essential for studying structures and dynamics in cells and tissues. However, due to limited imaging speed and depth of field, it has been challenging to perform live-cell imaging and single-particle tracking. Here we report a 2.5D fluorescence microscopy combined with highly inclined illumination beams, which significantly reduce not only the image acquisition time but also the out-of-focus background by ∼2-fold compared to epi-illumination. Instead of sequential z-scanning, our method projects a certain depth of volumetric information onto a 2D plane in a single shot using multi-layered glass for incoherent wavefront splitting, enabling high photon detection efficiency. We apply our method to multi-color immunofluorescence imaging and volumetric super-resolution imaging, covering ∼3-4 µm thickness of samples without z-scanning. Additionally, we demonstrate that our approach can substantially extend the observation time of single-particle tracking in living cells.

Multifocal fluorescence video-rate imaging of centimetre-wide arbitrarily shaped brain surfaces at micrometric resolution

Fluorescence microscopy allows for the high-throughput imaging of cellular activity across brain areas in mammals. However, capturing rapid cellular dynamics across the curved cortical surface is challenging, owing to trade-offs in image resolution, speed, field of view and depth of field. Here we report a technique for wide-field fluorescence imaging that leverages selective illumination and the integration of focal areas at different depths via a spinning disc with varying thickness to enable video-rate imaging of previously reconstructed centimetre-scale arbitrarily shaped surfaces at micrometre-scale resolution and at a depth of field of millimetres. By implementing the technique in a microscope capable of acquiring images at 1.68 billion pixels per second and resolving 16.8 billion voxels per second, we recorded neural activities and the trajectories of neutrophils in real time on curved cortical surfaces in live mice. The technique can be integrated into many microscopes and macroscopes, in both reflective and fluorescence modes, for the study of multiscale cellular interactions on arbitrarily shaped surfaces.

A head-mounted photoacoustic fiberscope for hemodynamic imaging in mobile mice

A miniaturized photoacoustic fiberscope has been developed, featuring a lateral resolution of 9 microns and a lightweight design at 4.5 grams. Engineered to capture hemodynamic processes at single-blood-vessel resolution at a rate of 0.2 Hz, this device represents an advancement in head-mounted tools for exploring intricate brain activities in mobile animals.

Bioluminescence as a functional tool for visualizing and controlling neuronal activity in vivo

The use of bioluminescence as a reporter for physiology in neuroscience is as old as the discovery of the calcium-dependent photon emission of aequorin. Over the years, luciferases have been largely replaced by fluorescent reporters, but recently, the field has seen a renaissance of bioluminescent probes, catalyzed by unique developments in imaging technology, bioengineering, and biochemistry to produce luciferases with previously unseen colors and intensity. This is not surprising as the advantages of bioluminescence make luciferases very attractive for noninvasive, longitudinal in vivo observations without the need of an excitation light source. Here, we review how the development of dedicated and specific sensor-luciferases afforded, among others, transcranial imaging of calcium and neurotransmitters, or cellular metabolites and physical quantities such as forces and membrane voltage. Further, the increased versatility and light output of luciferases have paved the way for a new field of functional bioluminescence optogenetics, in which the photon emission of the luciferase is coupled to the gating of a photosensor, e.g., a channelrhodopsin and we review how they have been successfully used to engineer synthetic neuronal connections. Finally, we provide a primer to consider important factors in setting up functional bioluminescence experiments, with a particular focus on the genetic model Caenorhabditis elegans, and discuss the leading challenges that the field needs to overcome to regain a competitive advantage over fluorescence modalities. Together, our paper caters to experienced users of bioluminescence as well as novices who would like to experience the advantages of luciferases in their own hand.

Light-field flow cytometry for high-resolution, volumetric and multiparametric 3D single-cell analysis

Imaging flow cytometry (IFC) combines flow cytometry and fluorescence microscopy to enable high-throughput, multiparametric single-cell analysis with rich spatial details. However, current IFC techniques remain limited in their ability to reveal subcellular information with a high 3D resolution, throughput, sensitivity, and instrumental simplicity. In this study, we introduce a light-field flow cytometer (LFC), an IFC system capable of high-content, single-shot, and multi-color acquisition of up to 5,750 cells per second with a near-diffraction-limited resolution of 400-600 nm in all three dimensions. The LFC system integrates optical, microfluidic, and computational strategies to facilitate the volumetric visualization of various 3D subcellular characteristics through convenient access to commonly used epi-fluorescence platforms. We demonstrate the effectiveness of LFC in assaying, analyzing, and enumerating intricate subcellular morphology, function, and heterogeneity using various phantoms and biological specimens. The advancement offered by the LFC system presents a promising methodological pathway for broad cell biological and translational discoveries, with the potential for widespread adoption in biomedical research.

Graded arc beam in light needle microscopy for axially resolved, rapid volumetric imaging without nonlinear processes

High-speed three-dimensional (3D) imaging is essential for revealing the structure and functions of biological specimens. Confocal laser scanning microscopy has been widely employed for this purpose. However, it requires a time-consuming image-stacking procedure. As a solution, we previously developed light needle microscopy using a Bessel beam with a wavefront-engineered approach [Biomed. Opt. Express13, 1702 (2022)10.1364/BOE.449329]. However, this method applies only to multiphoton excitation microscopy because of the requirement to reduce the sidelobes of the Bessel beam. Here, we introduce a beam that produces a needle spot while eluding the intractable artifacts due to the sidelobes. This beam can be adopted even in one-photon excitation fluorescence 3D imaging. The proposed method can achieve real-time, rapid 3D observation of 200-nm particles in water at a rate of over 50 volumes per second. In addition, fine structures, such as the spines of neurons in fixed mouse brain tissue, can be visualized in 3D from a single raster scan of the needle spot. The proposed method can be applied to various modalities in biological imaging, enabling rapid 3D image acquisition.

High-resolution 3D imaging in light-field microscopy through Stokes matrices and data fusion

The trade-off between the lateral and vertical resolution has long posed challenges to the efficient and widespread application of Fourier light-field microscopy, a highly scalable 3D imaging tool. Although existing methods for resolution enhancement can improve the measurement result to a certain extent, they come with limitations in terms of accuracy and applicable specimen types. To address these problems, this paper proposed a resolution enhancement scheme utilizing data fusion of polarization Stokes vectors and light-field information for Fourier light-field microscopy system. By introducing the surface normal vector information obtained from polarization measurement and integrating it with the light-field 3D point cloud data, 3D reconstruction results accuracy is highly improved in axial direction. Experimental results with a Fourier light-field 3D imaging microscope demonstrated a substantial enhancement of vertical resolution with a depth resolution to depth of field ratio of 0.19%. This represented approximately 44 times the improvement compared to the theoretical ratio before data fusion, enabling the system to access more detailed information with finer measurement accuracy for test samples. This work not only provides a feasible solution for breaking the limitations imposed by traditional light-field microscope hardware configurations but also offers superior 3D measurement approach in a more cost-effective and practical manner.

Highly flexible and compact volumetric endoscope by integrating multiple micro-imaging devices

A light-field endoscope can simultaneously capture the three-dimensional information of in situ lesions and enables single-shot quantitative depth perception with minimal invasion for improving surgical and diagnostic accuracy. However, due to oversized rigid probes, clinical applications of current techniques are limited by their cumbersome devices. To minimize the size and enhance the flexibility, here we report a highly flexible and compact volumetric endoscope by employing precision-machined multiple micro-imaging devices (MIRDs). To further protect the flexibility, the designed MIRD with a diameter and height of 5 mm is packaged in pliable polyamide, using soft data cables for data transmission. It achieves the optimal lateral resolvability of 31 µm and axial resolvability of 255 µm, with an imaging volume over 2.3 × 2.3 × 10 mm3. Our technique allows easy access to the organism interior through the natural entrance, which has been verified through observational experiments of the stomach and rectum of a rabbit. Together, we expect this device can assist in the removal of tumors and polyps as well as the identification of certain early cancers of the digestive tract.

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy

Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

Imagining the future of optical microscopy: everything, everywhere, all at once

The optical microscope has revolutionized biology since at least the 17th Century. Since then, it has progressed from a largely observational tool to a powerful bioanalytical platform. However, realizing its full potential to study live specimens is hindered by a daunting array of technical challenges. Here, we delve into the current state of live imaging to explore the barriers that must be overcome and the possibilities that lie ahead. We venture to envision a future where we can visualize and study everything, everywhere, all at once - from the intricate inner workings of a single cell to the dynamic interplay across entire organisms, and a world where scientists could access the necessary microscopy technologies anywhere.

Sparse deconvolution for background noise suppression with total variation regularization in light field microscopy

In this Letter, we present a method aiming at background noise removal in the 3D reconstruction of light field microscopy (LFM). Sparsity and Hessian regularization are taken as two prior knowledges to process the original light field image before 3D deconvolution. Due to the noise suppression function of total variation (TV) regularization, we add the TV regularization term to the 3D Richardson-Lucy (RL) deconvolution. By comparing the light field reconstruction results of our method with another state-of-the-art method that is also based on RL deconvolution, the proposed method shows improved performance in terms of removing background noise and detail enhancement. This method will be beneficial to the application of LFM in biological high-quality imaging.

Biomedical Optics Express recognizes the best paper prize winners: editorial

The Editor-in-Chief and Deputy Editor of Biomedical Optics Express introduce a new prize for the best paper published in the Journal between 2019 and 2021.

AutoDeconJ: a GPU-accelerated ImageJ plugin for 3D light-field deconvolution with optimal iteration numbers predicting

MOTIVATION

Light-field microscopy (LFM) is a compact solution to high-speed 3D fluorescence imaging. Usually, we need to do 3D deconvolution to the captured raw data. Although there are deep neural network methods that can accelerate the reconstruction process, the model is not universally applicable for all system parameters. Here, we develop AutoDeconJ, a GPU-accelerated ImageJ plugin for 4.4× faster and more accurate deconvolution of LFM data. We further propose an image quality metric for the deconvolution process, aiding in automatically determining the optimal number of iterations with higher reconstruction accuracy and fewer artifacts.

RESULTS

Our proposed method outperforms state-of-the-art light-field deconvolution methods in reconstruction time and optimal iteration numbers prediction capability. It shows better universality of different light-field point spread function (PSF) parameters than the deep learning method. The fast, accurate and general reconstruction performance for different PSF parameters suggests its potential for mass 3D reconstruction of LFM data.

AVAILABILITY AND IMPLEMENTATION

The codes, the documentation and example data are available on an open source at: https://github.com/Onetism/AutoDeconJ.git.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Low-voltage driving high-resistance liquid crystal micro-lens with electrically tunable depth of field for the light field imaging system

Light field imaging (LFI) based on Liquid crystal microlens array (LC MLAs) are emerging as a significant area for 3D imaging technology in the field of upcoming Internet of things and artificial intelligence era. However, in scenes of LFI through conventional MLAs, such as biological imaging and medicine imaging, the quality of imaging reconstruction will be severely reduced due to the limited depth of field. Here, we are proposed a low-voltage driving LC MLAs with electrically tunable depth of field (DOF) for the LFI system. An aluminum-doped zinc oxide (AZO) film was deposited on the top of the hole-patterned driven-electrode arrays and used as a high resistance (Hi-R) layer, a uniform gradient electric field was obtained across the sandwiched LC cell. Experimental results confirm that the proposed LC MLAs possess high-quality interference rings and tunable focal length at a lower working voltage. In addition, the focal lengths are tunable from 3.93 to 2.62 mm and the DOF are adjustable from 15.60 to 1.23 mm. The experiments demonstrated that the LFI system based on the proposed structure can clearly capture 3D information of the insets with enlarged depths by changing the working voltage and driving frequency, which indicates that the tunable DOF LC MLAs have a potential application prospects for the biological and medical imaging.

Background inhibited and speed-loss-free volumetric imaging in vivo based on structured-illumination Fourier light field microscopy

Benefiting from its advantages in fast volumetric imaging for recording biodynamics, Fourier light field microscopy (FLFM) has a wide range of applications in biomedical research, especially in neuroscience. However, the imaging quality of the FLFM is always deteriorated by both the out-of-focus background and the strong scattering in biological samples. Here we propose a structured-illumination and interleaved-reconstruction based Fourier light field microscopy (SI-FLFM), in which we can filter out the background fluorescence in FLFM without sacrificing imaging speed. We demonstrate the superiority of our SI-FLFM in high-speed, background-inhibited volumetric imaging of various biodynamics in larval zebrafish and mice in vivo. The signal-to-background ratio (SBR) is improved by tens of times. And the volumetric imaging speed can be up to 40 Hz, avoiding artifacts caused by temporal under-sampling in conventional structured illumination microscopy. These suggest that our SI-FLFM is suitable for applications of weak fluorescence signals but high imaging speed requirements.

Snapshot volumetric imaging with engineered point-spread functions

The biological world involves intracellular and intercellular interactions that occur at high speed, at multiple scales and in three dimensions. Acquiring 3D images, however, typically requires a compromise in either spatial or temporal resolution compared to 2D imaging. Conventional 2D fluorescence imaging provides high spatial resolution but requires plane-by-plane imaging of volumes. Conversely, snapshot methods such as light-field microscopy allow video-rate imaging, but at the cost of spatial resolution. Here we introduce 3D engineered point-spread function microscopy (3D-EPM), enabling snapshot imaging of real-world 3D extended biological structures while retaining the native resolution of the microscope in space and time. Our new computational recovery strategy is the key to volumetrically reconstructing arbitrary 3D structures from the information encapsulated in 2D raw EPM images. We validate our technique on both point-like and extended samples, and demonstrate its power by imaging the intracellular motion of chloroplasts undergoing cyclosis in a sample of Egeria densa. Our technique represents a generalised computational methodology for 3D image recovery which is readily adapted to a diverse range of existing microscopy platforms and engineered point-spread functions. We therefore expect it to find broad applicability in the study of rapid biological dynamics in 3D.

GANscan: continuous scanning microscopy using deep learning deblurring

Most whole slide imaging (WSI) systems today rely on the "stop-and-stare" approach, where, at each field of view, the scanning stage is brought to a complete stop before the camera snaps a picture. This procedure ensures that each image is free of motion blur, which comes at the expense of long acquisition times. In order to speed up the acquisition process, especially for large scanning areas, such as pathology slides, we developed an acquisition method in which the data is acquired continuously while the stage is moving at high speeds. Using generative adversarial networks (GANs), we demonstrate this ultra-fast imaging approach, referred to as GANscan, which restores sharp images from motion blurred videos. GANscan allows us to complete image acquisitions at 30x the throughput of stop-and-stare systems. This method is implemented on a Zeiss Axio Observer Z1 microscope, requires no specialized hardware, and accomplishes successful reconstructions at stage speeds of up to 5000 μm/s. We validate the proposed method by imaging H&E stained tissue sections. Our method not only retrieves crisp images from fast, continuous scans, but also adjusts for defocusing that occurs during scanning within +/- 5 μm. Using a consumer GPU, the inference runs at <20 ms/ image.

Single-Shot Light-Field Microscopy: An Emerging Tool for 3D Biomedical Imaging

3D microscopy is a useful tool to visualize the detailed structures and mechanisms of biomedical specimens. In particular, biophysical phenomena such as neural activity require fast 3D volumetric imaging because fluorescence signals degrade quickly. A light-field microscope (LFM) has recently attracted attention as a high-speed volumetric imaging technique by recording 3D information in a single-snapshot. This review highlighted recent progress in LFM techniques for 3D biomedical applications. In detail, various image reconstruction algorithms according to LFM configurations are explained, and several biomedical applications such as neuron activity localization, live-cell imaging, locomotion analysis, and single-molecule visualization are introduced. We also discuss deep learning-based LFMs to enhance image resolution and reduce reconstruction artifacts.

Fourier light-field imaging of human organoids with a hybrid point-spread function

Volumetric interrogation of the cellular morphology and dynamic processes of organoid systems with a high spatiotemporal resolution provides critical insights for understanding organogenesis, tissue homeostasis, and organ function. Fluorescence microscopy has emerged as one of the most vital and informative driving forces for probing the cellular complexity in organoid research. However, the underlying scanning mechanism of conventional imaging methods inevitably compromises the time resolution of volumetric acquisition, leading to increased photodamage and inability to capture fast cellular and tissue dynamic processes. Here, we report Fourier light-field microscopy using a hybrid point-spread function (hPSF-FLFM) for fast, volumetric, and high-resolution imaging of entire organoids. hPSF-FLFM transforms conventional 3D microscopy and enables exploration of less accessible spatiotemporally-challenging regimes for organoid research. To validate hPSF-FLFM, we demonstrate 3D imaging of rapid responses to extracellular physical cues such as osmotic and mechanical stresses on human induced pluripotent stem cells-derived colon organoids (hCOs). The system offers cellular (2-3 μm and 5-6 μm in x-y and z, respectively) and millisecond-scale spatiotemporal characterization of whole-organoid dynamic changes that span large imaging volumes (>900 μm × 900 μm × 200 μm in x, y, z, respectively). The hPSF-FLFM method provides a promising avenue to explore spatiotemporal-challenging cellular responses in a wide variety of organoid research.

Wavefront engineered light needle microscopy for axially resolved rapid volumetric imaging

Increasing the acquisition speed of three-dimensional volumetric images is important-particularly in biological imaging-to unveil the structural dynamics and functionalities of specimens in detail. In conventional laser scanning fluorescence microscopy, volumetric images are constructed from optical sectioning images sequentially acquired by changing the observation plane, limiting the acquisition speed. Here, we present a novel method to realize volumetric imaging from two-dimensional raster scanning of a light needle spot without sectioning, even in the traditional framework of laser scanning microscopy. Information from multiple axial planes is simultaneously captured using wavefront engineering for fluorescence signals, allowing us to readily survey the entire depth range while maintaining spatial resolution. This technique is applied to real-time and video-rate three-dimensional tracking of micrometer-sized particles, as well as the prompt visualization of thick fixed biological specimens, offering substantially faster volumetric imaging.

Binocular stereo-microscopy for deforming intact amoeba

A powerful and convenient method for measuring three-dimensional (3D) deformation of moving amoeboid cells will assist the progress of environmental and cytological studies as protists amoebae play a role in the fundamental environmental ecosystem. Here we develop an inexpensive and useful method for measuring 3D deformation of single protists amoeba through binocular microscopy and a newly proposed algorithm of stereo-scopy. From the movies taken from the left and right optical tubes of the binocular microscope, we detect the 3D positions of many intrinsic intracellular vesicles and reconstruct cellular surfaces of amoeboid cells in 3D space. Some observations of sampled behaviors are shown in a single-celled organism of Amoeba proteus. The resultant surface time series is then analyzed to obtain surface velocity, curvature and volume increasing rates of pseudo-pods for characterizing the movements of amoeboid cells. The limitations and errors of this method are also discussed.

Computational optical sectioning with an incoherent multiscale scattering model for light-field microscopy

Quantitative volumetric fluorescence imaging at high speed across a long term is vital to understand various cellular and subcellular behaviors in living organisms. Light-field microscopy provides a compact computational solution by imaging the entire volume in a tomographic way, while facing severe degradation in scattering tissue or densely-labelled samples. To address this problem, we propose an incoherent multiscale scattering model in a complete space for quantitative 3D reconstruction in complicated environments, which is called computational optical sectioning. Without the requirement of any hardware modifications, our method can be generally applied to different light-field schemes with reduction in background fluorescence, reconstruction artifacts, and computational costs, facilitating more practical applications of LFM in a broad community. We validate the superior performance by imaging various biological dynamics in Drosophila embryos, zebrafish larvae, and mice.

2.5D microscopy with polarization independent SLM for enhanced detection efficiency and aberration correction

Fast, volumetric imaging by fluorescence microscopy is essential in studying biological phenomena and cellular functions. Recently, single-shot 2.5D microscopy showed promising results for high-throughput quantitative subcellular analysis via extended depth of field imaging without sequential z-scanning; however, the detection efficiency was limited and it lacked depth-induced aberration correction. Here we report that a spatial light modulator (SLM) in a polarization insensitive configuration can significantly improve the detection efficiency of 2.5D microscopy, while also compensating for aberrations at large imaging depths caused by the refractive index mismatch between the sample and the immersion medium. We highlight the improved efficiency via quantitative single-molecule RNA imaging of mammalian cells with a 2-fold improvement in the fluorescence intensity compared to a conventional SLM-based microscopy. We demonstrate the aberration correction capabilities and extended depth of field by imaging thick specimens with fewer z-scanning steps.

GEOMScope: Large Field-of-view 3D Lensless Microscopy with Low Computational Complexity

Imaging systems with miniaturized device footprint, real-time processing speed and high resolution three-dimensional (3D) visualization are critical to broad biomedical applications such as endoscopy. Most of existing imaging systems rely on bulky lenses and mechanically refocusing to perform 3D imaging. Here, we demonstrate GEOMScope, a lensless single-shot 3D microscope that forms image through a single layer of thin microlens array and reconstructs objects through an innovative algorithm combining geometrical-optics-based pixel back projection and background suppressions. We verify the effectiveness of GEOMScope on resolution target, fluorescent particles and volumetric objects. Comparing to other widefield lensless imaging devices, we significantly reduce the required computational resource and increase the reconstruction speed by orders of magnitude. This enables us to image and recover large volume 3D object in high resolution with near real-time processing speed. Such a low computational complexity is attributed to the joint design of imaging optics and reconstruction algorithms, and a joint application of geometrical optics and machine learning in the 3D reconstruction. More broadly, the excellent performance of GEOMScope in imaging resolution, volume, and reconstruction speed implicates that geometrical optics could greatly benefit and play an important role in computational imaging.

PlenoptiCam v1.0: A Light-Field Imaging Framework

Light-field cameras play a vital role for rich 3D information retrieval in narrow range depth sensing applications. The key obstacle in composing light-fields from exposures taken by a plenoptic camera is to computationally calibrate, align and rearrange four-dimensional image data. Several attempts have been proposed to enhance the overall image quality by tailoring pipelines dedicated to particular plenoptic cameras and improving the consistency across viewpoints at the expense of high computational loads. The framework presented herein advances prior outcomes thanks to its novel micro image scale-space analysis for generic camera calibration independent of the lens specifications and its parallax-invariant, cost-effective viewpoint color equalization from optimal transport theory. Artifacts from the sensor and micro lens grid are compensated in an innovative way to enable superior quality in sub-aperture image extraction, computational refocusing and Scheimpflug rendering with sub-sampling capabilities. Benchmark comparisons using established image metrics suggest that our proposed pipeline outperforms state-of-the-art tool chains in the majority of cases. Results from a Wasserstein distance further show that our color transfer outdoes the existing transport methods. Our algorithms are released under an open-source license, offer cross-platform compatibility with few dependencies and different user interfaces. This makes the reproduction of results and experimentation with plenoptic camera technology convenient for peer researchers, developers, photographers, data scientists and others working in this field.

DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning

Light field microscopy (LFM) has been widely used for recording 3D biological dynamics at camera frame rate. However, LFM suffers from artifact contaminations due to the illness of the reconstruction problem via naïve Richardson-Lucy (RL) deconvolution. Moreover, the performance of LFM significantly dropped in low-light conditions due to the absence of sample priors. In this paper, we thoroughly analyze different kinds of artifacts and present a new LFM technique termed dictionary LFM (DiLFM) that substantially suppresses various kinds of reconstruction artifacts and improves the noise robustness with an over-complete dictionary. We demonstrate artifact-suppressed reconstructions in scattering samples such as Drosophila embryos and brains. Furthermore, we show our DiLFM can achieve robust blood cell counting in noisy conditions by imaging blood cell dynamic at 100 Hz and unveil more neurons in whole-brain calcium recording of zebrafish with low illumination power in vivo.

High-resolution light-field microscopy with patterned illumination

Light-field fluorescence microscopy can record large-scale population activity of neurons expressing genetically-encoded fluorescent indicators within volumes of tissue. Conventional light-field microscopy (LFM) suffers from poor lateral resolution when using wide-field illumination. Here, we demonstrate a structured-illumination light-field microscopy (SI-LFM) modality that enhances spatial resolution over the imaging volume. This modality increases resolution by illuminating sample volume with grating patterns that are invariant over the axial direction. The size of the SI-LFM point-spread-function (PSF) was approximately half the size of the conventional LFM PSF when imaging fluorescent beads. SI-LFM also resolved fine spatial features in lens tissue samples and fixed mouse retina samples. Finally, SI-LFM reported neural activity with approximately three times the signal-to-noise ratio of conventional LFM when imaging live zebrafish expressing a genetically encoded calcium sensor.

High-resolution Fourier light-field microscopy for volumetric multi-color live-cell imaging

Volumetric interrogation of the organization and processes of intracellular organelles and molecules in cellular systems with a high spatiotemporal resolution is essential for understanding cell physiology, development, and pathology. Here, we report high-resolution Fourier light-field microscopy (HR-FLFM) for fast and volumetric live-cell imaging. HR-FLFM transforms conventional cell microscopy and enables exploration of less accessible spatiotemporal-limiting regimes for single-cell studies. The results present a near-diffraction-limited resolution in all three dimensions, a five-fold extended focal depth to several micrometers, and a scanning-free volume acquisition time up to milliseconds. The system demonstrates instrumentation accessibility, low photo damage for continuous observation, and high compatibility with general cell assays. We anticipate HR-FLFM to offer a promising methodological pathway for investigating a wide range of intracellular processes and functions with exquisite spatiotemporal contextual details.

Deep learning-enhanced light-field imaging with continuous validation

Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence-enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.

Recurrent neural network-based volumetric fluorescence microscopy

Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments on C. elegans and nanobead samples, Recurrent-MZ is demonstrated to significantly increase the depth-of-field of a 63×/1.4NA objective lens, also providing a 30-fold reduction in the number of axial scans required to image the same sample volume. We further illustrated the generalization of this recurrent network for 3D imaging by showing its resilience to varying imaging conditions, including e.g., different sequences of input images, covering various axial permutations and unknown axial positioning errors. We also demonstrated wide-field to confocal cross-modality image transformations using Recurrent-MZ framework and performed 3D image reconstruction of a sample using a few wide-field 2D fluorescence images as input, matching confocal microscopy images of the same sample volume. Recurrent-MZ demonstrates the first application of recurrent neural networks in microscopic image reconstruction and provides a flexible and rapid volumetric imaging framework, overcoming the limitations of current 3D scanning microscopy tools.

Light-field microscopy for fast volumetric brain imaging

Recording neural activities over large populations is critical for a better understanding of the functional mechanisms of animal brains. Traditional optical imaging technologies for in vivo neural activity recording are usually limited in throughput and cannot cover a large imaging volume at high speed. Light-field microscopy features a highly parallelized imaging collection mechanism and can simultaneously record optical signals from different depths. Therefore, it can potentially increase the imaging throughput substantially. Furthermore, its unique instantaneous volumetric imaging capability enables the capture of highly dynamic processes, such as recording whole-animal neural activities in freely moving Caenorhabditis elegans and whole-brain neural activity in freely swimming larval zebrafish during prey capture. Here, we summarize the principles of and considerations in the practical implementation of light-field microscopy as currently applied in biological imaging experiments. We also discuss the strategies that light-field microscopy can employ when imaging thick tissues in the presence of scattering and background interference. Finally, we present a few examples of applying light-field microscopy in neuroscientific studies in several important animal models.

Six-pack holographic imaging for dynamic rejection of out-of-focus objects

Six-pack holography is adapted to reject out-of-focus objects in dynamic samples, using a single camera exposure and without any scanning. By illuminating the sample from six different angles in parallel using a low-coherence source, out-of-focus objects are laterally shifted in six different directions when projected onto the focal plane. Then pixel-wise averaging of the six reconstructed images creates a significantly clearer image, with rejection of out-of-focus objects. Dynamic imaging results are shown for swimming microalgae and flowing microbeads, including numerical refocusing by Fresnel propagation. The averaged images reduced the contribution of out-of-focus objects by up to 83% in comparison to standard holograms captured using the same light source, further improving the system sectioning capabilities. Both simulation and experimental results are presented.

wFLFM: enhancing the resolution of Fourier light-field microscopy using a hybrid wide-field image

We introduce wFLFM, an approach that enhances the resolution of Fourier light-field microscopy (FLFM) through a hybrid wide-field image. The system exploits the intrinsic compatibility of image formation between the on-axis FLFM elemental image and the wide-field image, allowing for minimal instrumental and computational complexity. The numerical and experimental results of wFLFM present a two- to three-fold improvement in the lateral resolution without compromising the 3D imaging capability in comparison with conventional FLFM.

Removal of ghost particles from the reconstruction of dusty plasma in integral photography by three-dimensional deconvolution

The integral photography and deconvolution techniques have been applied to identify the three-dimensional (3D) positions of particles levitating in plasma. Artifacts in the light field, i.e. ghost particles, are removed by collating between results of integral photography and direct Richardson-Lucy deconvolution (RLD). Our reconstruction system is tested with known target particles and it is found that it works well in the range of our dust experiment. By applying the integral photography and RLD techniques to the obtained experimental image, we identified the 3D positions of dust particles floating in a radio-frequency plasma. Ghost particles are eliminated from the results by deconvolution and we succeeded in obtaining the 3D structure of a dusty plasma from a single-exposure image obtained from one view port.

Modularized microscope based on parallel phase-shifting digital holography for imaging of living biospecimens

SIGNIFICANCE

Parallel phase-shifting digital holographic microscope (PPSDHM) is powerful for three-dimensional (3D) measurements of dynamic specimens. However, the PPSDHM reported previously was directly fixed on the optical bench and imposed difficulties case, thus it is required to modify the specification of the microscope or transport the microscope to another location.

AIM

We present a modularized PPSDHM. We construct the proposed PPSDHM and demonstrate the 3D measurement capability of the PPSDHM.

APPROACH

The PPSDHM was designed as an inverted microscope to record transparent objects and modularized by integrating the optical elements of the PPSDHM on an optical breadboard. To demonstrate the effectiveness of the PPSDHM, we recorded a 3D motion-picture of moving Volvoxes at 1000  frames  /  s and carried out 3D tracking of the Volvoxes.

RESULTS

The PPSDHM was practically realized and 3D images of objects were successfully reconstructed from holograms recorded with a single-shot exposure. The 3D trajectories of Volvoxes were obtained from the reconstructed images.

CONCLUSIONS

We established a modularized PPSDHM that is capable of 3D image acquisition by integrating the optical elements of the PPSDHM on an optical breadboard. The recording capability of 3D motion-pictures of dynamic specimens was experimentally demonstrated by the PPSDHM.

Design, characterization, and intracellular trafficking of biofunctionalized chitosan nanomicelles

The hydrophobically modified glycol chitosan (HGC) nanomicelle has received increasing attention as a promising platform for the delivery of chemotherapeutic drugs. To improve the tumor selectivity of HGC, here an avidin and biotin functionalization strategy was applied. The hydrodynamic diameter of the biotin-avidin-functionalized HGC (cy5.5-HGC-B4F) was observed to be 104.7 nm, and the surface charge was +3.1 mV. Confocal and structured illumination microscopy showed that at 0.1 mg/ml, cy5.5-HGC-B4F nanomicelles were distributed throughout the cytoplasm of MDA-MB-231 breast cancer cells after 2 h of exposure without significant cytotoxicity. To better understand the intracellular fate of the nanomicelles, entrapment studies were performed and demonstrated that some cy5.5-HGC-B4F nanomicelles were capable of escaping endocytic vesicles, likely via the proton sponge effect. Quantitative analysis of the movements of endosomes in living cells revealed that the addition of HGC greatly enhanced the motility of endosomal compartments, and the nanomicelles were transported by early and late endosomes from cell periphery to the perinuclear region. Our results validate the importance of using live-cell imaging to quantitatively assess the dynamics and mechanisms underlying the complex endocytic pathways of nanosized drug carriers.

An experimental method to characterize the relationship between aperture image and ray directions in microscope optics

We propose a direct experimental method to calibrate the relationship between ray directions in object space and their positions in the aperture plane of a light field microscope. The calibration improves the interpretation of light field images, which contain information from both types of image planes, the field plane and the aperture plane of the ray path in the microscope. Our method is based on the diffraction of line gratings of known periodicities and provides accurate results with subpixel resolution. The method can be custom-tailored to most any optical configuration, including standard light microscopy setups, whenever correct mapping between ray parameters in the object/image plane and the aperture plane is needed.

High-resolution 3D light-field imaging

SIGNIFICANCE

High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.

AIM

We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.

APPROACH

The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.

RESULTS

High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of 250  ×  250  ×  80  μm3 in the x, y, and z axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.

CONCLUSIONS

The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.

Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser

Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plane to encode 3D information, then computationally reconstructs the volume by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths; the random positions provide a larger FOV compared to a conventional MLA, and the diverse focal lengths improve the axial depth range. To predict system performance based on diffuser parameters, we, for the first time, establish a theoretical framework and design guidelines, which are verified by numerical simulations, and then build an experimental system that achieves < 3 µm lateral and 4 µm axial resolution over a 1000 × 1000 × 280 µm³ volume. Our diffuser design outperforms the MLA used in LFM, providing more uniform resolution over a larger volume, both laterally and axially.

Subcellular resolution three-dimensional light-field imaging with genetically encoded voltage indicators

Significance: Light-field microscopy (LFM) enables high signal-to-noise ratio (SNR) and light efficient volume imaging at fast frame rates. Voltage imaging with genetically encoded voltage indicators (GEVIs) stands to particularly benefit from LFM's volumetric imaging capability due to high required sampling rates and limited probe brightness and functional sensitivity. Aim: We demonstrate subcellular resolution GEVI light-field imaging in acute mouse brain slices resolving dendritic voltage signals in three spatial dimensions. Approach: We imaged action potential-induced fluorescence transients in mouse brain slices sparsely expressing the GEVI VSFP-Butterfly 1.2 in wide-field microscopy (WFM) and LFM modes. We compared functional signal SNR and localization between different LFM reconstruction approaches and between LFM and WFM. Results: LFM enabled three-dimensional (3-D) localization of action potential-induced fluorescence transients in neuronal somata and dendrites. Nonregularized deconvolution decreased SNR with increased iteration number compared to synthetic refocusing but increased axial and lateral signal localization. SNR was unaffected for LFM compared to WFM. Conclusions: LFM enables 3-D localization of fluorescence transients, therefore eliminating the need for structures to lie in a single focal plane. These results demonstrate LFM's potential for studying dendritic integration and action potential propagation in three spatial dimensions.

Diffuser-based computational imaging funduscope

Poor access to eye care is a major global challenge that could be ameliorated by low-cost, portable, and easy-to-use diagnostic technologies. Diffuser-based imaging has the potential to enable inexpensive, compact optical systems that can reconstruct a focused image of an object over a range of defocus errors. Here, we present a diffuser-based computational funduscope that reconstructs important clinical features of a model eye. Compared to existing diffuser-imager architectures, our system features an infinite-conjugate design by relaying the ocular lens onto the diffuser. This offers shift-invariance across a wide field-of-view (FOV) and an invariant magnification across an extended depth range. Experimentally, we demonstrate fundus image reconstruction over a 33° FOV and robustness to ±4D refractive error using a constant point-spread-function. Combined with diffuser-based wavefront sensing, this technology could enable combined ocular aberrometry and funduscopic screening through a single diffuser sensor.

What about computational super-resolution in fluorescence Fourier light field microscopy?

Recently, Fourier light field microscopy was proposed to overcome the limitations in conventional light field microscopy by placing a micro-lens array at the aperture stop of the microscope objective instead of the image plane. In this way, a collection of orthographic views from different perspectives are directly captured. When inspecting fluorescent samples, the sensitivity and noise of the sensors are a major concern and large sensor pixels are required to cope with low-light conditions, which implies under-sampling issues. In this context, we analyze the sampling patterns in Fourier light field microscopy to understand to what extent computational super-resolution can be triggered during deconvolution in order to improve the resolution of the 3D reconstruction of the imaged data.

Design of a high-resolution light field miniscope for volumetric imaging in scattering tissue

Integrating light field microscopy techniques with existing miniscope architectures has allowed for volumetric imaging of targeted brain regions in freely moving animals. However, the current design of light field miniscopes is limited by non-uniform resolution and long imaging path length. In an effort to overcome these limitations, this paper proposes an optimized Galilean-mode light field miniscope (Gali-MiniLFM), which achieves a more consistent resolution and a significantly shorter imaging path than its conventional counterparts. In addition, this paper provides a novel framework that incorporates the anticipated aberrations of the proposed Gali-MiniLFM into the point spread function (PSF) modeling. This more accurate PSF model can then be used in 3D reconstruction algorithms to further improve the resolution of the platform. Volumetric imaging in the brain necessitates the consideration of the effects of scattering. We conduct Monte Carlo simulations to demonstrate the robustness of the proposed Gali-MiniLFM for volumetric imaging in scattering tissue.

Geometry parameter calibration for focused plenoptic cameras

Due to the subtle structure, the exact geometry parameters of the focused plenoptic camera cannot be retrieved after packaging, which leads to inaccurate light field processing such as visible artifacts in the rendering images. This paper proposes a novel blind calibration method to calculate the geometry parameters for the focused plenoptic cameras with high precision. It translates the problem of deriving the value of the geometry parameters to be the problem of deriving the pixel patch-size of each micro-image used in subaperture image rendering based on the geometry projection of the relay imaging process in the focused plenoptic camera. Then, a dark image calibration algorithm is proposed to retrieve the position and the geometry parameters of the MLA for subaperture image rendering. A triple-level calibration board with random texture is designed to realize focus plane confirming blindly, to facilitate capturing light field images at different object distances via a single shot and to benefit intensity feature matching in determining the rendering patch size. The rendering patch-size is found by the proposed Gradient-SSIM-based fractional-pixel matching based on the geometry projection analysis. Experiments conducted on the simulated data and the real imaging system demonstrate that the proposed method can acquire the geometry parameters with high accuracy and is robust to different focused plenoptic cameras.

Fast and accurate sCMOS noise correction for fluorescence microscopy

The rapid development of scientific CMOS (sCMOS) technology has greatly advanced optical microscopy for biomedical research with superior sensitivity, resolution, field-of-view, and frame rates. However, for sCMOS sensors, the parallel charge-voltage conversion and different responsivity at each pixel induces extra readout and pattern noise compared to charge-coupled devices (CCD) and electron-multiplying CCD (EM-CCD) sensors. This can produce artifacts, deteriorate imaging capability, and hinder quantification of fluorescent signals, thereby compromising strategies to reduce photo-damage to live samples. Here, we propose a content-adaptive algorithm for the automatic correction of sCMOS-related noise (ACsN) for fluorescence microscopy. ACsN combines camera physics and layered sparse filtering to significantly reduce the most relevant noise sources in a sCMOS sensor while preserving the fine details of the signal. The method improves the camera performance, enabling fast, low-light and quantitative optical microscopy with video-rate denoising for a broad range of imaging conditions and modalities.

Scientific complementary metal-oxide semiconductor (sCMOS) cameras have advanced the imaging field, but they often suffer from additional noise compared to CCD sensors. Here the authors present a content-adaptive algorithm for the automatic correction of sCMOS-related noise for fluorescence microscopy.

Artifact-free deconvolution in light field microscopy

The sampling patterns of the light field microscope (LFM) are highly depth-dependent, which implies non-uniform recoverable lateral resolution across depth. Moreover, reconstructions using state-of-the-art approaches suffer from strong artifacts at axial ranges, where the LFM samples the light field at a coarse rate. In this work, we analyze the sampling patterns of the LFM, and introduce a flexible light field point spread function model (LFPSF) to cope with arbitrary LFM designs. We then propose a novel aliasing-aware deconvolution scheme to address the sampling artifacts. We demonstrate the high potential of the proposed method on real experimental data.

Fourier light-field microscopy

Observing the various anatomical and functional information that spans many spatiotemporal scales with high resolution provides deep understandings of the fundamentals of biological systems. Light-field microscopy (LFM) has recently emerged as a scanning-free, scalable method that allows for high-speed, volumetric imaging ranging from single-cell specimens to the mammalian brain. However, the prohibitive reconstruction artifacts and severe computational cost have thus far limited broader applications of LFM. To address the challenge, in this work, we report Fourier LFM (FLFM), a system that processes the light-field information through the Fourier domain. We established a complete theoretical and algorithmic framework that describes light propagation, image formation and system characterization of FLFM. Compared with conventional LFM, FLFM fundamentally mitigates the artifacts, allowing high-resolution imaging across a two- to three-fold extended depth. In addition, the system substantially reduces the reconstruction time by roughly two orders of magnitude. FLFM was validated by high-resolution, artifact-free imaging of various caliber and biological samples. Furthermore, we proposed a generic design principle for FLFM, as a highly scalable method to meet broader imaging needs across various spatial levels. We anticipate FLFM to be a particularly powerful tool for imaging diverse phenotypic and functional information, spanning broad molecular, cellular and tissue systems.

Phase-space deconvolution for light field microscopy

Light field microscopy, featuring with snapshot large-scale three-dimensional (3D) fluorescence imaging, has aroused great interests in various biological applications, especially for high-speed 3D calcium imaging. Traditional 3D deconvolution algorithms based on the beam propagation model facilitate high-resolution 3D reconstructions. However, such a high-precision model is not robust enough for the experimental data with different system errors such as optical aberrations and background fluorescence, which bring great periodic artifacts and reduce the image contrast. In order to solve this problem, here we propose a phase-space deconvolution method for light field microscopy, which fully exploits the smoothness prior in the phase-space domain. By modeling the imaging process in the phase-space domain, we convert the spatially-nonuniform point spread function (PSF) into a spatially-uniform one with a much smaller size. Experiments on various biological samples and resolution charts are demonstrated to verify the contrast enhancement with much fewer artifacts and 10-times less computational cost by our method without any hardware modifications required.