Deep-3D microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning

Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network

Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacerbates the condition of the inverse problem. Here, we develop a scattering simulator that models low-contrast target signals buried in heterogeneous strong background. We then train a deep neural network solely on synthetic data to descatter and reconstruct a 3D volume from a single-shot light-field measurement with low signal-to-background ratio (SBR). We apply this network to our previously developed computational miniature mesoscope and demonstrate the robustness of our deep learning algorithm on scattering phantoms with different scattering conditions. The network can robustly reconstruct emitters in 3D with a 2D measurement of SBR as low as 1.05 and as deep as a scattering length. We analyze fundamental tradeoffs based on network design factors and out-of-distribution data that affect the deep learning model's generalizability to real experimental data. Broadly, we believe that our simulator-based deep learning approach can be applied to a wide range of imaging through scattering techniques where experimental paired training data is lacking.

Optical tomography in a single camera frame using fringe-encoded deep-learning full-field OCT

Optical coherence tomography is a valuable tool for in vivo examination thanks to its superior combination of axial resolution, field-of-view and working distance. OCT images are reconstructed from several phases that are obtained by modulation/multiplexing of light wavelength or optical path. This paper shows that only one phase (and one camera frame) is sufficient for en face tomography. The idea is to encode a high-frequency fringe patterns into the selected layer of the sample using low-coherence interferometry. These patterns can then be efficiently extracted with a high-pass filter enhanced via deep learning networks to create the tomographic full-field OCT view. This brings 10-fold improvement in imaging speed, considerably reducing the phase errors and incoherent light artifacts related to in vivo movements. Moreover, this work opens a path for low-cost tomography with slow consumer cameras. Optically, the device resembles the conventional time-domain full-field OCT without incurring additional costs or a field-of-view/resolution reduction. The approach is validated by imaging in vivo cornea in human subjects. Open-source and easy-to-follow codes for data generation/training/inference with U-Net/Pix2Pix networks are provided to be used in a variety of image-to-image translation tasks.

Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network

Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacerbates the condition of the inverse problem. Here, we develop a scattering simulator that models low-contrast target signals buried in heterogeneous strong background. We then train a deep neural network solely on synthetic data to descatter and reconstruct a 3D volume from a single-shot light-field measurement with low signal-to-background ratio (SBR). We apply this network to our previously developed Computational Miniature Mesoscope and demonstrate the robustness of our deep learning algorithm on scattering phantoms with different scattering conditions. The network can robustly reconstruct emitters in 3D with a 2D measurement of SBR as low as 1.05 and as deep as a scattering length. We analyze fundamental tradeoffs based on network design factors and out-of-distribution data that affect the deep learning model's generalizability to real experimental data. Broadly, we believe that our simulator-based deep learning approach can be applied to a wide range of imaging through scattering techniques where experimental paired training data is lacking.

A novel multi-frame wavelet generative adversarial network for scattering reconstruction of structured illumination microscopy

Objective. Structured illumination microscopy (SIM) is widely used in various fields of life science research. In clinical practice, it has low phototoxicity, fast imaging speed and no special fluorescent markers. However, SIM is still affected by the scattering medium of biological tissues, resulting in insufficient resolution of the obtained images, which limits the development of life sciences. A novel multi-frame wavelet generation adversarial network (MWGAN) is proposed to improve the scattering reconstruction capability of SIM.Approach. MWGAN is based on two components derived from the original image. A generative adversarial network constructed by wavelet transform is trained to reconstruct some complex details in the cell structure. Multi-frame adversarial network is used to obtain the inter-frame information of the image and use the complementary information of the before and after frames to improve the quality of the model reconstruction.Results. To demonstrate the robustness of MWGAN, multiple low-quality SIM image datasets are tested. Compared with the state-of-the-art methods, the proposed method achieves superior performance in both of the subjective and objective evaluation.Conclusion. MWGAN is effective for improving the clarity of SIM images. Meanwhile, the SIM images reconstructed by multiple frames improve the reconstruction quality of complex regions and allow clearer and dynamic observation of cellular functions.

Experimental Validation of Shifted Position-Diffuse Reflectance Imaging (SP-DRI) on Optical Phantoms

Numerous diseases such as hemorrhage, sepsis or cardiogenic shock induce a heterogeneous perfusion of the capillaries. To detect such alterations in the human blood flow pattern, diagnostic devices must provide an appropriately high spatial resolution. Shifted position-diffuse reflectance imaging (SP-DRI) has the potential to do so; it is an all-optical diagnostic technique. So far, SP-DRI has mainly been developed using Monte Carlo simulations. The present study is therefore validating this algorithm experimentally on realistic optical phantoms with thread structures down to 10 μm in diameter; a SP-DRI sensor prototype was developed and realized by means of additive manufacturing. SP-DRI turned out to be functional within this experimental framework. The position of the structures within the optical phantoms become clearly visible using SP-DRI, and the structure thickness is reflected as modulation in the SP-DRI signal amplitude; this performed well for a shift along the x axis as well as along the y axis. Moreover, SP-DRI successfully masked the pronounced influence of the illumination cone on the data. The algorithm showed significantly superior to a mere raw data inspection. Within the scope of the study, the constructive design of the SP-DRI sensor prototype is discussed and potential for improvement is explored.

Volumetric imaging of fast cellular dynamics with deep learning enhanced bioluminescence microscopy

Bioluminescence microscopy is an appealing alternative to fluorescence microscopy, because it does not depend on external illumination, and consequently does neither produce spurious background autofluorescence, nor perturb intrinsically photosensitive processes in living cells and animals. The low photon emission of known luciferases, however, demands long exposure times that are prohibitive for imaging fast biological dynamics. To increase the versatility of bioluminescence microscopy, we present an improved low-light microscope in combination with deep learning methods to image extremely photon-starved samples enabling subsecond exposures for timelapse and volumetric imaging. We apply our method to image subcellular dynamics in mouse embryonic stem cells, epithelial morphology during zebrafish development, and DAF-16 FoxO transcription factor shuttling from the cytoplasm to the nucleus under external stress. Finally, we concatenate neural networks for denoising and light-field deconvolution to resolve intracellular calcium dynamics in three dimensions of freely moving Caenorhabditis elegans.