Deep learning optical-sectioning method

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.

Fast, multicolour optical sectioning over extended fields of view with patterned illumination and machine learning

Structured illumination can reject out-of-focus signal from a sample, enabling high-speed and high-contrast imaging over large areas with widefield detection optics. However, this optical sectioning technique is currently limited by image reconstruction artefacts and poor performance at low signal-to-noise ratios. We combine multicolour interferometric pattern generation with machine learning to achieve high-contrast, real-time reconstruction of image data that is robust to background noise and sample motion. We validate the method in silico and demonstrate imaging of diverse specimens, from fixed and live biological samples to synthetic biosystems, reconstructing data live at 11 Hz across a 44 × 44μm2 field of view, and demonstrate image acquisition speeds exceeding 154 Hz.

Optical tomography in a single camera frame using fringe-encoded deep-learning full-field OCT

Optical coherence tomography is a valuable tool for in vivo examination thanks to its superior combination of axial resolution, field-of-view and working distance. OCT images are reconstructed from several phases that are obtained by modulation/multiplexing of light wavelength or optical path. This paper shows that only one phase (and one camera frame) is sufficient for en face tomography. The idea is to encode a high-frequency fringe patterns into the selected layer of the sample using low-coherence interferometry. These patterns can then be efficiently extracted with a high-pass filter enhanced via deep learning networks to create the tomographic full-field OCT view. This brings 10-fold improvement in imaging speed, considerably reducing the phase errors and incoherent light artifacts related to in vivo movements. Moreover, this work opens a path for low-cost tomography with slow consumer cameras. Optically, the device resembles the conventional time-domain full-field OCT without incurring additional costs or a field-of-view/resolution reduction. The approach is validated by imaging in vivo cornea in human subjects. Open-source and easy-to-follow codes for data generation/training/inference with U-Net/Pix2Pix networks are provided to be used in a variety of image-to-image translation tasks.

Whole-brain Optical Imaging: A Powerful Tool for Precise Brain Mapping at the Mesoscopic Level

The mammalian brain is a highly complex network that consists of millions to billions of densely-interconnected neurons. Precise dissection of neural circuits at the mesoscopic level can provide important structural information for understanding the brain. Optical approaches can achieve submicron lateral resolution and achieve "optical sectioning" by a variety of means, which has the natural advantage of allowing the observation of neural circuits at the mesoscopic level. Automated whole-brain optical imaging methods based on tissue clearing or histological sectioning surpass the limitation of optical imaging depth in biological tissues and can provide delicate structural information in a large volume of tissues. Combined with various fluorescent labeling techniques, whole-brain optical imaging methods have shown great potential in the brain-wide quantitative profiling of cells, circuits, and blood vessels. In this review, we summarize the principles and implementations of various whole-brain optical imaging methods and provide some concepts regarding their future development.

Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy

One intrinsic yet critical issue that troubles the field of fluorescence microscopy ever since its introduction is the unmatched resolution in the lateral and axial directions (i.e., resolution anisotropy), which severely deteriorates the quality, reconstruction, and analysis of 3D volume images. By leveraging the natural anisotropy, we present a deep self-learning method termed Self-Net that significantly improves the resolution of axial images by using the lateral images from the same raw dataset as rational targets. By incorporating unsupervised learning for realistic anisotropic degradation and supervised learning for high-fidelity isotropic recovery, our method can effectively suppress the hallucination with substantially enhanced image quality compared to previously reported methods. In the experiments, we show that Self-Net can reconstruct high-fidelity isotropic 3D images from organelle to tissue levels via raw images from various microscopy platforms, e.g., wide-field, laser-scanning, or super-resolution microscopy. For the first time, Self-Net enables isotropic whole-brain imaging at a voxel resolution of 0.2 × 0.2 × 0.2 μm3, which addresses the last-mile problem of data quality in single-neuron morphology visualization and reconstruction with minimal effort and cost. Overall, Self-Net is a promising approach to overcoming the inherent resolution anisotropy for all classes of 3D fluorescence microscopy.

Superresolution structured illumination microscopy reconstruction algorithms: a review

Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.

Single-frame 3D lensless microscopic imaging via deep learning

Since the pollen of different species varies in shape and size, visualizing the 3-dimensional structure of a pollen grain can aid in its characterization. Lensless sensing is useful for reducing both optics footprint and cost, while the capability to image pollen grains in 3-dimensions using such a technique could be truly disruptive in the palynology, bioaerosol sensing, and ecology sectors. Here, we show the ability to employ deep learning to generate 3-dimensional images of pollen grains using a series of 2-dimensional images created from 2-dimensional scattering patterns. Using a microscope to obtain 3D Z-stack images of a pollen grain and a 520 nm laser to obtain scattering patterns from the pollen, a single scattering pattern per 3D image was obtained for each position of the pollen grain within the laser beam. In order to create a neural network to transform a single scattering pattern into different 2D images from the Z-stack, additional Z-axis information is required to be added to the scattering pattern. Information was therefore encoded into the scattering pattern image channels, such that the scattering pattern occupied the red channel, and a value indicating the position in the Z-axis occupied the green and blue channels. Following neural network training, 3D images were formed from collated generated 2D images. The volumes of the pollen grains were generated with a mean accuracy of ∼84%. The development of airborne-pollen sensors based on this technique could enable the collection of rich data that would be invaluable to scientists for understanding mechanisms of pollen production climate change and effects on the wider public health.

Background-free wide-field fluorescence imaging using edge detection combined with HiLo

Fluorescence microscopy has been widely used in the field of biological imaging, but the disturbance of background noise has always been an unavoidable phenomenon. To obtain a background free image, a virtual HiLo based on edge detection (V-HiLo-ED) method for background removing is proposed, which is different from the existing popular software algorithms that obtain the background-free image by subtracting the estimated background, but the background-free image is directly reconstructed by estimating the foreground. Compared with two other popular software-based methods, the wavelet-based background and noise subtraction algorithm (WBNS) and the rolling ball algorithm (RBA), the V-HiLo-ED owns a better quality on achieving background-free imaging. Compared with hardware-based method such as HiLo method, V-HiLo-ED exhibits almost the same performance but faster speed. In combination with light sheet microscopy, the V-HiLo-ED further improves the signal-to-noise ratio of images with thick light-sheet. These experiment results indicate that the V-HiLo-ED owns the potentiality in many other image applications such as endoscopy.

Single-shot optical sectioning microscopy based on structured illumination

In this Letter, we present a single-shot 3D-resolved structured illumination microscopy (SIM) based on a digital micromirror device (DMD), a galvanometric mirror, and the HiLo algorithm. During imaging, the DMD rapidly generates sinusoidal and plane illuminations in the focal region. By synchronizing the DMD with a galvanometric scanner and exploiting the unique data readout process of the camera, the emissions from the specimen under two different illuminations, i.e., structured and uniform illumination, are projected to different regions on a camera, achieving high-resolution single-exposure optical sectioning at the camera's limiting speed, i.e., 200 Hz, without sacrificing the resolution. A model has been developed to guide the design and optimization of the optical system. Imaging experiments on pollen and mouse kidney samples have been performed to verify the predicted performance. The results show that the single-shot SIM with the HiLo algorithm achieves comparable resolution to the standard two-shot HiLo method with a twofold speed enhancement, which may find important applications in biophotonics, e.g., visualizing high-speed biological events in vivo.

Deep-3D microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning

Confocal microscopy is a standard approach for obtaining volumetric images of a sample with high axial and lateral resolution, especially when dealing with scattering samples. Unfortunately, a confocal microscope is quite expensive compared to traditional microscopes. In addition, the point scanning in confocal microscopy leads to slow imaging speed and photobleaching due to the high dose of laser energy. In this paper, we demonstrate how the advances in machine learning can be exploited to "teach" a traditional wide-field microscope, one that's available in every lab, into producing 3D volumetric images like a confocal microscope. The key idea is to obtain multiple images with different focus settings using a wide-field microscope and use a 3D generative adversarial network (GAN) based neural network to learn the mapping between the blurry low-contrast image stacks obtained using a wide-field microscope and the sharp, high-contrast image stacks obtained using a confocal microscope. After training the network with widefield-confocal stack pairs, the network can reliably and accurately reconstruct 3D volumetric images that rival confocal images in terms of its lateral resolution, z-sectioning and image contrast. Our experimental results demonstrate generalization ability to handle unseen data, stability in the reconstruction results, high spatial resolution even when imaging thick (∼40 microns) highly-scattering samples. We believe that such learning-based microscopes have the potential to bring confocal imaging quality to every lab that has a wide-field microscope.

Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation

Structured illumination microscopy (SIM) reconstructs optically-sectioned images of a sample from multiple spatially-patterned wide-field images, but the traditional single non-patterned wide-field images are more inexpensively obtained since they do not require generation of specialized illumination patterns. In this work, we translated wide-field fluorescence microscopy images to optically-sectioned SIM images by a Pix2Pix conditional generative adversarial network (cGAN). Our model shows the capability of both 2D cross-modality image translation from wide-field images to optical sections, and further demonstrates potential to recover 3D optically-sectioned volumes from wide-field image stacks. The utility of the model was tested on a variety of samples including fluorescent beads and fresh human tissue samples.

DeepS: a web server for image optical sectioning and super resolution microscopy based on a deep learning framework

MOTIVATION

Microscopy technology plays important roles in many biological research fields. Solvent-cleared brain high-resolution (HR) 3D image reconstruction is an important microscopy application. However, 3D microscopy image generation is time-consuming and expensive. Therefore, we have developed a deep learning framework (DeepS) for both image optical sectioning and super resolution microscopy.

RESULTS

Using DeepS to perform super resolution solvent-cleared mouse brain microscopy 3D image yields improved performance in comparison with the standard image processing workflow. We have also developed a web server to allow online usage of DeepS. Users can train their own models with only one pair of training images using the transfer learning function of the web server.

AVAILABILITYAND IMPLEMENTATION

http://deeps.cibr.ac.cn.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Deep learning based one-shot optically-sectioned structured illumination microscopy for surface measurement

Optically-sectioned structured illumination microscopy (OS-SIM) is broadly used for biological imaging and engineering surface measurement owing to its simple, low-cost, scanning-free experimental setup and excellent optical sectioning capability. However, the efficiency of current optically-sectioned methods in OS-SIM is yet limited for surface measurement because a set of wide-field images under uniform or structured illumination are needed to derive an optical section at each scanning height. In this paper, a deep-learning-based one-shot optically-sectioned method, called Deep-OS-SIM, is proposed to improve the efficiency of OS-SIM for surface measurement. Specifically, we develop a convolutional neural network (CNN) to learn the statistical invariance of optical sectioning across structured illumination images. By taking full advantage of the high entropy properties of structured illumination images to train the CNN, fast convergence and low training error are achieved in our method even for low-textured surfaces. The well-trained CNN is then applied to a plane mirror for testing, demonstrating the ability of the method to reconstruct high-quality optical sectioning from only one instead of two or three raw structured illumination frames. Further measurement experiments on a standard step and milled surface show that the proposed method has similar accuracy to OS-SIM techniques but with higher imaging speed.

Kobitski A, Zhang W, Nienhaus GU. Wavelet-based background and noise subtraction for fluorescence microscopy images

Fluorescence microscopy images are inevitably contaminated by background intensity contributions. Fluorescence from out-of-focus planes and scattered light are important sources of slowly varying, low spatial frequency background, whereas background varying from pixel to pixel (high frequency noise) is introduced by the detection system. Here we present a powerful, easy-to-use software, wavelet-based background and noise subtraction (WBNS), which effectively removes both of these components. To assess its performance, we apply WBNS to synthetic images and compare the results quantitatively with the ground truth and with images processed by other background removal algorithms. We further evaluate WBNS on real images taken with a light-sheet microscope and a super-resolution stimulated emission depletion microscope. For both cases, we compare the WBNS algorithm with hardware-based background removal techniques and present a quantitative assessment of the results. WBNS shows an excellent performance in all these applications and significantly enhances the visual appearance of fluorescence images. Moreover, it may serve as a pre-processing step for further quantitative analysis.

Deep-learning-based whole-brain imaging at single-neuron resolution

Obtaining fine structures of neurons is necessary for understanding brain function. Simple and effective methods for large-scale 3D imaging at optical resolution are still lacking. Here, we proposed a deep-learning-based fluorescence micro-optical sectioning tomography (DL-fMOST) method for high-throughput, high-resolution whole-brain imaging. We utilized a wide-field microscope for imaging, a U-net convolutional neural network for real-time optical sectioning, and histological sectioning for exceeding the imaging depth limit. A 3D dataset of a mouse brain with a voxel size of 0.32 × 0.32 × 2 µm was acquired in 1.5 days. We demonstrated the robustness of DL-fMOST for mouse brains with labeling of different types of neurons.

Full-color optically-sectioned imaging by wide-field microscopy via deep-learning

Wide-field microscopy (WFM) is broadly used in experimental studies of biological specimens. However, combining the out-of-focus signals with the in-focus plane reduces the signal-to-noise ratio (SNR) and axial resolution of the image. Therefore, structured illumination microscopy (SIM) with white light illumination has been used to obtain full-color 3D images, which can capture high SNR optically-sectioned images with improved axial resolution and natural specimen colors. Nevertheless, this full-color SIM (FC-SIM) has a data acquisition burden for 3D-image reconstruction with a shortened depth-of-field, especially for thick samples such as insects and large-scale 3D imaging using stitching techniques. In this paper, we propose a deep-learning-based method for full-color WFM, i.e., FC-WFM-Deep, which can reconstruct high-quality full-color 3D images with an extended optical sectioning capability directly from the FC-WFM z-stack data. Case studies of different specimens with a specific imaging system are used to illustrate this method. Consequently, the image quality achievable with this FC-WFM-Deep method is comparable to the FC-SIM method in terms of 3D information and spatial resolution, while the reconstruction data size is 21-fold smaller and the in-focus depth is doubled. This technique significantly reduces the 3D data acquisition requirements without losing detail and improves the 3D imaging speed by extracting the optical sectioning in the depth-of-field. This cost-effective and convenient method offers a promising tool to observe high-precision color 3D spatial distributions of biological samples.

Optical Imaging in Brainsmatics

When neuroscience’s focus moves from molecular and cellular level to systems level, information technology mixes in and cultivates a new branch neuroinformatics. Especially under the investments of brain initiatives all around the world, brain atlases and connectomics are identified as the substructure to understand the brain. We think it is time to call for a potential interdisciplinary subject, brainsmatics, referring to brain-wide spatial informatics science and emphasizing on precise positioning information affiliated to brain-wide connectome, genome, proteome, transcriptome, metabolome, etc. Brainsmatics methodology includes tracing, surveying, visualizing, and analyzing brain-wide spatial information. Among all imaging techniques, optical imaging is the most appropriate solution to achieve whole-brain connectome in consistent single-neuron resolution. This review aims to introduce contributions of optical imaging to brainsmatics studies, especially the major strategies applied in tracing and surveying processes. After discussions on the state-of-the-art technology, the development objectives of optical imaging in brainsmatics field are suggested. We call for a global contribution to the brainsmatics field from all related communities such as neuroscientists, biologists, engineers, programmers, chemists, mathematicians, physicists, clinicians, pharmacists, etc. As the leading approach, optical imaging will, in turn, benefit from the prosperous development of brainsmatics.