1
|
Dai W, Wong IHM, Wong TTW. Exceeding the limit for microscopic image translation with a deep learning-based unified framework. PNAS NEXUS 2024; 3:pgae133. [PMID: 38601859 PMCID: PMC11004937 DOI: 10.1093/pnasnexus/pgae133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 03/19/2024] [Indexed: 04/12/2024]
Abstract
Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.
Collapse
Affiliation(s)
- Weixing Dai
- Department of Chemical and Biological Engineering, Translational and Advanced Bioimaging Laboratory, Hong Kong University of Science and Technology, Hong Kong 999077, China
| | - Ivy H M Wong
- Department of Chemical and Biological Engineering, Translational and Advanced Bioimaging Laboratory, Hong Kong University of Science and Technology, Hong Kong 999077, China
| | - Terence T W Wong
- Department of Chemical and Biological Engineering, Translational and Advanced Bioimaging Laboratory, Hong Kong University of Science and Technology, Hong Kong 999077, China
| |
Collapse
|
2
|
Ward EN, McClelland RM, Lamb JR, Rubio-Sánchez R, Christensen CN, Mazumder B, Kapsiani S, Mascheroni L, Di Michele L, Kaminski Schierle GS, Kaminski CF. Fast, multicolour optical sectioning over extended fields of view with patterned illumination and machine learning. BIOMEDICAL OPTICS EXPRESS 2024; 15:1074-1088. [PMID: 38404329 PMCID: PMC10890859 DOI: 10.1364/boe.510912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Revised: 12/22/2023] [Accepted: 01/09/2024] [Indexed: 02/27/2024]
Abstract
Structured illumination can reject out-of-focus signal from a sample, enabling high-speed and high-contrast imaging over large areas with widefield detection optics. However, this optical sectioning technique is currently limited by image reconstruction artefacts and poor performance at low signal-to-noise ratios. We combine multicolour interferometric pattern generation with machine learning to achieve high-contrast, real-time reconstruction of image data that is robust to background noise and sample motion. We validate the method in silico and demonstrate imaging of diverse specimens, from fixed and live biological samples to synthetic biosystems, reconstructing data live at 11 Hz across a 44 × 44μm2 field of view, and demonstrate image acquisition speeds exceeding 154 Hz.
Collapse
Affiliation(s)
- Edward N. Ward
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Rebecca M. McClelland
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Jacob R. Lamb
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Roger Rubio-Sánchez
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
- fabriCELL, Molecular Sciences Research Hub,
Imperial College London, London, W12 0BZ,
UK
| | - Charles N. Christensen
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Bismoy Mazumder
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Sofia Kapsiani
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Luca Mascheroni
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| | - Lorenzo Di Michele
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
- fabriCELL, Molecular Sciences Research Hub,
Imperial College London, London, W12 0BZ,
UK
| | | | - Clemens F. Kaminski
- Department of Chemical Engineering and
Biotechnology, University of Cambridge, Cambridge, CB3 0AS, UK
| |
Collapse
|
3
|
Mazlin V. Optical tomography in a single camera frame using fringe-encoded deep-learning full-field OCT. BIOMEDICAL OPTICS EXPRESS 2024; 15:222-236. [PMID: 38223177 PMCID: PMC10783898 DOI: 10.1364/boe.506664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 11/29/2023] [Accepted: 12/03/2023] [Indexed: 01/16/2024]
Abstract
Optical coherence tomography is a valuable tool for in vivo examination thanks to its superior combination of axial resolution, field-of-view and working distance. OCT images are reconstructed from several phases that are obtained by modulation/multiplexing of light wavelength or optical path. This paper shows that only one phase (and one camera frame) is sufficient for en face tomography. The idea is to encode a high-frequency fringe patterns into the selected layer of the sample using low-coherence interferometry. These patterns can then be efficiently extracted with a high-pass filter enhanced via deep learning networks to create the tomographic full-field OCT view. This brings 10-fold improvement in imaging speed, considerably reducing the phase errors and incoherent light artifacts related to in vivo movements. Moreover, this work opens a path for low-cost tomography with slow consumer cameras. Optically, the device resembles the conventional time-domain full-field OCT without incurring additional costs or a field-of-view/resolution reduction. The approach is validated by imaging in vivo cornea in human subjects. Open-source and easy-to-follow codes for data generation/training/inference with U-Net/Pix2Pix networks are provided to be used in a variety of image-to-image translation tasks.
Collapse
Affiliation(s)
- Viacheslav Mazlin
- Institut Langevin, ESPCI Paris, PSL University, CNRS, 1 rue Jussieu, 75005 Paris, France
- Quinze-Vingts National Eye Hospital, 28 Rue de Charenton, 75012 Paris, France
| |
Collapse
|
4
|
Jiang T, Gong H, Yuan J. Whole-brain Optical Imaging: A Powerful Tool for Precise Brain Mapping at the Mesoscopic Level. Neurosci Bull 2023; 39:1840-1858. [PMID: 37715920 PMCID: PMC10661546 DOI: 10.1007/s12264-023-01112-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Accepted: 05/08/2023] [Indexed: 09/18/2023] Open
Abstract
The mammalian brain is a highly complex network that consists of millions to billions of densely-interconnected neurons. Precise dissection of neural circuits at the mesoscopic level can provide important structural information for understanding the brain. Optical approaches can achieve submicron lateral resolution and achieve "optical sectioning" by a variety of means, which has the natural advantage of allowing the observation of neural circuits at the mesoscopic level. Automated whole-brain optical imaging methods based on tissue clearing or histological sectioning surpass the limitation of optical imaging depth in biological tissues and can provide delicate structural information in a large volume of tissues. Combined with various fluorescent labeling techniques, whole-brain optical imaging methods have shown great potential in the brain-wide quantitative profiling of cells, circuits, and blood vessels. In this review, we summarize the principles and implementations of various whole-brain optical imaging methods and provide some concepts regarding their future development.
Collapse
Affiliation(s)
- Tao Jiang
- Huazhong University of Science and Technology-Suzhou Institute for Brainsmatics, Jiangsu Industrial Technology Research Institute, Suzhou, 215123, China
| | - Hui Gong
- Huazhong University of Science and Technology-Suzhou Institute for Brainsmatics, Jiangsu Industrial Technology Research Institute, Suzhou, 215123, China
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Jing Yuan
- Huazhong University of Science and Technology-Suzhou Institute for Brainsmatics, Jiangsu Industrial Technology Research Institute, Suzhou, 215123, China.
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China.
| |
Collapse
|
5
|
Ning K, Lu B, Wang X, Zhang X, Nie S, Jiang T, Li A, Fan G, Wang X, Luo Q, Gong H, Yuan J. Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy. LIGHT, SCIENCE & APPLICATIONS 2023; 12:204. [PMID: 37640721 PMCID: PMC10462670 DOI: 10.1038/s41377-023-01230-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Revised: 07/04/2023] [Accepted: 07/12/2023] [Indexed: 08/31/2023]
Abstract
One intrinsic yet critical issue that troubles the field of fluorescence microscopy ever since its introduction is the unmatched resolution in the lateral and axial directions (i.e., resolution anisotropy), which severely deteriorates the quality, reconstruction, and analysis of 3D volume images. By leveraging the natural anisotropy, we present a deep self-learning method termed Self-Net that significantly improves the resolution of axial images by using the lateral images from the same raw dataset as rational targets. By incorporating unsupervised learning for realistic anisotropic degradation and supervised learning for high-fidelity isotropic recovery, our method can effectively suppress the hallucination with substantially enhanced image quality compared to previously reported methods. In the experiments, we show that Self-Net can reconstruct high-fidelity isotropic 3D images from organelle to tissue levels via raw images from various microscopy platforms, e.g., wide-field, laser-scanning, or super-resolution microscopy. For the first time, Self-Net enables isotropic whole-brain imaging at a voxel resolution of 0.2 × 0.2 × 0.2 μm3, which addresses the last-mile problem of data quality in single-neuron morphology visualization and reconstruction with minimal effort and cost. Overall, Self-Net is a promising approach to overcoming the inherent resolution anisotropy for all classes of 3D fluorescence microscopy.
Collapse
Affiliation(s)
- Kefu Ning
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
| | - Bolin Lu
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
| | - Xiaojun Wang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
- School of Biomedical Engineering, Hainan University, Haikou, China
| | - Xiaoyu Zhang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
| | - Shuo Nie
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
| | - Tao Jiang
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
| | - Anan Li
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
| | - Guoqing Fan
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
| | - Xiaofeng Wang
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
| | - Qingming Luo
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China
- School of Biomedical Engineering, Hainan University, Haikou, China
| | - Hui Gong
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China.
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China.
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China.
| | - Jing Yuan
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China.
- MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China.
- HUST-Suzhou Institute for Brainsmatics, Suzhou, China.
| |
Collapse
|
6
|
Chen X, Zhong S, Hou Y, Cao R, Wang W, Li D, Dai Q, Kim D, Xi P. Superresolution structured illumination microscopy reconstruction algorithms: a review. LIGHT, SCIENCE & APPLICATIONS 2023; 12:172. [PMID: 37433801 DOI: 10.1038/s41377-023-01204-4] [Citation(s) in RCA: 21] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 05/24/2023] [Accepted: 06/05/2023] [Indexed: 07/13/2023]
Abstract
Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.
Collapse
Affiliation(s)
- Xin Chen
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China
| | - Suyi Zhong
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China
| | - Yiwei Hou
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China
| | - Ruijie Cao
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China
| | - Wenyi Wang
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China
| | - Dong Li
- National Laboratory of Biomacromolecules, CAS Center for Excellence in Biomacromolecules, Institute of Biophysics, Chinese Academy of Sciences, Beijing, China
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
- Beijing Key Laboratory of Multidimension & Multiscale Computational Photography, Tsinghua University, Beijing, China
- Beijing Laboratory of Brain and Cognitive Intelligence, Beijing Municipal Education Commission, Beijing, China
| | - Donghyun Kim
- School of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-Ro, Seodaemun-Gu, Seoul, 03722, Korea
| | - Peng Xi
- Department of Biomedical Engineering, College of Future Technology, Peking University, Beijing, 100871, China.
- National Biomedical Imaging Center, Peking University, Beijing, 100871, China.
| |
Collapse
|
7
|
Grant-Jacob JA, Praeger M, Eason RW, Mills B. Single-frame 3D lensless microscopic imaging via deep learning. OPTICS EXPRESS 2022; 30:32621-32632. [PMID: 36242319 DOI: 10.1364/oe.464678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Accepted: 06/26/2022] [Indexed: 06/16/2023]
Abstract
Since the pollen of different species varies in shape and size, visualizing the 3-dimensional structure of a pollen grain can aid in its characterization. Lensless sensing is useful for reducing both optics footprint and cost, while the capability to image pollen grains in 3-dimensions using such a technique could be truly disruptive in the palynology, bioaerosol sensing, and ecology sectors. Here, we show the ability to employ deep learning to generate 3-dimensional images of pollen grains using a series of 2-dimensional images created from 2-dimensional scattering patterns. Using a microscope to obtain 3D Z-stack images of a pollen grain and a 520 nm laser to obtain scattering patterns from the pollen, a single scattering pattern per 3D image was obtained for each position of the pollen grain within the laser beam. In order to create a neural network to transform a single scattering pattern into different 2D images from the Z-stack, additional Z-axis information is required to be added to the scattering pattern. Information was therefore encoded into the scattering pattern image channels, such that the scattering pattern occupied the red channel, and a value indicating the position in the Z-axis occupied the green and blue channels. Following neural network training, 3D images were formed from collated generated 2D images. The volumes of the pollen grains were generated with a mean accuracy of ∼84%. The development of airborne-pollen sensors based on this technique could enable the collection of rich data that would be invaluable to scientists for understanding mechanisms of pollen production climate change and effects on the wider public health.
Collapse
|
8
|
Hu Y, Liang D, Wang J, Xuan Y, Zhao F, Liu J, Li R. Background-free wide-field fluorescence imaging using edge detection combined with HiLo. JOURNAL OF BIOPHOTONICS 2022; 15:e202200031. [PMID: 35488180 DOI: 10.1002/jbio.202200031] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Revised: 04/01/2022] [Accepted: 04/28/2022] [Indexed: 06/14/2023]
Abstract
Fluorescence microscopy has been widely used in the field of biological imaging, but the disturbance of background noise has always been an unavoidable phenomenon. To obtain a background free image, a virtual HiLo based on edge detection (V-HiLo-ED) method for background removing is proposed, which is different from the existing popular software algorithms that obtain the background-free image by subtracting the estimated background, but the background-free image is directly reconstructed by estimating the foreground. Compared with two other popular software-based methods, the wavelet-based background and noise subtraction algorithm (WBNS) and the rolling ball algorithm (RBA), the V-HiLo-ED owns a better quality on achieving background-free imaging. Compared with hardware-based method such as HiLo method, V-HiLo-ED exhibits almost the same performance but faster speed. In combination with light sheet microscopy, the V-HiLo-ED further improves the signal-to-noise ratio of images with thick light-sheet. These experiment results indicate that the V-HiLo-ED owns the potentiality in many other image applications such as endoscopy.
Collapse
Affiliation(s)
- Yuyao Hu
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- University Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing, China
| | - Dong Liang
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- School of Physics Science and Engineering, Tongji University, Shanghai, China
| | - Jing Wang
- Institute of Photonic Chips, University of Shanghai for Science and Technology, Shanghai, China
- Centre for Artificial-Intelligence Nanophotonics, School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Yaping Xuan
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- University Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing, China
| | - Fu Zhao
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- University Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing, China
| | - Jun Liu
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- University Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing, China
| | - Ruxin Li
- State Key Laboratory of High Field Laser Physics and CAS Center for Excellence in Ultra-intense Laser Science, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China
- University Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
9
|
Fu Z, Chen J, Liu G, Chen SC. Single-shot optical sectioning microscopy based on structured illumination. OPTICS LETTERS 2022; 47:814-817. [PMID: 35167532 DOI: 10.1364/ol.451267] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/16/2021] [Accepted: 01/04/2022] [Indexed: 06/14/2023]
Abstract
In this Letter, we present a single-shot 3D-resolved structured illumination microscopy (SIM) based on a digital micromirror device (DMD), a galvanometric mirror, and the HiLo algorithm. During imaging, the DMD rapidly generates sinusoidal and plane illuminations in the focal region. By synchronizing the DMD with a galvanometric scanner and exploiting the unique data readout process of the camera, the emissions from the specimen under two different illuminations, i.e., structured and uniform illumination, are projected to different regions on a camera, achieving high-resolution single-exposure optical sectioning at the camera's limiting speed, i.e., 200 Hz, without sacrificing the resolution. A model has been developed to guide the design and optimization of the optical system. Imaging experiments on pollen and mouse kidney samples have been performed to verify the predicted performance. The results show that the single-shot SIM with the HiLo algorithm achieves comparable resolution to the standard two-shot HiLo method with a twofold speed enhancement, which may find important applications in biophotonics, e.g., visualizing high-speed biological events in vivo.
Collapse
|
10
|
Li B, Tan S, Dong J, Lian X, Zhang Y, Ji X, Veeraraghavan A. Deep-3D microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning. BIOMEDICAL OPTICS EXPRESS 2022; 13:284-299. [PMID: 35154871 PMCID: PMC8803017 DOI: 10.1364/boe.444488] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/22/2021] [Revised: 12/02/2021] [Accepted: 12/03/2021] [Indexed: 06/14/2023]
Abstract
Confocal microscopy is a standard approach for obtaining volumetric images of a sample with high axial and lateral resolution, especially when dealing with scattering samples. Unfortunately, a confocal microscope is quite expensive compared to traditional microscopes. In addition, the point scanning in confocal microscopy leads to slow imaging speed and photobleaching due to the high dose of laser energy. In this paper, we demonstrate how the advances in machine learning can be exploited to "teach" a traditional wide-field microscope, one that's available in every lab, into producing 3D volumetric images like a confocal microscope. The key idea is to obtain multiple images with different focus settings using a wide-field microscope and use a 3D generative adversarial network (GAN) based neural network to learn the mapping between the blurry low-contrast image stacks obtained using a wide-field microscope and the sharp, high-contrast image stacks obtained using a confocal microscope. After training the network with widefield-confocal stack pairs, the network can reliably and accurately reconstruct 3D volumetric images that rival confocal images in terms of its lateral resolution, z-sectioning and image contrast. Our experimental results demonstrate generalization ability to handle unseen data, stability in the reconstruction results, high spatial resolution even when imaging thick (∼40 microns) highly-scattering samples. We believe that such learning-based microscopes have the potential to bring confocal imaging quality to every lab that has a wide-field microscope.
Collapse
Affiliation(s)
- Bowen Li
- Department of Automation & BNRist, Tsinghua University, Beijing, China
| | - Shiyu Tan
- Department of Electrical and Computer Engineering, Rice University, Houston, TX 77005, USA
| | - Jiuyang Dong
- Tsinghua Shenzhen International Graduate School, Shenzhen, China
| | - Xiaocong Lian
- Department of Automation & BNRist, Tsinghua University, Beijing, China
| | - Yongbing Zhang
- Harbin Institute of Technology (Shenzhen), Shenzhen, China
| | - Xiangyang Ji
- Department of Automation & BNRist, Tsinghua University, Beijing, China
| | - Ashok Veeraraghavan
- Department of Electrical and Computer Engineering, Rice University, Houston, TX 77005, USA
| |
Collapse
|
11
|
Zhuge H, Summa B, Hamm J, Brown JQ. Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation. BIOMEDICAL OPTICS EXPRESS 2021; 12:7526-7543. [PMID: 35003850 PMCID: PMC8713683 DOI: 10.1364/boe.439894] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 11/01/2021] [Accepted: 11/07/2021] [Indexed: 06/14/2023]
Abstract
Structured illumination microscopy (SIM) reconstructs optically-sectioned images of a sample from multiple spatially-patterned wide-field images, but the traditional single non-patterned wide-field images are more inexpensively obtained since they do not require generation of specialized illumination patterns. In this work, we translated wide-field fluorescence microscopy images to optically-sectioned SIM images by a Pix2Pix conditional generative adversarial network (cGAN). Our model shows the capability of both 2D cross-modality image translation from wide-field images to optical sections, and further demonstrates potential to recover 3D optically-sectioned volumes from wide-field image stacks. The utility of the model was tested on a variety of samples including fluorescent beads and fresh human tissue samples.
Collapse
Affiliation(s)
- Huimin Zhuge
- Department of Biomedical Engineering, Tulane University, 500 Lindy Boggs Center, New Orleans, LA 70118, USA
| | - Brian Summa
- Department of Computer Science, Tulane University, New Orleans, LA 70118, USA
| | - Jihun Hamm
- Department of Computer Science, Tulane University, New Orleans, LA 70118, USA
| | - J. Quincy Brown
- Department of Biomedical Engineering, Tulane University, 500 Lindy Boggs Center, New Orleans, LA 70118, USA
| |
Collapse
|
12
|
Zhu Q, Shao Y, Wang Z, Chen X, Li C, Liang Z, Jia M, Guo Q, Zhao H, Kong L, Zhang L. DeepS: a web server for image optical sectioning and super resolution microscopy based on a deep learning framework. Bioinformatics 2021; 37:3086-3087. [PMID: 33677518 DOI: 10.1093/bioinformatics/btab144] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2020] [Revised: 02/05/2021] [Accepted: 02/26/2021] [Indexed: 02/02/2023] Open
Abstract
MOTIVATION Microscopy technology plays important roles in many biological research fields. Solvent-cleared brain high-resolution (HR) 3D image reconstruction is an important microscopy application. However, 3D microscopy image generation is time-consuming and expensive. Therefore, we have developed a deep learning framework (DeepS) for both image optical sectioning and super resolution microscopy. RESULTS Using DeepS to perform super resolution solvent-cleared mouse brain microscopy 3D image yields improved performance in comparison with the standard image processing workflow. We have also developed a web server to allow online usage of DeepS. Users can train their own models with only one pair of training images using the transfer learning function of the web server. AVAILABILITYAND IMPLEMENTATION http://deeps.cibr.ac.cn. SUPPLEMENTARY INFORMATION Supplementary data are available at Bioinformatics online.
Collapse
Affiliation(s)
- Qingjie Zhu
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| | - Yi Shao
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| | - Zhicheng Wang
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
- Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
| | - Xingjun Chen
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
- Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
| | - Chunqiong Li
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| | - Zihan Liang
- Department of Bio-engineering, Southwest Jiaotong University, Chengdu, Sichuan 611756, China
| | - Mingyue Jia
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| | - Qingchun Guo
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| | - Hu Zhao
- Department of Comprehensive Dentistry, Texas A&M University, Dallas, TX 75246, USA
| | - Lei Kong
- Center for Bioinformatics, State Key Laboratory of Protein and Plant Gene Research, School of Life Sciences, Peking University, Beijing 100871, China
| | - Li Zhang
- Chinese Institute for Brain Research (CIBR), Beijing 102206, China
| |
Collapse
|
13
|
Chai C, Chen C, Liu X, Lei Z. Deep learning based one-shot optically-sectioned structured illumination microscopy for surface measurement. OPTICS EXPRESS 2021; 29:4010-4021. [PMID: 33770989 DOI: 10.1364/oe.415210] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/24/2020] [Accepted: 01/14/2021] [Indexed: 06/12/2023]
Abstract
Optically-sectioned structured illumination microscopy (OS-SIM) is broadly used for biological imaging and engineering surface measurement owing to its simple, low-cost, scanning-free experimental setup and excellent optical sectioning capability. However, the efficiency of current optically-sectioned methods in OS-SIM is yet limited for surface measurement because a set of wide-field images under uniform or structured illumination are needed to derive an optical section at each scanning height. In this paper, a deep-learning-based one-shot optically-sectioned method, called Deep-OS-SIM, is proposed to improve the efficiency of OS-SIM for surface measurement. Specifically, we develop a convolutional neural network (CNN) to learn the statistical invariance of optical sectioning across structured illumination images. By taking full advantage of the high entropy properties of structured illumination images to train the CNN, fast convergence and low training error are achieved in our method even for low-textured surfaces. The well-trained CNN is then applied to a plane mirror for testing, demonstrating the ability of the method to reconstruct high-quality optical sectioning from only one instead of two or three raw structured illumination frames. Further measurement experiments on a standard step and milled surface show that the proposed method has similar accuracy to OS-SIM techniques but with higher imaging speed.
Collapse
|
14
|
Hüpfel M, Yu. Kobitski A, Zhang W, Nienhaus GU. Wavelet-based background and noise subtraction for fluorescence microscopy images. BIOMEDICAL OPTICS EXPRESS 2021; 12:969-980. [PMID: 33680553 PMCID: PMC7901331 DOI: 10.1364/boe.413181] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/22/2020] [Revised: 12/27/2020] [Accepted: 01/15/2021] [Indexed: 05/20/2023]
Abstract
Fluorescence microscopy images are inevitably contaminated by background intensity contributions. Fluorescence from out-of-focus planes and scattered light are important sources of slowly varying, low spatial frequency background, whereas background varying from pixel to pixel (high frequency noise) is introduced by the detection system. Here we present a powerful, easy-to-use software, wavelet-based background and noise subtraction (WBNS), which effectively removes both of these components. To assess its performance, we apply WBNS to synthetic images and compare the results quantitatively with the ground truth and with images processed by other background removal algorithms. We further evaluate WBNS on real images taken with a light-sheet microscope and a super-resolution stimulated emission depletion microscope. For both cases, we compare the WBNS algorithm with hardware-based background removal techniques and present a quantitative assessment of the results. WBNS shows an excellent performance in all these applications and significantly enhances the visual appearance of fluorescence images. Moreover, it may serve as a pre-processing step for further quantitative analysis.
Collapse
Affiliation(s)
- Manuel Hüpfel
- Institute of Applied Physics, Karlsruhe Institute of Technology (KIT), Wolfgang-Gaede-Str. 1, 76131 Karlsruhe, Germany
| | - Andrei Yu. Kobitski
- Institute of Applied Physics, Karlsruhe Institute of Technology (KIT), Wolfgang-Gaede-Str. 1, 76131 Karlsruhe, Germany
| | - Weichun Zhang
- Institute of Applied Physics, Karlsruhe Institute of Technology (KIT), Wolfgang-Gaede-Str. 1, 76131 Karlsruhe, Germany
| | - G. Ulrich Nienhaus
- Institute of Applied Physics, Karlsruhe Institute of Technology (KIT), Wolfgang-Gaede-Str. 1, 76131 Karlsruhe, Germany
- Institute of Nanotechnology, Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany
- Institute of Biological and Chemical Systems, Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany
- Department of Physics, University of Illinois at Urbana-Champaign, 1110 W. Green Street, Urbana, IL 61801, USA
| |
Collapse
|
15
|
Ning K, Zhang X, Gao X, Jiang T, Wang H, Chen S, Li A, Yuan J. Deep-learning-based whole-brain imaging at single-neuron resolution. BIOMEDICAL OPTICS EXPRESS 2020; 11:3567-3584. [PMID: 33014552 PMCID: PMC7510917 DOI: 10.1364/boe.393081] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/17/2020] [Revised: 05/28/2020] [Accepted: 05/28/2020] [Indexed: 05/08/2023]
Abstract
Obtaining fine structures of neurons is necessary for understanding brain function. Simple and effective methods for large-scale 3D imaging at optical resolution are still lacking. Here, we proposed a deep-learning-based fluorescence micro-optical sectioning tomography (DL-fMOST) method for high-throughput, high-resolution whole-brain imaging. We utilized a wide-field microscope for imaging, a U-net convolutional neural network for real-time optical sectioning, and histological sectioning for exceeding the imaging depth limit. A 3D dataset of a mouse brain with a voxel size of 0.32 × 0.32 × 2 µm was acquired in 1.5 days. We demonstrated the robustness of DL-fMOST for mouse brains with labeling of different types of neurons.
Collapse
Affiliation(s)
- Kefu Ning
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
- These authors contributed equally to this work
| | - Xiaoyu Zhang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
- These authors contributed equally to this work
| | - Xuefei Gao
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
- HUST-Suzhou Institute for Brainsmatics, JITRI Institute for Brainsmatics, Suzhou 215000, China
| | - Tao Jiang
- HUST-Suzhou Institute for Brainsmatics, JITRI Institute for Brainsmatics, Suzhou 215000, China
| | - He Wang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
| | - Siqi Chen
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
| | - Anan Li
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
- HUST-Suzhou Institute for Brainsmatics, JITRI Institute for Brainsmatics, Suzhou 215000, China
| | - Jing Yuan
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Innovation Institute, Huazhong University of Science and Technology, Wuhan 430074, China
- HUST-Suzhou Institute for Brainsmatics, JITRI Institute for Brainsmatics, Suzhou 215000, China
| |
Collapse
|
16
|
Bai C, Qian J, Dang S, Peng T, Min J, Lei M, Dan D, Yao B. Full-color optically-sectioned imaging by wide-field microscopy via deep-learning. BIOMEDICAL OPTICS EXPRESS 2020; 11:2619-2632. [PMID: 32499948 PMCID: PMC7249807 DOI: 10.1364/boe.389852] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/05/2020] [Revised: 03/10/2020] [Accepted: 04/11/2020] [Indexed: 06/11/2023]
Abstract
Wide-field microscopy (WFM) is broadly used in experimental studies of biological specimens. However, combining the out-of-focus signals with the in-focus plane reduces the signal-to-noise ratio (SNR) and axial resolution of the image. Therefore, structured illumination microscopy (SIM) with white light illumination has been used to obtain full-color 3D images, which can capture high SNR optically-sectioned images with improved axial resolution and natural specimen colors. Nevertheless, this full-color SIM (FC-SIM) has a data acquisition burden for 3D-image reconstruction with a shortened depth-of-field, especially for thick samples such as insects and large-scale 3D imaging using stitching techniques. In this paper, we propose a deep-learning-based method for full-color WFM, i.e., FC-WFM-Deep, which can reconstruct high-quality full-color 3D images with an extended optical sectioning capability directly from the FC-WFM z-stack data. Case studies of different specimens with a specific imaging system are used to illustrate this method. Consequently, the image quality achievable with this FC-WFM-Deep method is comparable to the FC-SIM method in terms of 3D information and spatial resolution, while the reconstruction data size is 21-fold smaller and the in-focus depth is doubled. This technique significantly reduces the 3D data acquisition requirements without losing detail and improves the 3D imaging speed by extracting the optical sectioning in the depth-of-field. This cost-effective and convenient method offers a promising tool to observe high-precision color 3D spatial distributions of biological samples.
Collapse
Affiliation(s)
- Chen Bai
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
- These authors contributed equally to this work
| | - Jia Qian
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
- These authors contributed equally to this work
| | - Shipei Dang
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Tong Peng
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
| | - Junwei Min
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
| | - Ming Lei
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
| | - Dan Dan
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
| | - Baoli Yao
- State Key Laboratory of Transient Optics and Photonics, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119, China
| |
Collapse
|
17
|
Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat Methods 2019; 16:1323-1331. [DOI: 10.1038/s41592-019-0622-5] [Citation(s) in RCA: 110] [Impact Index Per Article: 22.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2019] [Accepted: 09/30/2019] [Indexed: 01/06/2023]
|
18
|
Abstract
When neuroscience’s focus moves from molecular and cellular level to systems level, information technology mixes in and cultivates a new branch neuroinformatics. Especially under the investments of brain initiatives all around the world, brain atlases and connectomics are identified as the substructure to understand the brain. We think it is time to call for a potential interdisciplinary subject, brainsmatics, referring to brain-wide spatial informatics science and emphasizing on precise positioning information affiliated to brain-wide connectome, genome, proteome, transcriptome, metabolome, etc. Brainsmatics methodology includes tracing, surveying, visualizing, and analyzing brain-wide spatial information. Among all imaging techniques, optical imaging is the most appropriate solution to achieve whole-brain connectome in consistent single-neuron resolution. This review aims to introduce contributions of optical imaging to brainsmatics studies, especially the major strategies applied in tracing and surveying processes. After discussions on the state-of-the-art technology, the development objectives of optical imaging in brainsmatics field are suggested. We call for a global contribution to the brainsmatics field from all related communities such as neuroscientists, biologists, engineers, programmers, chemists, mathematicians, physicists, clinicians, pharmacists, etc. As the leading approach, optical imaging will, in turn, benefit from the prosperous development of brainsmatics.
Collapse
|