1
|
Luo X, Lu Z, Jin M, Chen S, Yang J. Efficient high-resolution fluorescence projection imaging over an extended depth of field through optical hardware and deep learning optimizations. BIOMEDICAL OPTICS EXPRESS 2024; 15:3831-3847. [PMID: 38867796 PMCID: PMC11166417 DOI: 10.1364/boe.523312] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/07/2024] [Revised: 04/27/2024] [Accepted: 05/14/2024] [Indexed: 06/14/2024]
Abstract
Optical microscopy has witnessed notable advancements but has also become more costly and complex. Conventional wide field microscopy (WFM) has low resolution and shallow depth-of-field (DOF), which limits its applications in practical biological experiments. Recently, confocal and light sheet microscopy become major workhorses for biology that incorporate high-precision scanning to perform imaging within an extended DOF but at the sacrifice of expense, complexity, and imaging speed. Here, we propose deep focus microscopy, an efficient framework optimized both in hardware and algorithm to address the tradeoff between resolution and DOF. Our deep focus microscopy achieves large-DOF and high-resolution projection imaging by integrating a deep focus network (DFnet) into light field microscopy (LFM) setups. Based on our constructed dataset, deep focus microscopy features a significantly enhanced spatial resolution of ∼260 nm, an extended DOF of over 30 µm, and broad generalization across diverse sample structures. It also reduces the computational costs by four orders of magnitude compared to conventional LFM technologies. We demonstrate the excellent performance of deep focus microscopy in vivo, including long-term observations of cell division and migrasome formation in zebrafish embryos and mouse livers at high resolution without background contamination.
Collapse
Affiliation(s)
- Xin Luo
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Zhi Lu
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Manchang Jin
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Shuai Chen
- Department of Gastroenterology and Hepatology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Jingyu Yang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| |
Collapse
|
2
|
Xie H, Han X, Xiao G, Xu H, Zhang Y, Zhang G, Li Q, He J, Zhu D, Yu X, Dai Q. Multifocal fluorescence video-rate imaging of centimetre-wide arbitrarily shaped brain surfaces at micrometric resolution. Nat Biomed Eng 2024; 8:740-753. [PMID: 38057428 PMCID: PMC11250366 DOI: 10.1038/s41551-023-01155-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Accepted: 10/26/2023] [Indexed: 12/08/2023]
Abstract
Fluorescence microscopy allows for the high-throughput imaging of cellular activity across brain areas in mammals. However, capturing rapid cellular dynamics across the curved cortical surface is challenging, owing to trade-offs in image resolution, speed, field of view and depth of field. Here we report a technique for wide-field fluorescence imaging that leverages selective illumination and the integration of focal areas at different depths via a spinning disc with varying thickness to enable video-rate imaging of previously reconstructed centimetre-scale arbitrarily shaped surfaces at micrometre-scale resolution and at a depth of field of millimetres. By implementing the technique in a microscope capable of acquiring images at 1.68 billion pixels per second and resolving 16.8 billion voxels per second, we recorded neural activities and the trajectories of neutrophils in real time on curved cortical surfaces in live mice. The technique can be integrated into many microscopes and macroscopes, in both reflective and fluorescence modes, for the study of multiscale cellular interactions on arbitrarily shaped surfaces.
Collapse
Affiliation(s)
- Hao Xie
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
| | - Xiaofei Han
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Guihua Xiao
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Hanyun Xu
- Department of Neurosurgery, The First Medical Center of Chinese PLA General Hospital, Beijing, China
| | - Yuanlong Zhang
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Guoxun Zhang
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Qingwei Li
- School of Medicine, Tsinghua University, Beijing, China
| | - Jing He
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Dan Zhu
- Britton Chance Center for Biomedical Photonics - MoE Key Laboratory for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics - Advanced Biomedical Imaging Facility, Huazhong University of Science and Technology, Wuhan, China
| | - Xinguang Yu
- Department of Neurosurgery, The First Medical Center of Chinese PLA General Hospital, Beijing, China
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China.
| |
Collapse
|
3
|
Lu Z, Zuo S, Shi M, Fan J, Xie J, Xiao G, Yu L, Wu J, Dai Q. Long-term intravital subcellular imaging with confocal scanning light-field microscopy. Nat Biotechnol 2024:10.1038/s41587-024-02249-5. [PMID: 38802562 DOI: 10.1038/s41587-024-02249-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2023] [Accepted: 04/17/2024] [Indexed: 05/29/2024]
Abstract
Long-term observation of subcellular dynamics in living organisms is limited by background fluorescence originating from tissue scattering or dense labeling. Existing confocal approaches face an inevitable tradeoff among parallelization, resolution and phototoxicity. Here we present confocal scanning light-field microscopy (csLFM), which integrates axially elongated line-confocal illumination with the rolling shutter in scanning light-field microscopy (sLFM). csLFM enables high-fidelity, high-speed, three-dimensional (3D) imaging at near-diffraction-limit resolution with both optical sectioning and low phototoxicity. By simultaneous 3D excitation and detection, the excitation intensity can be reduced below 1 mW mm-2, with 15-fold higher signal-to-background ratio over sLFM. We imaged subcellular dynamics over 25,000 timeframes in optically challenging environments in different species, such as migrasome delivery in mouse spleen, retractosome generation in mouse liver and 3D voltage imaging in Drosophila. Moreover, csLFM facilitates high-fidelity, large-scale neural recording with reduced crosstalk, leading to high orientation selectivity to visual stimuli, similar to two-photon microscopy, which aids understanding of neural coding mechanisms.
Collapse
Affiliation(s)
- Zhi Lu
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
- Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing, China
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
- Zhejiang Hehu Technology, Hangzhou, China
- Hangzhou Zhuoxi Institute of Brain and Intelligence, Hangzhou, China
| | - Siqing Zuo
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
- Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing, China
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
| | - Minghui Shi
- State Key Laboratory of Membrane Biology, Tsinghua University-Peking University Joint Center for Life Sciences, Beijing Frontier Research Center for Biological Structure, School of Life Sciences, Tsinghua University, Beijing, China
| | - Jiaqi Fan
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Jingyu Xie
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Guihua Xiao
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Li Yu
- State Key Laboratory of Membrane Biology, Tsinghua University-Peking University Joint Center for Life Sciences, Beijing Frontier Research Center for Biological Structure, School of Life Sciences, Tsinghua University, Beijing, China.
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
- Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing, China.
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China.
- Shanghai AI Laboratory, Shanghai, China.
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
- Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing, China.
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
| |
Collapse
|
4
|
Chen S, Lu Z, Zhao Y, Xia L, Liu C, Zuo S, Jin M, Jia H, Li S, Zhang S, Yang B, Wang Z, Li J, Wang F, Yang C. Myeloid-Mas Signaling Modulates Pathogenic Crosstalk among MYC +CD63 + Endothelial Cells, MMP12 + Macrophages, and Monocytes in Acetaminophen-Induced Liver Injury. ADVANCED SCIENCE (WEINHEIM, BADEN-WURTTEMBERG, GERMANY) 2024; 11:e2306066. [PMID: 38350725 PMCID: PMC11040347 DOI: 10.1002/advs.202306066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 01/17/2024] [Indexed: 02/15/2024]
Abstract
Acetaminophen overdose is a leading cause of acute liver failure (ALF). Despite the pivotal role of the inflammatory microenvironment in the progression of advanced acetaminophen-induced liver injury (AILI), a comprehensive understanding of the underlying cellular interactions and molecular mechanisms remains elusive. Mas is a G protein-coupled receptor highly expressed by myeloid cells; however, its role in the AILI microenvironment remains to be elucidated. A multidimensional approach, including single-cell RNA sequencing, spatial transcriptomics, and hour-long intravital imaging, is employed to characterize the microenvironment in Mas1 deficient mice at the systemic and cell-specific levels. The characteristic landscape of mouse AILI models involves reciprocal cellular communication among MYC+CD63+ endothelial cells, MMP12+ macrophages, and monocytes, which is maintained by enhanced glycolysis and the NF-κB/TNF-α signaling pathway due to myeloid-Mas deficiency. Importantly, the pathogenic microenvironment is delineated in samples obtained from patients with ALF, demonstrating its clinical relevance. In summary, these findings greatly enhance the understanding of the microenvironment in advanced AILI and offer potential avenues for patient stratification and identification of novel therapeutic targets.
Collapse
Affiliation(s)
- Shuai Chen
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Zhi Lu
- Department of AutomationTsinghua UniversityBeijing100084China
- Institute for Brain and Cognitive SciencesTsinghua UniversityBeijing100084China
| | - Yudong Zhao
- Department of Liver Surgery, Renji Hospital, School of MedicineShanghai Jiao Tong UniversityShanghai200127China
| | - Lu Xia
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Chun Liu
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Siqing Zuo
- Department of AutomationTsinghua UniversityBeijing100084China
- Institute for Brain and Cognitive SciencesTsinghua UniversityBeijing100084China
| | - Manchang Jin
- Institute for Brain and Cognitive SciencesTsinghua UniversityBeijing100084China
- School of Electrical and Information EngineeringTianjin UniversityTianjin300072China
| | - Haoyu Jia
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Shanshan Li
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Shuo Zhang
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Bo Yang
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Zhijing Wang
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Jing Li
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| | - Fei Wang
- Division of GastroenterologySeventh Affiliated Hospital of Sun Yat‐sen UniversityShenzhen518107China
| | - Changqing Yang
- Department of Gastroenterology and HepatologyTongji Hospital, School of Medicine, Tongji UniversityShanghai200092China
| |
Collapse
|
5
|
Alido J, Greene J, Xue Y, Hu G, Gilmore M, Monk KJ, DiBenedictis BT, Davison IG, Tian L, Li Y. Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network. OPTICS EXPRESS 2024; 32:6241-6257. [PMID: 38439332 PMCID: PMC11018337 DOI: 10.1364/oe.514072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 01/16/2024] [Accepted: 01/16/2024] [Indexed: 03/06/2024]
Abstract
Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacerbates the condition of the inverse problem. Here, we develop a scattering simulator that models low-contrast target signals buried in heterogeneous strong background. We then train a deep neural network solely on synthetic data to descatter and reconstruct a 3D volume from a single-shot light-field measurement with low signal-to-background ratio (SBR). We apply this network to our previously developed computational miniature mesoscope and demonstrate the robustness of our deep learning algorithm on scattering phantoms with different scattering conditions. The network can robustly reconstruct emitters in 3D with a 2D measurement of SBR as low as 1.05 and as deep as a scattering length. We analyze fundamental tradeoffs based on network design factors and out-of-distribution data that affect the deep learning model's generalizability to real experimental data. Broadly, we believe that our simulator-based deep learning approach can be applied to a wide range of imaging through scattering techniques where experimental paired training data is lacking.
Collapse
Affiliation(s)
- Jeffrey Alido
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Joseph Greene
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Yujia Xue
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Guorong Hu
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Mitchell Gilmore
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Kevin J. Monk
- Department of Biology, Boston University, Boston, Massachusetts 02215, USA
| | - Brett T. DiBenedictis
- Department of Psychology and Brain Sciences, Boston University, Boston, Massachusetts 02215, USA
| | - Ian G. Davison
- Department of Biology, Boston University, Boston, Massachusetts 02215, USA
| | - Lei Tian
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
- Department of Biomedical Engineering, Boston University, Boston, Massachusetts, 02215, USA
| | - Yunzhe Li
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts, 02215, USA
- Current address: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, California, 94720, USA
| |
Collapse
|
6
|
Chai Y, Qi K, Wu Y, Li D, Tan G, Guo Y, Chu J, Mu Y, Shen C, Wen Q. All-optical interrogation of brain-wide activity in freely swimming larval zebrafish. iScience 2024; 27:108385. [PMID: 38205255 PMCID: PMC10776927 DOI: 10.1016/j.isci.2023.108385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Revised: 09/22/2023] [Accepted: 10/30/2023] [Indexed: 01/12/2024] Open
Abstract
We introduce an all-optical technique that enables volumetric imaging of brain-wide calcium activity and targeted optogenetic stimulation of specific brain regions in unrestrained larval zebrafish. The system consists of three main components: a 3D tracking module, a dual-color fluorescence imaging module, and a real-time activity manipulation module. Our approach uses a sensitive genetically encoded calcium indicator in combination with a long Stokes shift red fluorescence protein as a reference channel, allowing the extraction of Ca2+ activity from signals contaminated by motion artifacts. The method also incorporates rapid 3D image reconstruction and registration, facilitating real-time selective optogenetic stimulation of different regions of the brain. By demonstrating that selective light activation of the midbrain regions in larval zebrafish could reliably trigger biased turning behavior and changes of brain-wide neural activity, we present a valuable tool for investigating the causal relationship between distributed neural circuit dynamics and naturalistic behavior.
Collapse
Affiliation(s)
- Yuming Chai
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Kexin Qi
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Yubin Wu
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Daguang Li
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Guodong Tan
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Yuqi Guo
- Guangdong Provincial Key Laboratory of Biomedical Optical Imaging Technology and Center for Biomedical Optics and Molecular Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Jun Chu
- Guangdong Provincial Key Laboratory of Biomedical Optical Imaging Technology and Center for Biomedical Optics and Molecular Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Yu Mu
- Institute of Neuroscience, State Key Laboratory of Neuroscience, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China
| | - Chen Shen
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| | - Quan Wen
- Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Hefei National Research Center for Physical Sciences at the Microscale, Center for Integrative Imaging, University of Science and Technology of China, Hefei, China
| |
Collapse
|
7
|
Alido J, Greene J, Xue Y, Hu G, Li Y, Gilmore M, Monk KJ, Dibenedictis BT, Davison IG, Tian L. Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network. ARXIV 2023:arXiv:2303.12573v2. [PMID: 36994164 PMCID: PMC10055497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 03/31/2023]
Abstract
Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacerbates the condition of the inverse problem. Here, we develop a scattering simulator that models low-contrast target signals buried in heterogeneous strong background. We then train a deep neural network solely on synthetic data to descatter and reconstruct a 3D volume from a single-shot light-field measurement with low signal-to-background ratio (SBR). We apply this network to our previously developed Computational Miniature Mesoscope and demonstrate the robustness of our deep learning algorithm on scattering phantoms with different scattering conditions. The network can robustly reconstruct emitters in 3D with a 2D measurement of SBR as low as 1.05 and as deep as a scattering length. We analyze fundamental tradeoffs based on network design factors and out-of-distribution data that affect the deep learning model's generalizability to real experimental data. Broadly, we believe that our simulator-based deep learning approach can be applied to a wide range of imaging through scattering techniques where experimental paired training data is lacking.
Collapse
Affiliation(s)
- Jeffrey Alido
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Joseph Greene
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Yujia Xue
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Guorong Hu
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Yunzhe Li
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Mitchell Gilmore
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
| | - Kevin J. Monk
- Department of Biology, Boston University, Boston, MA 02215, USA
| | - Brett T. Dibenedictis
- Department of Psychology and Brain Sciences, Boston University, Boston, MA 02215, USA
| | - Ian G. Davison
- Department of Biology, Boston University, Boston, MA 02215, USA
| | - Lei Tian
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
- Department of Psychology and Brain Sciences, Boston University, Boston, MA 02215, USA
| |
Collapse
|
8
|
Zheng S, Koyama M, Mertz J. Multiplane HiLo microscopy with speckle illumination and non-local means denoising. JOURNAL OF BIOMEDICAL OPTICS 2023; 28:116502. [PMID: 38078150 PMCID: PMC10704089 DOI: 10.1117/1.jbo.28.11.116502] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 10/27/2023] [Accepted: 11/03/2023] [Indexed: 12/18/2023]
Abstract
Significance HiLo microscopy synthesizes an optically sectioned image from two images, one obtained with uniform and another with patterned illumination, such as laser speckle. Speckle-based HiLo has the advantage of being robust to aberrations but is susceptible to residual speckle noise that is difficult to control. We present a computational method to reduce this residual noise without undermining resolution. In addition, we improve the versatility of HiLo microscopy by enabling simultaneous multiplane imaging (here nine planes). Aim Our goal is to perform fast, high-contrast, multiplane imaging with a conventional camera-based fluorescence microscope. Approach Multiplane HiLo imaging is achieved with the use of a single camera and z-splitter prism. Speckle noise reduction is based on the application of a non-local means (NLM) denoising method to perform ensemble averaging of speckle grains. Results We demonstrate the capabilities of multiplane HiLo with NLM denoising both with synthesized data and by imaging cardiac and brain activity in zebrafish larvae at 40 Hz frame rates. Conclusions Multiplane HiLo microscopy aided by NLM denoising provides a simple tool for fast optically sectioned volumetric imaging that can be of general utility for fluorescence imaging applications.
Collapse
Affiliation(s)
- Shuqi Zheng
- Boston University, Department of Electrical and Computer Engineering, Boston, Massachusetts, United States
| | - Minoru Koyama
- University of Toronto, Department of Cell and Systems Biology, Scarborough, Ontario, Canada
| | - Jerome Mertz
- Boston University, Department of Biomedical Engineering, Boston, Massachusetts, United States
| |
Collapse
|
9
|
Wang S, Hu M, Sun B, Pang T, Zhuang Z, Chen T. Dependence of ghost on the incident light angle into dichroic mirror. JOURNAL OF BIOPHOTONICS 2023; 16:e202300190. [PMID: 37545092 DOI: 10.1002/jbio.202300190] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Revised: 07/17/2023] [Accepted: 08/03/2023] [Indexed: 08/08/2023]
Abstract
The dichroic mirror (DM) is a key component in microscope. We found a ghost in the reflection channel of a dual-channel fluorescence microscope and studied the relationship between the ghost and the incidence angle θ into the DM. The DM emission surface reflection generated ghost if the θ is not45 ° . We analyzed the distance and intensity relationship between the ghost and the primary image, which is θ -dependent and was demonstrated by imaging live cells and a stage micrometer. The ghost can be eliminated by placing the DM between objective and tube lens, but not between tube lens and detector, ensuring that the incident light into the DM is approximately parallel. Furthermore, the transmitted light of the DM is shifted towards a longer wavelength with increasing θ . Collectively, microscopists must carefully optimize the θ when designing a microscope to avoid the ghost.
Collapse
Affiliation(s)
- Shuo Wang
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| | - Min Hu
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| | - Beini Sun
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| | - Tian Pang
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| | - Zhengfei Zhuang
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| | - Tongsheng Chen
- Key Laboratory of Laser Life Science, Ministry of Education, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
- Guangdong Key Laboratory of Laser Life Science, College of Biophotonics, South China Normal University, Guangzhou, Guangdong, China
| |
Collapse
|
10
|
Ge S, Li X, Liu Z, Zhao J, Wang W, Li S, Zhang W. Polarization-multiplexed metasurface enabled tri-functional imaging. OPTICS LETTERS 2023; 48:5683-5686. [PMID: 37910733 DOI: 10.1364/ol.502632] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Accepted: 10/08/2023] [Indexed: 11/03/2023]
Abstract
Diffraction-limited focusing imaging, edge-enhanced imaging, and long depth of focus imaging offer crucial technical capabilities for applications such as biological microscopy and surface topography detection. To conveniently and quickly realize the microscopy imaging of different functions, the multifunctional integrated system of microscopy imaging has become an increasingly important research direction. However, conventional microscopes necessitate bulky optical components to switch between these functionalities, suffering from the system's complexity and unstability. Hence, solving the problem of integrating multiple functions within an optical system is a pressing need. In this work, we present an approach using a polarization-multiplexed tri-functional metasurface, capable of realizing the aforementioned imaging functions simply by changing the polarization state of the input and output light, enhancing the system structure's compactness and flexibility. This work offers a new avenue for multifunctional imaging, with potential applications in biomedicine and microscopy imaging.
Collapse
|
11
|
Affiliation(s)
- Xinyang Li
- Department of Automation, Tsinghua University, Beijing, China
| | - Yuanlong Zhang
- Department of Automation, Tsinghua University, Beijing, China
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing, China.
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China.
| |
Collapse
|
12
|
Yun H, Saavedra G, Garcia-Sucerquia J, Tolosa A, Martinez-Corral M, Sanchez-Ortiga E. Practical guide for setting up a Fourier light-field microscope. APPLIED OPTICS 2023; 62:4228-4235. [PMID: 37706910 DOI: 10.1364/ao.491369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Accepted: 04/26/2023] [Indexed: 09/15/2023]
Abstract
A practical guide for the easy implementation of a Fourier light-field microscope is reported. The Fourier light-field concept applied to microscopy allows the capture in real time of a series of 2D orthographic images of microscopic thick dynamic samples. Such perspective images contain spatial and angular information of the light-field emitted by the sample. A feature of this technology is the tight requirement of a double optical conjugation relationship, and also the requirement of NA matching. For these reasons, the Fourier light-field microscope being a non-complex optical system, a clear protocol on how to set up the optical elements accurately is needed. In this sense, this guide is aimed to simplify the implementation process, with an optical bench and off-the-shelf components. This will help the widespread use of this recent technology.
Collapse
|
13
|
Zhao Z, Zhou Y, Liu B, He J, Zhao J, Cai Y, Fan J, Li X, Wang Z, Lu Z, Wu J, Qi H, Dai Q. Two-photon synthetic aperture microscopy for minimally invasive fast 3D imaging of native subcellular behaviors in deep tissue. Cell 2023; 186:2475-2491.e22. [PMID: 37178688 DOI: 10.1016/j.cell.2023.04.016] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 02/21/2023] [Accepted: 04/10/2023] [Indexed: 05/15/2023]
Abstract
Holistic understanding of physio-pathological processes requires noninvasive 3D imaging in deep tissue across multiple spatial and temporal scales to link diverse transient subcellular behaviors with long-term physiogenesis. Despite broad applications of two-photon microscopy (TPM), there remains an inevitable tradeoff among spatiotemporal resolution, imaging volumes, and durations due to the point-scanning scheme, accumulated phototoxicity, and optical aberrations. Here, we harnessed the concept of synthetic aperture radar in TPM to achieve aberration-corrected 3D imaging of subcellular dynamics at a millisecond scale for over 100,000 large volumes in deep tissue, with three orders of magnitude reduction in photobleaching. With its advantages, we identified direct intercellular communications through migrasome generation following traumatic brain injury, visualized the formation process of germinal center in the mouse lymph node, and characterized heterogeneous cellular states in the mouse visual cortex, opening up a horizon for intravital imaging to understand the organizations and functions of biological systems at a holistic level.
Collapse
Affiliation(s)
- Zhifeng Zhao
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China; Hangzhou Zhuoxi Institute of Brain and Intelligence, Hangzhou 311100, China
| | - Yiliang Zhou
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China; Hangzhou Zhuoxi Institute of Brain and Intelligence, Hangzhou 311100, China
| | - Bo Liu
- Tsinghua-Peking Center for Life Sciences, Beijing 100084, China; Laboratory of Dynamic Immunobiology, Institute for Immunology, Tsinghua University, Beijing 100084, China; Department of Basic Medical Sciences, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Jing He
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
| | - Jiayin Zhao
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Yeyi Cai
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
| | - Jingtao Fan
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China
| | - Xinyang Li
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; Hangzhou Zhuoxi Institute of Brain and Intelligence, Hangzhou 311100, China; Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Zilin Wang
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Department of Anesthesiology, the First Medical Center, Chinese PLA General Hospital, Beijing 100853, China
| | - Zhi Lu
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China; Hangzhou Zhuoxi Institute of Brain and Intelligence, Hangzhou 311100, China
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China.
| | - Hai Qi
- Tsinghua-Peking Center for Life Sciences, Beijing 100084, China; Laboratory of Dynamic Immunobiology, Institute for Immunology, Tsinghua University, Beijing 100084, China; Department of Basic Medical Sciences, School of Medicine, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory for Immunological Research on Chronic Diseases, Tsinghua University, Beijing 100084, China; Beijing Frontier Research Center for Biological Structure, Tsinghua University, Beijing 100084, China.
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing 100084, China; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China; Beijing Key Laboratory of Multi-dimension & Multi-scale Computational Photography (MMCP), Tsinghua University, Beijing 100084, China; IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China.
| |
Collapse
|
14
|
Lu Z, Liu Y, Jin M, Luo X, Yue H, Wang Z, Zuo S, Zeng Y, Fan J, Pang Y, Wu J, Yang J, Dai Q. Virtual-scanning light-field microscopy for robust snapshot high-resolution volumetric imaging. Nat Methods 2023; 20:735-746. [PMID: 37024654 PMCID: PMC10172145 DOI: 10.1038/s41592-023-01839-6] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2022] [Accepted: 03/07/2023] [Indexed: 04/08/2023]
Abstract
High-speed three-dimensional (3D) intravital imaging in animals is useful for studying transient subcellular interactions and functions in health and disease. Light-field microscopy (LFM) provides a computational solution for snapshot 3D imaging with low phototoxicity but is restricted by low resolution and reconstruction artifacts induced by optical aberrations, motion and noise. Here, we propose virtual-scanning LFM (VsLFM), a physics-based deep learning framework to increase the resolution of LFM up to the diffraction limit within a snapshot. By constructing a 40 GB high-resolution scanning LFM dataset across different species, we exploit physical priors between phase-correlated angular views to address the frequency aliasing problem. This enables us to bypass hardware scanning and associated motion artifacts. Here, we show that VsLFM achieves ultrafast 3D imaging of diverse processes such as the beating heart in embryonic zebrafish, voltage activity in Drosophila brains and neutrophil migration in the mouse liver at up to 500 volumes per second.
Collapse
Affiliation(s)
- Zhi Lu
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Yu Liu
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Manchang Jin
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Xin Luo
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Huanjing Yue
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Zian Wang
- Department of Automation, Tsinghua University, Beijing, China
| | - Siqing Zuo
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Yunmin Zeng
- Department of Automation, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Jiaqi Fan
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
| | - Yanwei Pang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China.
| | - Jingyu Yang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China.
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China.
| |
Collapse
|
15
|
Zhai J, Jin C, Kong L. Compact, Hybrid Light-Sheet and Fourier Light-Field Microscopy with a Single Objective for High-Speed Volumetric Imaging In Vivo. J Phys Chem A 2023; 127:2873-2879. [PMID: 36926932 DOI: 10.1021/acs.jpca.3c00325] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/18/2023]
Abstract
Volumetric imaging of biodynamics at high spatiotemporal resolutions in vivo is vital in biomedical studies, in which Fourier light field microscopy (FLFM) is a promising technique. However, the commonly used wide-field illumination strategy in FLFM introduces intense out of depth-of-field background, which not only degrades the image quality, but also introduces reconstruction artifacts. Employing light sheet illumination is an effective way to alleviate the background and reduce photobleaching in light-field microscopy. Unfortunately, the introduction of light-sheet illumination often requires an extra objective and precise alignment, which increases the system complexity. Here, we propose the compact, hybrid light-sheet and FLFM (CLS-FLFM), which uses only a single objective to achieve both light-sheet illumination and Fourier light-field imaging simultaneously. With a micromirror under the objective, we focus the light sheet, which ensures selective-volume-illumination, on the imaging plane of the FLFM to perform volumetric imaging. We demonstrate the superior performance of CLS-FLFM in inhibiting background in both structural and dynamical imaging of larval zebrafish in vivo. We envision that CLS-FLFM finds wide applications in high-speed, background-inhibited volumetric imaging of biodynamics in vivo.
Collapse
Affiliation(s)
- Jiazhen Zhai
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Cheng Jin
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Lingjie Kong
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China.,IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
| |
Collapse
|
16
|
Cui Y, Zhang X, Li X, Lin J. Multiscale microscopy to decipher plant cell structure and dynamics. THE NEW PHYTOLOGIST 2023; 237:1980-1997. [PMID: 36477856 DOI: 10.1111/nph.18641] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2022] [Accepted: 11/09/2022] [Indexed: 06/17/2023]
Abstract
New imaging methodologies with high contrast and molecular specificity allow researchers to analyze dynamic processes in plant cells at multiple scales, from single protein and RNA molecules to organelles and cells, to whole organs and tissues. These techniques produce informative images and quantitative data on molecular dynamics to address questions that cannot be answered by conventional biochemical assays. Here, we review selected microscopy techniques, focusing on their basic principles and applications in plant science, discussing the pros and cons of each technique, and introducing methods for quantitative analysis. This review thus provides guidance for plant scientists in selecting the most appropriate techniques to decipher structures and dynamic processes at different levels, from protein dynamics to morphogenesis.
Collapse
Affiliation(s)
- Yaning Cui
- National Engineering Research Center of Tree Breeding and Ecological Restoration, Beijing Forestry University, Beijing, 100083, China
- College of Biological Sciences & Biotechnology, Beijing Forestry University, Beijing, 100083, China
| | - Xi Zhang
- National Engineering Research Center of Tree Breeding and Ecological Restoration, Beijing Forestry University, Beijing, 100083, China
- College of Biological Sciences & Biotechnology, Beijing Forestry University, Beijing, 100083, China
| | - Xiaojuan Li
- National Engineering Research Center of Tree Breeding and Ecological Restoration, Beijing Forestry University, Beijing, 100083, China
- College of Biological Sciences & Biotechnology, Beijing Forestry University, Beijing, 100083, China
| | - Jinxing Lin
- National Engineering Research Center of Tree Breeding and Ecological Restoration, Beijing Forestry University, Beijing, 100083, China
- College of Biological Sciences & Biotechnology, Beijing Forestry University, Beijing, 100083, China
| |
Collapse
|
17
|
Wang X, Xie P, Chen B, Zhang X. Chip-Based High-Dimensional Optical Neural Network. NANO-MICRO LETTERS 2022; 14:221. [PMID: 36374430 PMCID: PMC9663775 DOI: 10.1007/s40820-022-00957-8] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/18/2022] [Accepted: 10/03/2022] [Indexed: 05/16/2023]
Abstract
Parallel multi-thread processing in advanced intelligent processors is the core to realize high-speed and high-capacity signal processing systems. Optical neural network (ONN) has the native advantages of high parallelization, large bandwidth, and low power consumption to meet the demand of big data. Here, we demonstrate the dual-layer ONN with Mach-Zehnder interferometer (MZI) network and nonlinear layer, while the nonlinear activation function is achieved by optical-electronic signal conversion. Two frequency components from the microcomb source carrying digit datasets are simultaneously imposed and intelligently recognized through the ONN. We successfully achieve the digit classification of different frequency components by demultiplexing the output signal and testing power distribution. Efficient parallelization feasibility with wavelength division multiplexing is demonstrated in our high-dimensional ONN. This work provides a high-performance architecture for future parallel high-capacity optical analog computing.
Collapse
Affiliation(s)
- Xinyu Wang
- School of Future Technology, University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Peng Xie
- Department of Engineering Science, University of Oxford, Parks Road, Oxford, OX1 3PJ, UK.
| | - Bohan Chen
- Department of Engineering Science, University of Oxford, Parks Road, Oxford, OX1 3PJ, UK
| | - Xingcai Zhang
- School of Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, 02138, USA.
| |
Collapse
|
18
|
Wijesinghe P, Corsetti S, Chow DJX, Sakata S, Dunning KR, Dholakia K. Experimentally unsupervised deconvolution for light-sheet microscopy with propagation-invariant beams. LIGHT, SCIENCE & APPLICATIONS 2022; 11:319. [PMID: 36319636 PMCID: PMC9626625 DOI: 10.1038/s41377-022-00975-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 08/26/2022] [Accepted: 08/31/2022] [Indexed: 05/25/2023]
Abstract
Deconvolution is a challenging inverse problem, particularly in techniques that employ complex engineered point-spread functions, such as microscopy with propagation-invariant beams. Here, we present a deep-learning method for deconvolution that, in lieu of end-to-end training with ground truths, is trained using known physics of the imaging system. Specifically, we train a generative adversarial network with images generated with the known point-spread function of the system, and combine this with unpaired experimental data that preserve perceptual content. Our method rapidly and robustly deconvolves and super-resolves microscopy images, demonstrating a two-fold improvement in image contrast to conventional deconvolution methods. In contrast to common end-to-end networks that often require 1000-10,000s paired images, our method is experimentally unsupervised and can be trained solely on a few hundred regions of interest. We demonstrate its performance on light-sheet microscopy with propagation-invariant Airy beams in oocytes, preimplantation embryos and excised brain tissue, as well as illustrate its utility for Bessel-beam LSM. This method aims to democratise learned methods for deconvolution, as it does not require data acquisition outwith the conventional imaging protocol.
Collapse
Affiliation(s)
- Philip Wijesinghe
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK.
| | - Stella Corsetti
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK
| | - Darren J X Chow
- Robinson Research Institute, Adelaide Medical School, The University of Adelaide, Adelaide, SA, Australia
- Australian Research Council Centre of Excellence for Nanoscale BioPhotonics, The University of Adelaide, Adelaide, SA, Australia
- Institute for Photonics and Advanced Sensing, The University of Adelaide, Adelaide, SA, Australia
| | - Shuzo Sakata
- Strathclyde Institute of Pharmacy and Biomedical Sciences, University of Strathclyde, Glasgow, G4 0RE, UK
| | - Kylie R Dunning
- Robinson Research Institute, Adelaide Medical School, The University of Adelaide, Adelaide, SA, Australia
- Australian Research Council Centre of Excellence for Nanoscale BioPhotonics, The University of Adelaide, Adelaide, SA, Australia
- Institute for Photonics and Advanced Sensing, The University of Adelaide, Adelaide, SA, Australia
| | - Kishan Dholakia
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK.
- School of Biological Sciences, The University of Adelaide, Adelaide, SA, Australia.
- Department of Physics, College of Science, Yonsei University, Seoul, 03722, South Korea.
| |
Collapse
|
19
|
An integrated imaging sensor for aberration-corrected 3D photography. Nature 2022; 612:62-71. [PMID: 36261533 DOI: 10.1038/s41586-022-05306-8] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Accepted: 09/01/2022] [Indexed: 11/08/2022]
Abstract
Planar digital image sensors facilitate broad applications in a wide range of areas1-5, and the number of pixels has scaled up rapidly in recent years2,6. However, the practical performance of imaging systems is fundamentally limited by spatially nonuniform optical aberrations originating from imperfect lenses or environmental disturbances7,8. Here we propose an integrated scanning light-field imaging sensor, termed a meta-imaging sensor, to achieve high-speed aberration-corrected three-dimensional photography for universal applications without additional hardware modifications. Instead of directly detecting a two-dimensional intensity projection, the meta-imaging sensor captures extra-fine four-dimensional light-field distributions through a vibrating coded microlens array, enabling flexible and precise synthesis of complex-field-modulated images in post-processing. Using the sensor, we achieve high-performance photography up to a gigapixel with a single spherical lens without a data prior, leading to orders-of-magnitude reductions in system capacity and costs for optical imaging. Even in the presence of dynamic atmosphere turbulence, the meta-imaging sensor enables multisite aberration correction across 1,000 arcseconds on an 80-centimetre ground-based telescope without reducing the acquisition speed, paving the way for high-resolution synoptic sky surveys. Moreover, high-density accurate depth maps can be retrieved simultaneously, facilitating diverse applications from autonomous driving to industrial inspections.
Collapse
|
20
|
Cai Y, Wu J, Dai Q. Review on data analysis methods for mesoscale neural imaging in vivo. NEUROPHOTONICS 2022; 9:041407. [PMID: 35450225 PMCID: PMC9010663 DOI: 10.1117/1.nph.9.4.041407] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/30/2021] [Accepted: 03/23/2022] [Indexed: 06/14/2023]
Abstract
Significance: Mesoscale neural imaging in vivo has gained extreme popularity in neuroscience for its capacity of recording large-scale neurons in action. Optical imaging with single-cell resolution and millimeter-level field of view in vivo has been providing an accumulated database of neuron-behavior correspondence. Meanwhile, optical detection of neuron signals is easily contaminated by noises, background, crosstalk, and motion artifacts, while neural-level signal processing and network-level coordinate are extremely complicated, leading to laborious and challenging signal processing demands. The existing data analysis procedure remains unstandardized, which could be daunting to neophytes or neuroscientists without computational background. Aim: We hope to provide a general data analysis pipeline of mesoscale neural imaging shared between imaging modalities and systems. Approach: We divide the pipeline into two main stages. The first stage focuses on extracting high-fidelity neural responses at single-cell level from raw images, including motion registration, image denoising, neuron segmentation, and signal extraction. The second stage focuses on data mining, including neural functional mapping, clustering, and brain-wide network deduction. Results: Here, we introduce the general pipeline of processing the mesoscale neural images. We explain the principles of these procedures and compare different approaches and their application scopes with detailed discussions about the shortcomings and remaining challenges. Conclusions: There are great challenges and opportunities brought by the large-scale mesoscale data, such as the balance between fidelity and efficiency, increasing computational load, and neural network interpretability. We believe that global circuits on single-neuron level will be more extensively explored in the future.
Collapse
Affiliation(s)
- Yeyi Cai
- Tsinghua University, Department of Automation, Beijing, China
| | - Jiamin Wu
- Tsinghua University, Department of Automation, Beijing, China
| | - Qionghai Dai
- Tsinghua University, Department of Automation, Beijing, China
| |
Collapse
|
21
|
Benisty H, Song A, Mishne G, Charles AS. Review of data processing of functional optical microscopy for neuroscience. NEUROPHOTONICS 2022; 9:041402. [PMID: 35937186 PMCID: PMC9351186 DOI: 10.1117/1.nph.9.4.041402] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Accepted: 07/15/2022] [Indexed: 05/04/2023]
Abstract
Functional optical imaging in neuroscience is rapidly growing with the development of optical systems and fluorescence indicators. To realize the potential of these massive spatiotemporal datasets for relating neuronal activity to behavior and stimuli and uncovering local circuits in the brain, accurate automated processing is increasingly essential. We cover recent computational developments in the full data processing pipeline of functional optical microscopy for neuroscience data and discuss ongoing and emerging challenges.
Collapse
Affiliation(s)
- Hadas Benisty
- Yale Neuroscience, New Haven, Connecticut, United States
| | - Alexander Song
- Max Planck Institute for Intelligent Systems, Stuttgart, Germany
| | - Gal Mishne
- UC San Diego, Halıcığlu Data Science Institute, Department of Electrical and Computer Engineering and the Neurosciences Graduate Program, La Jolla, California, United States
| | - Adam S. Charles
- Johns Hopkins University, Kavli Neuroscience Discovery Institute, Center for Imaging Science, Department of Biomedical Engineering, Department of Neuroscience, and Mathematical Institute for Data Science, Baltimore, Maryland, United States
| |
Collapse
|
22
|
Zhai J, Shi R, Fan K, Kong L. Background inhibited and speed-loss-free volumetric imaging in vivo based on structured-illumination Fourier light field microscopy. Front Neurosci 2022; 16:1004228. [PMID: 36248666 PMCID: PMC9558295 DOI: 10.3389/fnins.2022.1004228] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2022] [Accepted: 09/14/2022] [Indexed: 11/17/2022] Open
Abstract
Benefiting from its advantages in fast volumetric imaging for recording biodynamics, Fourier light field microscopy (FLFM) has a wide range of applications in biomedical research, especially in neuroscience. However, the imaging quality of the FLFM is always deteriorated by both the out-of-focus background and the strong scattering in biological samples. Here we propose a structured-illumination and interleaved-reconstruction based Fourier light field microscopy (SI-FLFM), in which we can filter out the background fluorescence in FLFM without sacrificing imaging speed. We demonstrate the superiority of our SI-FLFM in high-speed, background-inhibited volumetric imaging of various biodynamics in larval zebrafish and mice in vivo. The signal-to-background ratio (SBR) is improved by tens of times. And the volumetric imaging speed can be up to 40 Hz, avoiding artifacts caused by temporal under-sampling in conventional structured illumination microscopy. These suggest that our SI-FLFM is suitable for applications of weak fluorescence signals but high imaging speed requirements.
Collapse
Affiliation(s)
- Jiazhen Zhai
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Ruheng Shi
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Kuikui Fan
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Lingjie Kong
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
- *Correspondence: Lingjie Kong,
| |
Collapse
|
23
|
Xue Y, Yang Q, Hu G, Guo K, Tian L. Deep-learning-augmented computational miniature mesoscope. OPTICA 2022; 9:1009-1021. [PMID: 36506462 PMCID: PMC9731182 DOI: 10.1364/optica.464700] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Accepted: 08/02/2022] [Indexed: 05/30/2023]
Abstract
Fluorescence microscopy is essential to study biological structures and dynamics. However, existing systems suffer from a trade-off between field of view (FOV), resolution, and system complexity, and thus cannot fulfill the emerging need for miniaturized platforms providing micron-scale resolution across centimeter-scale FOVs. To overcome this challenge, we developed a computational miniature mesoscope (CM2) that exploits a computational imaging strategy to enable single-shot, 3D high-resolution imaging across a wide FOV in a miniaturized platform. Here, we present CM2 V2, which significantly advances both the hardware and computation. We complement the 3 × 3 microlens array with a hybrid emission filter that improves the imaging contrast by 5×, and design a 3D-printed free-form collimator for the LED illuminator that improves the excitation efficiency by 3×. To enable high-resolution reconstruction across a large volume, we develop an accurate and efficient 3D linear shift-variant (LSV) model to characterize spatially varying aberrations. We then train a multimodule deep learning model called CM2Net, using only the 3D-LSV simulator. We quantify the detection performance and localization accuracy of CM2Net to reconstruct fluorescent emitters under different conditions in simulation. We then show that CM2Net generalizes well to experiments and achieves accurate 3D reconstruction across a ~7-mm FOV and 800-μm depth, and provides ~6-μm lateral and ~25-μm axial resolution. This provides an ~8× better axial resolution and ~1400× faster speed compared to the previous model-based algorithm. We anticipate this simple, low-cost computational miniature imaging system will be useful for many large-scale 3D fluorescence imaging applications.
Collapse
Affiliation(s)
- Yujia Xue
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts 02215, USA
| | - Qianwan Yang
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts 02215, USA
| | - Guorong Hu
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts 02215, USA
| | - Kehan Guo
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts 02215, USA
| | - Lei Tian
- Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts 02215, USA
- Department of Biomedical Engineering, Boston University, Boston, Massachusetts 02215, USA
- Neurophotonics Center, Boston University, Boston, Massachusetts 02215, USA
| |
Collapse
|
24
|
Zhu T, Guo Y, Zhang Y, Lu Z, Lin X, Fang L, Wu J, Dai Q. Noise-robust phase-space deconvolution for light-field microscopy. JOURNAL OF BIOMEDICAL OPTICS 2022; 27:076501. [PMID: 35883238 PMCID: PMC9319196 DOI: 10.1117/1.jbo.27.7.076501] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Accepted: 06/22/2022] [Indexed: 06/15/2023]
Abstract
SIGNIFICANCE Light-field microscopy has achieved success in various applications of life sciences that require high-speed volumetric imaging. However, existing light-field reconstruction algorithms degrade severely in low-light conditions, and the deconvolution process is time-consuming. AIM This study aims to develop a noise robustness phase-space deconvolution method with low computational costs. APPROACH We reformulate the light-field phase-space deconvolution model into the Fourier domain with random-subset ordering and total-variation (TV) regularization. Additionally, we build a time-division-based multicolor light-field microscopy and conduct the three-dimensional (3D) imaging of the heart beating in zebrafish larva at over 95 Hz with a low light dose. RESULTS We demonstrate that this approach reduces computational resources, brings a tenfold speedup, and achieves a tenfold improvement for the noise robustness in terms of SSIM over the state-of-the-art approach. CONCLUSIONS We proposed a phase-space deconvolution algorithm for 3D reconstructions in fluorescence imaging. Compared with the state-of-the-art method, we show significant improvement in both computational effectiveness and noise robustness; we further demonstrated practical application on zebrafish larva with low exposure and low light dose.
Collapse
Affiliation(s)
- Tianyi Zhu
- Tsinghua University, Tsinghua-Berkeley Shenzhen Institute, Beijing, China
| | - Yuduo Guo
- Tsinghua University, Tsinghua-Berkeley Shenzhen Institute, Beijing, China
| | - Yi Zhang
- Tsinghua University, Department of Automation, Beijing, China
| | - Zhi Lu
- Tsinghua University, Department of Automation, Beijing, China
| | - Xing Lin
- Tsinghua University, Department of Automation, Beijing, China
| | - Lu Fang
- Tsinghua University, Department of Electronic Engineering, Beijing, China
| | - Jiamin Wu
- Tsinghua University, Department of Automation, Beijing, China
| | - Qionghai Dai
- Tsinghua University, Department of Automation, Beijing, China
| |
Collapse
|
25
|
Lu Z, Cai Y, Nie Y, Yang Y, Wu J, Dai Q. A practical guide to scanning light-field microscopy with digital adaptive optics. Nat Protoc 2022; 17:1953-1979. [DOI: 10.1038/s41596-022-00703-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2021] [Accepted: 03/24/2022] [Indexed: 11/09/2022]
|
26
|
Galdón L, Saavedra G, Garcia-Sucerquia J, Martínez-Corral M, Sánchez-Ortiga E. Fourier lightfield microscopy: a practical design guide. APPLIED OPTICS 2022; 61:2558-2564. [PMID: 35471323 DOI: 10.1364/ao.453723] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/18/2022] [Accepted: 02/23/2022] [Indexed: 06/14/2023]
Abstract
In this work, a practical guide for the design of a Fourier lightfield microscope is reported. The fundamentals of the Fourier lightfield are presented and condensed on a set of contour plots from which the user can select the design values of the spatial resolution, the field of view, and the depth of field, as function of the specifications of the hardware of the host microscope. This work guides the reader to select the parameters of the infinity-corrected microscope objective, the optical relay lenses, the aperture stop, the microlens array, and the digital camera. A user-friendly graphic calculator is included to ease the design, even to those who are not familiar with the lightfield technology. The guide is aimed to simplify the design process of a Fourier lightfield microscope, which sometimes could be a daunting task, and in this way, to invite the widespread use of this technology. An example of a design and experimental results on imaging different types of samples is also presented.
Collapse
|
27
|
Galdon L, Yun H, Saavedra G, Garcia-Sucerquia J, Barreiro JC, Martinez-Corral M, Sanchez-Ortiga E. Handheld and Cost-Effective Fourier Lightfield Microscope. SENSORS 2022; 22:s22041459. [PMID: 35214359 PMCID: PMC8879591 DOI: 10.3390/s22041459] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Revised: 02/11/2022] [Accepted: 02/12/2022] [Indexed: 11/16/2022]
Abstract
In this work, the design, building, and testing of the most portable, easy-to-build, robust, handheld, and cost-effective Fourier Lightfield Microscope (FLMic) to date is reported. The FLMic is built by means of a surveillance camera lens and additional off-the-shelf optical elements, resulting in a cost-effective FLMic exhibiting all the regular sought features in lightfield microscopy, such as refocusing and gathering 3D information of samples by means of a single-shot approach. The proposed FLMic features reduced dimensions and light weight, which, combined with its low cost, turn the presented FLMic into a strong candidate for in-field application where 3D imaging capabilities are pursued. The use of cost-effective optical elements has a relatively low impact on the optical performance, regarding the figures dictated by the theory, while its price can be at least 100 times lower than that of a regular FLMic. The system operability is tested in both bright-field and fluorescent modes by imaging a resolution target, a honeybee wing, and a knot of dyed cotton fibers.
Collapse
Affiliation(s)
- Laura Galdon
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
| | - Hui Yun
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
| | - Genaro Saavedra
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
| | - Jorge Garcia-Sucerquia
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
- School of Physics, Universidad Nacional de Colombia, Medellin 050034, Colombia
| | - Juan C. Barreiro
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
| | - Manuel Martinez-Corral
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
| | - Emilio Sanchez-Ortiga
- 3D Imaging and Display Laboratory, Department of Optics, Universidad de Valencia, 46100 Burjassot, Spain; (L.G.); (H.Y.); (G.S.); (J.G.-S.); (J.C.B.); (M.M.-C.)
- Correspondence:
| |
Collapse
|