1
|
Han X, Lu X, Li PH, Wang S, Schalek R, Meirovitch Y, Lin Z, Adhinarta J, Murray KD, MacNiven LM, Berger DR, Wu Y, Fang T, Meral ES, Asraf S, Ploegh H, Pfister H, Wei D, Jain V, Trimmer JS, Lichtman JW. Multiplexed volumetric CLEM enabled by scFvs provides insights into the cytology of cerebellar cortex. Nat Commun 2024; 15:6648. [PMID: 39103318 DOI: 10.1038/s41467-024-50411-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 07/01/2024] [Indexed: 08/07/2024] Open
Abstract
Mapping neuronal networks is a central focus in neuroscience. While volume electron microscopy (vEM) can reveal the fine structure of neuronal networks (connectomics), it does not provide molecular information to identify cell types or functions. We developed an approach that uses fluorescent single-chain variable fragments (scFvs) to perform multiplexed detergent-free immunolabeling and volumetric-correlated-light-and-electron-microscopy on the same sample. We generated eight fluorescent scFvs targeting brain markers. Six fluorescent probes were imaged in the cerebellum of a female mouse, using confocal microscopy with spectral unmixing, followed by vEM of the same sample. The results provide excellent ultrastructure superimposed with multiple fluorescence channels. Using this approach, we documented a poorly described cell type, two types of mossy fiber terminals, and the subcellular localization of one type of ion channel. Because scFvs can be derived from existing monoclonal antibodies, hundreds of such probes can be generated to enable molecular overlays for connectomic studies.
Collapse
Affiliation(s)
- Xiaomeng Han
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA.
| | - Xiaotang Lu
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA.
| | | | - Shuohong Wang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Richard Schalek
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Yaron Meirovitch
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Zudi Lin
- School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Jason Adhinarta
- Computer Science Department, Boston College, Chestnut Hill, MA, USA
| | - Karl D Murray
- Department of Physiology and Membrane Biology, University of California Davis School of Medicine, Davis, CA, USA
| | - Leah M MacNiven
- Department of Physiology and Membrane Biology, University of California Davis School of Medicine, Davis, CA, USA
| | - Daniel R Berger
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Yuelong Wu
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Tao Fang
- Program of Cellular and Molecular Medicine, Boston Children's Hospital, Boston, MA, USA
| | | | - Shadnan Asraf
- School of Public Health, University of Massachusetts Amherst, Amherst, MA, USA
| | - Hidde Ploegh
- Program of Cellular and Molecular Medicine, Boston Children's Hospital, Boston, MA, USA
| | - Hanspeter Pfister
- Department of Physiology and Membrane Biology, University of California Davis School of Medicine, Davis, CA, USA
| | - Donglai Wei
- Computer Science Department, Boston College, Chestnut Hill, MA, USA
| | | | - James S Trimmer
- Department of Physiology and Membrane Biology, University of California Davis School of Medicine, Davis, CA, USA
| | - Jeff W Lichtman
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
2
|
Lu Y, Jiang Y, Wang F, Wu H, Hua Y. Electron Microscopic Mapping of Mitochondrial Morphology in the Cochlear Nerve Fibers. J Assoc Res Otolaryngol 2024:10.1007/s10162-024-00957-y. [PMID: 38937328 DOI: 10.1007/s10162-024-00957-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Accepted: 06/12/2024] [Indexed: 06/29/2024] Open
Abstract
To enable nervous system function, neurons are powered in a use-dependent manner by mitochondria undergoing morphological-functional adaptation. In a well-studied model system-the mammalian cochlea, auditory nerve fibers (ANFs) display distinct electrophysiological properties, which is essential for collectively sampling acoustic information of a large dynamic range. How exactly the associated mitochondrial networks are deployed in functionally differentiated ANFs remains scarcely interrogated. Here, we leverage volume electron microscopy and machine-learning-assisted image analysis to phenotype mitochondrial morphology and distribution along ANFs of full-length in the mouse cochlea inner spiral bundle. This reveals greater variance in mitochondrial size with increased ANF habenula to terminal path length. Particularly, we analyzed the ANF terminal-residing mitochondria, which are critical for local calcium uptake during sustained afferent activities. Our results suggest that terminal-specific enrichment of mitochondria, in addition to terminal size and overall mitochondrial abundance of the ANF, correlates with heterogenous mitochondrial contents of the terminal.
Collapse
Affiliation(s)
- Yan Lu
- Department of Otolaryngology-Head and Neck Surgery, Shanghai Ninth People's Hospital, Shanghai, China
- Ear Institute, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Key Laboratory of Translational Medicine on Ear and Nose Diseases, Shanghai, China
- Shanghai Institute of Precision Medicine, Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yi Jiang
- Shanghai Institute of Precision Medicine, Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Fangfang Wang
- Shanghai Institute of Precision Medicine, Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Hao Wu
- Department of Otolaryngology-Head and Neck Surgery, Shanghai Ninth People's Hospital, Shanghai, China
- Ear Institute, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Key Laboratory of Translational Medicine on Ear and Nose Diseases, Shanghai, China
- Shanghai Institute of Precision Medicine, Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yunfeng Hua
- Department of Otolaryngology-Head and Neck Surgery, Shanghai Ninth People's Hospital, Shanghai, China.
- Ear Institute, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Shanghai Key Laboratory of Translational Medicine on Ear and Nose Diseases, Shanghai, China.
- Shanghai Institute of Precision Medicine, Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
| |
Collapse
|
3
|
Zhang W, Yue Y, Pan H, Chen Z, Wang C, Pfister H, Wang W. Marching Windows: Scalable Mesh Generation for Volumetric Data With Multiple Materials. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2024; 30:1728-1742. [PMID: 36455093 PMCID: PMC10980537 DOI: 10.1109/tvcg.2022.3225526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Volumetric data abounds in medical imaging and other fields. With the improved imaging quality and the increased resolution, volumetric datasets are getting so large that the existing tools have become inadequate for processing and analyzing the data. Here we consider the problem of computing tetrahedral meshes to represent large volumetric datasets with labeled multiple materials, which are often encountered in medical imaging or microscopy optical slice tomography. Such tetrahedral meshes are a more compact and expressive geometric representation so are in demand for efficient visualization and simulation of the data, which are impossible if the original large volumetric data are used directly due to the large memory requirement. Existing methods for meshing volumetric data are not scalable for handling large datasets due to their sheer demand on excessively large run-time memory or failure to produce a tet-mesh that preserves the multi-material structure of the original volumetric data. In this article we propose a novel approach, called Marching Windows, that uses a moving window and a disk-swap strategy to reduce the run-time memory footprint, devise a new scheme that guarantees to preserve the topological structure of the original dataset, and adopt an error-guided optimization technique to improve both geometric approximation error and mesh quality. Extensive experiments show that our method is capable of processing very large volumetric datasets beyond the capability of the existing methods and producing tetrahedral meshes of high quality.
Collapse
|
4
|
Franco-Barranco D, Lin Z, Jang WD, Wang X, Shen Q, Yin W, Fan Y, Li M, Chen C, Xiong Z, Xin R, Liu H, Chen H, Li Z, Zhao J, Chen X, Pape C, Conrad R, Nightingale L, de Folter J, Jones ML, Liu Y, Ziaei D, Huschauer S, Arganda-Carreras I, Pfister H, Wei D. Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3956-3971. [PMID: 37768797 PMCID: PMC10753957 DOI: 10.1109/tmi.2023.3320497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/30/2023]
Abstract
In this paper, we present the results of the MitoEM challenge on mitochondria 3D instance segmentation from electron microscopy images, organized in conjunction with the IEEE-ISBI 2021 conference. Our benchmark dataset consists of two large-scale 3D volumes, one from human and one from rat cortex tissue, which are 1,986 times larger than previously used datasets. At the time of paper submission, 257 participants had registered for the challenge, 14 teams had submitted their results, and six teams participated in the challenge workshop. Here, we present eight top-performing approaches from the challenge participants, along with our own baseline strategies. Posterior to the challenge, annotation errors in the ground truth were corrected without altering the final ranking. Additionally, we present a retrospective evaluation of the scoring system which revealed that: 1) challenge metric was permissive with the false positive predictions; and 2) size-based grouping of instances did not correctly categorize mitochondria of interest. Thus, we propose a new scoring system that better reflects the correctness of the segmentation results. Although several of the top methods are compared favorably to our own baselines, substantial errors remain unsolved for mitochondria with challenging morphologies. Thus, the challenge remains open for submission and automatic evaluation, with all volumes available for download.
Collapse
Affiliation(s)
- Daniel Franco-Barranco
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, and also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain
| | - Zudi Lin
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Won-Dong Jang
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Xueying Wang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Qijia Shen
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU Oxford, U.K
| | - Wenjie Yin
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Yutian Fan
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Mingxing Li
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Chang Chen
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Zhiwei Xiong
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Rui Xin
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Hao Liu
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Huai Chen
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Zhili Li
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Jie Zhao
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Xuejin Chen
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Constantin Pape
- European Molecular Biology Laboratory (EMBL), 69117 Heidelberg, Germany. He is now with the Institute for Computer Science, Georg-August-Universität Göttingen, Göttingen, Germany
| | - Ryan Conrad
- Center for Molecular Microscopy, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda, MD 20892 USA, and also with the Cancer Research Technology Program, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | | | | | - Yanling Liu
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | - Dorsa Ziaei
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | - Ignacio Arganda-Carreras
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain, also with the IKERBASQUE, Basque Foundation for Science, 48009 Bilbao, Spain, and also with the Biofisika Institute, 48940 Leioa, Spain
| | - Hanspeter Pfister
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Donglai Wei
- Computer Science Department, Boston College, Chestnut Hill, MA 02467 USA
| |
Collapse
|
5
|
Lauenburg L, Lin Z, Zhang R, Santos MD, Huang S, Arganda-Carreras I, Boyden ES, Pfister H, Wei D. 3D Domain Adaptive Instance Segmentation via Cyclic Segmentation GANs. IEEE J Biomed Health Inform 2023; 27:4018-4027. [PMID: 37252868 PMCID: PMC10481620 DOI: 10.1109/jbhi.2023.3281332] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
3D instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming. Existing works segment a new modality by either deploying pre-trained models optimized on diverse training data or sequentially conducting image translation and segmentation with two relatively independent networks. In this work, we propose a novel Cyclic Segmentation Generative Adversarial Network (CySGAN) that conducts image translation and instance segmentation simultaneously using a unified network with weight sharing. Since the image translation layer can be removed at inference time, our proposed model does not introduce additional computational cost upon a standard segmentation model. For optimizing CySGAN, besides the CycleGAN losses for image translation and supervised losses for the annotated source domain, we also utilize self-supervised and segmentation-based adversarial objectives to enhance the model performance by leveraging unlabeled target domain images. We benchmark our approach on the task of 3D neuronal nuclei segmentation with annotated electron microscopy (EM) images and unlabeled expansion microscopy (ExM) data. The proposed CySGAN outperforms pre-trained generalist models, feature-level domain adaptation models, and the baselines that conduct image translation and segmentation sequentially. Our implementation and the newly collected, densely annotated ExM zebrafish brain nuclei dataset, named NucExM, are publicly available at https://connectomics-bazaar.github.io/proj/CySGAN/index.html.
Collapse
|
6
|
Han X, Lu X, Li PH, Wang S, Schalek R, Meirovitch Y, Lin Z, Adhinarta J, Berger D, Wu Y, Fang T, Meral ES, Asraf S, Ploegh H, Pfister H, Wei D, Jain V, Trimmer JS, Lichtman JW. Multiplexed volumetric CLEM enabled by antibody derivatives provides new insights into the cytology of the mouse cerebellar cortex. RESEARCH SQUARE 2023:rs.3.rs-3121892. [PMID: 37461609 PMCID: PMC10350204 DOI: 10.21203/rs.3.rs-3121892/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/25/2023]
Abstract
Mapping neuronal networks that underlie behavior has become a central focus in neuroscience. While serial section electron microscopy (ssEM) can reveal the fine structure of neuronal networks (connectomics), it does not provide the molecular information that helps identify cell types or their functional properties. Volumetric correlated light and electron microscopy (vCLEM) combines ssEM and volumetric fluorescence microscopy to incorporate molecular labeling into ssEM datasets. We developed an approach that uses small fluorescent single-chain variable fragment (scFv) immuno-probes to perform multiplexed detergent-free immuno-labeling and ssEM on the same samples. We generated eight such fluorescent scFvs that targeted useful markers for brain studies (green fluorescent protein, glial fibrillary acidic protein, calbindin, parvalbumin, voltage-gated potassium channel subfamily A member 2, vesicular glutamate transporter 1, postsynaptic density protein 95, and neuropeptide Y). To test the vCLEM approach, six different fluorescent probes were imaged in a sample of the cortex of a cerebellar lobule (Crus 1), using confocal microscopy with spectral unmixing, followed by ssEM imaging of the same sample. The results show excellent ultrastructure with superimposition of the multiple fluorescence channels. Using this approach we could document a poorly described cell type in the cerebellum, two types of mossy fiber terminals, and the subcellular localization of one type of ion channel. Because scFvs can be derived from existing monoclonal antibodies, hundreds of such probes can be generated to enable molecular overlays for connectomic studies.
Collapse
Affiliation(s)
- Xiaomeng Han
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Xiaotang Lu
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | | | - Shuohong Wang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Richard Schalek
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Yaron Meirovitch
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Zudi Lin
- School of Engineering and Applied Sciences, Harvard University, Cambridge, MA
| | - Jason Adhinarta
- Computer Science Department, Boston College, Chestnut Hill, MA
| | - Daniel Berger
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Yuelong Wu
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| | - Tao Fang
- Program of Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
| | | | - Shadnan Asraf
- School of Public Health, University of Massachusetts Amherst, Amherst, MA
| | - Hidde Ploegh
- Program of Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
| | - Hanspeter Pfister
- School of Engineering and Applied Sciences, Harvard University, Cambridge, MA
| | - Donglai Wei
- Computer Science Department, Boston College, Chestnut Hill, MA
| | | | - James S. Trimmer
- Department of Physiology and Membrane Biology, University of California Davis School of Medicine, Davis, CA
| | - Jeff W. Lichtman
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA
| |
Collapse
|
7
|
Gallusser B, Maltese G, Di Caprio G, Vadakkan TJ, Sanyal A, Somerville E, Sahasrabudhe M, O’Connor J, Weigert M, Kirchhausen T. Deep neural network automated segmentation of cellular structures in volume electron microscopy. J Cell Biol 2023; 222:e202208005. [PMID: 36469001 PMCID: PMC9728137 DOI: 10.1083/jcb.202208005] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 10/03/2022] [Accepted: 11/14/2022] [Indexed: 12/12/2022] Open
Abstract
Volume electron microscopy is an important imaging modality in contemporary cell biology. Identification of intracellular structures is a laborious process limiting the effective use of this potentially powerful tool. We resolved this bottleneck with automated segmentation of intracellular substructures in electron microscopy (ASEM), a new pipeline to train a convolutional neural network to detect structures of a wide range in size and complexity. We obtained dedicated models for each structure based on a small number of sparsely annotated ground truth images from only one or two cells. Model generalization was improved with a rapid, computationally effective strategy to refine a trained model by including a few additional annotations. We identified mitochondria, Golgi apparatus, endoplasmic reticulum, nuclear pore complexes, caveolae, clathrin-coated pits, and vesicles imaged by focused ion beam scanning electron microscopy. We uncovered a wide range of membrane-nuclear pore diameters within a single cell and derived morphological metrics from clathrin-coated pits and vesicles, consistent with the classical constant-growth assembly model.
Collapse
Affiliation(s)
- Benjamin Gallusser
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
- Institute of Bioengineering, School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Giorgio Maltese
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
| | - Giuseppe Di Caprio
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
- Department of Pediatrics, Harvard Medical School, Boston, MA
| | - Tegy John Vadakkan
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
| | - Anwesha Sanyal
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
- Department of Cell Biology, Harvard Medical School, Boston, MA
| | - Elliott Somerville
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
| | - Mihir Sahasrabudhe
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
- Université Paris-Saclay, CentraleSupélec, Mathématiques et Informatique pour la Complexité et les Systèmes, Gif-sur-Yvette, France
| | - Justin O’Connor
- Department of Biological Chemistry & Molecular Pharmacology, Harvard Medical School, Boston, MA
| | - Martin Weigert
- Institute of Bioengineering, School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Tom Kirchhausen
- Program in Cellular and Molecular Medicine, Boston Children’s Hospital, Boston, MA
- Department of Pediatrics, Harvard Medical School, Boston, MA
- Department of Cell Biology, Harvard Medical School, Boston, MA
| |
Collapse
|
8
|
Yang J, Shi R, Wei D, Liu Z, Zhao L, Ke B, Pfister H, Ni B. MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci Data 2023; 10:41. [PMID: 36658144 PMCID: PMC9852451 DOI: 10.1038/s41597-022-01721-8] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Accepted: 09/26/2022] [Indexed: 01/20/2023] Open
Abstract
We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D. All images are pre-processed into a small size of 28 × 28 (2D) or 28 × 28 × 28 (3D) with the corresponding classification labels so that no background knowledge is required for users. Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label). The resulting dataset, consisting of 708,069 2D images and 9,998 3D images in total, could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning. We benchmark several baseline methods on MedMNIST v2, including 2D/3D neural networks and open-source/commercial AutoML tools. The data and code are publicly available at https://medmnist.com/ .
Collapse
Affiliation(s)
- Jiancheng Yang
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| | - Rui Shi
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| | - Donglai Wei
- grid.208226.c0000 0004 0444 7053Boston College, Chestnut Hill, MA USA
| | - Zequan Liu
- grid.1957.a0000 0001 0728 696XRWTH Aachen University, Aachen, Germany
| | - Lin Zhao
- grid.8547.e0000 0001 0125 2443Department of Endocrinology and Metabolism, Fudan Institute of Metabolic Diseases, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Bilian Ke
- grid.16821.3c0000 0004 0368 8293Department of Ophthalmology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | | | - Bingbing Ni
- grid.16821.3c0000 0004 0368 8293Shanghai Jiao Tong University, Shanghai, China
| |
Collapse
|
9
|
Hong B, Liu J, Zhai H, Liu J, Shen L, Chen X, Xie Q, Han H. Joint reconstruction of neuron and ultrastructure via connectivity consensus in electron microscope volumes. BMC Bioinformatics 2022; 23:453. [PMID: 36316652 PMCID: PMC9623997 DOI: 10.1186/s12859-022-04991-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Accepted: 10/17/2022] [Indexed: 11/14/2022] Open
Abstract
Background Nanoscale connectomics, which aims to map the fine connections between neurons with synaptic-level detail, has attracted increasing attention in recent years. Currently, the automated reconstruction algorithms in electron microscope volumes are in great demand. Most existing reconstruction methodologies for cellular and subcellular structures are independent, and exploring the inter-relationships between structures will contribute to image analysis. The primary goal of this research is to construct a joint optimization framework to improve the accuracy and efficiency of neural structure reconstruction algorithms. Results In this investigation, we introduce the concept of connectivity consensus between cellular and subcellular structures based on biological domain knowledge for neural structure agglomeration problems. We propose a joint graph partitioning model for solving ultrastructural and neuronal connections to overcome the limitations of connectivity cues at different levels. The advantage of the optimization model is the simultaneous reconstruction of multiple structures in one optimization step. The experimental results on several public datasets demonstrate that the joint optimization model outperforms existing hierarchical agglomeration algorithms. Conclusions We present a joint optimization model by connectivity consensus to solve the neural structure agglomeration problem and demonstrate its superiority to existing methods. The intention of introducing connectivity consensus between different structures is to build a suitable optimization model that makes the reconstruction goals more consistent with biological plausible and domain knowledge. This idea can inspire other researchers to optimize existing reconstruction algorithms and other areas of biological data analysis.
Collapse
Affiliation(s)
- Bei Hong
- grid.410726.60000 0004 1797 8419School of Artificial Intelligence, School of Future Technology, University of Chinese Academy of Sciences, Beijing, China ,grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Jing Liu
- grid.410726.60000 0004 1797 8419School of Artificial Intelligence, School of Future Technology, University of Chinese Academy of Sciences, Beijing, China ,grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Hao Zhai
- grid.410726.60000 0004 1797 8419School of Artificial Intelligence, School of Future Technology, University of Chinese Academy of Sciences, Beijing, China ,grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Jiazheng Liu
- grid.410726.60000 0004 1797 8419School of Artificial Intelligence, School of Future Technology, University of Chinese Academy of Sciences, Beijing, China ,grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Lijun Shen
- grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Xi Chen
- grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Qiwei Xie
- grid.28703.3e0000 0000 9040 3743Research Base of Beijing Modern Manufacturing Development, Beijing University of Technology, Beijing, China
| | - Hua Han
- grid.410726.60000 0004 1797 8419School of Artificial Intelligence, School of Future Technology, University of Chinese Academy of Sciences, Beijing, China ,grid.9227.e0000000119573309National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China ,grid.507732.4CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China
| |
Collapse
|
10
|
Kievits AJ, Lane R, Carroll EC, Hoogenboom JP. How innovations in methodology offer new prospects for volume electron microscopy. J Microsc 2022; 287:114-137. [PMID: 35810393 PMCID: PMC9546337 DOI: 10.1111/jmi.13134] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Revised: 06/29/2022] [Accepted: 07/06/2022] [Indexed: 11/29/2022]
Abstract
Detailed knowledge of biological structure has been key in understanding biology at several levels of organisation, from organs to cells and proteins. Volume electron microscopy (volume EM) provides high resolution 3D structural information about tissues on the nanometre scale. However, the throughput rate of conventional electron microscopes has limited the volume size and number of samples that can be imaged. Recent improvements in methodology are currently driving a revolution in volume EM, making possible the structural imaging of whole organs and small organisms. In turn, these recent developments in image acquisition have created or stressed bottlenecks in other parts of the pipeline, like sample preparation, image analysis and data management. While the progress in image analysis is stunning due to the advent of automatic segmentation and server‐based annotation tools, several challenges remain. Here we discuss recent trends in volume EM, emerging methods for increasing throughput and implications for sample preparation, image analysis and data management.
Collapse
Affiliation(s)
- Arent J. Kievits
- Imaging Physics Delft University of Technology Delft 2624CJ The Netherlands
| | - Ryan Lane
- Imaging Physics Delft University of Technology Delft 2624CJ The Netherlands
| | | | | |
Collapse
|
11
|
Matejek B, Wei D, Chen T, Tsourakakis CE, Mitzenmacher M, Pfister H. Edge-colored directed subgraph enumeration on the connectome. Sci Rep 2022; 12:11349. [PMID: 35790766 PMCID: PMC9256670 DOI: 10.1038/s41598-022-15027-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2022] [Accepted: 06/16/2022] [Indexed: 11/24/2022] Open
Abstract
Following significant advances in image acquisition, synapse detection, and neuronal segmentation in connectomics, researchers have extracted an increasingly diverse set of wiring diagrams from brain tissue. Neuroscientists frequently represent these wiring diagrams as graphs with nodes corresponding to a single neuron and edges indicating synaptic connectivity. The edges can contain "colors" or "labels", indicating excitatory versus inhibitory connections, among other things. By representing the wiring diagram as a graph, we can begin to identify motifs, the frequently occurring subgraphs that correspond to specific biological functions. Most analyses on these wiring diagrams have focused on hypothesized motifs-those we expect to find. However, one of the goals of connectomics is to identify biologically-significant motifs that we did not previously hypothesize. To identify these structures, we need large-scale subgraph enumeration to find the frequencies of all unique motifs. Exact subgraph enumeration is a computationally expensive task, particularly in the edge-dense wiring diagrams. Furthermore, most existing methods do not differentiate between types of edges which can significantly affect the function of a motif. We propose a parallel, general-purpose subgraph enumeration strategy to count motifs in the connectome. Next, we introduce a divide-and-conquer community-based subgraph enumeration strategy that allows for enumeration per brain region. Lastly, we allow for differentiation of edges by types to better reflect the underlying biological properties of the graph. We demonstrate our results on eleven connectomes and publish for future analyses extensive overviews for the 26 trillion subgraphs enumerated that required approximately 9.25 years of computation time.
Collapse
Affiliation(s)
- Brian Matejek
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA.
- Computer Science Laboratory, SRI International, Washington, DC, USA.
| | - Donglai Wei
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
- Department of Computer Science, Boston College, Chestnut Hill, MA, USA
| | - Tianyi Chen
- Department of Computer Science, Boston University, Boston, MA, USA
| | - Charalampos E Tsourakakis
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
- Department of Computer Science, Boston University, Boston, MA, USA
- ISI Foundation, Turin, Italy
| | - Michael Mitzenmacher
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Hanspeter Pfister
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| |
Collapse
|
12
|
Kohl P, Greiner J, Rog-Zielinska EA. Electron microscopy of cardiac 3D nanodynamics: form, function, future. Nat Rev Cardiol 2022; 19:607-619. [PMID: 35396547 DOI: 10.1038/s41569-022-00677-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 02/04/2022] [Indexed: 11/09/2022]
Abstract
The 3D nanostructure of the heart, its dynamic deformation during cycles of contraction and relaxation, and the effects of this deformation on cell function remain largely uncharted territory. Over the past decade, the first inroads have been made towards 3D reconstruction of heart cells, with a native resolution of around 1 nm3, and of individual molecules relevant to heart function at a near-atomic scale. These advances have provided access to a new generation of data and have driven the development of increasingly smart, artificial intelligence-based, deep-learning image-analysis algorithms. By high-pressure freezing of cardiomyocytes with millisecond accuracy after initiation of an action potential, pseudodynamic snapshots of contraction-induced deformation of intracellular organelles can now be captured. In combination with functional studies, such as fluorescence imaging, exciting insights into cardiac autoregulatory processes at nano-to-micro scales are starting to emerge. In this Review, we discuss the progress in this fascinating new field to highlight the fundamental scientific insight that has emerged, based on technological breakthroughs in biological sample preparation, 3D imaging and data analysis; to illustrate the potential clinical relevance of understanding 3D cardiac nanodynamics; and to predict further progress that we can reasonably expect to see over the next 10 years.
Collapse
Affiliation(s)
- Peter Kohl
- Institute for Experimental Cardiovascular Medicine, University Heart Center and Faculty of Medicine, University of Freiburg, Freiburg, Germany.,Faculty of Engineering, University of Freiburg, Freiburg, Germany.,Centre for Integrative Biological Signalling Studies (CIBSS), University of Freiburg, Freiburg, Germany
| | - Joachim Greiner
- Institute for Experimental Cardiovascular Medicine, University Heart Center and Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Eva A Rog-Zielinska
- Institute for Experimental Cardiovascular Medicine, University Heart Center and Faculty of Medicine, University of Freiburg, Freiburg, Germany.
| |
Collapse
|
13
|
Pennington A, King ONF, Tun WM, Ho EML, Luengo I, Darrow MC, Basham M. SuRVoS 2: Accelerating Annotation and Segmentation for Large Volumetric Bioimage Workflows Across Modalities and Scales. Front Cell Dev Biol 2022; 10:842342. [PMID: 35433703 PMCID: PMC9011330 DOI: 10.3389/fcell.2022.842342] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2021] [Accepted: 02/22/2022] [Indexed: 12/29/2022] Open
Abstract
As sample preparation and imaging techniques have expanded and improved to include a variety of options for larger sized and numbers of samples, the bottleneck in volumetric imaging is now data analysis. Annotation and segmentation are both common, yet difficult, data analysis tasks which are required to bring meaning to the volumetric data. The SuRVoS application has been updated and redesigned to provide access to both manual and machine learning-based segmentation and annotation techniques, including support for crowd sourced data. Combining adjacent, similar voxels (supervoxels) provides a mechanism for speeding up segmentation both in the painting of annotation and by training a segmentation model on a small amount of annotation. The support for layers allows multiple datasets to be viewed and annotated together which, for example, enables the use of correlative data (e.g. crowd-sourced annotations or secondary imaging techniques) to guide segmentation. The ability to work with larger data on high-performance servers with GPUs has been added through a client-server architecture and the Pytorch-based image processing and segmentation server is flexible and extensible, and allows the implementation of deep learning-based segmentation modules. The client side has been built around Napari allowing integration of SuRVoS into an ecosystem for open-source image analysis while the server side has been built with cloud computing and extensibility through plugins in mind. Together these improvements to SuRVoS provide a platform for accelerating the annotation and segmentation of volumetric and correlative imaging data across modalities and scales.
Collapse
Affiliation(s)
| | | | - Win Min Tun
- Diamond Light Source Ltd., Didcot, United Kingdom
| | | | | | | | - Mark Basham
- Diamond Light Source Ltd., Didcot, United Kingdom
- The Rosalind Franklin Institute, Didcot, United Kingdom
- *Correspondence: Mark Basham,
| |
Collapse
|
14
|
Quesada J, Sathidevi L, Liu R, Ahad N, Jackson JM, Azabou M, Xiao J, Liding C, Jin M, Urzay C, Gray-Roncal W, Johnson EC, Dyer EL. MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 2022; 35:5299-5314. [PMID: 38414814 PMCID: PMC10898440] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 02/29/2024]
Abstract
There are multiple scales of abstraction from which we can describe the same image, depending on whether we are focusing on fine-grained details or a more global attribute of the image. In brain mapping, learning to automatically parse images to build representations of both small-scale features (e.g., the presence of cells or blood vessels) and global properties of an image (e.g., which brain region the image comes from) is a crucial and open challenge. However, most existing datasets and benchmarks for neuroanatomy consider only a single downstream task at a time. To bridge this gap, we introduce a new dataset, annotations, and multiple downstream tasks that provide diverse ways to readout information about brain structure and architecture from the same image. Our multi-task neuroimaging benchmark (MTNeuro) is built on volumetric, micrometer-resolution X-ray microtomography images spanning a large thalamocortical section of mouse brain, encompassing multiple cortical and subcortical regions. We generated a number of different prediction challenges and evaluated several supervised and self-supervised models for brain-region prediction and pixel-level semantic segmentation of microstructures. Our experiments not only highlight the rich heterogeneity of this dataset, but also provide insights into how self-supervised approaches can be used to learn representations that capture multiple attributes of a single image and perform well on a variety of downstream tasks. Datasets, code, and pre-trained baseline models are provided at: https://mtneuro.github.io/.
Collapse
Affiliation(s)
| | | | - Ran Liu
- Georgia Institute of Technology
| | | | | | | | | | | | | | | | | | | | | |
Collapse
|
15
|
Franco-Barranco D, Muñoz-Barrutia A, Arganda-Carreras I. Stable Deep Neural Network Architectures for Mitochondria Segmentation on Electron Microscopy Volumes. Neuroinformatics 2021; 20:437-450. [PMID: 34855126 PMCID: PMC9546980 DOI: 10.1007/s12021-021-09556-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/15/2021] [Indexed: 11/29/2022]
Abstract
Electron microscopy (EM) allows the identification of intracellular organelles such as mitochondria, providing insights for clinical and scientific studies. In recent years, a number of novel deep learning architectures have been published reporting superior performance, or even human-level accuracy, compared to previous approaches on public mitochondria segmentation datasets. Unfortunately, many of these publications make neither the code nor the full training details public, leading to reproducibility issues and dubious model comparisons. Thus, following a recent code of best practices in the field, we present an extensive study of the state-of-the-art architectures and compare them to different variations of U-Net-like models for this task. To unveil the impact of architectural novelties, a common set of pre- and post-processing operations has been implemented and tested with each approach. Moreover, an exhaustive sweep of hyperparameters has been performed, running each configuration multiple times to measure their stability. Using this methodology, we found very stable architectures and training configurations that consistently obtain state-of-the-art results in the well-known EPFL Hippocampus mitochondria segmentation dataset and outperform all previous works on two other available datasets: Lucchi++ and Kasthuri++. The code and its documentation are publicly available at https://github.com/danifranco/EM_Image_Segmentation.
Collapse
Affiliation(s)
- Daniel Franco-Barranco
- Donostia International Physics Center (DIPC), Donostia-San Sebastián, Spain. .,Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Donostia-San Sebastian, Spain.
| | - Arrate Muñoz-Barrutia
- Universidad Carlos III de Madrid, Leganés, Spain.,Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, Spain
| | - Ignacio Arganda-Carreras
- Donostia International Physics Center (DIPC), Donostia-San Sebastián, Spain.,Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Donostia-San Sebastian, Spain.,Ikerbasque, Basque Foundation for Science, Bilbao, Spain
| |
Collapse
|
16
|
Liu Y, Ji S. CleftNet: Augmented Deep Learning for Synaptic Cleft Detection From Brain Electron Microscopy. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:3507-3518. [PMID: 34129494 PMCID: PMC8674103 DOI: 10.1109/tmi.2021.3089547] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Detecting synaptic clefts is a crucial step to investigate the biological function of synapses. The volume electron microscopy (EM) allows the identification of synaptic clefts by photoing EM images with high resolution and fine details. Machine learning approaches have been employed to automatically predict synaptic clefts from EM images. In this work, we propose a novel and augmented deep learning model, known as CleftNet, for improving synaptic cleft detection from brain EM images. We first propose two novel network components, known as the feature augmentor and the label augmentor, for augmenting features and labels to improve cleft representations. The feature augmentor can fuse global information from inputs and learn common morphological patterns in clefts, leading to augmented cleft features. In addition, it can generate outputs with varying dimensions, making it flexible to be integrated in any deep network. The proposed label augmentor augments the label of each voxel from a value to a vector, which contains both the segmentation label and boundary label. This allows the network to learn important shape information and to produce more informative cleft representations. Based on the proposed feature augmentor and label augmentor, We build the CleftNet as a U-Net like network. The effectiveness of our methods is evaluated on both external and internal tasks. Our CleftNet currently ranks #1 on the external task of the CREMI open challenge. In addition, both quantitative and qualitative results in the internal tasks show that our method outperforms the baseline approaches significantly.
Collapse
|
17
|
Li Z, Chen X, Zhao J, Xiong Z. Contrastive Learning for Mitochondria Segmentation. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:3496-3500. [PMID: 34891993 DOI: 10.1109/embc46164.2021.9630350] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Mitochondria segmentation in electron microscopy images is essential in neuroscience. However, due to the image degradation during the imaging process, the large variety of mitochondrial structures, as well as the presence of noise, artifacts and other sub-cellular structures, mitochondria segmentation is very challenging. In this paper, we propose a novel and effective contrastive learning framework to learn a better feature representation from hard examples to improve segmentation. Specifically, we adopt a point sampling strategy to pick out representative pixels from hard examples in the training phase. Based on these sampled pixels, we introduce a pixel-wise label-based contrastive loss which consists of a similarity loss term and a consistency loss term. The similarity term can increase the similarity of pixels from the same class and the separability of pixels from different classes in feature space, while the consistency term is able to enhance the sensitivity of the 3D model to changes in image content from frame to frame. We demonstrate the effectiveness of our method on MitoEM dataset as well as FIB-SEM dataset and show better or on par with state-of-the-art results.
Collapse
|
18
|
Whole-cell organelle segmentation in volume electron microscopy. Nature 2021; 599:141-146. [PMID: 34616042 DOI: 10.1038/s41586-021-03977-3] [Citation(s) in RCA: 96] [Impact Index Per Article: 32.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2020] [Accepted: 08/31/2021] [Indexed: 12/12/2022]
Abstract
Cells contain hundreds of organelles and macromolecular assemblies. Obtaining a complete understanding of their intricate organization requires the nanometre-level, three-dimensional reconstruction of whole cells, which is only feasible with robust and scalable automatic methods. Here, to support the development of such methods, we annotated up to 35 different cellular organelle classes-ranging from endoplasmic reticulum to microtubules to ribosomes-in diverse sample volumes from multiple cell types imaged at a near-isotropic resolution of 4 nm per voxel with focused ion beam scanning electron microscopy (FIB-SEM)1. We trained deep learning architectures to segment these structures in 4 nm and 8 nm per voxel FIB-SEM volumes, validated their performance and showed that automatic reconstructions can be used to directly quantify previously inaccessible metrics including spatial interactions between cellular components. We also show that such reconstructions can be used to automatically register light and electron microscopy images for correlative studies. We have created an open data and open-source web repository, 'OpenOrganelle', to share the data, computer code and trained models, which will enable scientists everywhere to query and further improve automatic reconstruction of these datasets.
Collapse
|
19
|
Konishi K, Nonaka T, Takei S, Ohta K, Nishioka H, Suga M. Reducing Manual Operation Time to Obtain a Segmentation Learning Model for Volume Electron Microscopy Using Stepwise Deep Learning With Manual Correction. Microscopy (Oxf) 2021; 70:526-535. [PMID: 34259875 DOI: 10.1093/jmicro/dfab025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Revised: 06/18/2021] [Accepted: 07/14/2021] [Indexed: 11/13/2022] Open
Abstract
Three-dimensional (3D) observation of a biological sample using serial-section electron microscopy is widely used. However, organelle segmentation requires a significant amount of manual time. Therefore, several studies have been conducted to improve their efficiency. One such promising method is 3D deep learning (DL), which is highly accurate. However, the creation of training data for 3D DL still requires manual time and effort. In this study, we developed a highly efficient integrated image segmentation tool that includes stepwise DL with manual correction. The tool has four functions: efficient tracers for annotation, model training/inference for organelle segmentation using a lightweight convolutional neural network, efficient proofreading, and model refinement. We applied this tool to increase the training data step by step (stepwise annotation method) to segment the mitochondria in the cells of the cerebral cortex. We found that the stepwise annotation method reduced the manual operation time by one-third compared with that of the fully manual method, where all the training data were created manually. Moreover, we demonstrated that the F1 score, the metric of segmentation accuracy, was 0.9 by training the 3D DL model with these training data. The stepwise annotation method using this tool and the 3D DL model improved the segmentation efficiency for various organelles.
Collapse
Affiliation(s)
- Kohki Konishi
- Nikon Corporation, Research and Development Division, 471, Nagaodai, Sakae, Yokohama, Kanagawa 244-8533, JP
| | - Takao Nonaka
- Nikon Corporation, Optical Engineering Division, 471, Nagaodai, Sakae, Yokohama, Kanagawa 244-8533, JP
| | - Shunsuke Takei
- Nikon Corporation, Healthcare Business Unit, 471, Nagaodai, Sakae, Yokohama, Kanagawa 244-8533, JP
| | - Keisuke Ohta
- Kurume University, School of Medicine, 67 Asahi-machi, Kurume, Fukuoka 830-0011
| | - Hideo Nishioka
- JEOL Co. LTD., Application group, EM Business unit, 3-1-2 Musashino, Akishima, Tokyo 196-8558, JP
| | - Mitsuo Suga
- JEOL Co. LTD., Strategy Management Planning Office, 3-1-2, Musashino, Akishima, Tokyo 196-8558, JP
| |
Collapse
|
20
|
Luo Z, Wang Y, Liu S, Peng J. Hierarchical Encoder-Decoder With Soft Label-Decomposition for Mitochondria Segmentation in EM Images. Front Neurosci 2021; 15:687832. [PMID: 34248488 PMCID: PMC8264135 DOI: 10.3389/fnins.2021.687832] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Accepted: 05/31/2021] [Indexed: 11/13/2022] Open
Abstract
Semantic segmentation of mitochondria from electron microscopy (EM) images is an essential step to obtain reliable morphological statistics about mitochondria. However, automatically delineating plenty of mitochondria of varied shapes from complex backgrounds with sufficient accuracy is challenging. To address these challenges, we develop a hierarchical encoder-decoder network (HED-Net), which has a three-level nested U-shape architecture to capture rich contextual information. Given the irregular shape of mitochondria, we introduce a novel soft label-decomposition strategy to exploit shape knowledge in manual labels. Rather than simply using the ground truth label maps as the unique supervision in the model training, we introduce additional subcategory-aware supervision by softly decomposing each manual label map into two complementary label maps according to mitochondria's ovality. The three label maps are integrated with our HED-Net to supervise the model training. While the original label map guides the network to segment all the mitochondria of varied shapes, the auxiliary label maps guide the network to segment subcategories of mitochondria of circular shape and elliptic shape, respectively, which are much more manageable tasks. Extensive experiments on two public benchmarks show that our HED-Net performs favorably against state-of-the-art methods.
Collapse
Affiliation(s)
- Zhengrong Luo
- College of Computer Science and Technology, Huaqiao University, Xiamen, China
| | - Ye Wang
- School of Statistics, Huaqiao University, Xiamen, China
| | - Shikun Liu
- College of Computer Science and Technology, Huaqiao University, Xiamen, China
| | - Jialin Peng
- College of Computer Science and Technology, Huaqiao University, Xiamen, China
| |
Collapse
|