1
|
Kaynig V, Vazquez-Reina A, Knowles-Barley S, Roberts M, Jones TR, Kasthuri N, Miller E, Lichtman J, Pfister H. Large-scale automatic reconstruction of neuronal processes from electron microscopy images. Med Image Anal 2015; 22:77-88. [PMID: 25791436 DOI: 10.1016/j.media.2015.02.001] [Citation(s) in RCA: 47] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2014] [Revised: 11/02/2014] [Accepted: 02/06/2015] [Indexed: 01/14/2023]
Abstract
Automated sample preparation and electron microscopy enables acquisition of very large image data sets. These technical advances are of special importance to the field of neuroanatomy, as 3D reconstructions of neuronal processes at the nm scale can provide new insight into the fine grained structure of the brain. Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets. In this paper we present a pipeline that provides state-of-the art reconstruction performance while scaling to data sets in the GB-TB range. First, we train a random forest classifier on interactive sparse user annotations. The classifier output is combined with an anisotropic smoothing prior in a Conditional Random Field framework to generate multiple segmentation hypotheses per image. These segmentations are then combined into geometrically consistent 3D objects by segmentation fusion. We provide qualitative and quantitative evaluation of the automatic segmentation and demonstrate large-scale 3D reconstructions of neuronal processes from a 27,000 μm(3) volume of brain tissue over a cube of 30 μm in each dimension corresponding to 1000 consecutive image sections. We also introduce Mojo, a proofreading tool including semi-automated correction of merge errors based on sparse user scribbles.
Collapse
Affiliation(s)
- Verena Kaynig
- School of Engineering and Applied Sciences, Harvard University, United States
| | - Amelio Vazquez-Reina
- School of Engineering and Applied Sciences, Harvard University, United States; Department of Computer Science at Tufts University, United States
| | | | - Mike Roberts
- School of Engineering and Applied Sciences, Harvard University, United States
| | - Thouis R Jones
- School of Engineering and Applied Sciences, Harvard University, United States; Department of Molecular and Cellular Biology, Harvard University, United States
| | - Narayanan Kasthuri
- Department of Molecular and Cellular Biology, Harvard University, United States
| | - Eric Miller
- Department of Computer Science at Tufts University, United States
| | - Jeff Lichtman
- Department of Molecular and Cellular Biology, Harvard University, United States
| | - Hanspeter Pfister
- School of Engineering and Applied Sciences, Harvard University, United States
| |
Collapse
|
2
|
Liu T, Jones C, Seyedhosseini M, Tasdizen T. A modular hierarchical approach to 3D electron microscopy image segmentation. J Neurosci Methods 2014; 226:88-102. [PMID: 24491638 PMCID: PMC3970427 DOI: 10.1016/j.jneumeth.2014.01.022] [Citation(s) in RCA: 49] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2013] [Revised: 01/09/2014] [Accepted: 01/13/2014] [Indexed: 11/22/2022]
Abstract
The study of neural circuit reconstruction, i.e., connectomics, is a challenging problem in neuroscience. Automated and semi-automated electron microscopy (EM) image analysis can be tremendously helpful for connectomics research. In this paper, we propose a fully automatic approach for intra-section segmentation and inter-section reconstruction of neurons using EM images. A hierarchical merge tree structure is built to represent multiple region hypotheses and supervised classification techniques are used to evaluate their potentials, based on which we resolve the merge tree with consistency constraints to acquire final intra-section segmentation. Then, we use a supervised learning based linking procedure for the inter-section neuron reconstruction. Also, we develop a semi-automatic method that utilizes the intermediate outputs of our automatic algorithm and achieves intra-segmentation with minimal user intervention. The experimental results show that our automatic method can achieve close-to-human intra-segmentation accuracy and state-of-the-art inter-section reconstruction accuracy. We also show that our semi-automatic method can further improve the intra-segmentation accuracy.
Collapse
Affiliation(s)
- Ting Liu
- Scientific Computing and Imaging Institute, University of Utah, United States; School of Computing, University of Utah, United States
| | - Cory Jones
- Scientific Computing and Imaging Institute, University of Utah, United States; Department of Electrical and Computer Engineering, University of Utah, United States
| | - Mojtaba Seyedhosseini
- Scientific Computing and Imaging Institute, University of Utah, United States; Department of Electrical and Computer Engineering, University of Utah, United States
| | - Tolga Tasdizen
- Scientific Computing and Imaging Institute, University of Utah, United States; School of Computing, University of Utah, United States; Department of Electrical and Computer Engineering, University of Utah, United States.
| |
Collapse
|
3
|
Jagadeesh V, Manjunath BS, Anderson J, Jones BW, Marc R, Fisher SK. Robust segmentation based tracing using an adaptive wrapper for inducing priors. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2013; 22:4952-4963. [PMID: 23996562 DOI: 10.1109/tip.2013.2280002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
Segmentation based tracing algorithms detect the extent and borders of an object in a given frame IZ by propagating results from frames I1 ≤ z < Z. Although application specific tracers have been forthcoming, techniques that automatically adapt across applications have been less explored. We approach this problem by learning a prior model on topological dynamics that encourages segmentation transitions across frames that are most likely for a given application. Further, we augment a generic tracing technique with a locality sensitive prior derived from dense optic flow fields for deformation guidance. The proposed approach comprises two stages where the generic tracer initially yields multiple segmentation transitions when its parameters are perturbed, and the learnt topology prior subsequently propagates high scoring segmentations. Because the learnt topology model wraps around a generic tracer and adapts it by setting its free parameters, the need for careful parameter tuning is completely obviated. Through extensive experimental validation in surveillance, biological and medical image datasets, we verify the applicability of the proposed model while demonstrating good tracing performance under severe clutter.
Collapse
|
4
|
Jurrus E, Watanabe S, Giuly RJ, Paiva ARC, Ellisman MH, Jorgensen EM, Tasdizen T. Semi-automated neuron boundary detection and nonbranching process segmentation in electron microscopy images. Neuroinformatics 2013; 11:5-29. [PMID: 22644867 PMCID: PMC3914654 DOI: 10.1007/s12021-012-9149-y] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
Abstract
Neuroscientists are developing new imaging techniques and generating large volumes of data in an effort to understand the complex structure of the nervous system. The complexity and size of this data makes human interpretation a labor-intensive task. To aid in the analysis, new segmentation techniques for identifying neurons in these feature rich datasets are required. This paper presents a method for neuron boundary detection and nonbranching process segmentation in electron microscopy images and visualizing them in three dimensions. It combines both automated segmentation techniques with a graphical user interface for correction of mistakes in the automated process. The automated process first uses machine learning and image processing techniques to identify neuron membranes that deliniate the cells in each two-dimensional section. To segment nonbranching processes, the cell regions in each two-dimensional section are connected in 3D using correlation of regions between sections. The combination of this method with a graphical user interface specially designed for this purpose, enables users to quickly segment cellular processes in large volumes.
Collapse
Affiliation(s)
- Elizabeth Jurrus
- Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA.
| | | | | | | | | | | | | |
Collapse
|
5
|
Abstract
Advances in computational geometric modeling, imaging, and simulation let researchers build and test models of increasing complexity, generating unprecedented amounts of data. As recent research in biomedical applications illustrates, visualization will be critical in making this vast amount of data usable; it's also fundamental to understanding models of complex phenomena.
Collapse
|
6
|
Jaume S, Knobe K, Newton RR, Schlimbach F, Blower M, Reid RC. A multiscale parallel computing architecture for automated segmentation of the brain connectome. IEEE Trans Biomed Eng 2011; 59:35-8. [PMID: 21926011 DOI: 10.1109/tbme.2011.2168396] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
Several groups in neurobiology have embarked into deciphering the brain circuitry using large-scale imaging of a mouse brain and manual tracing of the connections between neurons. Creating a graph of the brain circuitry, also called a connectome, could have a huge impact on the understanding of neurodegenerative diseases such as Alzheimer's disease. Although considerably smaller than a human brain, a mouse brain already exhibits one billion connections and manually tracing the connectome of a mouse brain can only be achieved partially. This paper proposes to scale up the tracing by using automated image segmentation and a parallel computing approach designed for domain experts. We explain the design decisions behind our parallel approach and we present our results for the segmentation of the vasculature and the cell nuclei, which have been obtained without any manual intervention.
Collapse
Affiliation(s)
- Sylvain Jaume
- Department of Neurobiology, Harvard Medical School, Boston, MA 02115, USA.
| | | | | | | | | | | |
Collapse
|
7
|
Machines that learn to segment images: a crucial technology for connectomics. Curr Opin Neurobiol 2011; 20:653-66. [PMID: 20801638 PMCID: PMC2975605 DOI: 10.1016/j.conb.2010.07.004] [Citation(s) in RCA: 111] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2010] [Accepted: 07/07/2010] [Indexed: 11/21/2022]
Abstract
Connections between neurons can be found by checking whether synapses exist at points of contact, which in turn are determined by neural shapes. Finding these shapes is a special case of image segmentation, which is laborious for humans and would ideally be performed by computers. New metrics properly quantify the performance of a computer algorithm using its disagreement with 'true' segmentations of example images. New machine learning methods search for segmentation algorithms that minimize such metrics. These advances have reduced computer errors dramatically. It should now be faster for a human to correct the remaining errors than to segment an image manually. Further reductions in human effort are expected, and crucial for finding connectomes more complex than that of Caenorhabditis elegans.
Collapse
|
8
|
Roberts M, Jeong WK, Vázquez-Reina A, Unger M, Bischof H, Lichtman J, Pfister H. Neural Process Reconstruction from Sparse User Scribbles. LECTURE NOTES IN COMPUTER SCIENCE 2011; 14:621-8. [DOI: 10.1007/978-3-642-23623-5_78] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
9
|
Jurrus E, Paiva ARC, Watanabe S, Anderson JR, Jones BW, Whitaker RT, Jorgensen EM, Marc RE, Tasdizen T. Detection of neuron membranes in electron microscopy images using a serial neural network architecture. Med Image Anal 2010; 14:770-83. [PMID: 20598935 PMCID: PMC2930201 DOI: 10.1016/j.media.2010.06.002] [Citation(s) in RCA: 70] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2009] [Revised: 04/15/2010] [Accepted: 06/03/2010] [Indexed: 11/24/2022]
Abstract
Study of nervous systems via the connectome, the map of connectivities of all neurons in that system, is a challenging problem in neuroscience. Towards this goal, neurobiologists are acquiring large electron microscopy datasets. However, the shear volume of these datasets renders manual analysis infeasible. Hence, automated image analysis methods are required for reconstructing the connectome from these very large image collections. Segmentation of neurons in these images, an essential step of the reconstruction pipeline, is challenging because of noise, anisotropic shapes and brightness, and the presence of confounding structures. The method described in this paper uses a series of artificial neural networks (ANNs) in a framework combined with a feature vector that is composed of image intensities sampled over a stencil neighborhood. Several ANNs are applied in series allowing each ANN to use the classification context provided by the previous network to improve detection accuracy. We develop the method of serial ANNs and show that the learned context does improve detection over traditional ANNs. We also demonstrate advantages over previous membrane detection methods. The results are a significant step towards an automated system for the reconstruction of the connectome.
Collapse
Affiliation(s)
- Elizabeth Jurrus
- Scientific Computing and Imaging Institute, Salt Lake City, UT 84112, United States.
| | | | | | | | | | | | | | | | | |
Collapse
|
10
|
Kaynig V, Fuchs TJ, Buhmann JM. Geometrical consistent 3D tracing of neuronal processes in ssTEM data. ACTA ACUST UNITED AC 2010; 13:209-16. [PMID: 20879317 DOI: 10.1007/978-3-642-15745-5_26] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/21/2023]
Abstract
In neuroanatomy, automatic geometry extraction of neurons from electron microscopy images is becoming one of the main limiting factors in getting new insights into the functional structure of the brain. We propose a novel framework for tracing neuronal processes over serial sections for 3d reconstructions. The automatic processing pipeline combines the probabilistic output of a random forest classifier with geometrical consistency constraints which take the geometry of whole sections into account. Our experiments demonstrate significant improvement over grouping by Euclidean distance, reducing the split and merge error per object by a factor of two.
Collapse
Affiliation(s)
- Verena Kaynig
- Department of Computer Science, ETH Zurich, Switzerland.
| | | | | |
Collapse
|
11
|
Mishchenko Y, Hu T, Spacek J, Mendenhall J, Harris KM, Chklovskii DB. Ultrastructural analysis of hippocampal neuropil from the connectomics perspective. Neuron 2010; 67:1009-20. [PMID: 20869597 DOI: 10.1016/j.neuron.2010.08.014] [Citation(s) in RCA: 201] [Impact Index Per Article: 14.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/27/2010] [Indexed: 11/17/2022]
Abstract
Complete reconstructions of vertebrate neuronal circuits on the synaptic level require new approaches. Here, serial section transmission electron microscopy was automated to densely reconstruct four volumes, totaling 670 μm(3), from the rat hippocampus as proving grounds to determine when axo-dendritic proximities predict synapses. First, in contrast with Peters' rule, the density of axons within reach of dendritic spines did not predict synaptic density along dendrites because the fraction of axons making synapses was variable. Second, an axo-dendritic touch did not predict a synapse; nevertheless, the density of synapses along a hippocampal dendrite appeared to be a universal fraction, 0.2, of the density of touches. Finally, the largest touch between an axonal bouton and spine indicated the site of actual synapses with about 80% precision but would miss about half of all synapses. Thus, it will be difficult to predict synaptic connectivity using data sets missing ultrastructural details that distinguish between axo-dendritic touches and bona fide synapses.
Collapse
Affiliation(s)
- Yuriy Mishchenko
- Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | | | | | | | | | | |
Collapse
|
12
|
Mishchenko Y. Automation of 3D reconstruction of neural tissue from large volume of conventional serial section transmission electron micrographs. J Neurosci Methods 2008; 176:276-89. [PMID: 18834903 PMCID: PMC2948845 DOI: 10.1016/j.jneumeth.2008.09.006] [Citation(s) in RCA: 56] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2008] [Revised: 08/29/2008] [Accepted: 09/01/2008] [Indexed: 11/16/2022]
Abstract
We describe an approach for automation of the process of reconstruction of neural tissue from serial section transmission electron micrographs. Such reconstructions require 3D segmentation of individual neuronal processes (axons and dendrites) performed in densely packed neuropil. We first detect neuronal cell profiles in each image in a stack of serial micrographs with multi-scale ridge detector. Short breaks in detected boundaries are interpolated using anisotropic contour completion formulated in fuzzy-logic framework. Detected profiles from adjacent sections are linked together based on cues such as shape similarity and image texture. Thus obtained 3D segmentation is validated by human operators in computer-guided proofreading process. Our approach makes possible reconstructions of neural tissue at final rate of about 5 microm3/manh, as determined primarily by the speed of proofreading. To date we have applied this approach to reconstruct few blocks of neural tissue from different regions of rat brain totaling over 1000microm3, and used these to evaluate reconstruction speed, quality, error rates, and presence of ambiguous locations in neuropil ssTEM imaging data.
Collapse
Affiliation(s)
- Yuriy Mishchenko
- Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA.
| |
Collapse
|