Detection of neuron membranes in electron microscopy images using a serial neural network architecture

A neuron image segmentation method based Deep Boltzmann Machine and CV model

Neuron image segmentation has wide applications and important potential values for neuroscience research. Due to the complexity of the submicroscopic structure of neurons cells and the defects of the image quality such as anisotropy, boundary loss and blurriness in electron microscopy-based (EM) imaging, and one faces a challenge in efficient automated segmenting large-scale neuron image 3D datasets, which is an essential prerequisite front-end process for the reconstruction of neuron circuits. Here, a neuron image segmentation method by combining Chan-Vest (CV) model with Deep Boltzmann Machine (DBM) is proposed, and a generative model is used to model and generate the target shape, it take this as a prior information to add global target shape feature constraint to the energy function of CV model, and the shape priori information is fused to assist neuron image segmentation. We applied our method to two 3D-EM datasets from different types of nerve tissue and achieved the best performance consistently across two classical evaluation metrics of neuron segmentation accuracy, namely Variation of Information (VoI) and Adaptive Rand Index (ARI). Experimental results show that the fusion algorithm has high segmentation accuracy, strong robustness, and can characterize the sub-microstructure information of neuron images well.

ScanEV - A neural network-based tool for the automated detection of extracellular vesicles in TEM images

Transmission electron microscopy (TEM) is the most widely accepted method for visualization of extracellular vesicles (EVs), and particularly, exosomes. TEM images provide us with information about the size and morphology of the EVs. We have developed an online tool ScanEV (Scanner for the Extracellular Vesicles, available at https://bioeng.ru/scanev), for the rapid and automated processing of such images. ScanEV is based on a convolutional neural network; it detects the «cup-shaped» particles in the images and calculates their morphometric parameters. This tool will be useful for researchers who study EVs and use TEM for their characterization.

Supervised machine learning for coronary artery lumen segmentation in intravascular ultrasound images

Intravascular ultrasound (IVUS) has been widely used to capture cross sectional lumen frames of inner wall of coronary arteries. This kind of medical imaging modalities is capable of providing detailed and significant information of lumen contour shape, which is very important for clinical diagnosis and analysis of cardiovascular diseases. Numerous learning based techniques have recently become very popular for coronary artery segmentation due to their impressive results. In this work, a supervised machine learning method for coronary artery lumen segmentation with high accuracy and minimal user interaction is designed. The fully discriminative lumen segmentation method jointly learning a classifier the weak learners rely on and the features of the classifier is developed. Additionally, the theoretical supports of the Gradient Boosting framework used in this work and its quadratic approximation are presented. The proposed algorithm is tested on the public datasets of boundary detection of lumen in IVUS challenge held in MICCAI 2011 and achieves a higher average Jaccard similarity of 96.8% and a lower mean error distance of 0.55 (in Cartesian coordinates), which shows higher accuracy compared to the existing learning based methods. Moreover, three real patient IVUS datasets are used to evaluate the performance of the proposed coronary artery lumen segmentation algorithm, which is shown to achieve lower percent error of lumen area of 1.861% ± 0.965%, 1.968% ± 0.864%, and 1.671% ± 0.584%, respectively, compared to the manually measured lumen area (ground truth). The proposed lumen segmentation method is found to be superior to the latest learning based segmentation techniques. Given the efficiency and robustness, our method has great potential in IVUS images processing and coronary artery segmentation and quantification. NOVELTY STATEMENT: The main contributions are summarized in the following aspects: A detailed review of related work about learning based coronary artery lumen segmentation in intravascular ultrasound images is presented. A fully discriminative lumen segmentation method jointly learning a classifier our weak learners rely on and the features of the classifier is developed. The theoretical supports of the Gradient Boosting framework and its quadratic approximation used in this work are presented.

FCNN-based axon segmentation for convection-enhanced delivery optimization

PURPOSE

Glioblastoma multiforme treatment is a challenging task in clinical oncology. Convection- enhanced delivery (CED) is showing encouraging but still suboptimal results due to drug leakages. Numerical models can predict drug distribution within the brain, but require retrieving brain physical properties, such as the axon diameter distribution (ADD), through axon architecture analysis. The goal of this work was to provide an automatic, accurate and fast method for axon segmentation in electronic microscopy images based on fully convolutional neural network (FCNN) as to allow automatic ADD computation.

METHODS

The segmentation was performed using a residual FCNN inspired by U-Net and Resnet. The FCNN training was performed exploiting mini-batch gradient descent and the Adam optimizer. The Dice coefficient was chosen as loss function.

RESULTS

The proposed segmentation method achieved results comparable with already existing methods for axon segmentation in terms of Information Theoretic Scoring ([Formula: see text]) with a faster training (5 h on the deployed GPU) and without requiring heavy post-processing (testing time was 0.2 s with a non-optimized code). The ADDs computed from the segmented and ground-truth images were statistically equivalent.

CONCLUSIONS

The algorithm proposed in this work allowed fast and accurate axon segmentation and ADD computation, showing promising performance for brain microstructure analysis for CED delivery optimization.

An Adversarial and Densely Dilated Network for Connectomes Segmentation

Automatic reconstructing of neural circuits in the brain is one of the most crucial studies in neuroscience. Connectomes segmentation plays an important role in reconstruction from electron microscopy (EM) images; however, it is rather challenging due to highly anisotropic shapes with inferior quality and various thickness. In our paper, we propose a novel connectomes segmentation framework called adversarial and densely dilated network (ADDN) to address these issues. ADDN is based on the conditional Generative Adversarial Network (cGAN) structure which is the latest advance in machine learning with power to generate images similar to the ground truth especially when the training data is limited. Specifically, we design densely dilated network (DDN) as the segmentor to allow a deeper architecture and larger receptive fields for more accurate segmentation. Discriminator is trained to distinguish generated segmentation from manual segmentation. During training, such adversarial loss function is optimized together with dice loss. Extensive experimental results demonstrate that our ADDN is effective for such connectomes segmentation task, helping to retrieve more accurate segmentation and attenuate the blurry effects of generated boundary map. Our method obtains state-of-the-art performance while requiring less computation on ISBI 2012 EM dataset and mouse piriform cortex dataset.

Flexible Learning-Free Segmentation and Reconstruction of Neural Volumes

Imaging is a dominant strategy for data collection in neuroscience, yielding stacks of images that often scale to gigabytes of data for a single experiment. Machine learning algorithms from computer vision can serve as a pair of virtual eyes that tirelessly processes these images, automatically detecting and identifying microstructures. Unlike learning methods, our Flexible Learning-free Reconstruction of Imaged Neural volumes (FLoRIN) pipeline exploits structure-specific contextual clues and requires no training. This approach generalizes across different modalities, including serially-sectioned scanning electron microscopy (sSEM) of genetically labeled and contrast enhanced processes, spectral confocal reflectance (SCoRe) microscopy, and high-energy synchrotron X-ray microtomography (μCT) of large tissue volumes. We deploy the FLoRIN pipeline on newly published and novel mouse datasets, demonstrating the high biological fidelity of the pipeline’s reconstructions. FLoRIN reconstructions are of sufficient quality for preliminary biological study, for example examining the distribution and morphology of cells or extracting single axons from functional data. Compared to existing supervised learning methods, FLoRIN is one to two orders of magnitude faster and produces high-quality reconstructions that are tolerant to noise and artifacts, as is shown qualitatively and quantitatively.

Neuron Segmentation With High-Level Biological Priors

We present a novel approach to the problem of neuron segmentation in image volumes acquired by an electron microscopy. Existing methods, such as agglomerative or correlation clustering, rely solely on boundary evidence and have problems where such an evidence is lacking (e.g., incomplete staining) or ambiguous (e.g., co-located cell and mitochondria membranes). We investigate if these difficulties can be overcome by means of sparse region appearance cues that differentiate between pre- and postsynaptic neuron segments in mammalian neural tissue. We combine these cues with the traditional boundary evidence in the asymmetric multiway cut (AMWC) model, which simultaneously solves the partitioning and the semantic region labeling problems. We show that AMWC problems over superpixel graphs can be solved to global optimality with a cutting plane approach, and that the introduction of semantic class priors leads to significantly better segmentations.

Detection of axonal synapses in 3D two-photon images

Studies of structural plasticity in the brain often require the detection and analysis of axonal synapses (boutons). To date, bouton detection has been largely manual or semi-automated, relying on a step that traces the axons before detection the boutons. If tracing the axon fails, the accuracy of bouton detection is compromised. In this paper, we propose a new algorithm that does not require tracing the axon to detect axonal boutons in 3D two-photon images taken from the mouse cortex. To find the most appropriate techniques for this task, we compared several well-known algorithms for interest point detection and feature descriptor generation. The final algorithm proposed has the following main steps: (1) a Laplacian of Gaussian (LoG) based feature enhancement module to accentuate the appearance of boutons; (2) a Speeded Up Robust Features (SURF) interest point detector to find candidate locations for feature extraction; (3) non-maximum suppression to eliminate candidates that were detected more than once in the same local region; (4) generation of feature descriptors based on Gabor filters; (5) a Support Vector Machine (SVM) classifier, trained on features from labelled data, and was used to distinguish between bouton and non-bouton candidates. We found that our method achieved a Recall of 95%, Precision of 76%, and F1 score of 84% within a new dataset that we make available for accessing bouton detection. On average, Recall and F1 score were significantly better than the current state-of-the-art method, while Precision was not significantly different. In conclusion, in this article we demonstrate that our approach, which is independent of axon tracing, can detect boutons to a high level of accuracy, and improves on the detection performance of existing approaches. The data and code (with an easy to use GUI) used in this article are available from open source repositories.

Residual Deconvolutional Networks for Brain Electron Microscopy Image Segmentation

Accurate reconstruction of anatomical connections between neurons in the brain using electron microscopy (EM) images is considered to be the gold standard for circuit mapping. A key step in obtaining the reconstruction is the ability to automatically segment neurons with a precision close to human-level performance. Despite the recent technical advances in EM image segmentation, most of them rely on hand-crafted features to some extent that are specific to the data, limiting their ability to generalize. Here, we propose a simple yet powerful technique for EM image segmentation that is trained end-to-end and does not rely on prior knowledge of the data. Our proposed residual deconvolutional network consists of two information pathways that capture full-resolution features and contextual information, respectively. We showed that the proposed model is very effective in achieving the conflicting goals in dense output prediction; namely preserving full-resolution predictions and including sufficient contextual information. We applied our method to the ongoing open challenge of 3D neurite segmentation in EM images. Our method achieved one of the top results on this open challenge. We demonstrated the generality of our technique by evaluating it on the 2D neurite segmentation challenge dataset where consistently high performance was obtained. We thus expect our method to generalize well to other dense output prediction problems.

Multiscale Centerline Detection

Finding the centerline and estimating the radius of linear structures is a critical first step in many applications, ranging from road delineation in 2D aerial images to modeling blood vessels, lung bronchi, and dendritic arbors in 3D biomedical image stacks. Existing techniques rely either on filters designed to respond to ideal cylindrical structures or on classification techniques. The former tend to become unreliable when the linear structures are very irregular while the latter often has difficulties distinguishing centerline locations from neighboring ones, thus losing accuracy. We solve this problem by reformulating centerline detection in terms of a regression problem. We first train regressors to return the distances to the closest centerline in scale-space, and we apply them to the input images or volumes. The centerlines and the corresponding scale then correspond to the regressors local maxima, which can be easily identified. We show that our method outperforms state-of-the-art techniques for various 2D and 3D datasets. Moreover, our approach is very generic and also performs well on contour detection. We show an improvement above recent contour detection algorithms on the BSDS500 dataset.

Semantic Image Segmentation with Contextual Hierarchical Models

Semantic segmentation is the problem of assigning an object label to each pixel. It unifies the image segmentation and object recognition problems. The importance of using contextual information in semantic segmentation frameworks has been widely realized in the field. We propose a contextual framework, called contextual hierarchical model (CHM), which learns contextual information in a hierarchical framework for semantic segmentation. At each level of the hierarchy, a classifier is trained based on downsampled input images and outputs of previous levels. Our model then incorporates the resulting multi-resolution contextual information into a classifier to segment the input image at original resolution. This training strategy allows for optimization of a joint posterior probability at multiple resolutions through the hierarchy. Contextual hierarchical model is purely based on the input image patches and does not make use of any fragments or shape examples. Hence, it is applicable to a variety of problems such as object segmentation and edge detection. We demonstrate that CHM performs at par with state-of-the-art on Stanford background and Weizmann horse datasets. It also outperforms state-of-the-art edge detection methods on NYU depth dataset and achieves state-of-the-art on Berkeley segmentation dataset (BSDS 500).

Extracellular space preservation aids the connectomic analysis of neural circuits

Dense connectomic mapping of neuronal circuits is limited by the time and effort required to analyze 3D electron microscopy (EM) datasets. Algorithms designed to automate image segmentation suffer from substantial error rates and require significant manual error correction. Any improvement in segmentation error rates would therefore directly reduce the time required to analyze 3D EM data. We explored preserving extracellular space (ECS) during chemical tissue fixation to improve the ability to segment neurites and to identify synaptic contacts. ECS preserved tissue is easier to segment using machine learning algorithms, leading to significantly reduced error rates. In addition, we observed that electrical synapses are readily identified in ECS preserved tissue. Finally, we determined that antibodies penetrate deep into ECS preserved tissue with only minimal permeabilization, thereby enabling correlated light microscopy (LM) and EM studies. We conclude that preservation of ECS benefits multiple aspects of the connectomic analysis of neural circuits.

DOI:http://dx.doi.org/10.7554/eLife.08206.001

The brain consists of billions of neurons that are connected into many different circuits. Mapping the connections between these neurons could help researchers to understand how the nervous system works. A method commonly used to do so is to preserve samples of brain tissue in chemical fixatives, and then image thin slices of this tissue using powerful microscopes.

As each tissue sample contains many neurons, computer algorithms have been developed to analyze the microscope images and automatically identify the neurons and the connections they make. However, these algorithms often make 'segmentation errors' that researchers need to manually correct: for example, overlapping neurons may be counted as a single neuron, or a neuron may be marked into several segments. Correcting these errors is a time-consuming and tedious task that limits how much of the brain can be currently mapped. Future algorithm improvements will hopefully reduce the number of errors; Pallotto, Watkins et al. explored an alternative approach by making the images themselves easier to analyze using existing algorithms.

The chemicals used to preserve brain tissue often suck out the fluids that fill the spaces between the neurons, causing these 'extracellular spaces' to shrink. Pallotto, Watkins et al. have now developed a method of preserving tissue that maintains more space between the neurons, and used this method to preserve samples of mouse brain with different amounts of extracellular space. Pallotto, Watkins et al. found that the algorithm used to analyze the images of these samples made far fewer segmentation errors on samples that contained more extracellular space. It was also easier to identify the connections between different neurons in these samples. The next challenge will be to extend these methods to preserving extracellular space across whole brains.

DOI:http://dx.doi.org/10.7554/eLife.08206.002

An efficient conditional random field approach for automatic and interactive neuron segmentation

We present a new graphical-model-based method for automatic and interactive segmentation of neuron structures from electron microscopy (EM) images. For automated reconstruction, our learning based model selects a collection of nodes from a hierarchical merging tree as the proposed segmentation. More specifically, this is achieved by training a conditional random field (CRF) whose underlying graph is the watershed merging tree. The maximum a posteriori (MAP) prediction of the CRF is the output segmentation. Our results are comparable to the results of state-of-the-art methods. Furthermore, both the inference and the training are very efficient as the graph is tree-structured. The problem of neuron segmentation requires extremely high segmentation quality. Therefore, proofreading, namely, interactively correcting mistakes of the automatic method, is a necessary module in the pipeline. Based on our efficient tree-structured inference algorithm, we develop an interactive segmentation framework which only selects locations where the model is uncertain for a user to proofread. The uncertainty is measured by the marginals of the graphical model. Only giving a limited number of choices makes the user interaction very efficient. Based on user corrections, our framework modifies the merging tree and thus improves the segmentation globally.

A context-aware delayed agglomeration framework for electron microscopy segmentation

Electron Microscopy (EM) image (or volume) segmentation has become significantly important in recent years as an instrument for connectomics. This paper proposes a novel agglomerative framework for EM segmentation. In particular, given an over-segmented image or volume, we propose a novel framework for accurately clustering regions of the same neuron. Unlike existing agglomerative methods, the proposed context-aware algorithm divides superpixels (over-segmented regions) of different biological entities into different subsets and agglomerates them separately. In addition, this paper describes a “delayed” scheme for agglomerative clustering that postpones some of the merge decisions, pertaining to newly formed bodies, in order to generate a more confident boundary prediction. We report significant improvements attained by the proposed approach in segmentation accuracy over existing standard methods on 2D and 3D datasets.

Modeling brain circuitry over a wide range of scales

If we are ever to unravel the mysteries of brain function at its most fundamental level, we will need a precise understanding of how its component neurons connect to each other. Electron Microscopes (EM) can now provide the nanometer resolution that is needed to image synapses, and therefore connections, while Light Microscopes (LM) see at the micrometer resolution required to model the 3D structure of the dendritic network. Since both the topology and the connection strength are integral parts of the brain's wiring diagram, being able to combine these two modalities is critically important. In fact, these microscopes now routinely produce high-resolution imagery in such large quantities that the bottleneck becomes automated processing and interpretation, which is needed for such data to be exploited to its full potential. In this paper, we briefly review the Computer Vision techniques we have developed at EPFL to address this need. They include delineating dendritic arbors from LM imagery, segmenting organelles from EM, and combining the two into a consistent representation.

Large-scale automatic reconstruction of neuronal processes from electron microscopy images

Automated sample preparation and electron microscopy enables acquisition of very large image data sets. These technical advances are of special importance to the field of neuroanatomy, as 3D reconstructions of neuronal processes at the nm scale can provide new insight into the fine grained structure of the brain. Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets. In this paper we present a pipeline that provides state-of-the art reconstruction performance while scaling to data sets in the GB-TB range. First, we train a random forest classifier on interactive sparse user annotations. The classifier output is combined with an anisotropic smoothing prior in a Conditional Random Field framework to generate multiple segmentation hypotheses per image. These segmentations are then combined into geometrically consistent 3D objects by segmentation fusion. We provide qualitative and quantitative evaluation of the automatic segmentation and demonstrate large-scale 3D reconstructions of neuronal processes from a 27,000 μm(3) volume of brain tissue over a cube of 30 μm in each dimension corresponding to 1000 consecutive image sections. We also introduce Mojo, a proofreading tool including semi-automated correction of merge errors based on sparse user scribbles.

Automated method for the segmentation and morphometry of nerve fibers in large-scale CARS images of spinal cord tissue

A fully automated method for large-scale segmentation of nerve fibers from coherent anti-Stokes Raman scattering (CARS) microscopy images is presented. The method is specifically designed for CARS images of transverse cross sections of nervous tissue but is also suitable for use with standard light microscopy images. After a detailed description of the two-part segmentation algorithm, its accuracy is quantified by comparing the resulting binary images to manually segmented images. We then demonstrate the ability of our method to retrieve morphological data from CARS images of nerve tissue. Finally, we present the segmentation of a large mosaic of CARS images covering more than half the area of a mouse spinal cord cross section and show evidence of clusters of neurons with similar g-ratios throughout the spinal cord.

Input clustering and the microscale structure of local circuits

The recent development of powerful tools for high-throughput mapping of synaptic networks promises major advances in understanding brain function. One open question is how circuits integrate and store information. Competing models based on random vs. structured connectivity make distinct predictions regarding the dendritic addressing of synaptic inputs. In this article we review recent experimental tests of one of these models, the input clustering hypothesis. Across circuits, brain regions and species, there is growing evidence of a link between synaptic co-activation and dendritic location, although this finding is not universal. The functional implications of input clustering and future challenges are discussed.

A modular hierarchical approach to 3D electron microscopy image segmentation

The study of neural circuit reconstruction, i.e., connectomics, is a challenging problem in neuroscience. Automated and semi-automated electron microscopy (EM) image analysis can be tremendously helpful for connectomics research. In this paper, we propose a fully automatic approach for intra-section segmentation and inter-section reconstruction of neurons using EM images. A hierarchical merge tree structure is built to represent multiple region hypotheses and supervised classification techniques are used to evaluate their potentials, based on which we resolve the merge tree with consistency constraints to acquire final intra-section segmentation. Then, we use a supervised learning based linking procedure for the inter-section neuron reconstruction. Also, we develop a semi-automatic method that utilizes the intermediate outputs of our automatic algorithm and achieves intra-segmentation with minimal user intervention. The experimental results show that our automatic method can achieve close-to-human intra-segmentation accuracy and state-of-the-art inter-section reconstruction accuracy. We also show that our semi-automatic method can further improve the intra-segmentation accuracy.

Automatic Cell Detection in Bright-Field Microscope Images Using SIFT, Random Forests, and Hierarchical Clustering

We present a novel machine learning-based system for unstained cell detection in bright-field microscope images. The system is fully automatic since it requires no manual parameter tuning. It is also highly invariant with respect to illumination conditions and to the size and orientation of cells. Images from two adherent cell lines and one suspension cell line were used in the evaluation for a total number of more than 3500 cells. Besides real images, simulated images were also used in the evaluation. The detection error was between approximately zero and 15.5% which is a significantly superior performance compared to baseline approaches.

Image Segmentation with Cascaded Hierarchical Models and Logistic Disjunctive Normal Networks

Contextual information plays an important role in solving vision problems such as image segmentation. However, extracting contextual information and using it in an effective way remains a difficult problem. To address this challenge, we propose a multi-resolution contextual framework, called cascaded hierarchical model (CHM), which learns contextual information in a hierarchical framework for image segmentation. At each level of the hierarchy, a classifier is trained based on downsampled input images and outputs of previous levels. Our model then incorporates the resulting multi-resolution contextual information into a classifier to segment the input image at original resolution. We repeat this procedure by cascading the hierarchical framework to improve the segmentation accuracy. Multiple classifiers are learned in the CHM; therefore, a fast and accurate classifier is required to make the training tractable. The classifier also needs to be robust against overfitting due to the large number of parameters learned during training. We introduce a novel classification scheme, called logistic disjunctive normal networks (LDNN), which consists of one adaptive layer of feature detectors implemented by logistic sigmoid functions followed by two fixed layers of logical units that compute conjunctions and disjunctions, respectively. We demonstrate that LDNN outperforms state-of-theart classifiers and can be used in the CHM to improve object segmentation performance.

Multi-class multi-scale series contextual model for image segmentation

Contextual information has been widely used as a rich source of information to segment multiple objects in an image. A contextual model uses the relationships between the objects in a scene to facilitate object detection and segmentation. Using contextual information from different objects in an effective way for object segmentation, however, remains a difficult problem. In this paper, we introduce a novel framework, called multiclass multiscale (MCMS) series contextual model, which uses contextual information from multiple objects and at different scales for learning discriminative models in a supervised setting. The MCMS model incorporates cross-object and inter-object information into one probabilistic framework and thus is able to capture geometrical relationships and dependencies among multiple objects in addition to local information from each single object present in an image. We demonstrate that our MCMS model improves object segmentation performance in electron microscopy images and provides a coherent segmentation of multiple objects. Through speeding up the segmentation process, the proposed method will allow neurobiologists to move beyond individual specimens and analyze populations paving the way for understanding neurodegenerative diseases at the microscopic level.

Learning context cues for synapse segmentation

We present a new approach for the automated segmentation of synapses in image stacks acquired by electron microscopy (EM) that relies on image features specifically designed to take spatial context into account. These features are used to train a classifier that can effectively learn cues such as the presence of a nearby post-synaptic region. As a result, our algorithm successfully distinguishes synapses from the numerous other organelles that appear within an EM volume, including those whose local textural properties are relatively similar. Furthermore, as a by-product of the segmentation, our method flawlessly determines synaptic orientation, a crucial element in the interpretation of brain circuits. We evaluate our approach on three different datasets, compare it against the state-of-the-art in synapse segmentation and demonstrate our ability to reliably collect shape, density, and orientation statistics over hundreds of synapses.

WATERSHED MERGE FOREST CLASSIFICATION FOR ELECTRON MICROSCOPY IMAGE STACK SEGMENTATION

Automated electron microscopy (EM) image analysis techniques can be tremendously helpful for connectomics research. In this paper, we extend our previous work [1] and propose a fully automatic method to utilize inter-section information for intra-section neuron segmentation of EM image stacks. A watershed merge forest is built via the watershed transform with each tree representing the region merging hierarchy of one 2D section in the stack. A section classifier is learned to identify the most likely region correspondence between adjacent sections. The inter-section information from such correspondence is incorporated to update the potentials of tree nodes. We resolve the merge forest using these potentials together with consistency constraints to acquire the final segmentation of the whole stack. We demonstrate that our method leads to notable segmentation accuracy improvement by experimenting with two types of EM image data sets.

Machine learning of hierarchical clustering to segment 2D and 3D images

We aim to improve segmentation through the use of machine learning tools during region agglomeration. We propose an active learning approach for performing hierarchical agglomerative segmentation from superpixels. Our method combines multiple features at all scales of the agglomerative process, works for data with an arbitrary number of dimensions, and scales to very large datasets. We advocate the use of variation of information to measure segmentation accuracy, particularly in 3D electron microscopy (EM) images of neural tissue, and using this metric demonstrate an improvement over competing algorithms in EM and natural images.

Hierarchical level features based trainable segmentation for electron microscopy images

BACKGROUND

The neuronal electron microscopy images segmentation is the basic and key step to efficiently build the 3D brain structure and connectivity for a better understanding of central neural system. However, due to the visual complex appearance of neuronal structures, it is challenging to automatically segment membranes from the EM images.

METHODS

In this paper, we present a fast, efficient segmentation method for neuronal EM images that utilizes hierarchical level features based on supervised learning. Hierarchical level features are designed by combining pixel and superpixel information to describe the EM image. For pixels in a superpixel have similar characteristics, only part of them is automatically selected and used to reduce information redundancy. To each selected pixel, 34 dimensional features are extracted by traditional way. Each superpixel itself is viewed as a unit to extract 35 dimensional features with statistical method. Also, 3 dimensional context level features among multi superpixels are extracted. Above three kinds of features are combined as a feature vector, namely, hierarchical level features to use for segmentation. Random forest is used as classifier and is trained with hierarchical level features to perform segmentation.

RESULTS

In small sample condition and with low-dimensional features, the effectiveness of our method is verified on the data set of ISBI2012 EM Segmentation Challenge, and its rand error, warping error and pixel error attain to 0.106308715, 0.001200104 and 0.079132453, respectively.

CONCLUSIONS

Comparing to pixel level or superpixel level features, hierarchical level features have better discrimination ability and the proposed method is promising for membrane segmentation.

DP2: Distributed 3D image segmentation using micro-labor workforce

SUMMARY

This application note describes a new scalable semi-automatic approach, the Dual Point Decision Process, for segmentation of 3D structures contained in 3D microscopy. The segmentation problem is distributed to many individual workers such that each receives only simple questions regarding whether two points in an image are placed on the same object. A large pool of micro-labor workers available through Amazon's Mechanical Turk system provides the labor in a scalable manner.

AVAILABILITY AND IMPLEMENTATION

Python-based code for non-commercial use and test data are available in the source archive at https://sites.google.com/site/imagecrowdseg/.

CONTACT

rgiuly@ucsd.edu

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Semi-automated neuron boundary detection and nonbranching process segmentation in electron microscopy images

Neuroscientists are developing new imaging techniques and generating large volumes of data in an effort to understand the complex structure of the nervous system. The complexity and size of this data makes human interpretation a labor-intensive task. To aid in the analysis, new segmentation techniques for identifying neurons in these feature rich datasets are required. This paper presents a method for neuron boundary detection and nonbranching process segmentation in electron microscopy images and visualizing them in three dimensions. It combines both automated segmentation techniques with a graphical user interface for correction of mistakes in the automated process. The automated process first uses machine learning and image processing techniques to identify neuron membranes that deliniate the cells in each two-dimensional section. To segment nonbranching processes, the cell regions in each two-dimensional section are connected in 3D using correlation of regions between sections. The combination of this method with a graphical user interface specially designed for this purpose, enables users to quickly segment cellular processes in large volumes.

Watershed Merge Tree Classification for Electron Microscopy Image Segmentation

Automated segmentation of electron microscopy (EM) images is a challenging problem. In this paper, we present a novel method that utilizes a hierarchical structure and boundary classification for 2D neuron segmentation. With a membrane detection probability map, a watershed merge tree is built for the representation of hierarchical region merging from the watershed algorithm. A boundary classifier is learned with non-local image features to predict each potential merge in the tree, upon which merge decisions are made with consistency constraints to acquire the final segmentation. Independent of classifiers and decision strategies, our approach proposes a general framework for efficient hierarchical segmentation with statistical learning. We demonstrate that our method leads to a substantial improvement in segmentation accuracy.

Automatic identification and quantitative morphometry of unstained spinal nerve using molecular hyperspectral imaging technology

Quantitative observation of nerve fiber sections is often complemented by morphological analysis in both research and clinical condition. However, existing manual or semi-automated methods are tedious and labour intensive, fully automated morphometry methods are complicated as the information of color or gray images captured by traditional microscopy is limited. Moreover, most of the methods are time-consuming as the nerve sections need to be stained with some reagents before observation. To overcome these shortcomings, a molecular hyperspectral imaging system is developed and used to observe the spinal nerve sections. The molecular hyperspectral images contain both the structural and biochemical information of spinal nerve sections which is very useful for automatic identification and quantitative morphological analysis of nerve fibers. This characteristic makes it possible for researchers to observe the unstained spinal nerve and live cells in their native environment. To evaluate the performance of the new method, the molecular hyperspectral images were captured and the improved spectral angle mapper algorithm was proposed and used to segment the myelin contours. Then the morphological parameters such as myelin thickness and myelin area were calculated and evaluated. With these morphological parameters, the three dimension surface view images were drawn to help the investigators observe spinal nerve at different angles. The experiment results show that the hyperspectral based method has the potential to identify the spinal nerve more accurate than the traditional method as the new method contains both the spectral and spatial information of nerve sections.

Building retinal connectomes

Understanding vertebrate vision depends on knowing, in part, the complete network graph of at least one representative retina. Acquiring such graphs is the business of synaptic connectomics, emerging as a practical technology due to improvements in electron imaging platform control, management software for large-scale datasets, and availability of data storage. The optimal strategy for building complete connectomes uses transmission electron imaging with 2 nm or better resolution, molecular tags for cell identification, open-access data volumes for navigation, and annotation with open-source tools to build 3D cell libraries, complete network diagrams and connectivity databases. The first forays into retinal connectomics have shown that even nominally well-studied cells have much richer connection graphs than expected.

Neuronal tracing for connectomic studies

Reconstruction of the complete wiring diagram, or connectome, of a neural circuit provides an alternative approach to conventional circuit analysis. One major obstacle of connectomics lies in segmenting and tracing neuronal processes from the vast number of images obtained with optical or electron microscopy. Here I review recent progress in automated tracing algorithms for connectomic reconstruction with fluorescence and electron microscopy, and discuss the challenges to image analysis posed by novel optical imaging techniques.

Biomedical Visual Computing: Case Studies and Challenges

Advances in computational geometric modeling, imaging, and simulation let researchers build and test models of increasing complexity, generating unprecedented amounts of data. As recent research in biomedical applications illustrates, visualization will be critical in making this vast amount of data usable; it's also fundamental to understanding models of complex phenomena.

Method: automatic segmentation of mitochondria utilizing patch classification, contour pair classification, and automatically seeded level sets

Background

While progress has been made to develop automatic segmentation techniques for mitochondria, there remains a need for more accurate and robust techniques to delineate mitochondria in serial blockface scanning electron microscopic data. Previously developed texture based methods are limited for solving this problem because texture alone is often not sufficient to identify mitochondria. This paper presents a new three-step method, the Cytoseg process, for automated segmentation of mitochondria contained in 3D electron microscopic volumes generated through serial block face scanning electron microscopic imaging. The method consists of three steps. The first is a random forest patch classification step operating directly on 2D image patches. The second step consists of contour-pair classification. At the final step, we introduce a method to automatically seed a level set operation with output from previous steps.

Results

We report accuracy of the Cytoseg process on three types of tissue and compare it to a previous method based on Radon-Like Features. At step 1, we show that the patch classifier identifies mitochondria texture but creates many false positive pixels. At step 2, our contour processing step produces contours and then filters them with a second classification step, helping to improve overall accuracy. We show that our final level set operation, which is automatically seeded with output from previous steps, helps to smooth the results. Overall, our results show that use of contour pair classification and level set operations improve segmentation accuracy beyond patch classification alone. We show that the Cytoseg process performs well compared to another modern technique based on Radon-Like Features.

Conclusions

We demonstrated that texture based methods for mitochondria segmentation can be enhanced with multiple steps that form an image processing pipeline. While we used a random-forest based patch classifier to recognize texture, it would be possible to replace this with other texture identifiers, and we plan to explore this in future work.

Retinal implants: emergence of a multidisciplinary field

PURPOSE OF REVIEW

This review summarizes the current status of retinal prostheses, recent accomplishments, and major remaining research, engineering, and rehabilitation challenges.

RECENT FINDINGS

Retinal research, materials and biocompatibility studies, and clinical trials in patients blind from retinitis pigmentosa are representative of an emerging field with considerable promise and sobering challenges. A summary of progress in dozens of laboratories, companies, and clinics around the world is presented through a synopsis of relevant studies, not only to summarize the progress but also to convey the remarkable increase in interest, effort, and outside funding this field has enjoyed.

SUMMARY

At present, clinical applications of retinal implant technology are dominated by one or two groups/companies, but the field is wide open for others to take the lead through novel approaches in technology, tissue interfacing, information transfer paradigms, and rehabilitation. Where the field will go in the next few years is almost anybody's guess, but that it will move forward is a certainty.

Supervoxel-based segmentation of mitochondria in em image stacks with learned shape features

It is becoming increasingly clear that mitochondria play an important role in neural function. Recent studies show mitochondrial morphology to be crucial to cellular physiology and synaptic function and a link between mitochondrial defects and neuro-degenerative diseases is strongly suspected. Electron microscopy (EM), with its very high resolution in all three directions, is one of the key tools to look more closely into these issues but the huge amounts of data it produces make automated analysis necessary. State-of-the-art computer vision algorithms designed to operate on natural 2-D images tend to perform poorly when applied to EM data for a number of reasons. First, the sheer size of a typical EM volume renders most modern segmentation schemes intractable. Furthermore, most approaches ignore important shape cues, relying only on local statistics that easily become confused when confronted with noise and textures inherent in the data. Finally, the conventional assumption that strong image gradients always correspond to object boundaries is violated by the clutter of distracting membranes. In this work, we propose an automated graph partitioning scheme that addresses these issues. It reduces the computational complexity by operating on supervoxels instead of voxels, incorporates shape features capable of describing the 3-D shape of the target objects, and learns to recognize the distinctive appearance of true boundaries. Our experiments demonstrate that our approach is able to segment mitochondria at a performance level close to that of a human annotator, and outperforms a state-of-the-art 3-D segmentation technique.

Learning context cues for synapse segmentation in EM volumes

We present a new approach for the automated segmentation of excitatory synapses in image stacks acquired by electron microscopy. We rely on a large set of image features specifically designed to take spatial context into account and train a classifier that can effectively utilize cues such as the presence of a nearby post-synaptic region. As a result, our algorithm successfully distinguishes synapses from the numerous other organelles that appear within an EM volume, including those whose local textural properties are relatively similar. This enables us to achieve very high detection rates with very few false positives.

Carving: scalable interactive segmentation of neural volume electron microscopy images

Interactive segmentation algorithms should respond within seconds and require minimal user guidance. This is a challenge on 3D neural electron microscopy images. We propose a supervoxel-based energy function with a novel background prior that achieves these goals. This is verified by extensive experiments with a robot mimicking human interactions. A graphical user interface offering access to an open source implementation of these algorithms is made available.

Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images

We describe a protocol for fully automated detection and segmentation of asymmetric, presumed excitatory, synapses in serial electron microscopy images of the adult mammalian cerebral cortex, taken with the focused ion beam, scanning electron microscope (FIB/SEM). The procedure is based on interactive machine learning and only requires a few labeled synapses for training. The statistical learning is performed on geometrical features of 3D neighborhoods of each voxel and can fully exploit the high z-resolution of the data. On a quantitative validation dataset of 111 synapses in 409 images of 1948×1342 pixels with manual annotations by three independent experts the error rate of the algorithm was found to be comparable to that of the experts (0.92 recall at 0.89 precision). Our software offers a convenient interface for labeling the training data and the possibility to visualize and proofread the results in 3D. The source code, the test dataset and the ground truth annotation are freely available on the website http://www.ilastik.org/synapse-detection.

A semi-automated method for identifying and measuring myelinated nerve fibers in scanning electron microscope images

Diagnosing illnesses, developing and comparing treatment methods, and conducting research on the organization of the peripheral nervous system often require the analysis of peripheral nerve images to quantify the number, myelination, and size of axons in a nerve. Current methods that require manually labeling each axon can be extremely time-consuming as a single nerve can contain thousands of axons. To improve efficiency, we developed a computer-assisted axon identification and analysis method that is capable of analyzing and measuring sub-images covering the nerve cross-section, acquired using a scanning electron microscope. This algorithm performs three main procedures - it first uses cross-correlation to combine the acquired sub-images into a large image showing the entire nerve cross-section, then identifies and individually labels axons using a series of image intensity and shape criteria, and finally identifies and labels the myelin sheath of each axon using a region growing algorithm with the geometric centers of axons as seeds. To ensure accurate analysis of the image, we incorporated manual supervision to remove mislabeled axons and add missed axons. The typical user-assisted processing time for a two-megapixel image containing over 2000 axons was less than 1h. This speed was almost eight times faster than the time required to manually process the same image. Our method has proven to be well suited for identifying axons and their characteristics, and represents a significant time savings over traditional manual methods.

Machines that learn to segment images: a crucial technology for connectomics

Connections between neurons can be found by checking whether synapses exist at points of contact, which in turn are determined by neural shapes. Finding these shapes is a special case of image segmentation, which is laborious for humans and would ideally be performed by computers. New metrics properly quantify the performance of a computer algorithm using its disagreement with 'true' segmentations of example images. New machine learning methods search for segmentation algorithms that minimize such metrics. These advances have reduced computer errors dramatically. It should now be faster for a human to correct the remaining errors than to segment an image manually. Further reductions in human effort are expected, and crucial for finding connectomes more complex than that of Caenorhabditis elegans.

Detection of neuron membranes in electron microscopy images using multi-scale context and radon-like features

Automated neural circuit reconstruction through electron microscopy (EM) images is a challenging problem. In this paper, we present a novel method that exploits multi-scale contextual information together with Radon-like features (RLF) to learn a series of discriminative models. The main idea is to build a framework which is capable of extracting information about cell membranes from a large contextual area of an EM image in a computationally efficient way. Toward this goal, we extract RLF that can be computed efficiently from the input image and generate a scale-space representation of the context images that are obtained at the output of each discriminative model in the series. Compared to a single-scale model, the use of a multi-scale representation of the context image gives the subsequent classifiers access to a larger contextual area in an effective way. Our strategy is general and independent of the classifier and has the potential to be used in any context based framework. We demonstrate that our method outperforms the state-of-the-art algorithms in detection of neuron membranes in EM images.

Semi-automated reconstruction of neural circuits using electron microscopy

Reconstructing neuronal circuits at the level of synapses is a central problem in neuroscience, and the focus of the nascent field of connectomics. Previously used to reconstruct the C. elegans wiring diagram, serial-section transmission electron microscopy (ssTEM) is a proven technique for the task. However, to reconstruct more complex circuits, ssTEM will require the automation of image processing. We review progress in the processing of electron microscopy images and, in particular, a semi-automated reconstruction pipeline deployed at Janelia Farm. Drosophila circuits underlying identified behaviors are being reconstructed in the pipeline with the goal of generating a complete Drosophila connectome.