Segmentation of electron tomographic data sets using fuzzy set theory principles

RAZA: A Rapid 3D z-crossings algorithm to segment electron tomograms and extract organelles and macromolecules

Resolving the 3D architecture of cells to atomic resolution is one of the most ambitious challenges of cellular and structural biology. Central to this process is the ability to automate tomogram segmentation to identify sub-cellular components, facilitate molecular docking and annotate detected objects with associated metadata. Here we demonstrate that RAZA (Rapid 3D z-crossings algorithm) provides a robust, accurate, intuitive, fast, and generally applicable segmentation algorithm capable of detecting organelles, membranes, macromolecular assemblies and extrinsic membrane protein domains. RAZA defines each continuous contour within a tomogram as a discrete object and extracts a set of 3D structural fingerprints (major, middle and minor axes, surface area and volume), enabling selective, semi-automated segmentation and object extraction. RAZA takes advantage of the fact that the underlying algorithm is a true 3D edge detector, allowing the axes of a detected object to be defined, independent of its random orientation within a cellular tomogram. The selectivity of object segmentation and extraction can be controlled by specifying a user-defined detection tolerance threshold for each fingerprint parameter, within which segmented objects must fall and/or by altering the number of search parameters, to define morphologically similar structures. We demonstrate the capability of RAZA to selectively extract subgroups of organelles (mitochondria) and macromolecular assemblies (ribosomes) from cellular tomograms. Furthermore, the ability of RAZA to define objects and their contours, provides a basis for molecular docking and rapid tomogram annotation.

Convolutional neural networks for automated annotation of cellular cryo-electron tomograms

Cellular Electron Cryotomography (CryoET) offers the ability to look inside cells and observe macromolecules frozen in action. A primary challenge for this technique is identifying and extracting the molecular components within the crowded cellular environment. We introduce a method using neural networks to dramatically reduce the time and human effort required for subcellular annotation and feature extraction. Subsequent subtomogram classification and averaging yields in-situ structures of molecular components of interest.

Volume Segmentation and Analysis of Biological Materials Using SuRVoS (Super-region Volume Segmentation) Workbench

Segmentation is the process of isolating specific regions or objects within an imaged volume, so that further study can be undertaken on these areas of interest. When considering the analysis of complex biological systems, the segmentation of three-dimensional image data is a time consuming and labor intensive step. With the increased availability of many imaging modalities and with automated data collection schemes, this poses an increased challenge for the modern experimental biologist to move from data to knowledge. This publication describes the use of SuRVoS Workbench, a program designed to address these issues by providing methods to semi-automatically segment complex biological volumetric data. Three datasets of differing magnification and imaging modalities are presented here, each highlighting different strategies of segmenting with SuRVoS. Phase contrast X-ray tomography (microCT) of the fruiting body of a plant is used to demonstrate segmentation using model training, cryo electron tomography (cryoET) of human platelets is used to demonstrate segmentation using super- and megavoxels, and cryo soft X-ray tomography (cryoSXT) of a mammalian cell line is used to demonstrate the label splitting tools. Strategies and parameters for each datatype are also presented. By blending a selection of semi-automatic processes into a single interactive tool, SuRVoS provides several benefits. Overall time to segment volumetric data is reduced by a factor of five when compared to manual segmentation, a mainstay in many image processing fields. This is a significant savings when full manual segmentation can take weeks of effort. Additionally, subjectivity is addressed through the use of computationally identified boundaries, and splitting complex collections of objects by their calculated properties rather than on a case-by-case basis.

FIJI Macro 3D ART VeSElecT: 3D Automated Reconstruction Tool for Vesicle Structures of Electron Tomograms

Automatic image reconstruction is critical to cope with steadily increasing data from advanced microscopy. We describe here the Fiji macro 3D ART VeSElecT which we developed to study synaptic vesicles in electron tomograms. We apply this tool to quantify vesicle properties (i) in embryonic Danio rerio 4 and 8 days past fertilization (dpf) and (ii) to compare Caenorhabditis elegans N2 neuromuscular junctions (NMJ) wild-type and its septin mutant (unc-59(e261)). We demonstrate development-specific and mutant-specific changes in synaptic vesicle pools in both models. We confirm the functionality of our macro by applying our 3D ART VeSElecT on zebrafish NMJ showing smaller vesicles in 8 dpf embryos then 4 dpf, which was validated by manual reconstruction of the vesicle pool. Furthermore, we analyze the impact of C. elegans septin mutant unc-59(e261) on vesicle pool formation and vesicle size. Automated vesicle registration and characterization was implemented in Fiji as two macros (registration and measurement). This flexible arrangement allows in particular reducing false positives by an optional manual revision step. Preprocessing and contrast enhancement work on image-stacks of 1nm/pixel in x and y direction. Semi-automated cell selection was integrated. 3D ART VeSElecT removes interfering components, detects vesicles by 3D segmentation and calculates vesicle volume and diameter (spherical approximation, inner/outer diameter). Results are collected in color using the RoiManager plugin including the possibility of manual removal of non-matching confounder vesicles. Detailed evaluation considered performance (detected vesicles) and specificity (true vesicles) as well as precision and recall. We furthermore show gain in segmentation and morphological filtering compared to learning based methods and a large time gain compared to manual segmentation. 3D ART VeSElecT shows small error rates and its speed gain can be up to 68 times faster in comparison to manual annotation. Both automatic and semi-automatic modes are explained including a tutorial.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a "ground truth" for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.

Accurate membrane tracing in three-dimensional reconstructions from electron cryotomography data

The connection between the extracellular matrix and the cell is of major importance for mechanotransduction and mechanobiology. Electron cryo-tomography, in principle, enables better than nanometer-resolution analysis of these connections, but restrictions of data collection geometry hamper the accurate extraction of the ventral membrane location from these tomograms, an essential prerequisite for the analysis. Here, we introduce a novel membrane tracing strategy that enables ventral membrane extraction at high fidelity and extraordinary accuracy. The approach is based on detecting the boundary between the inside and the outside of the cell rather than trying to explicitly trace the membrane. Simulation studies show that over 99% of the membrane can be correctly modeled using this principle and the excellent match of visually identifiable membrane stretches with the extracted boundary of experimental data indicates that the accuracy is comparable for actual data.

Robust membrane detection based on tensor voting for electron tomography

Electron tomography enables three-dimensional (3D) visualization and analysis of the subcellular architecture at a resolution of a few nanometers. Segmentation of structural components present in 3D images (tomograms) is often necessary for their interpretation. However, it is severely hampered by a number of factors that are inherent to electron tomography (e.g. noise, low contrast, distortion). Thus, there is a need for new and improved computational methods to facilitate this challenging task. In this work, we present a new method for membrane segmentation that is based on anisotropic propagation of the local structural information using the tensor voting algorithm. The local structure at each voxel is then refined according to the information received from other voxels. Because voxels belonging to the same membrane have coherent structural information, the underlying global structure is strengthened. In this way, local information is easily integrated at a global scale to yield segmented structures. This method performs well under low signal-to-noise ratio typically found in tomograms of vitrified samples under cryo-tomography conditions and can bridge gaps present on membranes. The performance of the method is demonstrated by applications to tomograms of different biological samples and by quantitative comparison with standard template matching procedure.

Linear time distances between fuzzy sets with applications to pattern matching and classification

We present four novel point-to-set distances defined for fuzzy or gray-level image data, two based on integration over α-cuts and two based on the fuzzy distance transform. We explore their theoretical properties. Inserting the proposed point-to-set distances in existing definitions of set-to-set distances, among which are the Hausdorff distance and the sum of minimal distances, we define a number of distances between fuzzy sets. These set distances are directly applicable for comparing gray-level images or fuzzy segmented objects, but also for detecting patterns and matching parts of images. The distance measures integrate shape and intensity/membership of observed entities, providing a highly applicable tool for image processing and analysis. Performance evaluation of derived set distances in real image processing tasks is conducted and presented. It is shown that the considered distances have a number of appealing theoretical properties and exhibit very good performance in template matching and object classification for fuzzy segmented images as well as when applied directly on gray-level intensity images. Examples include recognition of hand written digits and identification of virus particles. The proposed set distances perform excellently on the MNIST digit classification task, achieving the best reported error rate for classification using only rigid body transformations and a kNN classifier.

Nhs: network-based hierarchical segmentation for cryo-electron microscopy density maps

Cryo-electron microscopy (cryo-EM) experiments yield low-resolution (3-30 Å) 3D-density maps of macromolecules. These density maps are segmented to identify structurally distinct proteins, protein domains, and subunits. Such partitioning aids the inference of protein motions and guides fitting of high-resolution atomistic structures. Cryo-EM density map segmentation has traditionally required tedious and subjective manual partitioning or semisupervised computational methods, whereas validation of resulting segmentations has remained an open problem in this field. We introduce a network-based hierarchical segmentation (Nhs) method, that provides a multi-scale partitioning, reflecting local and global clustering, while requiring no user input. This approach models each map as a graph, where map voxels constitute nodes and weighted edges connect neighboring voxels. Nhs initiates Markov diffusion (or random walk) on the weighted graph. As Markov probabilities homogenize through diffusion, an intrinsic segmentation emerges. We validate the segmentations with ground-truth maps based on atomistic models. When implemented on density maps in the 2010 Cryo-EM Modeling Challenge, Nhs efficiently and objectively partitions macromolecules into structurally and functionally relevant subregions at multiple scales.

Macromolecular structure modeling from 3D EM using VolRover 2.0

We review tools for structure identification and model-based refinement from three-dimensional electron microscopy implemented in our in-house software package, VOLROVER 2.0. For viral density maps with icosahedral symmetry, we segment the capsid, polymeric, and monomeric subunits using techniques based on automatic symmetry detection and multidomain fast marching. For large biomolecules without symmetry information, we again use our multidomain fast-marching method with manual or fit-based multiseeding to segment meaningful substructures. In either case, we subject the resulting segmented subunit to secondary structure detection when the EM resolution is sufficiently high, and rigid-body structure fitting when the corresponding X-ray structure is available. Secondary structure elements are identified by three techniques: our earlier volume-based and boundary-based skeletonization methods as well as a new method, currently in development, based on solving the grassfire flow equation. For rigid-body fitting, we adapt our earlier fast Fourier-based correlation scheme F2Dock. Our reported segmentation, secondary structure elements identification, and rigid-body fitting techniques, implemented in VOLROVER 2.0 are applied to the PSB 2011 cryo-EM modeling challenge data, and our results are briefly compared to similar results submitted from other research groups. The comparisons show that our techniques are equally capable of segmenting relatively accurate subunits from a viral or protein assembly, and that high segmentation quality leads in turn to higher-quality results of secondary structure elements identification and correlation-based rigid-body fitting. © 2012 Wiley Periodicals, Inc. Biopolymers 97: 709-731, 2012.

Parallel fuzzy connected image segmentation on GPU

PURPOSE

Image segmentation techniques using fuzzy connectedness (FC) principles have shown their effectiveness in segmenting a variety of objects in several large applications. However, one challenge in these algorithms has been their excessive computational requirements when processing large image datasets. Nowadays, commodity graphics hardware provides a highly parallel computing environment. In this paper, the authors present a parallel fuzzy connected image segmentation algorithm implementation on NVIDIA's compute unified device Architecture (CUDA) platform for segmenting medical image data sets.

METHODS

In the FC algorithm, there are two major computational tasks: (i) computing the fuzzy affinity relations and (ii) computing the fuzzy connectedness relations. These two tasks are implemented as CUDA kernels and executed on GPU. A dramatic improvement in speed for both tasks is achieved as a result.

RESULTS

Our experiments based on three data sets of small, medium, and large data size demonstrate the efficiency of the parallel algorithm, which achieves a speed-up factor of 24.4x, 18.1x, and 10.3x, correspondingly, for the three data sets on the NVIDIA Tesla C1060 over the implementation of the algorithm on CPU, and takes 0.25, 0.72, and 15.04 s, correspondingly, for the three data sets.

CONCLUSIONS

The authors developed a parallel algorithm of the widely used fuzzy connected image segmentation method on the NVIDIA GPUs, which are far more cost- and speed-effective than both cluster of workstations and multiprocessing systems. A near-interactive speed of segmentation has been achieved, even for the large data set.

GPU accelerated fuzzy connected image segmentation by using CUDA

Image segmentation techniques using fuzzy connectedness principles have shown their effectiveness in segmenting a variety of objects in several large applications in recent years. However, one problem of these algorithms has been their excessive computational requirements when processing large image datasets. Nowadays commodity graphics hardware provides high parallel computing power. In this paper, we present a parallel fuzzy connected image segmentation algorithm on Nvidia's Compute Unified Device Architecture (CUDA) platform for segmenting large medical image data sets. Our experiments based on three data sets with small, medium, and large data size demonstrate the efficiency of the parallel algorithm, which achieves a speed-up factor of 7.2x, 7.3x, and 14.4x, correspondingly, for the three data sets over the sequential implementation of fuzzy connected image segmentation algorithm on CPU.

Quantitative analysis of cryo-EM density map segmentation by watershed and scale-space filtering, and fitting of structures by alignment to regions

Cryo-electron microscopy produces 3D density maps of molecular machines, which consist of various molecular components such as proteins and RNA. Segmentation of individual components in such maps is a challenging task, and is mostly accomplished interactively. We present an approach based on the immersive watershed method and grouping of the resulting regions using progressively smoothed maps. The method requires only three parameters: the segmentation threshold, a smoothing step size, and the number of smoothing steps. We first apply the method to maps generated from molecular structures and use a quantitative metric to measure the segmentation accuracy. The method does not attain perfect accuracy, however it produces single or small groups of regions that roughly match individual proteins or subunits. We also present two methods for fitting of structures into density maps, based on aligning the structures with single regions or small groups of regions. The first method aligns centers and principal axes, whereas the second aligns centers and then rotates the structure to find the best fit. We describe both interactive and automated ways of using these two methods. Finally, we show segmentation and fitting results for several experimentally-obtained density maps.

Computer assisted biopsy of breast tumors

In this paper we report our preliminary results of the development of a computer assisted system for breast biopsy. The system is based on tracked ultrasound images of the breast. A three dimensional ultrasound volume is constructed from a set of tracked B-scan images acquired with a calibrated probe. The system has been designed to assist a radiologist during breast biopsy, and also as a training system for radiology residents. A semiautomatic classification algorithm was implemented to assist the user with the annotation of the tumor on an ultrasound volume. We report the development of the system prototype, tested on a physical phantom of a breast with a tumor, made of polivinil alcohol.

Methods for segmentation and interpretation of electron tomographic reconstructions

Electron tomography has become a powerful tool for revealing the molecular architecture of biological cells and tissues. In principle, electron tomography can provide high-resolution mapping of entire proteomes. The achievable resolution (3-8 nm) is capable of bridging the gap between live-cell imaging and atomic resolution structures. However, the relevant information is not readily accessible from the data and needs to be identified, extracted, and processed before it can be used. Because electron tomography imaging and image acquisition technologies have enjoyed major advances in the last few years and continue to increase data throughput, the need for approaches that allow automatic and objective interpretation of electron tomograms becomes more and more urgent. This chapter provides an overview of the state of the art in this field and attempts to identify the major bottlenecks that prevent approaches for interpreting electron tomography data to develop their full potential.

Requirements for the valid quantification of immunostains on tissue microarray materials using image analysis

Antibody-based proteomics applied to tissue microarray (TMA) technology provides a very efficient means of visualizing and locating antigen expression in large collections of normal and pathological tissue samples. To characterize antigen expression on TMAs, the use of image analysis methods avoids the effects of human subjectivity evidenced in manual microscopical analysis. Thus, these methods have the potential to significantly enhance both precision and reproducibility. Although some commercial systems include tools for the quantitative evaluation of immunohistochemistry-stained images, there exists no clear agreement on best practices to allow for correct and reproducible quantification results. Our study focuses on practical aspects regarding (i) image acquisition (ii) segmentation of staining and counterstaining areas and (iii) extraction of quantitative features. We illustrate our findings using a commercial system to quantify different immunohistochemistry markers targeting proteins with different expression patterns (cytoplasmic, nuclear or membranous) in colon cancer or brain tumor TMAs. Our investigations led us to identify several steps that we consider essential for standardizing computer-assisted immunostaining quantification experiments. In addition, we propose a data normalization process based on reference materials to be able to compare measurements between studies involving different TMAs. In conclusion, we recommend certain critical prerequisites that commercial or in-house systems should satisfy in order to permit valid immunostaining quantification.

Molecular cryo-electron tomography of vitreous tissue sections: current challenges

Electron tomography of vitreous tissue sections (tissue TOVIS) allows the study of the three-dimensional structure of molecular complexes in a near-native cellular context. Its usage is, however, limited by an unfortunate combination of noisy and incomplete data, by a technically demanding sample preparation procedure, and by a disposition for specimen degradation during data collection. Here we outline some major challenges as experienced from the application of TOVIS to human skin. We further consider a number of practical measures as well as theoretical approaches for its future development.

Interpretation of very low resolution X-ray electron-density maps using core objects

A novel approach to obtaining structural information from macromolecular X-ray data extending to resolutions as low as 20 A is presented. Following a simple map-segmentation procedure, the approximate shapes of the domains forming the structure are identified. A pattern-recognition comparative analysis of these shapes and those derived from the structures of domains from the PDB results in candidate structural models that can be used for a fit into the density map. It is shown that the placed candidate models can be employed for subsequent phase extension to higher resolution.

Identifying Components in 3D Density Maps of Protein Nanomachines by Multi-scale Segmentation

Segmentation of density maps obtained using cryo-electron microscopy (cryo-EM) is a challenging task, and is typically accomplished by time-intensive interactive methods. The goal of segmentation is to identify the regions inside the density map that correspond to individual components. We present a multi-scale segmentation method for accomplishing this task that requires very little user interaction. The method uses the concept of scale space, which is created by convolution of the input density map with a Gaussian filter. The latter process smoothes the density map. The standard deviation of the Gaussian filter is varied, with smaller values corresponding to finer scales and larger values to coarser scales. Each of the maps at different scales is segmented using the watershed method, which is very efficient, completely automatic, and does not require the specification of seed points. Some detail is lost in the smoothing process. A sharpening process reintroduces detail into the segmentation at the coarsest scale by using the segmentations at the finer scales. We apply the method to simulated density maps, where the exact segmentation (or ground truth) is known, and rigorously evaluate the accuracy of the resulting segmentations.

Chapter 2 Correlated light and electron microscopy/electron tomography of mitochondria in situ

Three-dimensional light microscopy and three-dimensional electron microscopy (electron tomography) separately provide very powerful tools to study cellular structure and physiology, including the structure and physiology of mitochondria. Fluorescence microscopy allows one to study processes in live cells with specific labels and stains that follow the movement of labeled proteins and changes within cellular compartments but does not have sufficient resolution to define the ultrastructure of intracellular organelles such as mitochondria. Electron microscopy and electron tomography provide the highest resolution currently available to study mitochondrial ultrastructure but cannot follow processes in living cells. We describe the combination of these two techniques in which fluorescence confocal microscopy is used to study structural and physiologic changes in mitochondria within apoptotic HeLa cells to define the apoptotic timeframe. Cells can then be selected at various stages of the apoptotic timeframe for examination at higher resolution by electron microscopy and electron tomography. This is a form of "virtual" 4-dimensional electron microscopy that has revealed interesting structural changes in the mitochondria of HeLa cells during apoptosis. The same techniques can be applied, with modification, to study other dynamic processes within cells in other experimental contexts.

Parallel Fuzzy Segmentation of Multiple Objects

The usefulness of fuzzy segmentation algorithms based on fuzzy connectedness principles has been established in numerous publications. New technologies are capable of producing larger-and-larger datasets and this causes the sequential implementations of fuzzy segmentation algorithms to be time-consuming. We have adapted a sequential fuzzy segmentation algorithm to multi-processor machines. We demonstrate the efficacy of such a distributed fuzzy segmentation algorithm by testing it with large datasets (of the order of 50 million points/voxels/items): a speed-up factor of approximately five over the sequential implementation seems to be the norm.