A hybrid 3D watershed algorithm incorporating gradient cues and object models for automatic segmentation of nuclei in confocal image stacks

UNSEG: unsupervised segmentation of cells and their nuclei in complex tissue samples

Multiplexed imaging technologies have made it possible to interrogate complex tumor microenvironments at sub-cellular resolution within their native spatial context. However, proper quantification of this complexity requires the ability to easily and accurately segment cells into their sub-cellular compartments. Within the supervised learning paradigm, deep learning based segmentation methods demonstrating human level performance have emerged. However, limited work has been done in developing such generalist methods within the label-free unsupervised context. Here we present an unsupervised segmentation (UNSEG) method that achieves deep learning level performance without requiring any training data. UNSEG leverages a Bayesian-like framework and the specificity of nucleus and cell membrane markers to construct an a posteriori probability estimate of each pixel belonging to the nucleus, cell membrane, or background. It uses this estimate to segment each cell into its nuclear and cell-membrane compartments. We show that UNSEG is more internally consistent and better at generalizing to the complexity of tissue morphology than current deep learning methods. This allows UNSEG to unambiguously identify the cytoplasmic compartment of a cell, which we employ to demonstrate its use in an exemplar biological scenario. Within the UNSEG framework, we also introduce a new perturbed watershed algorithm capable of stably and automatically segmenting a cluster of cell nuclei into individual cell nuclei that increases the accuracy of classical watershed. Perturbed watershed can also be used as a standalone algorithm that researchers can incorporate within their supervised or unsupervised learning approaches to extend classical watershed, particularly in the multiplexed imaging context. Finally, as part of developing UNSEG, we have generated a high-quality annotated gastrointestinal tissue (GIT) dataset, which we anticipate will be useful for the broader research community. We demonstrate the efficacy of UNSEG on the GIT dataset, publicly available datasets, and on a range of practical scenarios. In these contexts, we also discuss the possibility of bias inherent in quantification of segmentation accuracy based on F 1 score. Segmentation, despite its long antecedents, remains a challenging problem, particularly in the context of tissue samples. UNSEG, an easy-to-use algorithm, provides an unsupervised approach to overcome this bottleneck, and as we discuss, can help improve deep learning based segmentation methods by providing a bridge between unsupervised and supervised learning paradigms.

A reinterpretation of the gap fraction of tree crowns from the perspectives of computer graphics and porous media theory

The gap fraction (GF) of vegetative canopies is an important property related to the contained bulk of reproductive elements and woody facets within the tree crown volume. This work was developed from the perspectives of porous media theory and computer graphics techniques, considering the vegetative elements in the canopy as a solid matrix and treating the gaps between them as pores to guide volume-based GF_vol calculations. Woody components and individual leaves were extracted from terrestrial laser scanning data. The concept of equivalent leaf thickness describing the degrees of leaf curling and drooping was proposed to construct hexagonal prisms properly enclosing the scanned points of each leaf, and cylinder models were adopted to fit each branch segment, enabling the calculation of the equivalent leaf and branch volumes within the crown. Finally, the volume-based GF_vol of the tree crown following the definition of the void fraction in porous media theory was calculated as one minus the ratio of the total plant leaf and branch volume to the canopy volume. This approach was tested on five tree species and a forest plot with variable canopy architecture, yielding an estimated maximum volume-based GF_vol of 0.985 for a small crepe myrtle and a minimal volume-based GF_vol of 0.953 for a sakura tree. The 3D morphology of each compositional element in the tree canopy was geometrically defined and the canopy was considered a porous structure to conduct GF_vol calculations based on multidisciplinary theory.

A comprehensive review of computational and image analysis techniques for quantitative evaluation of striated muscle tissue architecture

Unbiased evaluation of morphology is crucial to understanding development, mechanics, and pathology of striated muscle tissues. Indeed, the ability of striated muscles to contract and the strength of their contraction is dependent on their tissue-, cellular-, and cytoskeletal-level organization. Accordingly, the study of striated muscles often requires imaging and assessing aspects of their architecture at multiple different spatial scales. While an expert may be able to qualitatively appraise tissues, it is imperative to have robust, repeatable tools to quantify striated myocyte morphology and behavior that can be used to compare across different labs and experiments. There has been a recent effort to define the criteria used by experts to evaluate striated myocyte architecture. In this review, we will describe metrics that have been developed to summarize distinct aspects of striated muscle architecture in multiple different tissues, imaged with various modalities. Additionally, we will provide an overview of metrics and image processing software that needs to be developed. Importantly to any lab working on striated muscle platforms, characterization of striated myocyte morphology using the image processing pipelines discussed in this review can be used to quantitatively evaluate striated muscle tissues and contribute to a robust understanding of the development and mechanics of striated muscles.

Spatial statistics is a comprehensive tool for quantifying cell neighbor relationships and biological processes via tissue image analysis

Automated microscopy and computational image analysis has transformed cell biology, providing quantitative, spatially resolved information on cells and their constituent molecules from the sub-micron to the whole-organ scale. Here we explore the application of spatial statistics to the cellular relationships within tissue microscopy data and discuss how spatial statistics offers cytometry a powerful yet underused mathematical tool set for which the required data are readily captured using standard protocols and microscopy equipment. We also highlight the often-overlooked need to carefully consider the structural heterogeneity of tissues in terms of the applicability of different statistical measures and their accuracy and demonstrate how spatial analyses offer a great deal more than just basic quantification of biological variance. Ultimately, we highlight how statistical modeling can help reveal the hierarchical spatial processes that connect the properties of individual cells to the establishment of biological function.

Adaptive tracking algorithm for trajectory analysis of cells and layer-by-layer assessment of motility dynamics

Tracking biological objects such as cells or subcellular components imaged with time-lapse microscopy enables us to understand the molecular principles about the dynamics of cell behaviors. However, automatic object detection, segmentation and extracting trajectories remain as a rate-limiting step due to intrinsic challenges of video processing. This paper presents an adaptive tracking algorithm (Adtari) that automatically finds the optimum search radius and cell linkages to determine trajectories in consecutive frames. A critical assumption in most tracking studies is that displacement remains unchanged throughout the movie and cells in a few frames are usually analyzed to determine its magnitude. Tracking errors and inaccurate association of cells may occur if the user does not correctly evaluate the value or prior knowledge is not present on cell movement. The key novelty of our method is that minimum intercellular distance and maximum displacement of cells between frames are dynamically computed and used to determine the threshold distance. Since the space between cells is highly variable in a given frame, our software recursively alters the magnitude to determine all plausible matches in the trajectory analysis. Our method therefore eliminates a major preprocessing step where a constant distance was used to determine the neighbor cells in tracking methods. Cells having multiple overlaps and splitting events were further evaluated by using the shape attributes including perimeter, area, ellipticity and distance. The features were applied to determine the closest matches by minimizing the difference in their magnitudes. Finally, reporting section of our software were used to generate instant maps by overlaying cell features and trajectories. Adtari was validated by using videos with variable signal-to-noise, contrast ratio and cell density. We compared the adaptive tracking with constant distance and other methods to evaluate performance and its efficiency. Our algorithm yields reduced mismatch ratio, increased ratio of whole cell track, higher frame tracking efficiency and allows layer-by-layer assessment of motility to characterize single-cells. Adaptive tracking provides a reliable, accurate, time efficient and user-friendly open source software that is well suited for analysis of 2D fluorescence microscopy video datasets.

A disector-based framework for the automatic optical fractionator

Stereology-based methods provide the current state-of-the-art approaches for accurate quantification of numbers and other morphometric parameters of biological objects in stained tissue sections. The advent of artificial intelligence (AI)-based deep learning (DL) offers the possibility of improving throughput by automating the collection of stereology data. We have recently shown that DL can effectively achieve comparable accuracy to manual stereology but with higher repeatability, improved throughput, and less variation due to human factors by quantifying the total number of immunostained cells at their maximal profile of focus in extended depth of field (EDF) images. In the first of two novel contributions in this work, we propose a semi-automatic approach using a handcrafted Adaptive Segmentation Algorithm (ASA) to automatically generate ground truth on EDF images for training our deep learning (DL) models to automatically count cells using unbiased stereology methods. This update increases the amount of training data, thereby improving the accuracy and efficiency of automatic cell counting methods, without a requirement for extra expert time. The second contribution of this work is a Multi-channel Input and Multi-channel Output (MIMO) method using a U-Net deep learning architecture for automatic cell counting in a stack of z-axis images (also known as disector stacks). This DL-based digital automation of the ordinary optical fractionator ensures accurate counts through spatial separation of stained cells in the z-plane, thereby avoiding false negatives from overlapping cells in EDF images without the shortcomings of 3D and recurrent DL models. The contribution overcomes the issue of under-counting errors with EDF images due to overlapping cells in the z-plane (masking). We demonstrate the practical applications of these advances with automatic disector-based estimates of the total number of NeuN-immunostained neurons in a mouse neocortex. In summary, this work provides the first demonstration of automatic estimation of a total cell number in tissue sections using a combination of deep learning and the disector-based optical fractionator method.

Mapping fetal brain development based on automated segmentation and 4D brain atlasing

Fetal brain MRI has become an important tool for in utero assessment of brain development and disorders. However, quantitative analysis of fetal brain MRI remains difficult, partially due to the limited tools for automated preprocessing and the lack of normative brain templates. In this paper, we proposed an automated pipeline for fetal brain extraction, super-resolution reconstruction, and fetal brain atlasing to quantitatively map in utero fetal brain development during mid-to-late gestation in a Chinese population. First, we designed a U-net convolutional neural network for automated fetal brain extraction, which achieved an average accuracy of 97%. We then generated a developing fetal brain atlas, using an iterative linear and nonlinear registration approach. Based on the 4D spatiotemporal atlas, we quantified the morphological development of the fetal brain between 23 and 36 weeks of gestation. The proposed pipeline enabled the fully automated volumetric reconstruction for clinically available fetal brain MRI data, and the 4D fetal brain atlas provided normative templates for the quantitative characterization of fetal brain development, especially in the Chinese population.

A method for quantification of vesicular compartments within cells using 3D reconstructed confocal z-stacks: Comparison of ImageJ and Imaris to count early endosomes within basal forebrain cholinergic neurons

BACKGROUND

Phenotypic changes in vesicular compartments are an early pathological hallmark of many peripheral and central diseases. For example, accurate assessment of early endosome pathology is crucial to the study of Down syndrome (DS) and Alzheimer's disease (AD), as well as other neurological disorders with endosomal-lysosomal pathology.

NEW METHOD

We describe a method for quantification of immunolabeled early endosomes within transmitter-identified basal forebrain cholinergic neurons (BFCNs) using 3-dimensional (3D) reconstructed confocal z-stacks employing Imaris software.

RESULTS

Quantification of 3D reconstructed z-stacks was performed using two different image analysis programs: ImageJ and Imaris. We found ImageJ consistently overcounted the number of early endosomes present within individual BFCNs. Difficulty separating densely packed early endosomes within defined BFCNs was observed in ImageJ compared to Imaris.

COMPARISON WITH EXISTING METHODS

Previous methods quantifying endosomal-lysosomal pathology relied on confocal microscopy images taken in a single plane of focus. Since early endosomes are distributed throughout the soma and neuronal processes of BFCNs, critical insight into the abnormal early endosome phenotype may be lost as a result of analyzing only a single image of the perikaryon. Rather than relying on a representative sampling, this protocol enables precise, direct quantification of all immunolabeled vesicles within a defined cell of interest.

CONCLUSIONS

Imaris is an ideal program for accurately counting punctate vesicles in the context of dual label confocal microscopy. Superior image resolution and detailed algorithms offered by Imaris make precise and rigorous quantification of individual early endosomes dispersed throughout a BFCN in 3D space readily achievable.

Electron tomography and immunogold labeling of plant cells

Electron microscopy enables the imaging of organelles and macromolecular complexes within cells at nanometer scale resolution. Electron tomography of biological samples, either in vitrified ice or fixed and embedded in resin, provides three-dimensional structural information of relatively small volumes (a few cubic microns) of cells at axial resolutions of 1-7nm. This chapter discusses approaches for plant sample preparation by high-pressure freezing/freeze-substitution and resin-embedding for electron tomography and immunogold labeling using transmission electron microscopy.

High-Throughput Segmentation of Tiled Biological Structures using Random-Walk Distance Transforms

Various 3D imaging techniques are routinely used to examine biological materials, the results of which are usually a stack of grayscale images. In order to quantify structural aspects of the biological materials, however, they must first be extracted from the dataset in a process called segmentation. If the individual structures to be extracted are in contact or very close to each other, distance-based segmentation methods utilizing the Euclidean distance transform are commonly employed. Major disadvantages of the Euclidean distance transform, however, are its susceptibility to noise (very common in biological data), which often leads to incorrect segmentations (i.e., poor separation of objects of interest), and its limitation of being only effective for roundish objects. In the present work, we propose an alternative distance transform method, the random-walk distance transform, and demonstrate its effectiveness in high-throughput segmentation of three microCT datasets of biological tilings (i.e., structures composed of a large number of similar repeating units). In contrast to the Euclidean distance transform, the random-walk approach represents the global, rather than the local, geometric character of the objects to be segmented and, thus, is less susceptible to noise. In addition, it is directly applicable to structures with anisotropic shape characteristics. Using three case studies—tessellated cartilage from a stingray, the dermal endoskeleton of a starfish, and the prismatic layer of a bivalve mollusc shell—we provide a typical workflow for the segmentation of tiled structures, describe core image processing concepts that are underused in biological research, and show that for each study system, large amounts of biologically-relevant data can be rapidly segmented, visualized, and analyzed.

In vitro monitoring of photoinduced necrosis in HeLa cells using digital holographic microscopy and machine learning

Digital holographic microscopy supplemented with the developed cell segmentation and machine learning and classification algorithms is implemented for quantitative description of the dynamics of cellular necrosis induced by photodynamic treatment in vitro. It is demonstrated that the developed algorithms operating with a set of optical, morphological, and physiological parameters of cells, obtained from their phase images, can be used for automatic distinction between live and necrotic cells. The developed classifier provides high accuracy of about 95.5% and allows for calculation of survival rates in the course of cell death.

FocAn: automated 3D analysis of DNA repair foci in image stacks acquired by confocal fluorescence microscopy

Background

Phosphorylated histone H2AX, also known as γH2AX, forms μm-sized nuclear foci at the sites of DNA double-strand breaks (DSBs) induced by ionizing radiation and other agents. Due to their specificity and sensitivity, γH2AX immunoassays have become the gold standard for studying DSB induction and repair. One of these assays relies on the immunofluorescent staining of γH2AX followed by microscopic imaging and foci counting. During the last years, semi- and fully automated image analysis, capable of fast detection and quantification of γH2AX foci in large datasets of fluorescence images, are gradually replacing the traditional method of manual foci counting. A major drawback of the non-commercial software for foci counting (available so far) is that they are restricted to 2D-image data. In practice, these algorithms are useful for counting the foci located close to the midsection plane of the nucleus, while the out-of-plane foci are neglected.

Results

To overcome the limitations of 2D foci counting, we present a freely available ImageJ-based plugin (FocAn) for automated 3D analysis of γH2AX foci in z-image stacks acquired by confocal fluorescence microscopy. The image-stack processing algorithm implemented in FocAn is capable of automatic 3D recognition of individual cell nuclei and γH2AX foci, as well as evaluation of the total foci number per cell nucleus. The FocAn algorithm consists of two parts: nucleus identification and foci detection, each employing specific sequences of auto local thresholding in combination with watershed segmentation techniques. We validated the FocAn algorithm using fluorescence-labeled γH2AX in two glioblastoma cell lines, irradiated with 2 Gy and given up to 24 h post-irradiation for repair. We found that the data obtained with FocAn agreed well with those obtained with an already available software (FoCo) and manual counting. Moreover, FocAn was capable of identifying overlapping foci in 3D space, which ensured accurate foci counting even at high DSB density of up to ~ 200 DSB/nucleus.

Conclusions

FocAn is freely available an open-source 3D foci analyzer. The user-friendly algorithm FocAn requires little supervision and can automatically count the amount of DNA-DSBs, i.e. fluorescence-labeled γH2AX foci, in 3D image stacks acquired by laser-scanning microscopes without additional nuclei staining.

DeepSynth: Three-dimensional nuclear segmentation of biological images using neural networks trained with synthetic data

The scale of biological microscopy has increased dramatically over the past ten years, with the development of new modalities supporting collection of high-resolution fluorescence image volumes spanning hundreds of microns if not millimeters. The size and complexity of these volumes is such that quantitative analysis requires automated methods of image processing to identify and characterize individual cells. For many workflows, this process starts with segmentation of nuclei that, due to their ubiquity, ease-of-labeling and relatively simple structure, make them appealing targets for automated detection of individual cells. However, in the context of large, three-dimensional image volumes, nuclei present many challenges to automated segmentation, such that conventional approaches are seldom effective and/or robust. Techniques based upon deep-learning have shown great promise, but enthusiasm for applying these techniques is tempered by the need to generate training data, an arduous task, particularly in three dimensions. Here we present results of a new technique of nuclear segmentation using neural networks trained on synthetic data. Comparisons with results obtained using commonly-used image processing packages demonstrate that DeepSynth provides the superior results associated with deep-learning techniques without the need for manual annotation.

Three-dimensional GPU-accelerated active contours for automated localization of cells in large images

Cell segmentation in microscopy is a challenging problem, since cells are often asymmetric and densely packed. Successful cell segmentation algorithms rely identifying seed points, and are highly sensitive to variablility in cell size. In this paper, we present an efficient and highly parallel formulation for symmetric three-dimensional contour evolution that extends previous work on fast two-dimensional snakes. We provide a formulation for optimization on 3D images, as well as a strategy for accelerating computation on consumer graphics hardware. The proposed software takes advantage of Monte-Carlo sampling schemes in order to speed up convergence and reduce thread divergence. Experimental results show that this method provides superior performance for large 2D and 3D cell localization tasks when compared to existing methods on large 3D brain images.

In vitro and in vivo evaluations of nanocrystalline Zn-doped carbonated hydroxyapatite/alginate microspheres: zinc and calcium bioavailability and bone regeneration

Background: Zinc-doped hydroxyapatite has been proposed as a graft biomaterial for bone regeneration. However, the effect of zinc on osteoconductivity is still controversial, since the release and resorption of calcium, phosphorus, and zinc in graft-implanted defects have rarely been studied.

Methods: Microspheres containing alginate and either non-doped carbonated hydroxyapatite (cHA) or nanocrystalline 3.2 wt% zinc-doped cHA (Zn-cHA) were implanted in critical-sized calvarial defects in Wistar rats for 1, 3, and 6 months. Histological and histomorphometric analyses were performed to evaluate the volume density of newly formed bone, residual biomaterial, and connective tissue formation. Biomaterial degradation was characterized by transmission electron microscopy (TEM) and synchrotron radiation-based X-ray microfluorescence (SR-µXRF), which enabled the elemental mapping of calcium, phosphorus, and zinc on the microsphere-implanted defects at 6 months post-implantation.

Results: The bone repair was limited to regions close to the preexistent bone, whereas connective tissue occupied the major part of the defect. Moreover, no significant difference in the amount of new bone formed was found between the two microsphere groups. TEM analysis revealed the degradation of the outer microsphere surface with detachment of the nanoparticle aggregates. According to SR-µXRF, both types of microspheres released high amounts of calcium, phosphorus, and zinc, distributed throughout the defective region. The cHA microsphere surface strongly adsorbed the zinc from organic constituents of the biological fluid, and phosphorus was resorbed more quickly than calcium. In the Zn-cHA group, zinc and calcium had similar release profiles, indicating a stoichiometric dissolution of these elements and non-preferential zinc resorption.

Conclusions: The nanometric size of cHA and Zn-cHA was a decisive factor in accelerating the in vivo availability of calcium and zinc. The high calcium and zinc accumulation in the defect, which was not cleared by the biological medium, played a critical role in inhibiting osteoconduction and thus impairing bone repair.

Practical Considerations in Particle and Object Tracking and Analysis

The rapid advancement of live-cell imaging technologies has enabled biologists to generate high-dimensional data to follow biological movement at the microscopic level. Yet, the "perceived" ease of use of modern microscopes has led to challenges whereby sub-optimal data are commonly generated that cannot support quantitative tracking and analysis as a result of various ill-advised decisions made during image acquisition. Even optimally acquired images often require further optimization through digital processing before they can be analyzed. In writing this article, we presume our target audience to be biologists with a foundational understanding of digital image acquisition and processing, who are seeking to understand the essential steps for particle/object tracking experiments. It is with this targeted readership in mind that we review the basic principles of image-processing techniques as well as analysis strategies commonly used for tracking experiments. We conclude this technical survey with a discussion of how movement behavior can be mathematically modeled and described. © 2019 by John Wiley & Sons, Inc.

Separating Touching Cells Using Pixel Replicated Elliptical Shape Models

One of the most important and error-prone tasks in biological image analysis is the segmentation of touching or overlapping cells. Particularly for optical microscopy, including transmitted light and confocal fluorescence microscopy, there is often no consistent discriminative information to separate cells that touch or overlap. It is desired to partition touching foreground pixels into cells using the binary threshold image information only, and optionally incorporating gradient information. The most common approaches for segmenting touching and overlapping cells in these scenarios are based on the watershed transform. We describe a new approach called pixel replication for the task of segmenting elliptical objects that touch or overlap. Pixel replication uses the image Euclidean distance transform in combination with Gaussian mixture models to better exploit practically effective optimization for delineating objects with elliptical decision boundaries. Pixel replication improves significantly on commonly used methods based on watershed transforms, or based on fitting Gaussian mixtures directly to the thresholded image data. Pixel replication works equivalently on both 2-D and 3-D image data, and naturally combines information from multi-channel images. The accuracy of the proposed technique is measured using both the segmentation accuracy on simulated ellipse data and the tracking accuracy on validated stem cell tracking results extracted from hundreds of live-cell microscopy image sequences. Pixel replication is shown to be significantly more accurate compared with other approaches. Variance relationships are derived, allowing a more practically effective Gaussian mixture model to extract cell boundaries for data generated from the threshold image using the uniform elliptical distribution and from the distance transform image using the triangular elliptical distribution.

Segmentation of Heavily Clustered Nuclei from Histopathological Images

Automated cell nucleus segmentation is the key to gain further insight into cell features and functionality which support computer-aided pathology in early diagnosis of diseases such as breast cancer and brain tumour. Despite considerable advances in automated segmentation, it still remains a challenging task to split heavily clustered nuclei due to intensity variations caused by noise and uneven absorption of stains. To address this problem, we propose a novel method applicable to variety of histopathological images stained for different proteins, with high speed, accuracy and level of automation. Our algorithm is initiated by applying a new locally adaptive thresholding method on watershed regions. Followed by a new splitting technique based on multilevel thresholding and the watershed algorithm to separate clustered nuclei. Finalized by a model-based merging step to eliminate oversegmentation and a model-based correction step to improve segmentation results and eliminate small objects. We have applied our method to three image datasets: breast cancer stained for hematoxylin and eosin (H&E), Drosophila Kc167 cells stained for DNA to label nuclei, and mature neurons stained for NeuN. Evaluated results show our method outperforms the state-of-the-art methods in terms of accuracy, precision, F1-measure, and computational time.

Three-Dimensional Embryonic Image Segmentation and Registration Based on Shape Index and Ellipsoid-Fitting Method

Quantitative analysis based on three-dimensional differential interference contrast (DIC) images is currently the mainstream in analyzing gene functions involved in early cell fate specifications. Segmentation and registration are the two most important steps in analysis. Many image segmentation methods have poor performance on embryonic DIC images because of the interference of egg shells, blurs, and nonuniform intensity background. A novel segmentation method is presented based on the shape index (SI) of local intensity variation in DIC images. To compute the SI, the intensity values of a local neighborhood are regarded as z coordinates over x-y planes. After calculating the SI for each pixel by evaluating the shape of intensity surface over the corresponding local neighborhood, SI thresholding is used to detect cytoplasm granules within embryonic boundaries. As a scalar and rotation invariant, the SI is both insensitive to directional changes and different ranges of intensity variations. Embryonic registration methods are usually based on the orientation of vertebrate anteroposterior (AP) axes computed from segmented embryonic boundaries. Due to the blur of marginal slices in DIC images, usually the segmented boundaries are incomplete, which may make the computed AP axes shift away from the correct orientation when using the principal component analysis method. A method calculating AP axes based on ellipsoid-fitting is proposed, which can achieve high accuracy when handling incomplete segmented boundaries. Using Worm Developmental Dynamics Database, we evaluated the performance of the proposed segmentation method and the computation of AP axes. Experimental results show that the two methods outperform existing methods.

VOTING-BASED SEGMENTATION OF OVERLAPPING NUCLEI IN CLARITY IMAGES

New tissue-clearing techniques and improvements in optical microscopy have rapidly advanced capabilities to acquire volumetric imagery of neural tissue at resolutions of one micron or better. As sizes for data collections increase, accurate automatic segmentation of cell nuclei becomes increasingly important for quantitative analysis of imaged tissue. We present a cell nucleus segmentation method that is formulated as a parameter estimation problem with the goal of determining the count, shapes, and locations of nuclei that most accurately describe an image. We applied our new voting-based approach to fluorescence confocal microscopy images of neural tissue stained with DAPI, which highlights nuclei. Compared to manual counting of cells in three DAPI images, our method outperformed three existing approaches. On a manually labeled high-resolution DAPI image, our method also outperformed those methods and achieved a cell count accuracy of 98.99% and mean Dice coefficient of 0.6498.

Segmentation of clustered cells in negative phase contrast images with integrated light intensity and cell shape information

Automated cell segmentation plays a key role in characterisations of cell behaviours for both biology research and clinical practices. Currently, the segmentation of clustered cells still remains as a challenge and is the main reason for false segmentation. In this study, the emphasis was put on the segmentation of clustered cells in negative phase contrast images. A new method was proposed to combine both light intensity and cell shape information through the construction of grey-weighted distance transform (GWDT) within preliminarily segmented areas. With the constructed GWDT, the clustered cells can be detected and then separated with a modified region skeleton-based method. Moreover, a contour expansion operation was applied to get optimised detection of cell boundaries. In this paper, the working principle and detailed procedure of the proposed method are described, followed by the evaluation of the method on clustered cell segmentation. Results show that the proposed method achieves an improved performance in clustered cell segmentation compared with other methods, with 85.8% and 97.16% accuracy rate for clustered cells and all cells, respectively.

Modeling, validation and verification of three-dimensional cell-scaffold contacts from terabyte-sized images

Background

Cell-scaffold contact measurements are derived from pairs of co-registered volumetric fluorescent confocal laser scanning microscopy (CLSM) images (z-stacks) of stained cells and three types of scaffolds (i.e., spun coat, large microfiber, and medium microfiber). Our analysis of the acquired terabyte-sized collection is motivated by the need to understand the nature of the shape dimensionality (1D vs 2D vs 3D) of cell-scaffold interactions relevant to tissue engineers that grow cells on biomaterial scaffolds.

Results

We designed five statistical and three geometrical contact models, and then down-selected them to one from each category using a validation approach based on physically orthogonal measurements to CLSM. The two selected models were applied to 414 z-stacks with three scaffold types and all contact results were visually verified. A planar geometrical model for the spun coat scaffold type was validated from atomic force microscopy images by computing surface roughness of 52.35 nm ±31.76 nm which was 2 to 8 times smaller than the CLSM resolution. A cylindrical model for fiber scaffolds was validated from multi-view 2D scanning electron microscopy (SEM) images. The fiber scaffold segmentation error was assessed by comparing fiber diameters from SEM and CLSM to be between 0.46% to 3.8% of the SEM reference values. For contact verification, we constructed a web-based visual verification system with 414 pairs of images with cells and their segmentation results, and with 4968 movies with animated cell, scaffold, and contact overlays. Based on visual verification by three experts, we report the accuracy of cell segmentation to be 96.4% with 94.3% precision, and the accuracy of cell-scaffold contact for a statistical model to be 62.6% with 76.7% precision and for a geometrical model to be 93.5% with 87.6% precision.

Conclusions

The novelty of our approach lies in (1) representing cell-scaffold contact sites with statistical intensity and geometrical shape models, (2) designing a methodology for validating 3D geometrical contact models and (3) devising a mechanism for visual verification of hundreds of 3D measurements. The raw and processed data are publicly available from https://isg.nist.gov/deepzoomweb/data/ together with the web -based verification system.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-017-1928-x) contains supplementary material, which is available to authorized users.

Auto-Context Convolutional Neural Network (Auto-Net) for Brain Extraction in Magnetic Resonance Imaging

Brain extraction or whole brain segmentation is an important first step in many of the neuroimage analysis pipelines. The accuracy and the robustness of brain extraction, therefore, are crucial for the accuracy of the entire brain analysis process. The state-of-the-art brain extraction techniques rely heavily on the accuracy of alignment or registration between brain atlases and query brain anatomy, and/or make assumptions about the image geometry, and therefore have limited success when these assumptions do not hold or image registration fails. With the aim of designing an accurate, learning-based, geometry-independent, and registration-free brain extraction tool, in this paper, we present a technique based on an auto-context convolutional neural network (CNN), in which intrinsic local and global image features are learned through 2-D patches of different window sizes. We consider two different architectures: 1) a voxelwise approach based on three parallel 2-D convolutional pathways for three different directions (axial, coronal, and sagittal) that implicitly learn 3-D image information without the need for computationally expensive 3-D convolutions and 2) a fully convolutional network based on the U-net architecture. Posterior probability maps generated by the networks are used iteratively as context information along with the original image patches to learn the local shape and connectedness of the brain to extract it from non-brain tissue. The brain extraction results we have obtained from our CNNs are superior to the recently reported results in the literature on two publicly available benchmark data sets, namely, LPBA40 and OASIS, in which we obtained the Dice overlap coefficients of 97.73% and 97.62%, respectively. Significant improvement was achieved via our auto-context algorithm. Furthermore, we evaluated the performance of our algorithm in the challenging problem of extracting arbitrarily oriented fetal brains in reconstructed fetal brain magnetic resonance imaging (MRI) data sets. In this application, our voxelwise auto-context CNN performed much better than the other methods (Dice coefficient: 95.97%), where the other methods performed poorly due to the non-standard orientation and geometry of the fetal brain in MRI. Through training, our method can provide accurate brain extraction in challenging applications. This, in turn, may reduce the problems associated with image registration in segmentation tasks.

Touching Soma Segmentation Based on the Rayburst Sampling Algorithm

Neuronal soma segmentation is essential for morphology quantification analysis. Rapid advances in light microscope imaging techniques have generated such massive amounts of data that time-consuming manual methods cannot meet requirements for high throughput. However, touching soma segmentation is still a challenge for automatic segmentation methods. In this paper, we propose a soma segmentation method that combines the Rayburst sampling algorithm and ellipsoid fitting. The improved Rayburst sampling algorithm is used to detect the soma surface; the ellipsoid fitting method then refines jagged sampled soma surface to generate smooth ellipsoidal shapes for efficient analysis. In experiments, we validated the proposed method by applying it to datasets from the fluorescence micro-optical sectioning tomography (fMOST) system. The results indicate that the proposed method is comparable to the manual segmented gold standard with accurate soma segmentation at a relatively high speed. The proposed method can be extended to large-scale image stacks in the future.

Automatically tracking neurons in a moving and deforming brain

Advances in optical neuroimaging techniques now allow neural activity to be recorded with cellular resolution in awake and behaving animals. Brain motion in these recordings pose a unique challenge. The location of individual neurons must be tracked in 3D over time to accurately extract single neuron activity traces. Recordings from small invertebrates like C. elegans are especially challenging because they undergo very large brain motion and deformation during animal movement. Here we present an automated computer vision pipeline to reliably track populations of neurons with single neuron resolution in the brain of a freely moving C. elegans undergoing large motion and deformation. 3D volumetric fluorescent images of the animal's brain are straightened, aligned and registered, and the locations of neurons in the images are found via segmentation. Each neuron is then assigned an identity using a new time-independent machine-learning approach we call Neuron Registration Vector Encoding. In this approach, non-rigid point-set registration is used to match each segmented neuron in each volume with a set of reference volumes taken from throughout the recording. The way each neuron matches with the references defines a feature vector which is clustered to assign an identity to each neuron in each volume. Finally, thin-plate spline interpolation is used to correct errors in segmentation and check consistency of assigned identities. The Neuron Registration Vector Encoding approach proposed here is uniquely well suited for tracking neurons in brains undergoing large deformations. When applied to whole-brain calcium imaging recordings in freely moving C. elegans, this analysis pipeline located 156 neurons for the duration of an 8 minute recording and consistently found more neurons more quickly than manual or semi-automated approaches.

Automating cell detection and classification in human brain fluorescent microscopy images using dictionary learning and sparse coding

BACKGROUND

Immunofluorescence (IF) plays a major role in quantifying protein expression in situ and understanding cell function. It is widely applied in assessing disease mechanisms and in drug discovery research. Automation of IF analysis can transform studies using experimental cell models. However, IF analysis of postmortem human tissue relies mostly on manual interaction, often subjected to low-throughput and prone to error, leading to low inter and intra-observer reproducibility. Human postmortem brain samples challenges neuroscientists because of the high level of autofluorescence caused by accumulation of lipofuscin pigment during aging, hindering systematic analyses. We propose a method for automating cell counting and classification in IF microscopy of human postmortem brains. Our algorithm speeds up the quantification task while improving reproducibility.

NEW METHOD

Dictionary learning and sparse coding allow for constructing improved cell representations using IF images. These models are input for detection and segmentation methods. Classification occurs by means of color distances between cells and a learned set.

RESULTS

Our method successfully detected and classified cells in 49 human brain images. We evaluated our results regarding true positive, false positive, false negative, precision, recall, false positive rate and F1 score metrics. We also measured user-experience and time saved compared to manual countings.

COMPARISON WITH EXISTING METHODS

We compared our results to four open-access IF-based cell-counting tools available in the literature. Our method showed improved accuracy for all data samples.

CONCLUSION

The proposed method satisfactorily detects and classifies cells from human postmortem brain IF images, with potential to be generalized for applications in other counting tasks.

Automatic extraction of cell nuclei from H&E-stained histopathological images

Extraction of cell nuclei from hematoxylin and eosin (H&E)-stained histopathological images is an essential preprocessing step in computerized image analysis for disease detection, diagnosis, and prognosis. We present an automated cell nuclei segmentation approach that works with H&E-stained images. A color deconvolution algorithm was first applied to the image to get the hematoxylin channel. Using a morphological operation and thresholding technique on the hematoxylin channel image, candidate target nuclei and background regions were detected, which were then used as markers for a marker-controlled watershed transform segmentation algorithm. Moreover, postprocessing was conducted to split the touching nuclei. For each segmented region from the previous steps, the regional maximum value positions were identified as potential nuclei centers. These maximum values were further grouped into [Formula: see text]-clusters, and the locations within each cluster were connected with the minimum spanning tree technique. Then, these connected positions were utilized as new markers for a watershed segmentation approach. The final number of nuclei at each region was determined by minimizing an objective function that iterated all of the possible [Formula: see text]-values. The proposed method was applied to the pathological images of the tumor tissues from The Cancer Genome Atlas study. Experimental results show that the proposed method can lead to promising results in terms of segmentation accuracy and separation of touching nuclei.

A novel toolbox to investigate tissue spatial organization applied to the study of the islets of Langerhans

Thanks to the development of new 3D Imaging techniques, volumetric data of thick samples, especially tissues, are commonly available. Several algorithms were proposed to analyze cells or nuclei in tissues, however these tools are limited to two dimensions. Within any given tissue, cells are not likely to be organized randomly and as such have specific patterns of cell-cell interaction forming complex communication networks. In this paper, we propose a new set of tools as an approach to segment and analyze tissues in 3D with single cell resolution. This new tool box can identify and compute the geographical location of single cells and analyze the potential physical interactions between different cell types and in 3D. As a proof-of-principle, we applied our methodology to investigation of the cyto-architecture of the islets of Langerhans in mice and monkeys. The results obtained here are a significant improvement in current methodologies and provides new insight into the organization of alpha cells and their cellular interactions within the islet’s cellular framework.

Quantification of neuronal density across cortical depth using automated 3D analysis of confocal image stacks

A new framework for measuring densities of immunolabeled neurons across cortical layers was implemented that combines a confocal microscopy sampling strategy with automated analysis of 3D image stacks. Its utility was demonstrated by quantifying neuronal density in macaque cortical areas V1 and V2. A series of overlapping confocal image stacks were acquired, each spanning from the pial surface to the white matter. DAPI channel images were automatically thresholded, and contiguous regions that included multiple clumped nuclear profiles were split using k-means clustering of image pixels for a set of candidate k values determined based on the clump's area; the most likely candidate segmentation was selected based on criteria that capture expected nuclear profile shape and size. The centroids of putative nuclear profiles estimated from 2D images were then grouped across z planes in an image stack to identify the positions of nuclei in x-y-z. 3D centroids falling outside user-specified exclusion boundaries were deleted, nuclei were classified by the presence or absence of signal in a channel corresponding to an immunolabeled antigen (e.g., the pan-neuronal marker NeuN) at the nuclear centroid location, and the set of classified cells was combined across image stacks to estimate density across cortical depth. The method was validated by comparison with conventional stereological methods. The average neuronal density across cortical layers was 230 × 10³ neurons per mm³ in V1 and 130 × 10³ neurons per mm³ in V2. The method is accurate, flexible, and general enough to measure densities of neurons of various molecularly identified types.

3D Clumped Cell Segmentation Using Curvature Based Seeded Watershed

Image segmentation is an important process that separates objects from the background and also from each other. Applied to cells, the results can be used for cell counting which is very important in medical diagnosis and treatment, and biological research that is often used by scientists and medical practitioners. Segmenting 3D confocal microscopy images containing cells of different shapes and sizes is still challenging as the nuclei are closely packed. The watershed transform provides an efficient tool in segmenting such nuclei provided a reasonable set of markers can be found in the image. In the presence of low-contrast variation or excessive noise in the given image, the watershed transform leads to over-segmentation (a single object is overly split into multiple objects). The traditional watershed uses the local minima of the input image and will characteristically find multiple minima in one object unless they are specified (marker-controlled watershed). An alternative to using the local minima is by a supervised technique called seeded watershed, which supplies single seeds to replace the minima for the objects. Consequently, the accuracy of a seeded watershed algorithm relies on the accuracy of the predefined seeds. In this paper, we present a segmentation approach based on the geometric morphological properties of the ‘landscape’ using curvatures. The curvatures are computed as the eigenvalues of the Shape matrix, producing accurate seeds that also inherit the original shape of their respective cells. We compare with some popular approaches and show the advantage of the proposed method.

An Automatic Learning-Based Framework for Robust Nucleus Segmentation

Computer-aided image analysis of histopathology specimens could potentially provide support for early detection and improved characterization of diseases such as brain tumor, pancreatic neuroendocrine tumor (NET), and breast cancer. Automated nucleus segmentation is a prerequisite for various quantitative analyses including automatic morphological feature computation. However, it remains to be a challenging problem due to the complex nature of histopathology images. In this paper, we propose a learning-based framework for robust and automatic nucleus segmentation with shape preservation. Given a nucleus image, it begins with a deep convolutional neural network (CNN) model to generate a probability map, on which an iterative region merging approach is performed for shape initializations. Next, a novel segmentation algorithm is exploited to separate individual nuclei combining a robust selection-based sparse shape model and a local repulsive deformable model. One of the significant benefits of the proposed framework is that it is applicable to different staining histopathology images. Due to the feature learning characteristic of the deep CNN and the high level shape prior modeling, the proposed method is general enough to perform well across multiple scenarios. We have tested the proposed algorithm on three large-scale pathology image datasets using a range of different tissue and stain preparations, and the comparative experiments with recent state of the arts demonstrate the superior performance of the proposed approach.

Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Digital pathology and microscopy image analysis is widely used for comprehensive studies of cell morphology or tissue structure. Manual assessment is labor intensive and prone to interobserver variations. Computer-aided methods, which can significantly improve the objectivity and reproducibility, have attracted a great deal of interest in recent literature. Among the pipeline of building a computer-aided diagnosis system, nucleus or cell detection and segmentation play a very important role to describe the molecular morphological information. In the past few decades, many efforts have been devoted to automated nucleus/cell detection and segmentation. In this review, we provide a comprehensive summary of the recent state-of-the-art nucleus/cell segmentation approaches on different types of microscopy images including bright-field, phase-contrast, differential interference contrast, fluorescence, and electron microscopies. In addition, we discuss the challenges for the current methods and the potential future work of nucleus/cell detection and segmentation.

A method for the evaluation of thousands of automated 3D stem cell segmentations

There is no segmentation method that performs perfectly with any dataset in comparison to human segmentation. Evaluation procedures for segmentation algorithms become critical for their selection. The problems associated with segmentation performance evaluations and visual verification of segmentation results are exaggerated when dealing with thousands of three-dimensional (3D) image volumes because of the amount of computation and manual inputs needed. We address the problem of evaluating 3D segmentation performance when segmentation is applied to thousands of confocal microscopy images (z-stacks). Our approach is to incorporate experimental imaging and geometrical criteria, and map them into computationally efficient segmentation algorithms that can be applied to a very large number of z-stacks. This is an alternative approach to considering existing segmentation methods and evaluating most state-of-the-art algorithms. We designed a methodology for 3D segmentation performance characterization that consists of design, evaluation and verification steps. The characterization integrates manual inputs from projected surrogate 'ground truth' of statistically representative samples and from visual inspection into the evaluation. The novelty of the methodology lies in (1) designing candidate segmentation algorithms by mapping imaging and geometrical criteria into algorithmic steps, and constructing plausible segmentation algorithms with respect to the order of algorithmic steps and their parameters, (2) evaluating segmentation accuracy using samples drawn from probability distribution estimates of candidate segmentations and (3) minimizing human labour needed to create surrogate 'truth' by approximating z-stack segmentations with 2D contours from three orthogonal z-stack projections and by developing visual verification tools. We demonstrate the methodology by applying it to a dataset of 1253 mesenchymal stem cells. The cells reside on 10 different types of biomaterial scaffolds, and are stained for actin and nucleus yielding 128 460 image frames (on average, 125 cells/scaffold × 10 scaffold types × 2 stains × 51 frames/cell). After constructing and evaluating six candidates of 3D segmentation algorithms, the most accurate 3D segmentation algorithm achieved an average precision of 0.82 and an accuracy of 0.84 as measured by the Dice similarity index where values greater than 0.7 indicate a good spatial overlap. A probability of segmentation success was 0.85 based on visual verification, and a computation time was 42.3 h to process all z-stacks. While the most accurate segmentation technique was 4.2 times slower than the second most accurate algorithm, it consumed on average 9.65 times less memory per z-stack segmentation.

High-throughput histopathological image analysis via robust cell segmentation and hashing

Computer-aided diagnosis of histopathological images usually requires to examine all cells for accurate diagnosis. Traditional computational methods may have efficiency issues when performing cell-level analysis. In this paper, we propose a robust and scalable solution to enable such analysis in a real-time fashion. Specifically, a robust segmentation method is developed to delineate cells accurately using Gaussian-based hierarchical voting and repulsive balloon model. A large-scale image retrieval approach is also designed to examine and classify each cell of a testing image by comparing it with a massive database, e.g., half-million cells extracted from the training dataset. We evaluate this proposed framework on a challenging and important clinical use case, i.e., differentiation of two types of lung cancers (the adenocarcinoma and squamous carcinoma), using thousands of lung microscopic tissue images extracted from hundreds of patients. Our method has achieved promising accuracy and running time by searching among half-million cells .

Cell recognition based on topological sparse coding for microscopy imaging of focused ultrasound treatment

BACKGROUND

Ultrasound is considered a reliable, widely available, non-invasive, and inexpensive imaging technique for assessing and detecting the development phases of cancer; both in vivo and ex vivo, and for understanding the effects on cell cycle and viability after ultrasound treatment.

METHODS

Based on the topological continuity characteristics, and that adjacent points or areas represent similar features, we propose a topological penalized convex objective function of sparse coding, to recognize similar cell phases.

RESULTS

This method introduces new features using a deep learning method of sparse coding with topological continuity characteristics. Large-scale comparison tests demonstrate that the RAW can outperform SIFT GIST and HoG as the input features with this method, achieving higher sensitivity, specificity, F1 score, and accuracy.

CONCLUSIONS

Experimental results show that the proposed topological sparse coding technique is valid and effective for extracting new features, and the proposed system was effective for cell recognition of microscopy images of theMDA-MB-231 cell line. This method allows features from sparse coding learning methods to have topological continuity characteristics, and the RAW features are more applicable for the deep learning of the topological sparse coding method than SIFT GIST and HoG.

Survey statistics of automated segmentations applied to optical imaging of mammalian cells

BACKGROUND

The goal of this survey paper is to overview cellular measurements using optical microscopy imaging followed by automated image segmentation. The cellular measurements of primary interest are taken from mammalian cells and their components. They are denoted as two- or three-dimensional (2D or 3D) image objects of biological interest. In our applications, such cellular measurements are important for understanding cell phenomena, such as cell counts, cell-scaffold interactions, cell colony growth rates, or cell pluripotency stability, as well as for establishing quality metrics for stem cell therapies. In this context, this survey paper is focused on automated segmentation as a software-based measurement leading to quantitative cellular measurements.

METHODS

We define the scope of this survey and a classification schema first. Next, all found and manually filteredpublications are classified according to the main categories: (1) objects of interests (or objects to be segmented), (2) imaging modalities, (3) digital data axes, (4) segmentation algorithms, (5) segmentation evaluations, (6) computational hardware platforms used for segmentation acceleration, and (7) object (cellular) measurements. Finally, all classified papers are converted programmatically into a set of hyperlinked web pages with occurrence and co-occurrence statistics of assigned categories.

RESULTS

The survey paper presents to a reader: (a) the state-of-the-art overview of published papers about automated segmentation applied to optical microscopy imaging of mammalian cells, (b) a classification of segmentation aspects in the context of cell optical imaging, (c) histogram and co-occurrence summary statistics about cellular measurements, segmentations, segmented objects, segmentation evaluations, and the use of computational platforms for accelerating segmentation execution, and (d) open research problems to pursue.

CONCLUSIONS

The novel contributions of this survey paper are: (1) a new type of classification of cellular measurements and automated segmentation, (2) statistics about the published literature, and (3) a web hyperlinked interface to classification statistics of the surveyed papers at https://isg.nist.gov/deepzoomweb/resources/survey/index.html.

A three-dimensional image processing program for accurate, rapid, and semi-automated segmentation of neuronal somata with dense neurite outgrowth

Three-dimensional (3-D) image analysis techniques provide a powerful means to rapidly and accurately assess complex morphological and functional interactions between neural cells. Current software-based identification methods of neural cells generally fall into two applications: (1) segmentation of cell nuclei in high-density constructs or (2) tracing of cell neurites in single cell investigations. We have developed novel methodologies to permit the systematic identification of populations of neuronal somata possessing rich morphological detail and dense neurite arborization throughout thick tissue or 3-D in vitro constructs. The image analysis incorporates several novel automated features for the discrimination of neurites and somata by initially classifying features in 2-D and merging these classifications into 3-D objects; the 3-D reconstructions automatically identify and adjust for over and under segmentation errors. Additionally, the platform provides for software-assisted error corrections to further minimize error. These features attain very accurate cell boundary identifications to handle a wide range of morphological complexities. We validated these tools using confocal z-stacks from thick 3-D neural constructs where neuronal somata had varying degrees of neurite arborization and complexity, achieving an accuracy of ≥95%. We demonstrated the robustness of these algorithms in a more complex arena through the automated segmentation of neural cells in ex vivo brain slices. These novel methods surpass previous techniques by improving the robustness and accuracy by: (1) the ability to process neurites and somata, (2) bidirectional segmentation correction, and (3) validation via software-assisted user input. This 3-D image analysis platform provides valuable tools for the unbiased analysis of neural tissue or tissue surrogates within a 3-D context, appropriate for the study of multi-dimensional cell-cell and cell-extracellular matrix interactions.