Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Automated Decision Support System for Detection of Leukemia from Peripheral Blood Smear Images

Peripheral blood smear analysis plays a vital role in diagnosing many diseases including cancer. Leukemia is a type of cancer which begins in bone marrow and results in increased number of white blood cells in peripheral blood. Unusual variations in appearance of white blood cells indicate leukemia. In this paper, an automated method for detection of leukemia using image processing approach is proposed. In the present study, 1159 images of different brightness levels and color shades were acquired from Leishman stained peripheral blood smears. SVM classifier was used for classification of white blood cells into normal and abnormal, and also for detection of leukemic WBCs from the abnormal class. Classification of the normal white blood cells into five sub-types was performed using NN classifier. Overall classification accuracy of 98.8% was obtained using the combination of NN and SVM.

Fluorescence anisotropy imaging in drug discovery

Non-invasive measurement of drug-target engagement can provide critical insights in the molecular pharmacology of small molecule drugs. Fluorescence polarization/fluorescence anisotropy measurements are commonly employed in protein/cell screening assays. However, the expansion of such measurements to the in vivo setting has proven difficult until recently. With the advent of high-resolution fluorescence anisotropy microscopy it is now possible to perform kinetic measurements of intracellular drug distribution and target engagement in commonly used mouse models. In this review we discuss the background, current advances and future perspectives in intravital fluorescence anisotropy measurements to derive pharmacokinetic and pharmacodynamic measurements in single cells and whole organs.

Society of Toxicologic Pathology Digital Pathology and Image Analysis Special Interest Group Article*: Opinion on the Application of Artificial Intelligence and Machine Learning to Digital Toxicologic Pathology

Toxicologic pathology is transitioning from analog to digital methods. This transition seems inevitable due to a host of ongoing social and medical technological forces. Of these, artificial intelligence (AI) and in particular machine learning (ML) are globally disruptive, rapidly growing sectors of technology whose impact on the long-established field of histopathology is quickly being realized. The development of increasing numbers of algorithms, peering ever deeper into the histopathological space, has demonstrated to the scientific community that AI pathology platforms are now poised to truly impact the future of precision and personalized medicine. However, as with all great technological advances, there are implementation and adoption challenges. This review aims to define common and relevant AI and ML terminology, describe data generation and interpretation, outline current and potential future business cases, discuss validation and regulatory hurdles, and most importantly, propose how overcoming the challenges of this burgeoning technology may shape toxicologic pathology for years to come, enabling pathologists to contribute even more effectively to answering scientific questions and solving global health issues. [Box: see text]

An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images

BACKGROUND

Since nuclei segmentation in histopathology images can provide key information for identifying the presence or stage of a disease, the images need to be assessed carefully. However, color variation in histopathology images, and various structures of nuclei are two major obstacles in accurately segmenting and analyzing histopathology images. Several machine learning methods heavily rely on hand-crafted features which have limitations due to manual thresholding.

RESULTS

To obtain robust results, deep learning based methods have been proposed. Deep convolutional neural networks (DCNN) used for automatically extracting features from raw image data have been proven to achieve great performance. Inspired by such achievements, we propose a nuclei segmentation method based on DCNNs. To normalize the color of histopathology images, we use a deep convolutional Gaussian mixture color normalization model which is able to cluster pixels while considering the structures of nuclei. To segment nuclei, we use Mask R-CNN which achieves state-of-the-art object segmentation performance in the field of computer vision. In addition, we perform multiple inference as a post-processing step to boost segmentation performance. We evaluate our segmentation method on two different datasets. The first dataset consists of histopathology images of various organ while the other consists histopathology images of the same organ. Performance of our segmentation method is measured in various experimental setups at the object-level and the pixel-level. In addition, we compare the performance of our method with that of existing state-of-the-art methods. The experimental results show that our nuclei segmentation method outperforms the existing methods.

CONCLUSIONS

We propose a nuclei segmentation method based on DCNNs for histopathology images. The proposed method which uses Mask R-CNN with color normalization and multiple inference post-processing provides robust nuclei segmentation results. Our method also can facilitate downstream nuclei morphological analyses as it provides high-quality features extracted from histopathology images.

Automated integrated system for stained neuron detection: An end-to-end framework with a high negative predictive rate

BACKGROUND AND OBJECTIVE

Mapping the architecture of the brain is essential for identifying the neural computations that affect behavior. Traditionally in histology, stained objects in tissue slices are hand-marked under a microscope in a manually intensive, time-consuming process. An integrated hardware and software system is needed to automate image acquisition, image processing, and object detection. Such a system would enable high throughput tissue analysis to rapidly map an entire brain.

METHODS

We demonstrate an automated system to detect neurons using a monkey brain slice immunohistochemically stained for retrogradely labeled neurons. The proposed system obtains a reconstructed image of the sample, and stained neurons are detected in three steps. First, the reconstructed image is pre-processed using adaptive histogram equalization. Second, candidates for stained neurons are segmented from each region via marker-controlled watershed transformation (MCWT) using maximally stable extremal regions (MSERs). Third, the candidates are categorized as neurons or non-neurons using deep transfer learning via pre-trained convolutional neural networks (CNN).

RESULTS

The proposed MCWT algorithm was compared qualitatively against MorphLibJ and an IHC analysis tool, while our unified classification algorithm was evaluated quantitatively using ROC analyses. The proposed classification system was first compared with five previously developed layers (AlexNet, VGG-16, VGG-19, GoogleNet, and ResNet). A comparison with conventional multi-stage frameworks followed using six off-the-shelf classifiers [Bayesian network (BN), support vector machines (SVM), decision tree (DT), bagging (BAG), AdaBoost (ADA), and logistic regression (LR)] and two descriptors (LBP and HOG). The system achieved a 0.918 F1-score with an 86.6% negative prediction value. Remarkably, other metrics such as precision, recall, and F-scores surpassed the 90% threshold compared to traditional methods.

CONCLUSIONS

We demonstrate a fully automated, integrated hardware and software system for rapidly acquiring focused images and identifying neurons from a stained brain slice. This system could be adapted for the identification of stained features of any biological tissue.

Towards pixel-to-pixel deep nucleus detection in microscopy images

BACKGROUND

Nucleus is a fundamental task in microscopy image analysis and supports many other quantitative studies such as object counting, segmentation, tracking, etc. Deep neural networks are emerging as a powerful tool for biomedical image computing; in particular, convolutional neural networks have been widely applied to nucleus/cell detection in microscopy images. However, almost all models are tailored for specific datasets and their applicability to other microscopy image data remains unknown. Some existing studies casually learn and evaluate deep neural networks on multiple microscopy datasets, but there are still several critical, open questions to be addressed.

RESULTS

We analyze the applicability of deep models specifically for nucleus detection across a wide variety of microscopy image data. More specifically, we present a fully convolutional network-based regression model and extensively evaluate it on large-scale digital pathology and microscopy image datasets, which consist of 23 organs (or cancer diseases) and come from multiple institutions. We demonstrate that for a specific target dataset, training with images from the same types of organs might be usually necessary for nucleus detection. Although the images can be visually similar due to the same staining technique and imaging protocol, deep models learned with images from different organs might not deliver desirable results and would require model fine-tuning to be on a par with those trained with target data. We also observe that training with a mixture of target and other/non-target data does not always mean a higher accuracy of nucleus detection, and it might require proper data manipulation during model training to achieve good performance.

CONCLUSIONS

We conduct a systematic case study on deep models for nucleus detection in a wide variety of microscopy images, aiming to address several important but previously understudied questions. We present and extensively evaluate an end-to-end, pixel-to-pixel fully convolutional regression network and report a few significant findings, some of which might have not been reported in previous studies. The model performance analysis and observations would be helpful to nucleus detection in microscopy images.

Learning to detect lymphocytes in immunohistochemistry with deep learning

The immune system is of critical importance in the development of cancer. The evasion of destruction by the immune system is one of the emerging hallmarks of cancer. We have built a dataset of 171,166 manually annotated CD3⁺ and CD8⁺ cells, which we used to train deep learning algorithms for automatic detection of lymphocytes in histopathology images to better quantify immune response. Moreover, we investigate the effectiveness of four deep learning based methods when different subcompartments of the whole-slide image are considered: normal tissue areas, areas with immune cell clusters, and areas containing artifacts. We have compared the proposed methods in breast, colon and prostate cancer tissue slides collected from nine different medical centers. Finally, we report the results of an observer study on lymphocyte quantification, which involved four pathologists from different medical centers, and compare their performance with the automatic detection. The results give insights on the applicability of the proposed methods for clinical use. U-Net obtained the highest performance with an F1-score of 0.78 and the highest agreement with manual evaluation (κ=0.72), whereas the average pathologists agreement with reference standard was κ=0.64. The test set and the automatic evaluation procedure are publicly available at lyon19.grand-challenge.org.

LeukocyteMask: An automated localization and segmentation method for leukocyte in blood smear images using deep neural networks

Digital pathology and microscope image analysis is widely used in comprehensive studies of cell morphology. Identification and analysis of leukocytes in blood smear images, acquired from bright field microscope, are vital for diagnosing many diseases such as hepatitis, leukaemia and acquired immune deficiency syndrome (AIDS). The major challenge for robust and accurate identification and segmentation of leukocyte in blood smear images lays in the large variations of cell appearance such as size, colour and shape of cells, the adhesion between leukocytes (white blood cells, WBCs) and erythrocytes (red blood cells, RBCs), and the emergence of substantial dyeing impurities in blood smear images. In this paper, an end-to-end leukocyte localization and segmentation method is proposed, named LeukocyteMask, in which pixel-level prior information is utilized for supervisor training of a deep convolutional neural network, which is then employed to locate the region of interests (ROI) of leukocyte, and finally segmentation mask of leukocyte is obtained based on the extracted ROI by forward propagation of the network. Experimental results validate the effectiveness of the propose method and both the quantitative and qualitative comparisons with existing methods indicate that LeukocyteMask achieves a state-of-the-art performance for the segmentation of leukocyte in terms of robustness and accuracy .

Cell segmentation methods for label-free contrast microscopy: review and comprehensive comparison

BACKGROUND

Because of its non-destructive nature, label-free imaging is an important strategy for studying biological processes. However, routine microscopic techniques like phase contrast or DIC suffer from shadow-cast artifacts making automatic segmentation challenging. The aim of this study was to compare the segmentation efficacy of published steps of segmentation work-flow (image reconstruction, foreground segmentation, cell detection (seed-point extraction) and cell (instance) segmentation) on a dataset of the same cells from multiple contrast microscopic modalities.

RESULTS

We built a collection of routines aimed at image segmentation of viable adherent cells grown on the culture dish acquired by phase contrast, differential interference contrast, Hoffman modulation contrast and quantitative phase imaging, and we performed a comprehensive comparison of available segmentation methods applicable for label-free data. We demonstrated that it is crucial to perform the image reconstruction step, enabling the use of segmentation methods originally not applicable on label-free images. Further we compared foreground segmentation methods (thresholding, feature-extraction, level-set, graph-cut, learning-based), seed-point extraction methods (Laplacian of Gaussians, radial symmetry and distance transform, iterative radial voting, maximally stable extremal region and learning-based) and single cell segmentation methods. We validated suitable set of methods for each microscopy modality and published them online.

CONCLUSIONS

We demonstrate that image reconstruction step allows the use of segmentation methods not originally intended for label-free imaging. In addition to the comprehensive comparison of methods, raw and reconstructed annotated data and Matlab codes are provided.

Machine learning approaches for pathologic diagnosis

Machine learning techniques, especially deep learning techniques such as convolutional neural networks, have been successfully applied to general image recognitions since their overwhelming performance at the 2012 ImageNet Large Scale Visual Recognition Challenge. Recently, such techniques have also been applied to various medical, including histopathological, images to assist the process of medical diagnosis. In some cases, deep learning-based algorithms have already outperformed experienced pathologists for recognition of histopathological images. However, pathological images differ from general images in some aspects, and thus, machine learning of histopathological images requires specialized learning methods. Moreover, many pathologists are skeptical about the ability of deep learning technology to accurately recognize histopathological images because what the learned neural network recognizes is often indecipherable to humans. In this review, we first introduce various applications incorporating machine learning developed to assist the process of pathologic diagnosis, and then describe machine learning problems related to histopathological image analysis, and review potential ways to solve these problems.

Split and Merge Watershed: a two-step method for cell segmentation in fluorescence microscopy images

The development of advanced techniques in medical imaging has allowed scanning of the human body to microscopic levels, making research on cell behavior more complex and more in-depth. Recent studies have focused on cellular heterogeneity since cell-to-cell differences are always present in the cell population and this variability contains valuable information. However, identifying each cell is not an easy task because, in the images acquired from the microscope, there are clusters of cells that are touching one another. Therefore, the segmentation stage is a problem of considerable difficulty in cell image processing. Although several methods for cell segmentation are described in the literature, they have drawbacks in terms of over-segmentation, under-segmentation or misidentification. Consequently, our main motivation in studying cell segmentation was to develop a new method to achieve a good tradeoff between accurately identifying all relevant elements and not inserting segmentation artifacts. This article presents a new method for cell segmentation in fluorescence microscopy images. The proposed approach combines the well-known Marker-Controlled Watershed algorithm (MC-Watershed) with a new, two-step method based on Watershed, Split and Merge Watershed (SM-Watershed): in the first step, or split phase, the algorithm identifies the clusters using inherent characteristics of the cell, such as size and convexity, and separates them using watershed. In the second step, or the merge stage, it identifies the over-segmented regions using proper features of the cells and eliminates the divisions. Before applying our two-step method, the input image is first preprocessed, and the MC-Watershed algorithm is used to generate an initial segmented image. However, this initial result may not be suitable for subsequent tasks, such as cell count or feature extraction, because not all cells are separated, and some cells may be mistakenly confused with the background. Thus, our proposal corrects this issue with its two-step process, reaching a high performance, a suitable tradeoff between over-segmentation and under-segmentation and preserving the shape of the cell, without the need of any labeled data or relying on machine learning processes. The latter is advantageous over state-of-the-art techniques that in order to achieve similar results require labeled data, which may not be available for all of the domains. Two cell datasets were used to validate this approach, and the results were compared with other methods in the literature, using traditional metrics and quality visual assessment. We obtained 90% of average visual accuracy and an F-index higher than 80%. This proposal outperforms other techniques for cell separation, achieving an acceptable balance between over-segmentation and under-segmentation, which makes it suitable for several applications in cell identification, such as virus infection analysis, high-content cell screening, drug discovery, and morphometry.

Practical Considerations in Particle and Object Tracking and Analysis

The rapid advancement of live-cell imaging technologies has enabled biologists to generate high-dimensional data to follow biological movement at the microscopic level. Yet, the "perceived" ease of use of modern microscopes has led to challenges whereby sub-optimal data are commonly generated that cannot support quantitative tracking and analysis as a result of various ill-advised decisions made during image acquisition. Even optimally acquired images often require further optimization through digital processing before they can be analyzed. In writing this article, we presume our target audience to be biologists with a foundational understanding of digital image acquisition and processing, who are seeking to understand the essential steps for particle/object tracking experiments. It is with this targeted readership in mind that we review the basic principles of image-processing techniques as well as analysis strategies commonly used for tracking experiments. We conclude this technical survey with a discussion of how movement behavior can be mathematically modeled and described. © 2019 by John Wiley & Sons, Inc.

Digital pathology and artificial intelligence

In modern clinical practice, digital pathology has a crucial role and is increasingly a technological requirement in the scientific laboratory environment. The advent of whole-slide imaging, availability of faster networks, and cheaper storage solutions has made it easier for pathologists to manage digital slide images and share them for clinical use. In parallel, unprecedented advances in machine learning have enabled the synergy of artificial intelligence and digital pathology, which offers image-based diagnosis possibilities that were once limited only to radiology and cardiology. Integration of digital slides into the pathology workflow, advanced algorithms, and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enable true utilisation and integration of knowledge that is beyond human limits and boundaries, and we believe there is clear potential for artificial intelligence breakthroughs in the pathology setting. In this Review, we discuss advancements in digital slide-based image diagnosis for cancer along with some challenges and opportunities for artificial intelligence in digital pathology.

Digital Staining of High-Definition Fourier Transform Infrared (FT-IR) Images Using Deep Learning

Histological stains, such as hematoxylin and eosin (H&E), are routinely used in clinical diagnosis and research. While these labels offer a high degree of specificity, throughput is limited by the need for multiple samples. Traditional histology stains, such as immunohistochemical labels, also rely only on protein expression and cannot quantify small molecules and metabolites that may aid in diagnosis. Finally, chemical stains and dyes permanently alter the tissue, making downstream analysis impossible. Fourier transform infrared (FT-IR) spectroscopic imaging has shown promise for label-free characterization of important tissue phenotypes and can bypass the need for many chemical labels. Fourier transform infrared classification commonly leverages supervised learning, requiring human annotation that is tedious and prone to errors. One alternative is digital staining, which leverages machine learning to map IR spectra to a corresponding chemical stain. This replaces human annotation with computer-aided alignment. Previous work relies on alignment of adjacent serial tissue sections. Since the tissue samples are not identical at the cellular level, this technique cannot be applied to high-definition FT-IR images. In this paper, we demonstrate that cellular-level mapping can be accomplished using identical samples for both FT-IR and chemical labels. In addition, higher-resolution results can be achieved using a deep convolutional neural network that integrates spatial and spectral features.

Cone Photoreceptor Cell Segmentation and Diameter Measurement on Adaptive Optics Images Using Circularly Constrained Active Contour Model

Purpose

Cone photoreceptor cells can be noninvasively imaged in the living human eye by using nonconfocal adaptive optics scanning ophthalmoscopy split detection. Existing metrics, such as cone density and spacing, are based on simplifying cone photoreceptors to single points. The purposes of this study were to introduce a computer-aided approach for segmentation of cone photoreceptors, to apply this technique to create a normal database of cone diameters, and to demonstrate its use in the context of existing metrics.

Methods

Cone photoreceptor segmentation is achieved through a circularly constrained active contour model (CCACM). Circular templates and image gradients attract active contours toward cone photoreceptor boundaries. Automated segmentation from in vivo human subject data was compared to ground truth established by manual segmentation. Cone diameters computed from curated data (automated segmentation followed by manual removal of errors) were compared with histology and published data.

Results

Overall, there was good agreement between automated and manual segmentations and between diameter measurements (n = 5191 cones) and published histologic data across retinal eccentricities ranging from 1.35 to 6.35 mm (temporal). Interestingly, cone diameter was correlated to both cone density and cone spacing (negatively and positively, respectively; P < 0.01 for both). Application of the proposed automated segmentation to images from a patient with late-onset retinal degeneration revealed the presence of enlarged cones above individual reticular pseudodrusen (average 23.0% increase, P < 0.05).

Conclusions

CCACM can accurately segment cone photoreceptors on split detection images across a range of eccentricities. Metrics derived from this automated segmentation of adaptive optics retinal images can provide new insights into retinal diseases.

Methods for Segmentation and Classification of Digital Microscopy Tissue Images

High-resolution microscopy images of tissue specimens provide detailed information about the morphology of normal and diseased tissue. Image analysis of tissue morphology can help cancer researchers develop a better understanding of cancer biology. Segmentation of nuclei and classification of tissue images are two common tasks in tissue image analysis. Development of accurate and efficient algorithms for these tasks is a challenging problem because of the complexity of tissue morphology and tumor heterogeneity. In this paper we present two computer algorithms; one designed for segmentation of nuclei and the other for classification of whole slide tissue images. The segmentation algorithm implements a multiscale deep residual aggregation network to accurately segment nuclear material and then separate clumped nuclei into individual nuclei. The classification algorithm initially carries out patch-level classification via a deep learning method, then patch-level statistical and morphological features are used as input to a random forest regression model for whole slide image classification. The segmentation and classification algorithms were evaluated in the MICCAI 2017 Digital Pathology challenge. The segmentation algorithm achieved an accuracy score of 0.78. The classification algorithm achieved an accuracy score of 0.81. These scores were the highest in the challenge.

Deep Convolutional Hashing for Low-Dimensional Binary Embedding of Histopathological Images

Compact binary representations of histopa-thology images using hashing methods provide efficient approximate nearest neighbor search for direct visual query in large-scale databases. They can be utilized to measure the probability of the abnormality of the query image based on the retrieved similar cases, thereby providing support for medical diagnosis. They also allow for efficient managing of large-scale image databases because of a low storage requirement. However, the effectiveness of binary representations heavily relies on the visual descriptors that represent the semantic information in the histopathological images. Traditional approaches with hand-crafted visual descriptors might fail due to significant variations in image appearance. Recently, deep learning architectures provide promising solutions to address this problem using effective semantic representations. In this paper, we propose a deep convolutional hashing method that can be trained "point-wise" to simultaneously learn both semantic and binary representations of histopathological images. Specifically, we propose a convolutional neural network that introduces a latent binary encoding (LBE) layer for low-dimensional feature embedding to learn binary codes. We design a joint optimization objective function that encourages the network to learn discriminative representations from the label information, and reduce the gap between the real-valued low-dimensional embedded features and desired binary values. The binary encoding for new images can be obtained by forward propagating through the network and quantizing the output of the LBE layer. Experimental results on a large-scale histopathological image dataset demonstrate the effectiveness of the proposed method.

Pixel-to-Pixel Learning With Weak Supervision for Single-Stage Nucleus Recognition in Ki67 Images

OBJECTIVE

Nucleus recognition is a critical yet challenging step in histopathology image analysis, for example, in Ki67 immunohistochemistry stained images. Although many automated methods have been proposed, most use a multi-stage processing pipeline to categorize nuclei, leading to cumbersome, low-throughput, and error-prone assessments. To address this issue, we propose a novel deep fully convolutional network for single-stage nucleus recognition.

METHODS

Instead of conducting direct pixel-wise classification, we formulate nucleus identification as a deep structured regression model. For each input image, it produces multiple proximity maps, each of which corresponds to one nucleus category and exhibits strong responses in central regions of the nuclei. In addition, by taking into consideration the nucleus distribution in histopathology images, we further introduce an auxiliary task, region of interest (ROI) extraction, to assist and boost the nucleus quantification with weak ROI annotation. The proposed network can be learned in an end-to-end, pixel-to-pixel manner for simultaneous nucleus detection and classification.

RESULTS

We have evaluated this network on a pancreatic neuroendocrine tumor Ki67 image dataset, and the experiments demonstrate that our method outperforms recent state-of-the-art approaches.

CONCLUSION

We present a new, pixel-to-pixel deep neural network with two sibling branches for effective nucleus recognition and observe that learning with another relevant task, ROI extraction, can further boost individual nucleus localization and classification.

SIGNIFICANCE

Our method provides a clean, single-stage nucleus recognition pipeline for histopathology image analysis, especially a new perspective for Ki67 image quantification, which would potentially benefit individual object quantification in whole-slide images.

Joint Region and Nucleus Segmentation for Characterization of Tumor Infiltrating Lymphocytes in Breast Cancer

Histologic assessment of stromal tumor infiltrating lymphocytes (sTIL) as a surrogate of the host immune response has been shown to be prognostic and potentially chemo-predictive in triple-negative and HER2-positive breast cancers. The current practice of manual assessment is prone to intra- and inter-observer variability. Furthermore, the inter-play of sTILs, tumor cells, other microenvironment mediators, their spatial relationships, quantity, and other image-based features have yet to be determined exhaustively and systemically. Towards analysis of these aspects, we developed a deep learning based method for joint region-level and nucleus-level segmentation and classification of breast cancer H&E tissue whole slide images. Our proposed method simultaneously identifies tumor, fibroblast, and lymphocyte nuclei, along with key histologic region compartments including tumor and stroma. We also show how the resultant segmentation masks can be combined with seeding approaches to yield accurate nucleus classifications. Furthermore, we outline a simple workflow for calibrating computational scores to human scores for consistency. The pipeline identifies key compartments with high accuracy (Dice= overall: 0.78, tumor: 0.83, and fibroblasts: 0.77). ROC AUC for nucleus classification is high at 0.89 (micro-average), 0.89 (lymphocytes), 0.90 (tumor), and 0.78 (fibroblasts). Spearman correlation between computational sTIL and pathologist consensus is high (R=0.73, p<0.001) and is higher than inter-pathologist correlation (R=0.66, p<0.001). Both manual and computational sTIL scores successfully stratify patients by clinical progression outcomes.

Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images

We propose a sparse Convolutional Autoencoder (CAE) for simultaneous nucleus detection and feature extraction in histopathology tissue images. Our CAE detects and encodes nuclei in image patches in tissue images into sparse feature maps that encode both the location and appearance of nuclei. A primary contribution of our work is the development of an unsupervised detection network by using the characteristics of histopathology image patches. The pretrained nucleus detection and feature extraction modules in our CAE can be fine-tuned for supervised learning in an end-to-end fashion. We evaluate our method on four datasets and achieve state-of-the-art results. In addition, we are able to achieve comparable performance with only 5% of the fully- supervised annotation cost.

Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network

Detection of nuclei is an important step in phenotypic profiling of 1) histology sections imaged in bright field; and 2) colony formation of the 3-D cell culture models that are imaged using confocal microscopy. It is shown that feature-based representation of the original image improves color decomposition (CD) and subsequent nuclear detection using convolutional neural networks independent of the imaging modality. The feature-based representation utilizes the Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Moreover, in the case of samples imaged in bright field, the LoG response also provides the necessary initial statistics for CD using nonnegative matrix factorization. Several permutations of input data representations and network architectures are evaluated to show that by coupling improved CD and the LoG response of this representation, detection of nuclei is advanced. In particular, the frequencies of detection of nuclei with the vesicular or necrotic phenotypes, or poor staining, are improved. The overall system has been evaluated against manually annotated images, and the F-scores for alternative representations and architectures are reported.

Immune cell constitution in bone marrow microenvironment predicts outcome in adult ALL

As novel immunological treatments are gaining a foothold in the treatment of acute lymphoblastic leukemia (ALL), it is elemental to examine ALL immunobiology in more detail. We used multiplexed immunohistochemistry (mIHC) to study the immune contexture in adult precursor B cell ALL bone marrow (BM). In addition, we developed a multivariate risk prediction model that stratified a poor survival group based on clinical parameters and mIHC data. We analyzed BM biopsy samples of ALL patients (n = 52) and healthy controls (n = 14) using mIHC with 30 different immunophenotype markers and computerized image analysis. In ALL BM, the proportions of M1-like macrophages, granzyme B+CD57+CD8+ T cells, and CD27+ T cells were decreased, whereas the proportions of myeloid-derived suppressor cells and M2-like macrophages were increased. Also, the expression of checkpoint molecules PD1 and CTLA4 was elevated. In the multivariate model, age, platelet count, and the proportion of PD1+TIM3+ double-positive CD4+ T cells differentiated a poor survival group. These results were validated by flow cytometry in a separate cohort (n = 31). In conclusion, the immune cell contexture in ALL BM differs from healthy controls. CD4+PD1+TIM3+ T cells were independent predictors of poor outcome in our multivariate risk model, suggesting that PD1 might serve as an attractive immuno-oncological target in B-ALL.

Ultrafast cell edge detection by line-scan time-stretch microscopy

Ultrafast time-stretch imaging technique recently attracts an increasing interest for applications in cell classification due to high throughput and high sensitivity. A novel imaging modality of time-stretch imaging technique for edge detection is proposed. Edge detection based on the directional derivative is realized using differential detection. As the image processing is mainly implemented in the physical layer, the computation complexity of edge extraction is significantly reduced. An imaging system for edge detection with the scan rate of 77.76 MHz is experimentally demonstrated. Resolution target is first measured to verify the feasibility of the edge extraction. Furthermore, various cells, including red blood cells, lung cancer cells and breast cancer cells, are detected. The edges of cancerous cells present in a completely different form. The imaging system for edge detection would be a good candidate for high-throughput cell classification.

Three-Dimensional Embryonic Image Segmentation and Registration Based on Shape Index and Ellipsoid-Fitting Method

Quantitative analysis based on three-dimensional differential interference contrast (DIC) images is currently the mainstream in analyzing gene functions involved in early cell fate specifications. Segmentation and registration are the two most important steps in analysis. Many image segmentation methods have poor performance on embryonic DIC images because of the interference of egg shells, blurs, and nonuniform intensity background. A novel segmentation method is presented based on the shape index (SI) of local intensity variation in DIC images. To compute the SI, the intensity values of a local neighborhood are regarded as z coordinates over x-y planes. After calculating the SI for each pixel by evaluating the shape of intensity surface over the corresponding local neighborhood, SI thresholding is used to detect cytoplasm granules within embryonic boundaries. As a scalar and rotation invariant, the SI is both insensitive to directional changes and different ranges of intensity variations. Embryonic registration methods are usually based on the orientation of vertebrate anteroposterior (AP) axes computed from segmented embryonic boundaries. Due to the blur of marginal slices in DIC images, usually the segmented boundaries are incomplete, which may make the computed AP axes shift away from the correct orientation when using the principal component analysis method. A method calculating AP axes based on ellipsoid-fitting is proposed, which can achieve high accuracy when handling incomplete segmented boundaries. Using Worm Developmental Dynamics Database, we evaluated the performance of the proposed segmentation method and the computation of AP axes. Experimental results show that the two methods outperform existing methods.

Contour-Seed Pairs Learning-Based Framework for Simultaneously Detecting and Segmenting Various Overlapping Cells/Nuclei in Microscopy Images

In this paper, we propose a novel contour-seed pairs learning-based framework for robust and automated cell/nucleus segmentation. Automated granular object segmentation in microscopy images has significant clinical importance for pathology grading of the cell carcinoma and gene expression. The focus of the past literature is dominated by either segmenting a certain type of cells/nuclei or simply splitting the clustered objects without contours inference of them. Our method addresses these issues by formulating the detection and segmentation tasks in terms of a unified regression problem, where a cascade sparse regression chain model is trained and then applied to return object locations and entire boundaries of clustered objects. In particular, we first learn a set of online convolutional features in each layer. Then, in the proposed cascade sparse regression chain, with the input from the learned features, we iteratively update the locations and clustered object boundaries until convergence. In this way, the boundary evidences of each individual object can be easily delineated and be further fed to a complete contour inference procedure optimized by the minimum description length principle. For any probe image, our method enables to analyze free-lying and overlapping cells with complex shapes. Experimental results show that the proposed method is very generic and performs well on contour inferences of various cell/nucleus types. Compared with the current segmentation techniques, our approach achieves state-of-the-art performances on four challenging datasets, i.e., the kidney renal cell carcinoma histopathology dataset, Drosophila Kc167 cellular dataset, differential interference contrast red blood cell dataset, and cervical cytology dataset.

Twenty Years of Digital Pathology: An Overview of the Road Travelled, What is on the Horizon, and the Emergence of Vendor-Neutral Archives

Almost 20 years have passed since the commercial introduction of whole-slide imaging (WSI) scanners. During this time, the creation of various WSI devices with the ability to digitize an entire glass slide has transformed the field of pathology. Parallel advances in computational technology and storage have permitted rapid processing of large-scale WSI datasets. This article provides an overview of important past and present efforts related to WSI. An account of how the virtual microscope evolved from the need to visualize and manage satellite data for earth science applications is provided. The article also discusses important milestones beginning from the first WSI scanner designed by Bacus to the Food and Drug Administration approval of the first digital pathology system for primary diagnosis in surgical pathology. As pathology laboratories commit to going fully digitalize, the need has emerged to include WSIs into an enterprise-level vendor-neutral archive (VNA). The different types of VNAs available are reviewed as well as how best to implement them and how pathology can benefit from participating in this effort. Differences between traditional image algorithms that extract pixel-, object-, and semantic-level features versus deep learning methods are highlighted. The need for large-scale data management, analysis, and visualization in computational pathology is also addressed.

Segmentation of Total Cell Area in Brightfield Microscopy Images

Segmentation is one of the most important steps in microscopy image analysis. Unfortunately, most of the methods use fluorescence images for this task, which is not suitable for analysis that requires a knowledge of area occupied by cells and an experimental design that does not allow necessary labeling. In this protocol, we present a simple method, based on edge detection and morphological operations, that separates total area occupied by cells from the background using only brightfield channel image. The resulting segmented picture can be further used as a mask for fluorescence quantification and other analyses. The whole procedure is carried out in open source software Fiji.

Analysis of Cellular Feature Differences of Astrocytomas with Distinct Mutational Profiles Using Digitized Histopathology Images

Cellular phenotypic features derived from histopathology images are the basis of pathologic diagnosis and are thought to be related to underlying molecular profiles. Due to overwhelming cell numbers and population heterogeneity, it remains challenging to quantitatively compute and compare features of cells with distinct molecular signatures. In this study, we propose a self-reliant and efficient analysis framework that supports quantitative analysis of cellular phenotypic difference across distinct molecular groups. To demonstrate efficacy, we quantitatively analyze astrocytomas that are molecularly characterized as either Isocitrate Dehydrogenase (IDH) mutant (MUT) or wildtype (WT) using imaging data from The Cancer Genome Atlas database. Representative cell instances that are phenotypically different between these two groups are retrieved after segmentation, feature computation, data pruning, dimensionality reduction, and unsupervised clustering. Our analysis is generic and can be applied to a wide set of cell-based biomedical research.

Melanoma and Melanocyte Identification from Hyperspectral Pathology Images Using Object-Based Multiscale Analysis

Pathological skin imaging analysis is identified as an efficient technique to diagnose melanoma and provide necessary information for treatment. Automatic detection of melanoma and melanocytes in the epidermis area can be a challenging task as a result of the variability of melanocytes and similarity among cytological components. In order to develop a practical and reliable approach to address the issue, this paper proposed a melanoma and melanocyte detection method based on hyperspectral pathology images. Given the abundant and related spectral and spatial information associated with the hyperspectral skin pathological image, an object-based method was first used to construct the image into the object level; then a multiscale descriptor was performed to extract specific features of melanoma and melanocytes. A quantitative evaluation of 100 scenes of hyperspectral pathology images from 49 patients showed the optimal accuracy, sensitivity, and specificity of 94.29%, 95.57%, and 93.15%, respectively. The results can be interpreted that hyperspectral pathology imaging techniques help to detect the melanoma and melanocytes effectively and provide useful information for further segmentation and classification.

Deep Learning in Microscopy Image Analysis: A Survey

Computerized microscopy image analysis plays an important role in computer aided diagnosis and prognosis. Machine learning techniques have powered many aspects of medical investigation and clinical practice. Recently, deep learning is emerging as a leading machine learning tool in computer vision and has attracted considerable attention in biomedical image analysis. In this paper, we provide a snapshot of this fast-growing field, specifically for microscopy image analysis. We briefly introduce the popular deep neural networks and summarize current deep learning achievements in various tasks, such as detection, segmentation, and classification in microscopy image analysis. In particular, we explain the architectures and the principles of convolutional neural networks, fully convolutional networks, recurrent neural networks, stacked autoencoders, and deep belief networks, and interpret their formulations or modelings for specific tasks on various microscopy images. In addition, we discuss the open challenges and the potential trends of future research in microscopy image analysis using deep learning.

Morphologically constrained and data informed cell segmentation of budding yeast

Motivation

Although high-content image cytometry is becoming increasingly routine, processing the large amount of data acquired during time-lapse experiments remains a challenge. The majority of approaches for automated single-cell segmentation focus on flat, uniform fields of view covered with a single layer of cells. In the increasingly popular microfluidic devices that trap individual cells for long term imaging, these conditions are not met. Consequently, most techniques for segmentation perform poorly. Although potentially constraining the generalizability of software, incorporating information about the microfluidic features, flow of media and the morphology of the cells can substantially improve performance.

Results

Here we present DISCO (Data Informed Segmentation of Cell Objects), a framework for using the physical constraints imposed by microfluidic traps, the shape based morphological constraints of budding yeast and temporal information about cell growth and motion to allow tracking and segmentation of cells in microfluidic devices. Using manually curated datasets, we demonstrate substantial improvements in both tracking and segmentation when compared with existing software.

Availability and implementation

The MATLAB code for the algorithm and for measuring performance is available at https://github.com/pswain/segmentation-software and the test images and the curated ground-truth results used for comparing the algorithms are available at http://datashare.is.ed.ac.uk/handle/10283/2002.

Contact

mcrane2@uw.edu.

Supplementary information

Supplementary data are available at Bioinformatics online.

Relationship between the Ki67 index and its area based approximation in breast cancer

BACKGROUND

The Ki67 Index has been extensively studied as a prognostic biomarker in breast cancer. However, its clinical adoption is largely hampered by the lack of a standardized method to assess Ki67 that limits inter-laboratory reproducibility. It is important to standardize the computation of the Ki67 Index before it can be effectively used in clincial practice.

METHOD

In this study, we develop a systematic approach towards standardization of the Ki67 Index. We first create the ground truth consisting of tumor positive and tumor negative nuclei by registering adjacent breast tissue sections stained with Ki67 and H&E. The registration is followed by segmentation of positive and negative nuclei within tumor regions from Ki67 images. The true Ki67 Index is then approximated with a linear model of the area of positive to the total area of tumor nuclei.

RESULTS

When tested on 75 images of Ki67 stained breast cancer biopsies, the proposed method resulted in an average root mean square error of 3.34. In comparison, an expert pathologist resulted in an average root mean square error of 9.98 and an existing automated approach produced an average root mean square error of 5.64.

CONCLUSIONS

We show that it is possible to approximate the true Ki67 Index accurately without detecting individual nuclei and also statically demonstrate the weaknesses of commonly adopted approaches that use both tumor and non-tumor regions together while compensating for the latter with higher order approximations.

Automated fluorescence intensity and gradient analysis enables detection of rare fluorescent mutant cells deep within the tissue of RaDR mice

Homologous recombination (HR) events are key drivers of cancer-promoting mutations, and the ability to visualize these events in situ provides important information regarding mutant cell type, location, and clonal expansion. We have previously created the Rosa26 Direct Repeat (RaDR) mouse model wherein HR at an integrated substrate gives rise to a fluorescent cell. To fully leverage this in situ approach, we need better ways to quantify rare fluorescent cells deep within tissues. Here, we present a robust, automated event quantification algorithm that uses image intensity and gradient features to detect fluorescent cells in deep tissue specimens. To analyze the performance of our algorithm, we simulate fluorescence behavior in tissue using Monte Carlo methods. Importantly, this approach reduces the potential for bias in manual counting and enables quantification of samples with highly dense HR events. Using this approach, we measured the relative frequency of HR within a chromosome and between chromosomes and found that HR within a chromosome is more frequent, which is consistent with the close proximity of sister chromatids. Our approach is both objective and highly rapid, providing a powerful tool, not only to researchers interested in HR, but also to many other researchers who are similarly using fluorescence as a marker for understanding mammalian biology in tissues.

Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes

BACKGROUND

Nuclear segmentation is an important step for profiling aberrant regions of histology sections. If nuclear segmentation can be resolved, then new biomarkers of nuclear phenotypes and their organization can be predicted for the application of precision medicine. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), nuclear phenotypes (e.g., vesicular, aneuploidy), and overlapping nuclei. The problem is further complicated as a result of variations in sample preparation (e.g., fixation, staining). Our hypothesis is that (i) deep learning techniques can learn complex phenotypic signatures that rise in tumor sections, and (ii) fusion of different representations (e.g., regions, boundaries) contributes to improved nuclear segmentation.

RESULTS

We have demonstrated that training of deep encoder-decoder convolutional networks overcomes complexities associated with multiple nuclear phenotypes, where we evaluate alternative architecture of deep learning for an improved performance against the simplicity of the design. In addition, improved nuclear segmentation is achieved by color decomposition and combining region- and boundary-based features through a fusion network. The trained models have been evaluated against approximately 19,000 manually annotated nuclei, and object-level Precision, Recall, F1-score and Standard Error are reported with the best F1-score being 0.91. Raw training images, annotated images, processed images, and source codes are released as a part of the Additional file 1.

CONCLUSIONS

There are two intrinsic barriers in nuclear segmentation to H&E stained images, which correspond to the diversity of nuclear phenotypes and perceptual boundaries between adjacent cells. We demonstrate that (i) the encoder-decoder architecture can learn complex phenotypes that include the vesicular type; (ii) delineation of overlapping nuclei is enhanced by fusion of region- and edge-based networks; (iii) fusion of ENets produces an improved result over the fusion of UNets; and (iv) fusion of networks is better than multitask learning. We suggest that our protocol enables processing a large cohort of whole slide images for applications in precision medicine.

Comparison of Different Classifiers with Active Learning to Support Quality Control in Nucleus Segmentation in Pathology Images

Segmentation of nuclei in whole slide tissue images is a common methodology in pathology image analysis. Most segmentation algorithms are sensitive to input algorithm parameters and the characteristics of input images (tissue morphology, staining, etc.). Because there can be large variability in the color, texture, and morphology of tissues within and across cancer types (heterogeneity can exist even within a tissue specimen), it is likely that a set of input parameters will not perform well across multiple images. It is, therefore, highly desired, and necessary in some cases, to carry out a quality control of segmentation results. This work investigates the application of machine learning in this process. We report on the application of active learning for segmentation quality assessment for pathology images and compare three classification methods, Support Vector Machine (SVM), Random Forest (RF) and Convolutional Neural Network (CNN), for their performance improvement and efficiency.

Deep Learning Nuclei Detection in Digitized Histology Images by Superpixels

Background

Advances in image analysis and computational techniques have facilitated automatic detection of critical features in histopathology images. Detection of nuclei is critical for squamous epithelium cervical intraepithelial neoplasia (CIN) classification into normal, CIN1, CIN2, and CIN3 grades.

Methods

In this study, a deep learning (DL)-based nuclei segmentation approach is investigated based on gathering localized information through the generation of superpixels using a simple linear iterative clustering algorithm and training with a convolutional neural network.

Results

The proposed approach was evaluated on a dataset of 133 digitized histology images and achieved an overall nuclei detection (object-based) accuracy of 95.97%, with demonstrated improvement over imaging-based and clustering-based benchmark techniques.

Conclusions

The proposed DL-based nuclei segmentation Method with superpixel analysis has shown improved segmentation results in comparison to state-of-the-art methods.

Machine Learning Methods for Histopathological Image Analysis

Abundant accumulation of digital histopathological images has led to the increased demand for their analysis, such as computer-aided diagnosis using machine learning techniques. However, digital pathological images and related tasks have some issues to be considered. In this mini-review, we introduce the application of digital pathological image analysis using machine learning algorithms, address some problems specific to such analysis, and propose possible solutions.