An Automatic Learning-Based Framework for Robust Nucleus Segmentation

ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning

Machine learning algorithms hold the promise to effectively automate the analysis of histopathological images that are routinely generated in clinical practice. Any machine learning method used in the clinical diagnostic process has to be extremely accurate and, ideally, provide a measure of uncertainty for its predictions. Such accurate and reliable classifiers need enough labelled data for training, which requires time-consuming and costly manual annotation by pathologists. Thus, it is critical to minimise the amount of data needed to reach the desired accuracy by maximising the efficiency of training. We propose an accurate, reliable and active (ARA) image classification framework and introduce a new Bayesian Convolutional Neural Network (ARA-CNN) for classifying histopathological images of colorectal cancer. The model achieves exceptional classification accuracy, outperforming other models trained on the same dataset. The network outputs an uncertainty measurement for each tested image. We show that uncertainty measures can be used to detect mislabelled training samples and can be employed in an efficient active learning workflow. Using a variational dropout-based entropy measure of uncertainty in the workflow speeds up the learning process by roughly 45%. Finally, we utilise our model to segment whole-slide images of colorectal tissue and compute segmentation-based spatial statistics.

Microvascularity detection and quantification in glioma: a novel deep-learning-based framework

Microvascularity is highly correlated with the grading and subtyping of gliomas, making this one of its most important histological features. Accurate quantitative analysis of microvessels is helpful for the development of a targeted therapy for antiangiogenesis. The deep-learning algorithm is by far the most effective segmentation and detection model and enables location and recognition of complex microvascular networks in large images obtained from hematoxylin and eosin (H&E) stained specimens. We proposed an automated deep-learning-based method to detect and quantify the microvascularity in glioma and applied it to comprehensive clinical analyses. A total of 350 glioma patients were enrolled in our study, for which digitalized imaging of H&E stained slides were reviewed, molecular diagnosis was performed and follow-up was investigated. The microvascular features were compared according to their histologic types, molecular types, and patients' prognosis. The results show that the proposed method can quantify microvascular characteristics automatically and effectively. Significant increases of microvascular density and microvascular area were observed in glioblastomas (95% p < 0.001 in density, 170% p < 0.001 in area) in comparison with other histologic types; increases were also observed in cases with TERT-mut only (68% p < 0.001 in density, 54% p < 0.001 in area) compared with other molecular types. Survival analysis showed that microvascular features can be used to cluster cases into two groups with different survival periods (hazard ratio [HR] 2.843, log-rank <0.001), which indicates the quantified microvascular features may potentially be alternative signatures for revealing patients' prognosis. This deep-learning-based method may be a useful tool in routine clinical practice for precise diagnosis and antiangiogenic treatment.

DeNeRD: high-throughput detection of neurons for brain-wide analysis with deep learning

Mapping the structure of the mammalian brain at cellular resolution is a challenging task and one that requires capturing key anatomical features at the appropriate level of analysis. Although neuroscientific methods have managed to provide significant insights at the micro and macro level, in order to obtain a whole-brain analysis at a cellular resolution requires a meso-scopic approach. A number of methods can be currently used to detect and count cells, with, nevertheless, significant limitations when analyzing data of high complexity. To overcome some of these constraints, we introduce a fully automated Artificial Intelligence (AI)-based method for whole-brain image processing to Detect Neurons in different brain Regions during Development (DeNeRD). We demonstrate a high performance of our deep neural network in detecting neurons labeled with different genetic markers in a range of imaging planes and imaging modalities.

Towards pixel-to-pixel deep nucleus detection in microscopy images

BACKGROUND

Nucleus is a fundamental task in microscopy image analysis and supports many other quantitative studies such as object counting, segmentation, tracking, etc. Deep neural networks are emerging as a powerful tool for biomedical image computing; in particular, convolutional neural networks have been widely applied to nucleus/cell detection in microscopy images. However, almost all models are tailored for specific datasets and their applicability to other microscopy image data remains unknown. Some existing studies casually learn and evaluate deep neural networks on multiple microscopy datasets, but there are still several critical, open questions to be addressed.

RESULTS

We analyze the applicability of deep models specifically for nucleus detection across a wide variety of microscopy image data. More specifically, we present a fully convolutional network-based regression model and extensively evaluate it on large-scale digital pathology and microscopy image datasets, which consist of 23 organs (or cancer diseases) and come from multiple institutions. We demonstrate that for a specific target dataset, training with images from the same types of organs might be usually necessary for nucleus detection. Although the images can be visually similar due to the same staining technique and imaging protocol, deep models learned with images from different organs might not deliver desirable results and would require model fine-tuning to be on a par with those trained with target data. We also observe that training with a mixture of target and other/non-target data does not always mean a higher accuracy of nucleus detection, and it might require proper data manipulation during model training to achieve good performance.

CONCLUSIONS

We conduct a systematic case study on deep models for nucleus detection in a wide variety of microscopy images, aiming to address several important but previously understudied questions. We present and extensively evaluate an end-to-end, pixel-to-pixel fully convolutional regression network and report a few significant findings, some of which might have not been reported in previous studies. The model performance analysis and observations would be helpful to nucleus detection in microscopy images.

A segmentation method combining probability map and boundary based on multiple fully convolutional networks and repetitive training

Cell nuclei image segmentation technology can help researchers observe each cell's stress response to drug treatment. However, it is still a challenge to accurately segment the adherent cell nuclei. At present, image segmentation based on a fully convolutional network (FCN) is attracting researchers' attention. We propose a multiple FCN architecture and repetitive training (M-FCN-RT) method to learn features of cell nucleus images. In M-FCN-RT, the multiple FCN (M-FCN) architecture is composed of several single FCNs (S-FCNs) with the same structure, and each FCN is used to learn the specific features of image datasets. In this paper, the M-FCN contains three FCNs; FCN_1-2, FCN₃ and FCN_B. FCN_1-2 and FCN₃ are respectively used to learn the spatial features of cell nuclei for generating probability maps to indicate nucleus regions of an image; FCN_B (boundary FCN) is used to learn the edge features of cell nuclei for generating the nucleus boundary. For the training of each FCN, we propose a repetitive training (RT) method to improve the classification accuracy of the model. To segment cell nuclei, we finally propose an algorithm combining the probability map and boundary (PMB) to segment the adherent nuclei. This paper uses a public opening nucleus image dataset to train, verify and evaluate the proposed M-FCN-RT and PMB methods. Our M-FCN-RT method achieves a high Dice similarity coefficient (DSC) of 92.11%, 95.64% and 87.99% on the three types of sub-datasets respectively for probability maps. In addition, segmentation experimental results show the PMB method is more effective and efficient compared with other methods.

Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images

Identifying nuclei is often a critical first step in analyzing microscopy images of cells and classical image processing algorithms are most commonly used for this task. Recent developments in deep learning can yield superior accuracy, but typical evaluation metrics for nucleus segmentation do not satisfactorily capture error modes that are relevant in cellular images. We present an evaluation framework to measure accuracy, types of errors, and computational efficiency; and use it to compare deep learning strategies and classical approaches. We publicly release a set of 23,165 manually annotated nuclei and source code to reproduce experiments and run the proposed evaluation methodology. Our evaluation framework shows that deep learning improves accuracy and can reduce the number of biologically relevant errors by half. © 2019 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of International Society for Advancement of Cytometry.

Prior Information Guided Regularized Deep Learning for Cell Nucleus Detection

Cell nuclei detection is a challenging research topic because of limitations in cellular image quality and diversity of nuclear morphology, i.e., varying nuclei shapes, sizes, and overlaps between multiple cell nuclei. This has been a topic of enduring interest with promising recent success shown by deep learning methods. These methods train convolutional neural networks (CNNs) with a training set of input images and known, labeled nuclei locations. Many such methods are supplemented by spatial or morphological processing. Using a set of canonical cell nuclei shapes, prepared with the help of a domain expert, we develop a new approach that we call shape priors (SPs) with CNNs (SPs-CNN). We further extend the network to introduce an SP layer and then allowing it to become trainable (i.e., optimizable). We call this network as tunable SP-CNN (TSP-CNN). In summary, we present new network structures that can incorporate "expected behavior" of nucleus shapes via two components: learnable layers that perform the nucleus detection and a fixed processing part that guides the learning with prior information. Analytically, we formulate two new regularization terms that are targeted at: 1) learning the shapes and 2) reducing false positives while simultaneously encouraging detection inside the cell nucleus boundary. Experimental results on two challenging datasets reveal that the proposed SP-CNN and TSP-CNN can outperform the state-of-the-art alternatives.

RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification

The whole slide histopathology images (WSIs) play a critical role in gastric cancer diagnosis. However, due to the large scale of WSIs and various sizes of the abnormal area, how to select informative regions and analyze them are quite challenging during the automatic diagnosis process. The multi-instance learning based on the most discriminative instances can be of great benefit for whole slide gastric image diagnosis. In this paper, we design a recalibrated multi-instance deep learning method (RMDL) to address this challenging problem. We first select the discriminative instances, and then utilize these instances to diagnose diseases based on the proposed RMDL approach. The designed RMDL network is capable of capturing instance-wise dependencies and recalibrating instance features according to the importance coefficient learned from the fused features. Furthermore, we build a large whole-slide gastric histopathology image dataset with detailed pixel-level annotations. Experimental results on the constructed gastric dataset demonstrate the significant improvement on the accuracy of our proposed framework compared with other state-of-the-art multi-instance learning methods. Moreover, our method is general and can be extended to other diagnosis tasks of different cancer types based on WSIs.

Cancer Diagnosis Using Deep Learning: A Bibliographic Review

In this paper, we first describe the basics of the field of cancer diagnosis, which includes steps of cancer diagnosis followed by the typical classification methods used by doctors, providing a historical idea of cancer classification techniques to the readers. These methods include Asymmetry, Border, Color and Diameter (ABCD) method, seven-point detection method, Menzies method, and pattern analysis. They are used regularly by doctors for cancer diagnosis, although they are not considered very efficient for obtaining better performance. Moreover, considering all types of audience, the basic evaluation criteria are also discussed. The criteria include the receiver operating characteristic curve (ROC curve), Area under the ROC curve (AUC), F1 score, accuracy, specificity, sensitivity, precision, dice-coefficient, average accuracy, and Jaccard index. Previously used methods are considered inefficient, asking for better and smarter methods for cancer diagnosis. Artificial intelligence and cancer diagnosis are gaining attention as a way to define better diagnostic tools. In particular, deep neural networks can be successfully used for intelligent image analysis. The basic framework of how this machine learning works on medical imaging is provided in this study, i.e., pre-processing, image segmentation and post-processing. The second part of this manuscript describes the different deep learning techniques, such as convolutional neural networks (CNNs), generative adversarial models (GANs), deep autoencoders (DANs), restricted Boltzmann's machine (RBM), stacked autoencoders (SAE), convolutional autoencoders (CAE), recurrent neural networks (RNNs), long short-term memory (LTSM), multi-scale convolutional neural network (M-CNN), multi-instance learning convolutional neural network (MIL-CNN). For each technique, we provide Python codes, to allow interested readers to experiment with the cited algorithms on their own diagnostic problems. The third part of this manuscript compiles the successfully applied deep learning models for different types of cancers. Considering the length of the manuscript, we restrict ourselves to the discussion of breast cancer, lung cancer, brain cancer, and skin cancer. The purpose of this bibliographic review is to provide researchers opting to work in implementing deep learning and artificial neural networks for cancer diagnosis a knowledge from scratch of the state-of-the-art achievements.

A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images

This paper addresses the task of nuclei segmentation in high-resolution histopathology images. We propose an automatic end-to-end deep neural network algorithm for segmentation of individual nuclei. A nucleus-boundary model is introduced to predict nuclei and their boundaries simultaneously using a fully convolutional neural network. Given a color-normalized image, the model directly outputs an estimated nuclei map and a boundary map. A simple, fast, and parameter-free post-processing procedure is performed on the estimated nuclei map to produce the final segmented nuclei. An overlapped patch extraction and assembling method is also designed for seamless prediction of nuclei in large whole-slide images. We also show the effectiveness of data augmentation methods for nuclei segmentation task. Our experiments showed our method outperforms prior state-of-the-art methods. Moreover, it is efficient that one 1000×1000 image can be segmented in less than 5 s. This makes it possible to precisely segment the whole-slide image in acceptable time. The source code is available at https://github.com/easycui/nuclei_segmentation . Graphical Abstract The neural network for nuclei segmentation.

Quick Leukocyte Nucleus Segmentation in Leukocyte Counting

The leukocyte nucleus quick segmentation is one of the key techniques in leukocyte real-time online scanning of human blood smear. We propose a quick leukocyte nucleus segmentation method based on the component difference in RGB color space. By analyzing the captured microscopic images of the peripheral blood smears from the autoscanning microscope, it is found that the difference values between B component and G component (B − G values) in the regions of the leukocyte nuclei and the platelets are much bigger than those in the other regions, even in the regions including the stains. So, the B − G values can segment the leukocyte nuclei and the platelets with an appropriate empirical threshold because the platelets are much smaller than the leukocyte nuclei, so the leukocyte nuclei can be segmented by size filtering. Also, only an 8 bit subtraction operation is performed for the B − G values, and it can improve the leukocyte nucleus segmentation speed significantly. Experimental results show that the proposed method performs well for the five types of leukocyte segmentation with a quick speed. It is very suitable for the real-time peripheral blood smear autoscanning test application. In addition, the five types of leukocytes can be counted accurately.

Three-dimensional GPU-accelerated active contours for automated localization of cells in large images

Cell segmentation in microscopy is a challenging problem, since cells are often asymmetric and densely packed. Successful cell segmentation algorithms rely identifying seed points, and are highly sensitive to variablility in cell size. In this paper, we present an efficient and highly parallel formulation for symmetric three-dimensional contour evolution that extends previous work on fast two-dimensional snakes. We provide a formulation for optimization on 3D images, as well as a strategy for accelerating computation on consumer graphics hardware. The proposed software takes advantage of Monte-Carlo sampling schemes in order to speed up convergence and reduce thread divergence. Experimental results show that this method provides superior performance for large 2D and 3D cell localization tasks when compared to existing methods on large 3D brain images.

Digital pathology and artificial intelligence

In modern clinical practice, digital pathology has a crucial role and is increasingly a technological requirement in the scientific laboratory environment. The advent of whole-slide imaging, availability of faster networks, and cheaper storage solutions has made it easier for pathologists to manage digital slide images and share them for clinical use. In parallel, unprecedented advances in machine learning have enabled the synergy of artificial intelligence and digital pathology, which offers image-based diagnosis possibilities that were once limited only to radiology and cardiology. Integration of digital slides into the pathology workflow, advanced algorithms, and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enable true utilisation and integration of knowledge that is beyond human limits and boundaries, and we believe there is clear potential for artificial intelligence breakthroughs in the pathology setting. In this Review, we discuss advancements in digital slide-based image diagnosis for cancer along with some challenges and opportunities for artificial intelligence in digital pathology.

NHL Pathological Image Classification Based on Hierarchical Local Information and GoogLeNet-Based Representations

Background

Accurate classification for different non-Hodgkin lymphomas (NHL) is one of the main challenges in clinical pathological diagnosis due to its intrinsic complexity. Therefore, this paper proposes an effective classification model for three types of NHL pathological images, including mantle cell lymphoma (MCL), follicular lymphoma (FL), and chronic lymphocytic leukemia (CLL).

Methods

There are three main parts with respect to our model. First, NHL pathological images stained by hematoxylin and eosin (H&E) are transferred into blue ratio (BR) and Lab spaces, respectively. Then specific patch-level textural and statistical features are extracted from BR images and color features are obtained from Lab images both using a hierarchical way, yielding a set of hand-crafted representations corresponding to different image spaces. A random forest classifier is subsequently trained for patch-level classification. Second, H&E images are cropped and fed into a pretrained google inception net (GoogLeNet) for learning high-level representations and a softmax classifier is used for patch-level classification. Finally, three image-level classification strategies based on patch-level results are discussed including a novel method for calculating the weighted sum of patch results. Different classification results are fused at both feature 1 and image levels to obtain a more satisfactory result.

Results

The proposed model is evaluated on a public IICBU Malignant Lymphoma Dataset and achieves an improved overall accuracy of 0.991 and area under the receiver operating characteristic curve of 0.998.

Conclusion

The experimentations demonstrate the significantly increased classification performance of the proposed model, indicating that it is a suitable classification approach for NHL pathological images.

Effective nuclei segmentation with sparse shape prior and dynamic occlusion constraint for glioblastoma pathology images

We propose a segmentation method for nuclei in glioblastoma histopathologic images based on a sparse shape prior guided variational level set framework. By spectral clustering and sparse coding, a set of shape priors is exploited to accommodate complicated shape variations. We automate the object contour initialization by a seed detection algorithm and deform contours by minimizing an energy functional that incorporates a shape term in a sparse shape prior representation, an adaptive contour occlusion penalty term, and a boundary term encouraging contours to converge to strong edges. As a result, our approach is able to deal with mutual occlusions and detect contours of multiple intersected nuclei simultaneously. Our method is applied to several whole-slide histopathologic image datasets for nuclei segmentation. The proposed method is compared with other state-of-the-art methods and demonstrates good accuracy for nuclei detection and segmentation, suggesting its promise to support biomedical image-based investigations.

Segmentation of Heavily Clustered Nuclei from Histopathological Images

Automated cell nucleus segmentation is the key to gain further insight into cell features and functionality which support computer-aided pathology in early diagnosis of diseases such as breast cancer and brain tumour. Despite considerable advances in automated segmentation, it still remains a challenging task to split heavily clustered nuclei due to intensity variations caused by noise and uneven absorption of stains. To address this problem, we propose a novel method applicable to variety of histopathological images stained for different proteins, with high speed, accuracy and level of automation. Our algorithm is initiated by applying a new locally adaptive thresholding method on watershed regions. Followed by a new splitting technique based on multilevel thresholding and the watershed algorithm to separate clustered nuclei. Finalized by a model-based merging step to eliminate oversegmentation and a model-based correction step to improve segmentation results and eliminate small objects. We have applied our method to three image datasets: breast cancer stained for hematoxylin and eosin (H&E), Drosophila Kc167 cells stained for DNA to label nuclei, and mature neurons stained for NeuN. Evaluated results show our method outperforms the state-of-the-art methods in terms of accuracy, precision, F1-measure, and computational time.

An Enhancement of Deep Learning Algorithm for Brain Tumor Segmentation Using Kernel Based CNN with M-SVM

The brain tumor can be created by uncontrollable increase of abnormal cells in tissue of brain and it has two kinds of tumors: one is benign and another one is malignant tumor. The benign brain tumor does not affect the adjacent normal and healthy tissue but the malignant tumor can affect the neighboring tissues of brain that can lead to the death of person. An early detection of brain tumor can be required to protect the survival of patients. Usually, the brain tumor is detected using MRI scanning method. However, the radiologists are not providing the effective tumor segmentation in MRI image due to the irregular shape of tumors and position of tumor in the brain. Accurate brain tumor segmentation is needed to locate the tumor and it is used to give the correct treatment for a patient and it provides the key to the doctor who must execute the surgery for patient. In this paper, a novel deep learning algorithm (kernel based CNN) with M-SVM is presented to segment the tumor automatically and efficiently. This presented work contains several steps that are preprocessing, feature extraction, image classification and tumor segmentation of brain. The MRI image is smoothed and enhanced by Laplacian of Gaussian filtering method (LoG) with Contrast Limited Adaptive Histrogram Equalization (CLAHE) and feature can be extracted from it based on tumor shape position, shape and surface features in brain. Consequently, the image classification is done using M-SVM depending on the selected features. From MRI image, the tumor is segmented with help of kernel based CNN method.. Experimental results of proposed method can show that this presented technique can executes brain tumor segmentation accurately reaching almost 84% in evaluation with existing algorithms.

Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation

PURPOSE

Reliable automated segmentation of the prostate is indispensable for image-guided prostate interventions. However, the segmentation task is challenging due to inhomogeneous intensity distributions, variation in prostate anatomy, among other problems. Manual segmentation can be time-consuming and is subject to inter- and intraobserver variation. We developed an automated deep learning-based method to address this technical challenge.

METHODS

We propose a three-dimensional (3D) fully convolutional networks (FCN) with deep supervision and group dilated convolution to segment the prostate on magnetic resonance imaging (MRI). In this method, a deeply supervised mechanism was introduced into a 3D FCN to effectively alleviate the common exploding or vanishing gradients problems in training deep models, which forces the update process of the hidden layer filters to favor highly discriminative features. A group dilated convolution which aggregates multiscale contextual information for dense prediction was proposed to enlarge the effective receptive field of convolutional neural networks, which improve the prediction accuracy of prostate boundary. In addition, we introduced a combined loss function including cosine and cross entropy, which measures similarity and dissimilarity between segmented and manual contours, to further improve the segmentation accuracy. Prostate volumes manually segmented by experienced physicians were used as a gold standard against which our segmentation accuracy was measured.

RESULTS

The proposed method was evaluated on an internal dataset comprising 40 T2-weighted prostate MR volumes. Our method achieved a Dice similarity coefficient (DSC) of 0.86 ± 0.04, a mean surface distance (MSD) of 1.79 ± 0.46 mm, 95% Hausdorff distance (95%HD) of 7.98 ± 2.91 mm, and absolute relative volume difference (aRVD) of 15.65 ± 10.82. A public dataset (PROMISE12) including 50 T2-weighted prostate MR volumes was also employed to evaluate our approach. Our method yielded a DSC of 0.88 ± 0.05, MSD of 1.02 ± 0.35 mm, 95% HD of 9.50 ± 5.11 mm, and aRVD of 8.93 ± 7.56.

CONCLUSION

We developed a novel deeply supervised deep learning-based approach with a group dilated convolution to automatically segment the MRI prostate, demonstrated its clinical feasibility, and validated its accuracy against manual segmentation. The proposed technique could be a useful tool for image-guided interventions in prostate cancer.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Combined morphologic and metabolic pipeline for Positron emission tomography/computed tomography based radiotherapy response evaluation in locally advanced pancreatic adenocarcinoma

•

A novel morphologic and metabolic combined pipeline for PA response evaluation.

•

The derived metric outperformed traditional imaging metrics in risk stratification.

•

May serve as a new image biomarker to characterize heterogeneous tumor response.

Background and purpose

Adaptive radiation planning for pancreatic adenocarcinoma (PA) relies on accurate treatment response assessment, while traditional response evaluation criteria inefficiently characterize tumors with complex morphological features or intrinsically low metabolism. To better assess treatment response of PA, we quantify and compare regional morphological and metabolic features of the 3D pre- and post-radiation therapy (RT) tumor models.

Materials and methods

Thirty-one PA patients with pre and post-RT Positron emission tomography/computed tomography (PET/CT) scans were evaluated. 3D meshes of pre- and post-RT tumors were generated and registered to establish vertex-wise correspondence. To assess tumor response, Mahalanobis distances (M_dist|_Fusion) between pre- and post-RT tumor surfaces with anatomic and metabolic fused vectors were calculated for each patient. M_dist|_Fusion was evaluated by overall survival (OS) prediction and survival risk classification. As a comparison, the same analyses were conducted on traditional imaging/physiological predictors, and distances measurements based on metabolic and morphological features only.

Results

Among all the imaging/physiological parameters, M_dist|_Fusion was shown to be the best predictor of OS (HR = 0.52, p = 0.008), while other parameters failed to reach significance. Moreover, M_dist|_Fusion outperformed traditional morphologic and metabolic measurements in patient risk stratification, either alone (HR = 11.51, p < 0.001) or combined with age (HR = 9.04, p < 0.001).

Conclusions

We introduced a PET/CT-based novel morphologic and metabolic pipeline for response evaluation in locally advanced PA. The fused M_dist|_Fusion outperformed traditional morphologic, metabolic, and physiological measurements in OS prediction and risk stratification. The novel fusion model may serve as a new imaging-marker to more accurately characterize the heterogeneous tumor RT response.

PCSeg: Color model driven probabilistic multiphase level set based tool for plasma cell segmentation in multiple myeloma

Plasma cell segmentation is the first stage of a computer assisted automated diagnostic tool for multiple myeloma (MM). Owing to large variability in biological cell types, a method for one cell type cannot be applied directly on the other cell types. In this paper, we present PCSeg Tool for plasma cell segmentation from microscopic medical images. These images were captured from bone marrow aspirate slides of patients with MM. PCSeg has a robust pipeline consisting of a pre-processing step, the proposed modified multiphase level set method followed by post-processing steps including the watershed and circular Hough transform to segment clusters of cells of interest and to remove unwanted cells. Our modified level set method utilizes prior information about the probability densities of regions of interest (ROIs) in the color spaces and provides a solution to the minimal-partition problem to segment ROIs in one of the level sets of a two-phase level set formulation. PCSeg tool is tested on a number of microscopic images and provides good segmentation results on single cells as well as efficient segmentation of plasma cell clusters.

Contour-Seed Pairs Learning-Based Framework for Simultaneously Detecting and Segmenting Various Overlapping Cells/Nuclei in Microscopy Images

In this paper, we propose a novel contour-seed pairs learning-based framework for robust and automated cell/nucleus segmentation. Automated granular object segmentation in microscopy images has significant clinical importance for pathology grading of the cell carcinoma and gene expression. The focus of the past literature is dominated by either segmenting a certain type of cells/nuclei or simply splitting the clustered objects without contours inference of them. Our method addresses these issues by formulating the detection and segmentation tasks in terms of a unified regression problem, where a cascade sparse regression chain model is trained and then applied to return object locations and entire boundaries of clustered objects. In particular, we first learn a set of online convolutional features in each layer. Then, in the proposed cascade sparse regression chain, with the input from the learned features, we iteratively update the locations and clustered object boundaries until convergence. In this way, the boundary evidences of each individual object can be easily delineated and be further fed to a complete contour inference procedure optimized by the minimum description length principle. For any probe image, our method enables to analyze free-lying and overlapping cells with complex shapes. Experimental results show that the proposed method is very generic and performs well on contour inferences of various cell/nucleus types. Compared with the current segmentation techniques, our approach achieves state-of-the-art performances on four challenging datasets, i.e., the kidney renal cell carcinoma histopathology dataset, Drosophila Kc167 cellular dataset, differential interference contrast red blood cell dataset, and cervical cytology dataset.

MRI Brain Tumour Segmentation Using Hybrid Clustering and Classification by Back Propagation Algorithm

Generally the segmentation refers, the partitioning of an image into smaller regions to identify or locate the region of abnormality. Even though image segmentation is the challenging task in medical applications, due to contrary image, local observations of an image, noise image, non uniform texture of the images and so on. Many techniques are available for image segmentation, but still it requires to introduce an efficient, fast medical image segmentation methods. This research article introduces an efficient image segmentation method based on K means clustering integrated with a spatial Fuzzy C means clustering algorithms. The suggested technique combines the advantages of the two methods. K means segmentation requires minimum computation time, but spatial Fuzzy C means provides high accuracy for image segmentation. The performance of the proposed method is evaluated in terms of accuracy, PSNR and processing time. It also provides good implementation results for MRI brain image segmentation with high accuracy and minimal execution time. After completing the segmentation the of abnormal part of the input MRI brain image, it is compulsory to classify the image is normal or abnormal. There are many classifiers like a self organizing map, Back propagation algorithm, support vector machine etc., The algorithm helps to classify the abnormalities like benign or malignant brain tumour in case of MRI brain image. The abnormality is detected based on the extracted features from an input image. Discrete wavelet transform helps to find the hidden information from the MRI brain image. The extracted features are trained by Back Propagation Algorithm to classify the abnormalities of MRI brain image.

Deep dense multi-path neural network for prostate segmentation in magnetic resonance imaging

PURPOSE

We propose an approach of 3D convolutional neural network to segment the prostate in MR images.

METHODS

A 3D deep dense multi-path convolutional neural network that follows the framework of the encoder-decoder design is proposed. The encoder is built based upon densely connected layers that learn the high-level feature representation of the prostate. The decoder interprets the features and predicts the whole prostate volume by utilizing a residual layout and grouped convolution. A set of sub-volumes of MR images, centered at the prostate, is generated and fed into the proposed network for training purpose. The performance of the proposed network is compared to previously reported approaches.

RESULTS

Two independent datasets were employed to assess the proposed network. In quantitative evaluations, the proposed network achieved 95.11 and 89.01 Dice coefficients for the two datasets. The segmentation results were robust to variations in MR images. In comparison experiments, the segmentation performance of the proposed network was comparable to the previously reported approaches. In qualitative evaluations, the segmentation results by the proposed network were well matched to the ground truth provided by human experts.

CONCLUSIONS

The proposed network is capable of segmenting the prostate in an accurate and robust manner. This approach can be applied to other types of medical images.

A novel MRI segmentation method using CNN-based correction network for MRI-guided adaptive radiotherapy

PURPOSE

The purpose of this study was to expedite the contouring process for MRI-guided adaptive radiotherapy (MR-IGART), a convolutional neural network (CNN) deep-learning (DL) model is proposed to accurately segment the liver, kidneys, stomach, bowel and duodenum in 3D MR images.

METHODS

Images and structure contours for 120 patients were collected retrospectively. Treatment sites included pancreas, liver, stomach, adrenal gland, and prostate. The proposed DL model contains a voxel-wise label prediction CNN and a correction network which consists of two sub-networks. The prediction CNN and sub-networks in the correction network each includes a dense block which consists of twelve densely connected convolutional layers. The correction network was designed to improve the voxel-wise labeling accuracy of a CNN by learning and enforcing implicit anatomical constraints in the segmentation process. Its sub-networks learn to fix the erroneous classification of its previous network by taking as input both the original images and the softmax probability maps generated from its previous sub-network. The parameters of each sub-network were trained independently using piecewise training. The model was trained on 100 datasets, validated on 10 datasets and tested on the remaining 10 datasets. Dice coefficient, Hausdorff distance (HD) were calculated to evaluate the segmentation accuracy.

RESULTS

The proposed DL model was able to segment the organs with good accuracy. The correction network outperformed the conditional random field (CRF), a most comparable method that is usually applied as a post-processing step. For the 10 testing patients, the average Dice coefficients were 95.3 ± 0.73, 93.1 ± 2.22, 85.0 ± 3.75, 86.6 ± 2.69, and 65.5 ± 8.90 for liver, kidneys, stomach, bowel, and duodenum, respectively. The mean Hausdorff Distance (HD) were 5.41 ± 2.34, 6.23 ± 4.59, 6.88 ± 4.89, 5.90 ± 4.05, and 7.99 ± 6.84 mm, respectively. Manual contouring, as to correct the automatic segmentation results, was four times as fast as manual contouring from scratch.

CONCLUSION

The proposed method can automatically segment the liver, kidneys, stomach, bowel, and duodenum in 3D MR images with good accuracy. It is useful to expedite the manual contouring for MR-IGART.

Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning

Visual inspection of histopathology slides is one of the main methods used by pathologists to assess the stage, type and subtype of lung tumors. Adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) are the most prevalent subtypes of lung cancer, and their distinction requires visual inspection by an experienced pathologist. In this study, we trained a deep convolutional neural network (inception v3) on whole-slide images obtained from The Cancer Genome Atlas to accurately and automatically classify them into LUAD, LUSC or normal lung tissue. The performance of our method is comparable to that of pathologists, with an average area under the curve (AUC) of 0.97. Our model was validated on independent datasets of frozen tissues, formalin-fixed paraffin-embedded tissues and biopsies. Furthermore, we trained the network to predict the ten most commonly mutated genes in LUAD. We found that six of them-STK11, EGFR, FAT1, SETBP1, KRAS and TP53-can be predicted from pathology images, with AUCs from 0.733 to 0.856 as measured on a held-out population. These findings suggest that deep-learning models can assist pathologists in the detection of cancer subtype or gene mutations. Our approach can be applied to any cancer type, and the code is available at https://github.com/ncoudray/DeepPATH .

Fast unsupervised nuclear segmentation and classification scheme for automatic allred cancer scoring in immunohistochemical breast tissue images

BACKGROUND AND OBJECTIVE

This paper presents an improved scheme able to perform accurate segmentation and classification of cancer nuclei in immunohistochemical (IHC) breast tissue images in order to provide quantitative evaluation of estrogen or progesterone (ER/PR) receptor status that will assist pathologists in cancer diagnostic process.

METHODS

The proposed segmentation method is based on adaptive local thresholding and an enhanced morphological procedure, which are applied to extract all stained nuclei regions and to split overlapping nuclei. In fact, a new segmentation approach is presented here for cell nuclei detection from the IHC image using a modified Laplacian filter and an improved watershed algorithm. Stromal cells are then removed from the segmented image using an adaptive criterion in order to get fast tumor nuclei recognition. Finally, unsupervised classification of cancer nuclei is obtained by the combination of four common color separation techniques for a subsequent Allred cancer scoring.

RESULTS

Experimental results on various IHC tissue images of different cancer affected patients, demonstrate the effectiveness of the proposed scheme when compared to the manual scoring of pathological experts. A statistical analysis is performed on the whole image database between immuno-score of manual and automatic method, and compared with the scores that have reached using other state-of-art segmentation and classification strategies. According to the performance evaluation, we recorded more than 98% for both accuracy of detected nuclei and image cancer scoring over the truths provided by experienced pathologists which shows the best correlation with the expert's score (Pearson's correlation coefficient = 0.993, p-value < 0.005) and the lowest computational total time of 72.3 s/image (±1.9) compared to recent studied methods.

CONCLUSIONS

The proposed scheme can be easily applied for any histopathological diagnostic process that needs stained nuclear quantification and cancer grading. Moreover, the reduced processing time and manual interactions of our procedure can facilitate its implementation in a real-time device to construct a fully online evaluation system of IHC tissue images.

Deep Learning in Microscopy Image Analysis: A Survey

Computerized microscopy image analysis plays an important role in computer aided diagnosis and prognosis. Machine learning techniques have powered many aspects of medical investigation and clinical practice. Recently, deep learning is emerging as a leading machine learning tool in computer vision and has attracted considerable attention in biomedical image analysis. In this paper, we provide a snapshot of this fast-growing field, specifically for microscopy image analysis. We briefly introduce the popular deep neural networks and summarize current deep learning achievements in various tasks, such as detection, segmentation, and classification in microscopy image analysis. In particular, we explain the architectures and the principles of convolutional neural networks, fully convolutional networks, recurrent neural networks, stacked autoencoders, and deep belief networks, and interpret their formulations or modelings for specific tasks on various microscopy images. In addition, we discuss the open challenges and the potential trends of future research in microscopy image analysis using deep learning.

Deep learning nuclei detection: A simple approach can deliver state-of-the-art results

BACKGROUND

Deep convolutional neural networks have become a widespread tool for the detection of nuclei in histopathology images. Many implementations share a basic approach that includes generation of an intermediate map indicating the presence of a nucleus center, which we refer to as PMap. Nevertheless, these implementations often still differ in several parameters, resulting in different detection qualities.

METHODS

We identified several essential parameters and configured the basic PMap approach using combinations of them. We thoroughly evaluated and compared various configurations on multiple datasets with respect to detection quality, efficiency and training effort.

RESULTS

Post-processing of the PMap was found to have the largest impact on detection quality. Also, two different network architectures were identified that improve either detection quality or runtime performance. The best-performing configuration yields f1-measures of 0.816 on H&E stained images of colorectal adenocarcinomas and 0.819 on Ki-67 stained images of breast tumor tissue. On average, it was fully trained in less than 15,000 iterations and processed 4.15 megapixels per second at prediction time.

CONCLUSIONS

The basic PMap approach is greatly affected by certain parameters. Our evaluation provides guidance on their impact and best settings. When configured properly, this simple and efficient approach can yield equal detection quality as more complex and time-consuming state-of-the-art approaches.

Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes

BACKGROUND

Nuclear segmentation is an important step for profiling aberrant regions of histology sections. If nuclear segmentation can be resolved, then new biomarkers of nuclear phenotypes and their organization can be predicted for the application of precision medicine. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), nuclear phenotypes (e.g., vesicular, aneuploidy), and overlapping nuclei. The problem is further complicated as a result of variations in sample preparation (e.g., fixation, staining). Our hypothesis is that (i) deep learning techniques can learn complex phenotypic signatures that rise in tumor sections, and (ii) fusion of different representations (e.g., regions, boundaries) contributes to improved nuclear segmentation.

RESULTS

We have demonstrated that training of deep encoder-decoder convolutional networks overcomes complexities associated with multiple nuclear phenotypes, where we evaluate alternative architecture of deep learning for an improved performance against the simplicity of the design. In addition, improved nuclear segmentation is achieved by color decomposition and combining region- and boundary-based features through a fusion network. The trained models have been evaluated against approximately 19,000 manually annotated nuclei, and object-level Precision, Recall, F1-score and Standard Error are reported with the best F1-score being 0.91. Raw training images, annotated images, processed images, and source codes are released as a part of the Additional file 1.

CONCLUSIONS

There are two intrinsic barriers in nuclear segmentation to H&E stained images, which correspond to the diversity of nuclear phenotypes and perceptual boundaries between adjacent cells. We demonstrate that (i) the encoder-decoder architecture can learn complex phenotypes that include the vesicular type; (ii) delineation of overlapping nuclei is enhanced by fusion of region- and edge-based networks; (iii) fusion of ENets produces an improved result over the fusion of UNets; and (iv) fusion of networks is better than multitask learning. We suggest that our protocol enables processing a large cohort of whole slide images for applications in precision medicine.

Deep Learning Nuclei Detection in Digitized Histology Images by Superpixels

Background

Advances in image analysis and computational techniques have facilitated automatic detection of critical features in histopathology images. Detection of nuclei is critical for squamous epithelium cervical intraepithelial neoplasia (CIN) classification into normal, CIN1, CIN2, and CIN3 grades.

Methods

In this study, a deep learning (DL)-based nuclei segmentation approach is investigated based on gathering localized information through the generation of superpixels using a simple linear iterative clustering algorithm and training with a convolutional neural network.

Results

The proposed approach was evaluated on a dataset of 133 digitized histology images and achieved an overall nuclei detection (object-based) accuracy of 95.97%, with demonstrated improvement over imaging-based and clustering-based benchmark techniques.

Conclusions

The proposed DL-based nuclei segmentation Method with superpixel analysis has shown improved segmentation results in comparison to state-of-the-art methods.

SetSVM: An Approach to Set Classification in Nuclei-Based Cancer Detection

Due to the importance of nuclear structure in cancer diagnosis, several predictive models have been described for diagnosing a wide variety of cancers based on nuclear morphology. In many computer-aided diagnosis (CAD) systems, cancer detection tasks can be generally formulated as set classification problems, which can not be directly solved by classifying single instances. In this paper, we propose a novel set classification approach SetSVM to build a predictive model by considering any nuclei set as a whole without specific assumptions. SetSVM features highly discriminative power in cancer detection challenges in the sense that it not only optimizes the classifier decision boundary but also transfers discriminative information to set representation learning. During model training, these two processes are unified in the support vector machine (SVM) maximum separation margin problem. Experiment results show that SetSVM provides significant improvements compared with five commonly used approaches in cancer detection tasks utilizing 260 patients in total across three different cancer types, namely, thyroid cancer, liver cancer, and melanoma. In addition, we show that SetSVM enables visual interpretation of discriminative nuclear characteristics representing the nuclei set. These features make SetSVM a potentially practical tool in building accurate and interpretable CAD systems for cancer detection.

Efficient and robust cell detection: A structured regression approach

Efficient and robust cell detection serves as a critical prerequisite for many subsequent biomedical image analysis methods and computer-aided diagnosis (CAD). It remains a challenging task due to touching cells, inhomogeneous background noise, and large variations in cell sizes and shapes. In addition, the ever-increasing amount of available datasets and the high resolution of whole-slice scanned images pose a further demand for efficient processing algorithms. In this paper, we present a novel structured regression model based on a proposed fully residual convolutional neural network for efficient cell detection. For each testing image, our model learns to produce a dense proximity map that exhibits higher responses at locations near cell centers. Our method only requires a few training images with weak annotations (just one dot indicating the cell centroids). We have extensively evaluated our method using four different datasets, covering different microscopy staining methods (e.g., H & E or Ki-67 staining) or image acquisition techniques (e.g., bright-filed image or phase contrast). Experimental results demonstrate the superiority of our method over existing state of the art methods in terms of both detection accuracy and running time.

Deep Deconvolutional Neural Network for Target Segmentation of Nasopharyngeal Cancer in Planning Computed Tomography Images

BACKGROUND

Radiotherapy is one of the main treatment methods for nasopharyngeal carcinoma (NPC). It requires exact delineation of the nasopharynx gross tumor volume (GTVnx), the metastatic lymph node gross tumor volume (GTVnd), the clinical target volume (CTV), and organs at risk in the planning computed tomography images. However, this task is time-consuming and operator dependent. In the present study, we developed an end-to-end deep deconvolutional neural network (DDNN) for segmentation of these targets.

METHODS

The proposed DDNN is an end-to-end architecture enabling fast training and testing. It consists of two important components: an encoder network and a decoder network. The encoder network was used to extract the visual features of a medical image and the decoder network was used to recover the original resolution by deploying deconvolution. A total of 230 patients diagnosed with NPC stage I or stage II were included in this study. Data from 184 patients were chosen randomly as a training set to adjust the parameters of DDNN, and the remaining 46 patients were the test set to assess the performance of the model. The Dice similarity coefficient (DSC) was used to quantify the segmentation results of the GTVnx, GTVnd, and CTV. In addition, the performance of DDNN was compared with the VGG-16 model.

RESULTS

The proposed DDNN method outperformed the VGG-16 in all the segmentation. The mean DSC values of DDNN were 80.9% for GTVnx, 62.3% for the GTVnd, and 82.6% for CTV, whereas VGG-16 obtained 72.3, 33.7, and 73.7% for the DSC values, respectively.

CONCLUSION

DDNN can be used to segment the GTVnx and CTV accurately. The accuracy for the GTVnd segmentation was relatively low due to the considerable differences in its shape, volume, and location among patients. The accuracy is expected to increase with more training data and combination of MR images. In conclusion, DDNN has the potential to improve the consistency of contouring and streamline radiotherapy workflows, but careful human review and a considerable amount of editing will be required.

NeuroSeg: automated cell detection and segmentation for in vivo two-photon Ca2+ imaging data

Two-photon Ca²⁺ imaging has become a popular approach for monitoring neuronal population activity with cellular or subcellular resolution in vivo. This approach allows for the recording of hundreds to thousands of neurons per animal and thus leads to a large amount of data to be processed. In particular, manually drawing regions of interest is the most time-consuming aspect of data analysis. However, the development of automated image analysis pipelines, which will be essential for dealing with the likely future deluge of imaging data, remains a major challenge. To address this issue, we developed NeuroSeg, an open-source MATLAB program that can facilitate the accurate and efficient segmentation of neurons in two-photon Ca²⁺ imaging data. We proposed an approach using a generalized Laplacian of Gaussian filter to detect cells and weighting-based segmentation to separate individual cells from the background. We tested this approach on an in vivo two-photon Ca²⁺ imaging dataset obtained from mouse cortical neurons with differently sized view fields. We show that this approach exhibits superior performance for cell detection and segmentation compared with the existing published tools. In addition, we integrated the previously reported, activity-based segmentation into our approach and found that this combined method was even more promising. The NeuroSeg software, including source code and graphical user interface, is freely available and will be a useful tool for in vivo brain activity mapping.

Automatic segmentation of the clinical target volume and organs at risk in the planning CT for rectal cancer using deep dilated convolutional neural networks

PURPOSE

Delineation of the clinical target volume (CTV) and organs at risk (OARs) is very important for radiotherapy but is time-consuming and prone to inter-observer variation. Here, we proposed a novel deep dilated convolutional neural network (DDCNN)-based method for fast and consistent auto-segmentation of these structures.

METHODS

Our DDCNN method was an end-to-end architecture enabling fast training and testing. Specifically, it employed a novel multiple-scale convolutional architecture to extract multiple-scale context features in the early layers, which contain the original information on fine texture and boundaries and which are very useful for accurate auto-segmentation. In addition, it enlarged the receptive fields of dilated convolutions at the end of networks to capture complementary context features. Then, it replaced the fully connected layers with fully convolutional layers to achieve pixel-wise segmentation. We used data from 278 patients with rectal cancer for evaluation. The CTV and OARs were delineated and validated by senior radiation oncologists in the planning computed tomography (CT) images. A total of 218 patients chosen randomly were used for training, and the remaining 60 for validation. The Dice similarity coefficient (DSC) was used to measure segmentation accuracy.

RESULTS

Performance was evaluated on segmentation of the CTV and OARs. In addition, the performance of DDCNN was compared with that of U-Net. The proposed DDCNN method outperformed the U-Net for all segmentations, and the average DSC value of DDCNN was 3.8% higher than that of U-Net. Mean DSC values of DDCNN were 87.7% for the CTV, 93.4% for the bladder, 92.1% for the left femoral head, 92.3% for the right femoral head, 65.3% for the intestine, and 61.8% for the colon. The test time was 45 s per patient for segmentation of all the CTV, bladder, left and right femoral heads, colon, and intestine. We also assessed our approaches and results with those in the literature: our system showed superior performance and faster speed.

CONCLUSIONS

These data suggest that DDCNN can be used to segment the CTV and OARs accurately and efficiently. It was invariant to the body size, body shape, and age of the patients. DDCNN could improve the consistency of contouring and streamline radiotherapy workflows.

Low Dimensional Representation of Fisher Vectors for Microscopy Image Classification

Microscopy image classification is important in various biomedical applications, such as cancer subtype identification, and protein localization for high content screening. To achieve automated and effective microscopy image classification, the representative and discriminative capability of image feature descriptors is essential. To this end, in this paper, we propose a new feature representation algorithm to facilitate automated microscopy image classification. In particular, we incorporate Fisher vector (FV) encoding with multiple types of local features that are handcrafted or learned, and we design a separation-guided dimension reduction method to reduce the descriptor dimension while increasing its discriminative capability. Our method is evaluated on four publicly available microscopy image data sets of different imaging types and applications, including the UCSB breast cancer data set, MICCAI 2015 CBTC challenge data set, and IICBU malignant lymphoma, and RNAi data sets. Our experimental results demonstrate the advantage of the proposed low-dimensional FV representation, showing consistent performance improvement over the existing state of the art and the commonly used dimension reduction techniques.

A Dataset and a Technique for Generalized Nuclear Segmentation for Computational Pathology

Nuclear segmentation in digital microscopic tissue images can enable extraction of high-quality features for nuclear morphometrics and other analysis in computational pathology. Conventional image processing techniques, such as Otsu thresholding and watershed segmentation, do not work effectively on challenging cases, such as chromatin-sparse and crowded nuclei. In contrast, machine learning-based segmentation can generalize across various nuclear appearances. However, training machine learning algorithms requires data sets of images, in which a vast number of nuclei have been annotated. Publicly accessible and annotated data sets, along with widely agreed upon metrics to compare techniques, have catalyzed tremendous innovation and progress on other image classification problems, particularly in object recognition. Inspired by their success, we introduce a large publicly accessible data set of hematoxylin and eosin (H&E)-stained tissue images with more than 21000 painstakingly annotated nuclear boundaries, whose quality was validated by a medical doctor. Because our data set is taken from multiple hospitals and includes a diversity of nuclear appearances from several patients, disease states, and organs, techniques trained on it are likely to generalize well and work right out-of-the-box on other H&E-stained images. We also propose a new metric to evaluate nuclear segmentation results that penalizes object- and pixel-level errors in a unified manner, unlike previous metrics that penalize only one type of error. We also propose a segmentation technique based on deep learning that lays a special emphasis on identifying the nuclear boundaries, including those between the touching or overlapping nuclei, and works well on a diverse set of test images.

An Advanced Deep Learning Approach for Ki-67 Stained Hotspot Detection and Proliferation Rate Scoring for Prognostic Evaluation of Breast Cancer

Being a non-histone protein, Ki-67 is one of the essential biomarkers for the immunohistochemical assessment of proliferation rate in breast cancer screening and grading. The Ki-67 signature is always sensitive to radiotherapy and chemotherapy. Due to random morphological, color and intensity variations of cell nuclei (immunopositive and immunonegative), manual/subjective assessment of Ki-67 scoring is error-prone and time-consuming. Hence, several machine learning approaches have been reported; nevertheless, none of them had worked on deep learning based hotspots detection and proliferation scoring. In this article, we suggest an advanced deep learning model for computerized recognition of candidate hotspots and subsequent proliferation rate scoring by quantifying Ki-67 appearance in breast cancer immunohistochemical images. Unlike existing Ki-67 scoring techniques, our methodology uses Gamma mixture model (GMM) with Expectation-Maximization for seed point detection and patch selection and deep learning, comprises with decision layer, for hotspots detection and proliferation scoring. Experimental results provide 93% precision, 0.88% recall and 0.91% F-score value. The model performance has also been compared with the pathologists’ manual annotations and recently published articles. In future, the proposed deep learning framework will be highly reliable and beneficial to the junior and senior pathologists for fast and efficient Ki-67 scoring.

DeepPap: Deep Convolutional Networks for Cervical Cell Classification

Automation-assisted cervical screening via Pap smear or liquid-based cytology (LBC) is a highly effective cell imaging based cancer detection tool, where cells are partitioned into "abnormal" and "normal" categories. However, the success of most traditional classification methods relies on the presence of accurate cell segmentations. Despite sixty years of research in this field, accurate segmentation remains a challenge in the presence of cell clusters and pathologies. Moreover, previous classification methods are only built upon the extraction of hand-crafted features, such as morphology and texture. This paper addresses these limitations by proposing a method to directly classify cervical cells-without prior segmentation-based on deep features, using convolutional neural networks (ConvNets). First, the ConvNet is pretrained on a natural image dataset. It is subsequently fine-tuned on a cervical cell dataset consisting of adaptively resampled image patches coarsely centered on the nuclei. In the testing phase, aggregation is used to average the prediction scores of a similar set of image patches. The proposed method is evaluated on both Pap smear and LBC datasets. Results show that our method outperforms previous algorithms in classification accuracy (98.3%), area under the curve (0.99) values, and especially specificity (98.3%), when applied to the Herlev benchmark Pap smear dataset and evaluated using five-fold cross validation. Similar superior performances are also achieved on the HEMLBC (H&E stained manual LBC) dataset. Our method is promising for the development of automation-assisted reading systems in primary cervical screening.

Training Nuclei Detection Algorithms with Simple Annotations

BACKGROUND

Generating good training datasets is essential for machine learning-based nuclei detection methods. However, creating exhaustive nuclei contour annotations, to derive optimal training data from, is often infeasible.

METHODS

We compared different approaches for training nuclei detection methods solely based on nucleus center markers. Such markers contain less accurate information, especially with regard to nuclear boundaries, but can be produced much easier and in greater quantities. The approaches use different automated sample extraction methods to derive image positions and class labels from nucleus center markers. In addition, the approaches use different automated sample selection methods to improve the detection quality of the classification algorithm and reduce the run time of the training process. We evaluated the approaches based on a previously published generic nuclei detection algorithm and a set of Ki-67-stained breast cancer images.

RESULTS

A Voronoi tessellation-based sample extraction method produced the best performing training sets. However, subsampling of the extracted training samples was crucial. Even simple class balancing improved the detection quality considerably. The incorporation of active learning led to a further increase in detection quality.

CONCLUSIONS

With appropriate sample extraction and selection methods, nuclei detection algorithms trained on the basis of simple center marker annotations can produce comparable quality to algorithms trained on conventionally created training sets.

Joint multiple fully connected convolutional neural network with extreme learning machine for hepatocellular carcinoma nuclei grading

Accurate cell grading of cancerous tissue pathological image is of great importance in medical diagnosis and treatment. This paper proposes a joint multiple fully connected convolutional neural network with extreme learning machine (MFC-CNN-ELM) architecture for hepatocellular carcinoma (HCC) nuclei grading. First, in preprocessing stage, each grayscale image patch with the fixed size is obtained using center-proliferation segmentation (CPS) method and the corresponding labels are marked under the guidance of three pathologists. Next, a multiple fully connected convolutional neural network (MFC-CNN) is designed to extract the multi-form feature vectors of each input image automatically, which considers multi-scale contextual information of deep layer maps sufficiently. After that, a convolutional neural network extreme learning machine (CNN-ELM) model is proposed to grade HCC nuclei. Finally, a back propagation (BP) algorithm, which contains a new up-sample method, is utilized to train MFC-CNN-ELM architecture. The experiment comparison results demonstrate that our proposed MFC-CNN-ELM has superior performance compared with related works for HCC nuclei grading. Meanwhile, external validation using ICPR 2014 HEp-2 cell dataset shows the good generalization of our MFC-CNN-ELM architecture.