Multi-class texture analysis in colorectal cancer histology

Computerized Classification of Prostate Cancer Gleason Scores from Whole Slide Images

Histological Gleason grading of tumor patterns is one of the most powerful prognostic predictors in prostate cancer. However, manual analysis and grading performed by pathologists are typically subjective and time-consuming. In this paper, we present an automatic technique for Gleason grading of prostate cancer from H&E stained whole slide pathology images using a set of novel completed and statistical local binary pattern (CSLBP) descriptors. First, the technique divides the whole slide image (WSI) into a set of small image tiles, where salient tumor tiles with high nuclei densities are selected for analysis. The CSLBP texture features that encode pixel intensity variations from circularly surrounding neighborhoods are extracted from salient image tiles to characterize different Gleason patterns. Finally, the CSLBP texture features computed from all tiles are integrated and utilized by the multi-class support vector machine (SVM) that assigns patient slides with different Gleason scores such as 6, 7, or ≥ 8. Experiments have been performed on 312 different patient cases selected from the cancer genome atlas (TCGA) and have achieved superior performances over state-of-the-art texture descriptors and baseline methods including deep learning models for prostate cancer Gleason grading.

Hist-Immune signature: a prognostic factor in colorectal cancer using immunohistochemical slide image analysis

Computerized image analysis for whole-slide images has been shown to improve efficiency, accuracy, and consistency in histopathology evaluations. We aimed to assess whether immunohistochemistry (IHC) image quantitative features can reflect the immune status and provide prognostic information for colorectal cancer patients. A fully automated pipeline was designed to extract histogram features from IHC digital images in a training set (N = 243). A Hist-Immune signature was generated with selected features using the LASSO Cox model. The results were validated using internal (N = 147) and external (N = 76) validation sets. The five-feature-based Hist-Immune signature was significantly associated with overall survival in training (HR 2.72, 95% CI 1.68-4.41, P < .001), internal (2.86, 1.28-6.39, 0.010), and external (2.30, 1.02-6.16, 0.044) validation sets. The full model constructed by integrating the Hist-Immune signature and clinicopathological factors had good discrimination ability (C-index 0.727, 95% CI 0.678-0.776), confirmed using internal (0.703, 0.621-0.784) and external (0.756, 0.653-0.859) validation sets. Our findings indicate that the Hist-Immune signature constructed based on the quantitative features could reflect the immune status of patients with colorectal cancer, which might advocate change in risk stratification and consequent precision medicine.

Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study

As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large sets of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning. We investigated the notion of similarity and dissimilarity in pathology whole-slide images and compared different setups from unsupervised and semi-supervised to supervised learning in our experiments. Additionally, different approaches were tested, applying few-shot learning on two publicly available pathology image datasets. We achieved high accuracy and generalization when the learned representations were applied to two different pathology datasets.

A Multiscale Approach for Whole-Slide Image Segmentation of five Tissue Classes in Urothelial Carcinoma Slides

In pathology labs worldwide, we see an increasing number of tissue samples that need to be assessed without the same increase in the number of pathologists. Computational pathology, where digital scans of histological samples called whole-slide images (WSI) are processed by computational tools, can be of help for the pathologists and is gaining research interests. Most research effort has been given to classify slides as being cancerous or not, localization of cancerous regions, and to the “big-four” in cancer: breast, lung, prostate, and bowel. Urothelial carcinoma, the most common form of bladder cancer, is expensive to follow up due to a high risk of recurrence, and grading systems have a high degree of inter- and intra-observer variability. The tissue samples of urothelial carcinoma contain a mixture of damaged tissue, blood, stroma, muscle, and urothelium, where it is mainly muscle and urothelium that is diagnostically relevant. A coarse segmentation of these tissue types would be useful to i) guide pathologists to the diagnostic relevant areas of the WSI, and ii) use as input in a computer-aided diagnostic (CAD) system. However, little work has been done on segmenting tissue types in WSIs, and on computational pathology for urothelial carcinoma in particular. In this work, we are using convolutional neural networks (CNN) for multiscale tile-wise classification and coarse segmentation, including both context and detail, by using three magnification levels: 25x, 100x, and 400x. 28 models were trained on weakly labeled data from 32 WSIs, where the best model got an F1-score of 96.5% across six classes. The multiscale models were consistently better than the single-scale models, demonstrating the benefit of combining multiple scales. No tissue-class ground-truth for complete WSIs exist, but the best models were used to segment seven unseen WSIs where the results were manually inspected by a pathologist and are considered as very promising.

Heterogeneity Measurement of Cardiac Tissues Leveraging Uncertainty Information from Image Segmentation

Identifying arrhythmia substrates and quantifying their heterogeneity has great potential to provide critical guidance for radio frequency ablation. However, quantitative analysis of heterogeneity on cardiac optical coherence tomography (OCT) images is lacking. In this paper, we conduct the first study on quantifying cardiac tissue heterogeneity from human OCT images. Our proposed method applies a dropout-based Monte Carlo sampling technique to measure the model uncertainty. The heterogeneity information is extracted by decoupling the intra/inter-tissue heterogeneity and tissue boundary uncertainty from the uncertainty measurement. We empirically demonstrate that our model can highlight the subtle features from OCT images, and the heterogeneity information extracted is positively correlated with the tissue heterogeneity information from corresponding histology images.

Multiplex Cellular Communities in Multi-Gigapixel Colorectal Cancer Histology Images for Tissue Phenotyping

In computational pathology, automated tissue phenotyping in cancer histology images is a fundamental tool for profiling tumor microenvironments. Current tissue phenotyping methods use features derived from image patches which may not carry biological significance. In this work, we propose a novel multiplex cellular community-based algorithm for tissue phenotyping integrating cell-level features within a graph-based hierarchical framework. We demonstrate that such integration offers better performance compared to prior deep learning and texture-based methods as well as to cellular community based methods using uniplex networks. To this end, we construct celllevel graphs using texture, alpha diversity and multi-resolution deep features. Using these graphs, we compute cellular connectivity features which are then employed for the construction of a patch-level multiplex network. Over this network, we compute multiplex cellular communities using a novel objective function. The proposed objective function computes a low-dimensional subspace from each cellular network and subsequently seeks a common low-dimensional subspace using the Grassmann manifold. We evaluate our proposed algorithm on three publicly available datasets for tissue phenotyping, demonstrating a significant improvement over existing state-of-the-art methods.

Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images

Pathologists are responsible for cancer type diagnoses from histopathological cancer tissues. However, it is known that microscopic examination is tedious and time-consuming. In recent years, a long list of machine learning approaches to image classification and whole-slide segmentation has been developed to support pathologists. Although many showed exceptional performances, the majority of them are not able to rationalize their decisions. In this study, we developed an explainable classifier to support decision making for medical diagnoses. The proposed model does not provide an explanation about the causality between the input and the decisions, but offers a human-friendly explanation about the plausibility of the decision. Cumulative Fuzzy Class Membership Criterion (CFCMC) explains its decisions in three ways: through a semantical explanation about the possibilities of misclassification, showing the training sample responsible for a certain prediction and showing training samples from conflicting classes. In this paper, we explain about the mathematical structure of the classifier, which is not designed to be used as a fully automated diagnosis tool but as a support system for medical experts. We also report on the accuracy of the classifier against real world histopathological data for colorectal cancer. We also tested the acceptability of the system through clinical trials by 14 pathologists. We show that the proposed classifier is comparable to state of the art neural networks in accuracy, but more importantly it is more acceptable to be used by human experts as a diagnosis tool in the medical domain.

Ensemble of convolutional neural networks for bioimage classification

This work presents a system based on an ensemble of Convolutional Neural Networks (CNNs) and descriptors for bioimage classification that has been validated on different datasets of color images. The proposed system represents a very simple yet effective way of boosting the performance of trained CNNs by composing multiple CNNs into an ensemble and combining scores by sum rule. Several types of ensembles are considered, with different CNN topologies along with different learning parameter sets. The proposed system not only exhibits strong discriminative power but also generalizes well over multiple datasets thanks to the combination of multiple descriptors based on different feature types, both learned and handcrafted. Separate classifiers are trained for each descriptor, and the entire set of classifiers is combined by sum rule. Results show that the proposed system obtains state-of-the-art performance across four different bioimage and medical datasets. The MATLAB code of the descriptors will be available at https://github.com/LorisNanni.

A New Deep Learning Model Selection Method for Colorectal Cancer Classification

Deep learning is one of the most commonly used techniques in computer-aided diagnosis systems. Their exploitation for histopathological image analysis is important because of the complex morphology of whole slide images. However, the main limitation of these methods is the restricted number of available medical images, which can lead to an overfitting problem. Many studies have suggested the use of static ensemble learning methods to address this issue. This article aims to propose a new dynamic ensemble deep learning method. First, it generates a set of models based on the transfer learning strategy from deep neural networks. Then, the relevant subset of models is selected by the particle swarm optimization algorithm and combined by voting or averaging methods. The proposed approach was tested on a histopathological dataset for colorectal cancer classification, based on seven types of CNNs. The method has achieved accurate results (94.52%) by the Resnet121 model and the voting strategy, which provides important insights into the efficiency of dynamic ensembling in deep learning.

Characterization of interstitial diffuse fibrosis patterns using texture analysis of myocardial native T1 mapping

Background

The pattern of myocardial fibrosis differs significantly between different cardiomyopathies. Fibrosis in hypertrophic cardiomyopathy (HCM) is characteristically as patchy and regional but in dilated cardiomyopathy (DCM) as diffuse and global. We sought to investigate if texture analyses on myocardial native T₁ mapping can differentiate between fibrosis patterns in patients with HCM and DCM.

Methods

We prospectively acquired native myocardial T₁ mapping images for 321 subjects (55±15 years, 70% male): 65 control, 116 HCM, and 140 DCM patients. To quantify different fibrosis patterns, four sets of texture descriptors were used to extract 152 texture features from native T₁ maps. Seven features were sequentially selected to identify HCM- and DCM-specific patterns in 70% of data (training dataset). Pattern reproducibility and generalizability were tested on the rest of data (testing dataset) using support vector machines (SVM) and regression models.

Results

Pattern-derived texture features were capable to identify subjects in HCM, DCM, and controls cohorts with 202/237(85.2%) accuracy of all subjects in the training dataset using 10-fold cross-validation on SVM (AUC = 0.93, 0.93, and 0.93 for controls, HCM and DCM, respectively), while pattern-independent global native T₁ mapping was poorly capable to identify those subjects with 121/237(51.1%) accuracy (AUC = 0.78, 0.51, and 0.74) (P<0.001 for all). The pattern-derived features were reproducible with excellent intra- and inter-observer reliability and generalizable on the testing dataset with 75/84(89.3%) accuracy.

Conclusion

Texture analysis of myocardial native T₁ mapping can characterize fibrosis patterns in HCM and DCM patients and provides additional information beyond average native T₁ values.

Learning from irregularly sampled data for endomicroscopy super-resolution: a comparative study of sparse and dense approaches

Purpose

Probe-based confocal laser endomicroscopy (pCLE) enables performing an optical biopsy via a probe. pCLE probes consist of multiple optical fibres arranged in a bundle, which taken together generate signals in an irregularly sampled pattern. Current pCLE reconstruction is based on interpolating irregular signals onto an over-sampled Cartesian grid, using a naive linear interpolation. It was shown that convolutional neural networks (CNNs) could improve pCLE image quality. Yet classical CNNs may be suboptimal in regard to irregular data.

Methods

We compare pCLE reconstruction and super-resolution (SR) methods taking irregularly sampled or reconstructed pCLE images as input. We also propose to embed a Nadaraya–Watson (NW) kernel regression into the CNN framework as a novel trainable CNN layer. We design deep learning architectures allowing for reconstructing high-quality pCLE images directly from the irregularly sampled input data. We created synthetic sparse pCLE images to evaluate our methodology.

Results

The results were validated through an image quality assessment based on a combination of the following metrics: peak signal-to-noise ratio and the structural similarity index. Our analysis indicates that both dense and sparse CNNs outperform the reconstruction method currently used in the clinic.

Conclusion

The main contributions of our study are a comparison of sparse and dense approach in pCLE image reconstruction. We also implement trainable generalised NW kernel regression as a novel sparse approach. We also generated synthetic data for training pCLE SR.

Electronic supplementary material

The online version of this article (10.1007/s11548-020-02170-7) contains supplementary material, which is available to authorized users.

Classification of childhood medulloblastoma into WHO-defined multiple subtypes based on textural analysis

Childhood medulloblastoma is a case of a childhood brain tumour that requires close attention due to the low survival rate. Effective prognosis depends a lot on accurate detection of its subtype. The present study proposes a texture-based computer-aided categorization of childhood medulloblastoma samples. According to the World Health Organization, it has four subtypes (desmoplastic, classic, nodular and large). Classification is done in two levels: (i) normal and abnormal and (ii) its four subtypes. The system is evaluated on indigenous patient samples collected from the region. The main objective of database generation is to create a data set of childhood medulloblastoma samples since there exists no available benchmark data set. The proposed framework for automated classification is based on the architectural property and the distribution of cells. Five texture features were extracted for the feature set, namely: grey-level co-occurrence matrix, grey-level run length matrix, first-order histogram features, local binary pattern and Tamura features. The performance of each feature set was evaluated, both individually and in combinations, using five different classifiers. Fivefold cross-validation was used for training and testing the data set. Experiments on both individual feature sets and combinations (best-2, best-3, best-4 and all-5) of feature sets were evaluated based on the accuracy of performance. It was revealed that the combined best-4 feature set resulted in the highest accuracy of 91.3%. The precision, recall and specificity were 0.913, 0.913 and 0.97, respectively. Significantly, it implied that the all-5 feature set is not necessary to have a useful classification. Feature reduction by principal component analysis resulted in increased accuracy of 96.7%. LAY DESCRIPTION: Childhood medulloblastoma is a case of childhood brain tumour that requires high attention due to a low survival rate. Effective prognosis depends a lot on accurate detection of its subtype. The present study proposes a texture-based computer-aided categorization of childhood medulloblastoma samples. According to the World Health Organization (W.H.O), it has four subtypes (desmoplastic, classic, nodular, and large). Classification is done in two levels: i) normal and abnormal ii) its four subtypes. The system is evaluated on indigenous patient samples collected from the region. The main objective of database generation is to create a data set of childhood medulloblastoma samples since there exists no available benchmark data set. The proposed framework is a model for the automatic classification of the samples. The tissue samples obtained post-operation by doctors are converted into images, and then necessary algorithms are applied so that certain features describing each group of the image are known and studied for classification. Later these images are classified using the image features into the subtypes of abnormal samples.

Cellular community detection for tissue phenotyping in colorectal cancer histology images

Classification of various types of tissue in cancer histology images based on the cellular compositions is an important step towards the development of computational pathology tools for systematic digital profiling of the spatial tumor microenvironment. Most existing methods for tissue phenotyping are limited to the classification of tumor and stroma and require large amount of annotated histology images which are often not available. In the current work, we pose the problem of identifying distinct tissue phenotypes as finding communities in cellular graphs or networks. First, we train a deep neural network for cell detection and classification into five distinct cellular components. Considering the detected nuclei as nodes, potential cell-cell connections are assigned using Delaunay triangulation resulting in a cell-level graph. Based on this cell graph, a feature vector capturing potential cell-cell connection of different types of cells is computed. These feature vectors are used to construct a patch-level graph based on chi-square distance. We map patch-level nodes to the geometric space by representing each node as a vector of geodesic distances from other nodes in the network and iteratively drifting the patch nodes in the direction of positive density gradients towards maximum density regions. The proposed algorithm is evaluated on a publicly available dataset and another new large-scale dataset consisting of 280K patches of seven tissue phenotypes. The estimated communities have significant biological meanings as verified by the expert pathologists. A comparison with current state-of-the-art methods reveals significant performance improvement in tissue phenotyping.

Stochastic Selection of Activation Layers for Convolutional Neural Networks

In recent years, the field of deep learning has achieved considerable success in pattern recognition, image segmentation, and many other classification fields. There are many studies and practical applications of deep learning on images, video, or text classification. Activation functions play a crucial role in discriminative capabilities of the deep neural networks and the design of new "static" or "dynamic" activation functions is an active area of research. The main difference between "static" and "dynamic" functions is that the first class of activations considers all the neurons and layers as identical, while the second class learns parameters of the activation function independently for each layer or even each neuron. Although the "dynamic" activation functions perform better in some applications, the increased number of trainable parameters requires more computational time and can lead to overfitting. In this work, we propose a mixture of "static" and "dynamic" activation functions, which are stochastically selected at each layer. Our idea for model design is based on a method for changing some layers along the lines of different functional blocks of the best performing CNN models, with the aim of designing new models to be used as stand-alone networks or as a component of an ensemble. We propose to replace each activation layer of a CNN (usually a ReLU layer) by a different activation function stochastically drawn from a set of activation functions: in this way, the resulting CNN has a different set of activation function layers.

Label-free multiphoton imaging allows brain tumor recognition based on texture analysis-a study of 382 tumor patients

Background

Label-free multiphoton microscopy has been suggested for intraoperative recognition and delineation of brain tumors. For any future clinical application, appropriate approaches for image acquisition and analysis have to be developed. Moreover, an evaluation of the reliability of the approach, taking into account inter- and intrapatient variability, is needed.

Methods

Coherent anti-Stokes Raman scattering (CARS), two-photon excited fluorescence (TPEF), and second-harmonic generation were acquired on cryosections of brain tumors of 382 patients and 28 human nontumor brain samples. Texture parameters of those images were calculated and used as input for linear discriminant analysis.

Results

The combined analysis of texture parameters of the CARS and TPEF signal proved to be most suited for the discrimination of nontumor brain versus brain tumors (low- and high-grade astrocytoma, oligodendroglioma, glioblastoma, recurrent glioblastoma, brain metastases of lung, colon, renal, and breast cancer and of malignant melanoma) leading to a correct rate of 96% (sensitivity: 96%, specificity: 100%). To approximate the clinical setting, the results were validated on 42 fresh, unfixed tumor biopsies. 82% of the tumors and, most important, all of the nontumor samples were correctly recognized. An image resolution of 1 µm was sufficient to distinguish brain tumors and nontumor brain. Moreover, the vast majority of single fields of view of each patient’s sample were correctly classified with high probabilities, which is important for clinical translation.

Conclusion

Label-free multiphoton imaging might allow fast and accurate intraoperative delineation of primary and secondary brain tumors in combination with endoscopic systems.

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

Pathologists are responsible for rapidly providing a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. In addition to on-site colleagues, there is an active worldwide community of pathologists on social media for complementary opinions. Such access to pathologists worldwide has the capacity to improve diagnostic accuracy and generate broader consensus on next steps in patient care. From Twitter we curate 13,626 images from 6,351 tweets from 25 pathologists from 13 countries. We supplement the Twitter data with 113,161 images from 1,074,484 PubMed articles. We develop machine learning and deep learning models to (i) accurately identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. Area Under Receiver Operating Characteristic (AUROC) is 0.805-0.996 for these tasks. We repurpose the disease classifier to search for similar disease states given an image and clinical covariates. We report precision@k = 1 = 0.7618 ± 0.0018 (chance 0.397 ± 0.004, mean ±stdev ). The classifiers find that texture and tissue are important clinico-visual features of disease. Deep features trained only on natural images (e.g., cats and dogs) substantially improved search performance, while pathology-specific deep features and cell nuclei features further improved search to a lesser extent. We implement a social media bot (@pathobot on Twitter) to use the trained classifiers to aid pathologists in obtaining real-time feedback on challenging cases. If a social media post containing pathology text and images mentions the bot, the bot generates quantitative predictions of disease state (normal/artifact/infection/injury/nontumor, preneoplastic/benign/low-grade-malignant-potential, or malignant) and lists similar cases across social media and PubMed. Our project has become a globally distributed expert system that facilitates pathological diagnosis and brings expertise to underserved regions or hospitals with less expertise in a particular disease. This is the first pan-tissue pan-disease (i.e., from infection to malignancy) method for prediction and search on social media, and the first pathology study prospectively tested in public on social media. We will share data through http://pathobotology.org . We expect our project to cultivate a more connected world of physicians and improve patient care worldwide.

ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning

Machine learning algorithms hold the promise to effectively automate the analysis of histopathological images that are routinely generated in clinical practice. Any machine learning method used in the clinical diagnostic process has to be extremely accurate and, ideally, provide a measure of uncertainty for its predictions. Such accurate and reliable classifiers need enough labelled data for training, which requires time-consuming and costly manual annotation by pathologists. Thus, it is critical to minimise the amount of data needed to reach the desired accuracy by maximising the efficiency of training. We propose an accurate, reliable and active (ARA) image classification framework and introduce a new Bayesian Convolutional Neural Network (ARA-CNN) for classifying histopathological images of colorectal cancer. The model achieves exceptional classification accuracy, outperforming other models trained on the same dataset. The network outputs an uncertainty measurement for each tested image. We show that uncertainty measures can be used to detect mislabelled training samples and can be employed in an efficient active learning workflow. Using a variational dropout-based entropy measure of uncertainty in the workflow speeds up the learning process by roughly 45%. Finally, we utilise our model to segment whole-slide images of colorectal tissue and compute segmentation-based spatial statistics.

Colorization of H&E stained tissue using Deep Learning

Histopathology is a critical tool in the diagnosis and stratification of cancer. Digital Pathology involves the scanning of stained and fixed tissue samples to produce highresolution images that can be used for computer-aided diagnosis and research. A common challenge in digital pathology related to the quality and characteristics of staining, which can vary widely from center to center and also within the same institution depending on the age of the stain and other human factors. In this paper we examine the use of deep learning models for colorizing H&E stained tissue images and compare the results with traditional image processing/statistical approaches that have been developed for standardizing or normalizing histopathology images. We adapt existing deep learning models that have been developed for colorizing natural images and compare the results with models developed specifically for digital pathology. Our results show that deep learning approaches can standardize the colorization of H&E images. The performance as measured by the chi-square statistic shows that the deep learning approach can be nearly as good as current state-of-the art normalization methods.

Texture Segmentation: An Objective Comparison between Five Traditional Algorithms and a Deep-Learning U-Net Architecture

This paper compares a series of traditional and deep learning methodologies for the segmentation of textures. Six well-known texture composites first published by Randen and Husøy were used to compare traditional segmentation techniques (co-occurrence, filtering, local binary patterns, watershed, multiresolution sub-band filtering) against a deep-learning approach based on the U-Net architecture. For the latter, the effects of depth of the network, number of epochs and different optimisation algorithms were investigated. Overall, the best results were provided by the deep-learning approach. However, the best results were distributed within the parameters, and many configurations provided results well below the traditional techniques.

Semi-automated analysis of digital whole slides from humanized lung-cancer xenograft models for checkpoint inhibitor response prediction

We propose a deep learning workflow for the classification of hematoxylin and eosin stained histological whole-slide images of non-small-cell lung cancer. The workflow includes automatic extraction of meta-features for the characterization of the tumor. We show that the tissue-classification produces state-of-the-art results with an average F1-score of 83%. Manual supervision indicates that experts, in practice, accept a far higher percentage of predictions. Furthermore, the extracted meta-features are validated via visualization revealing relevant biomedical relations between the different tissue classes. In a hypothetical decision-support scenario, these meta-features can be used to discriminate the tumor response with regard to available treatment options with an estimated accuracy of 84%. This workflow supports large-scale analysis of tissue obtained in preclinical animal experiments, enables reproducible quantification of tissue classes and immune system markers, and paves the way towards discovery of novel features predicting response in translational immune-oncology research.

Aryl hydrocarbon receptor nuclear translocator-like (ARNTL/BMAL1) is associated with bevacizumab resistance in colorectal cancer via regulation of vascular endothelial growth factor A

BACKGROUND

The identification of new biomarkers and the development of novel, targetable contexts of vulnerability are of urgent clinical need in drug-resistant metastatic colorectal cancer (mCRC). Aryl-Hydrocarbon-Receptor-Nuclear-Translocator-Like (ARNTL/BMAL1) is a circadian clock-regulated transcription factor promoting expression of genes involved in angiogenesis and tumour progression. We hypothesised that BMAL1 increases expression of the vascular endothelial growth factor A VEGFA gene and, thereby, confers resistance to anti-angiogenic therapy with bevacizumab (Beva), a clinically used antibody for neutralization of VEGFA.

METHODS

PCR and immunohistochemistry were employed to assess BMAL1 expression in mice (C57BL/6 J Apcmin/+; BALB/c nu/nu xenografts) and CRC patients under combination chemotherapy with Beva. BMAL1 single nucleotide gene polymorphisms (SNPs) were analysed by DNA-microarray in clinical samples. BMAL1 functions were studied in human CRC cell lines using colorimetric growth, DNA-binding and reporter assays.

FINDINGS

In murine CRCs, high BMAL1 expression correlated with poor preclinical response to Beva treatment. In CRC patients' tumours (n = 74), high BMAL1 expression was associated with clinical non-response to combination chemotherapy with Beva (*p = .0061) and reduced progression-free survival (PFS) [*p = .0223, Hazard Ratio (HR) = 1.69]. BMAL1 SNPs also correlated with shorter PFS (rs7396943, rs7938307, rs2279287) and overall survival (OS) [rs11022780, *p = .014, HR = 1.61]. Mechanistically, Nuclear-Receptor-Subfamily-1-Group-D-Member-1 (NR1D1/REVERBA) bound a - 672 bp Retinoic-Acid-Receptor-Related-Orphan-Receptor-Alpha-responsive-element (RORE) adjacent to a BMAL1 DNA-binding motif (E-box) in the VEGFA gene promoter, resulting in increased VEGFA synthesis and proliferation of human CRC cell lines.

INTERPRETATION

BMAL1 was associated with Beva resistance in CRC. Inhibition of REVERBA-BMAL1 signalling may prevent resistance to anti-angiogenic therapy. FUND: This work was in part supported by the European Commission Seventh Framework Programme (Contract No. 278981 [ANGIOPREDICT]).

Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer

Microsatellite instability determines whether patients with gastrointestinal cancer respond exceptionally well to immunotherapy. However, in clinical practice, not every patient is tested for MSI, because this requires additional genetic or immunohistochemical tests. Here we show that deep residual learning can predict MSI directly from H&E histology, which is ubiquitously available. This approach has the potential to provide immunotherapy to a much broader subset of patients with gastrointestinal cancer.

Application of Gray Level Co-Occurrence Matrix Analysis as a New Method for Enzyme Histochemistry Quantification

Enzyme histochemistry is a valuable histological method which provides a connection between morphology, activity, and spatial localization of investigated enzymes. Even though the method relies purely on arbitrary evaluations performed by the human eye, it is still wildly accepted and used in histo(patho)logy. Texture analysis emerged as an excellent tool for image quantification of subtle differences reflected in both spatial discrepancies and gray level values of pixels. The current study of texture analysis utilizes the gray-level co-occurrence matrix as a method for quantification of differences between ecto-5'-nucleotidase activities in healthy hippocampal tissue and tissue with marked neurodegeneration. We used the angular second moment, contrast (CON), correlation, inverse difference moment (INV), and entropy for texture analysis and receiver operating characteristic analysis with immunoblot and qualitative assessment of enzyme histochemistry as a validation. Our results strongly argue that co-occurrence matrix analysis could be used for the determination of fine differences in the enzyme activities with the possibility to ascribe those differences to regions or specific cell types. In addition, it emerged that INV and CON are especially useful parameters for this type of enzyme histochemistry analysis. We concluded that texture analysis is a reliable method for quantification of this descriptive technique, thus removing biases and adding it a quantitative dimension.

Signature maps for automatic identification of prostate cancer from colorimetric analysis of H&E- and IHC-stained histopathological specimens

Prostate cancer (PCa) is a major cause of cancer death among men. The histopathological examination of post-surgical prostate specimens and manual annotation of PCa not only allow for detailed assessment of disease characteristics and extent, but also supply the ground truth for developing of computer-aided diagnosis (CAD) systems for PCa detection before definitive treatment. As manual cancer annotation is tedious and subjective, there have been a number of publications describing methods for automating the procedure via the analysis of digitized whole-slide images (WSIs). However, these studies have focused only on the analysis of WSIs stained with hematoxylin and eosin (H&E), even though there is additional information that could be obtained from immunohistochemical (IHC) staining. In this work, we propose a framework for automating the annotation of PCa that is based on automated colorimetric analysis of both H&E and IHC WSIs stained with a triple-antibody cocktail against high-molecular weight cytokeratin (HMWCK), p63, and α-methylacyl CoA racemase (AMACR). The analysis outputs were then used to train a regression model to estimate the distribution of cancerous epithelium within slides. The approach yielded an AUC of 0.951, sensitivity of 87.1%, and specificity of 90.7% as compared to slide-level annotations, and generalized well to cancers of all grades.

Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features

Tumor segmentation in whole-slide images of histology slides is an important step towards computer-assisted diagnosis. In this work, we propose a tumor segmentation framework based on the novel concept of persistent homology profiles (PHPs). For a given image patch, the homology profiles are derived by efficient computation of persistent homology, which is an algebraic tool from homology theory. We propose an efficient way of computing topological persistence of an image, alternative to simplicial homology. The PHPs are devised to distinguish tumor regions from their normal counterparts by modeling the atypical characteristics of tumor nuclei. We propose two variants of our method for tumor segmentation: one that targets speed without compromising accuracy and the other that targets higher accuracy. The fast version is based on a selection of exemplar image patches from a convolution neural network (CNN) and patch classification by quantifying the divergence between the PHPs of exemplars and the input image patch. Detailed comparative evaluation shows that the proposed algorithm is significantly faster than competing algorithms while achieving comparable results. The accurate version combines the PHPs and high-level CNN features and employs a multi-stage ensemble strategy for image patch labeling. Experimental results demonstrate that the combination of PHPs and CNN features outperform competing algorithms. This study is performed on two independently collected colorectal datasets containing adenoma, adenocarcinoma, signet, and healthy cases. Collectively, the accurate tumor segmentation produces the highest average patch-level F1-score, as compared with competing algorithms, on malignant and healthy cases from both the datasets. Overall the proposed framework highlights the utility of persistent homology for histopathology image analysis.

Dealing with Lack of Training Data for Convolutional Neural Networks: The Case of Digital Pathology

Thanks to their capability to learn generalizable descriptors directly from images, deep Convolutional Neural Networks (CNNs) seem the ideal solution to most pattern recognition problems. On the other hand, to learn the image representation, CNNs need huge sets of annotated samples that are unfeasible in many every-day scenarios. This is the case, for example, of Computer-Aided Diagnosis (CAD) systems for digital pathology, where additional challenges are posed by the high variability of the cancerous tissue characteristics. In our experiments, state-of-the-art CNNs trained from scratch on histological images were less accurate and less robust to variability than a traditional machine learning framework, highlighting all the issues of fully training deep networks with limited data from real patients. To solve this problem, we designed and compared three transfer learning frameworks, leveraging CNNs pre-trained on non-medical images. This approach obtained very high accuracy, requiring much less computational resource for the training. Our findings demonstrate that transfer learning is a solution to the automated classification of histological samples and solves the problem of designing accurate and computationally-efficient CAD systems with limited training data.

Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

BACKGROUND

For virtually every patient with colorectal cancer (CRC), hematoxylin-eosin (HE)-stained tissue slides are available. These images contain quantitative information, which is not routinely used to objectively extract prognostic biomarkers. In the present study, we investigated whether deep convolutional neural networks (CNNs) can extract prognosticators directly from these widely available images.

METHODS AND FINDINGS

We hand-delineated single-tissue regions in 86 CRC tissue slides, yielding more than 100,000 HE image patches, and used these to train a CNN by transfer learning, reaching a nine-class accuracy of >94% in an independent data set of 7,180 images from 25 CRC patients. With this tool, we performed automated tissue decomposition of representative multitissue HE images from 862 HE slides in 500 stage I-IV CRC patients in the The Cancer Genome Atlas (TCGA) cohort, a large international multicenter collection of CRC tissue. Based on the output neuron activations in the CNN, we calculated a "deep stroma score," which was an independent prognostic factor for overall survival (OS) in a multivariable Cox proportional hazard model (hazard ratio [HR] with 95% confidence interval [CI]: 1.99 [1.27-3.12], p = 0.0028), while in the same cohort, manual quantification of stromal areas and a gene expression signature of cancer-associated fibroblasts (CAFs) were only prognostic in specific tumor stages. We validated these findings in an independent cohort of 409 stage I-IV CRC patients from the "Darmkrebs: Chancen der Verhütung durch Screening" (DACHS) study who were recruited between 2003 and 2007 in multiple institutions in Germany. Again, the score was an independent prognostic factor for OS (HR 1.63 [1.14-2.33], p = 0.008), CRC-specific OS (HR 2.29 [1.5-3.48], p = 0.0004), and relapse-free survival (RFS; HR 1.92 [1.34-2.76], p = 0.0004). A prospective validation is required before this biomarker can be implemented in clinical workflows.

CONCLUSIONS

In our retrospective study, we show that a CNN can assess the human tumor microenvironment and predict prognosis directly from histopathological images.

Multidimensional and fuzzy sample entropy (SampEnMF) for quantifying H&E histological images of colorectal cancer

In this study, we propose to use a method based on the combination of sample entropy with multiscale and multidimensional approaches, along with a fuzzy function. The model was applied to quantify and classify H&E histological images of colorectal cancer. The multiscale approach was defined by analysing windows of different sizes and variations in tolerance for determining pattern similarity. The multidimensional strategy was performed by considering each pixel in the colour image as an n-dimensional vector, which was analysed from the Minkowski distance. The fuzzy strategy was a Gaussian function used to verify the pertinence of the distances between windows. The result was a method capable of computing similarities between pixels contained in windows of various sizes, as well as the information present in the colour channels. The power of quantification was tested in a public colorectal image dataset, which was composed of both benign and malignant classes. The results were given as inputs for classifiers of different categories and analysed by applying the k-fold cross-validation and holdout methods. The derived performances indicate that the proposed association was capable of distinguishing the benign and malignant groups, with values that surpassed those results obtained with important techniques available in the Literature. The best performance was an AUC value of 0.983, an important result, mainly when we consider the difficulties of clinical practice for the diagnosis of the colorectal cancer.

An Analysis in Tissue Classification for Colorectal Cancer Histology Using Convolution Neural Network and Colour Models

Computer vision-based identification of different tissue categories in histological images is a critical application of the computer-assisted diagnosis (CAD). Computer-assisted diagnosis systems support to reduce the cost and increase the efficiency of this process. Traditional image classification approaches depend on feature extraction methods designed for a specific problem based on domain information. Deep learning approaches are becoming important alternatives with advance of machine learning technologies to overcome the numerous difficulties of the feature-based approaches. A method for the classification of histological images of human colorectal cancer containing seven different types of tissue using convolutional neural network (CNN) is proposed in this article. The method is evaluated using four different colour models in absence and presence of Gaussian noise. The highest classification accuracies are achieved with HVI colour model, which is 95.8% in nonexistence and 78.5% in existence of noise respectively.

Topography of cancer-associated immune cells in human solid tumors

Lymphoid and myeloid cells are abundant in the tumor microenvironment, can be quantified by immunohistochemistry and shape the disease course of human solid tumors. Yet, there is no comprehensive understanding of spatial immune infiltration patterns (‘topography’) across cancer entities and across various immune cell types. In this study, we systematically measure the topography of multiple immune cell types in 965 histological tissue slides from N = 177 patients in a pan-cancer cohort. We provide a definition of inflamed (‘hot’), non-inflamed (‘cold’) and immune excluded patterns and investigate how these patterns differ between immune cell types and between cancer types. In an independent cohort of N = 287 colorectal cancer patients, we show that hot, cold and excluded topographies for effector lymphocytes (CD8) and tumor-associated macrophages (CD163) alone are not prognostic, but that a bivariate classification system can stratify patients. Our study adds evidence to consider immune topographies as biomarkers for patients with solid tumors.

Segmentation of glandular epithelium in colorectal tumours to automatically compartmentalise IHC biomarker quantification: A deep learning approach

In this paper, we propose a method for automatically annotating slide images from colorectal tissue samples. Our objective is to segment glandular epithelium in histological images from tissue slides submitted to different staining techniques, including usual haematoxylin-eosin (H&E) as well as immunohistochemistry (IHC). The proposed method makes use of Deep Learning and is based on a new convolutional network architecture. Our method achieves better performances than the state of the art on the H&E images of the GlaS challenge contest, whereas it uses only the haematoxylin colour channel extracted by colour deconvolution from the RGB images in order to extend its applicability to IHC. The network only needs to be fine-tuned on a small number of additional examples to be accurate on a new IHC dataset. Our approach also includes a new method of data augmentation to achieve good generalisation when working with different experimental conditions and different IHC markers. We show that our methodology enables to automate the compartmentalisation of the IHC biomarker analysis, results concurring highly with manual annotations.

Gastric Pathology Image Classification Using Stepwise Fine-Tuning for Deep Neural Networks

Deep learning using convolutional neural networks (CNNs) is a distinguished tool for many image classification tasks. Due to its outstanding robustness and generalization, it is also expected to play a key role to facilitate advanced computer-aided diagnosis (CAD) for pathology images. However, the shortage of well-annotated pathology image data for training deep neural networks has become a major issue at present because of the high-cost annotation upon pathologist's professional observation. Faced with this problem, transfer learning techniques are generally used to reinforcing the capacity of deep neural networks. In order to further boost the performance of the state-of-the-art deep neural networks and alleviate insufficiency of well-annotated data, this paper presents a novel stepwise fine-tuning-based deep learning scheme for gastric pathology image classification and establishes a new type of target-correlative intermediate datasets. Our proposed scheme is deemed capable of making the deep neural network imitating the pathologist's perception manner and of acquiring pathology-related knowledge in advance, but with very limited extra cost in data annotation. The experiments are conducted with both well-annotated gastric pathology data and the proposed target-correlative intermediate data on several state-of-the-art deep neural networks. The results congruously demonstrate the feasibility and superiority of our proposed scheme for boosting the classification performance.

Histopathological image classification with bilinear convolutional neural networks

The computer-aided quantitative analysis for histopathological images has attracted considerable attention. The stain decomposition on histopathological images is usually recommended to address the issue of co-localization or aliasing of tissue substances. Although the convolutional neural networks (CNN) is a popular deep learning algorithm for various tasks on histopathological image analysis, it is only directly performed on histopathological images without considering stain decomposition. The bilinear CNN (BCNN) is a new CNN model for fine-grained classification. BCNN consists of two CNNs, whose convolutional-layer outputs are multiplied with outer product at each spatial location. In this work, we propose a novel BCNN-based method for classification of histopathological images, which first decomposes histopathological images into hematoxylin and eosin stain components, and then perform BCNN on the decomposed images to fuse and improve the feature representation performance. The experimental results on the colorectal cancer histopathological image dataset with eight classes indicate that the proposed BCNN-based algorithm is superior to the traditional CNN.

Using spectral imaging for the analysis of abnormalities for colorectal cancer: When is it helpful?

The spectral imaging technique has been shown to provide more discriminative information than the RGB images and has been proposed for a range of problems. There are many studies demonstrating its potential for the analysis of histopathology images for abnormality detection but there have been discrepancies among previous studies as well. Many multispectral based methods have been proposed for histopathology images but the significance of the use of whole multispectral cube versus a subset of bands or a single band is still arguable. We performed comprehensive analysis using individual bands and different subsets of bands to determine the effectiveness of spectral information for determining the anomaly in colorectal images. Our multispectral colorectal dataset consists of four classes, each represented by infra-red spectrum bands in addition to the visual spectrum bands. We performed our analysis of spectral imaging by stratifying the abnormalities using both spatial and spectral information. For our experiments, we used a combination of texture descriptors with an ensemble classification approach that performed best on our dataset. We applied our method to another dataset and got comparable results with those obtained using the state-of-the-art method and convolutional neural network based method. Moreover, we explored the relationship of the number of bands with the problem complexity and found that higher number of bands is required for a complex task to achieve improved performance. Our results demonstrate a synergy between infra-red and visual spectrum by improving the classification accuracy (by 6%) on incorporating the infra-red representation. We also highlight the importance of how the dataset should be divided into training and testing set for evaluating the histopathology image-based approaches, which has not been considered in previous studies on multispectral histopathology images.

Multiclass classification for skin cancer profiling based on the integration of heterogeneous gene expression series

Most of the research studies developed applying microarray technology to the characterization of different pathological states of any disease may fail in reaching statistically significant results. This is largely due to the small repertoire of analysed samples, and to the limitation in the number of states or pathologies usually addressed. Moreover, the influence of potential deviations on the gene expression quantification is usually disregarded. In spite of the continuous changes in omic sciences, reflected for instance in the emergence of new Next-Generation Sequencing-related technologies, the existing availability of a vast amount of gene expression microarray datasets should be properly exploited. Therefore, this work proposes a novel methodological approach involving the integration of several heterogeneous skin cancer series, and a later multiclass classifier design. This approach is thus a way to provide the clinicians with an intelligent diagnosis support tool based on the use of a robust set of selected biomarkers, which simultaneously distinguishes among different cancer-related skin states. To achieve this, a multi-platform combination of microarray datasets from Affymetrix and Illumina manufacturers was carried out. This integration is expected to strengthen the statistical robustness of the study as well as the finding of highly-reliable skin cancer biomarkers. Specifically, the designed operation pipeline has allowed the identification of a small subset of 17 differentially expressed genes (DEGs) from which to distinguish among 7 involved skin states. These genes were obtained from the assessment of a number of potential batch effects on the gene expression data. The biological interpretation of these genes was inspected in the specific literature to understand their underlying information in relation to skin cancer. Finally, in order to assess their possible effectiveness in cancer diagnosis, a cross-validation Support Vector Machines (SVM)-based classification including feature ranking was performed. The accuracy attained exceeded the 92% in overall recognition of the 7 different cancer-related skin states. The proposed integration scheme is expected to allow the co-integration with other state-of-the-art technologies such as RNA-seq.

Bioimage Classification with Handcrafted and Learned Features

Bioimage classification is increasingly becoming more important in many biological studies including those that require accurate cell phenotype recognition, subcellular localization, and histopathological classification. In this paper, we present a new General Purpose (GenP) bioimage classification method that can be applied to a large range of classification problems. The GenP system we propose is an ensemble that combines multiple texture features (both handcrafted and learned descriptors) for superior and generalizable discriminative power. Our ensemble obtains a boosting of performance by combining local features, dense sampling features, and deep learning features. Each descriptor is used to train a different Support Vector Machine that is then combined by sum rule. We evaluate our method on a diverse set of bioimage classification tasks each represented by a benchmark database, including some of those available in the IICBU 2008 database. Each bioimage classification task represents a typical subcellular, cellular, and tissue level classification problem. Our evaluation on these datasets demonstrates that the proposed GenP bioimage ensemble obtains state-of-the-art performance without any ad-hoc dataset tuning of the parameters (thereby avoiding any risk of overfitting/overtraining). To reproduce the experiments reported in this paper, the MATLAB code of all the descriptors is available at https://github.com/LorisNanni and https://www.dropbox.com/s/bguw035yrqz0pwp/ElencoCode.docx?dl=0.

Genomics and emerging biomarkers for immunotherapy of colorectal cancer

Colorectal cancer (CRC) is a common and lethal disease with a high therapeutic need. For most patients with metastatic CRC, chemotherapy is the only viable option. Currently, immunotherapy is restricted to the particular genetic subgroup of mismatch-repair deficient (MMRd)/microsatellite instable (MSI) CRC. Anti-PD1 therapy was recently FDA-approved as a second-line treatment in this subgroup. However, in a metastatic setting, these MMRd/MSI tumors are vastly outnumbered by mismatch-repair proficient (MMRp)/microsatellite stable (MSS) tumors. These MMRp/MSS tumors do not meaningfully respond to any traditional immunotherapy approach including checkpoint blockade, adoptive cell transfer and vaccination. This resistance to immunotherapy is due to a complex tumor microenvironment that counteracts antitumor immunity through a combination of poorly antigenic tumor cells and an immunosuppressive tumor microenvironment. To find ways of overcoming immunotherapy resistance in the majority of CRC patients, it is necessary to analyze the immunological makeup in an in-depth and personalized way and in the context of their tumor genetic makeup. Flexible, biomarker-guided early-phase immunotherapy trials are needed to optimize this workflow. In this review, we detail key mechanisms for immune evasion and emerging immune biomarkers for personalized immunotherapy in CRC. Also, we present a template for biomarker-guided clinical trials that are needed to move new immunotherapy approaches closer to clinical application.

Machine Learning Methods for Histopathological Image Analysis

Abundant accumulation of digital histopathological images has led to the increased demand for their analysis, such as computer-aided diagnosis using machine learning techniques. However, digital pathological images and related tasks have some issues to be considered. In this mini-review, we introduce the application of digital pathological image analysis using machine learning algorithms, address some problems specific to such analysis, and propose possible solutions.

SDL: Saliency-Based Dictionary Learning Framework for Image Similarity

In image classification, obtaining adequate data to learn a robust classifier has often proven to be difficult in several scenarios. Classification of histological tissue images for health care analysis is a notable application in this context due to the necessity of surgery, biopsy or autopsy. To adequately exploit limited training data in classification, we propose a saliency guided dictionary learning method and subsequently an image similarity technique for histo-pathological image classification. Salient object detection from images aids in the identification of discriminative image features. We leverage the saliency values for the local image regions to learn a dictionary and respective sparse codes for an image, such that the more salient features are reconstructed with smaller error. The dictionary learned from an image gives a compact representation of the image itself and is capable of representing images with similar content, with comparable sparse codes. We employ this idea to design a similarity measure between a pair of images, where local image features of one image, are encoded with the dictionary learned from the other and vice versa. To effectively utilize the learned dictionary, we take into account the contribution of each dictionary atom in the sparse codes to generate a global image representation for image comparison. The efficacy of the proposed method was evaluated using three tissue data sets that consist of mammalian kidney, lung and spleen tissue, breast cancer, and colon cancer tissue images. From the experiments, we observe that our methods outperform the state of the art with an increase of 14.2% in the average classification accuracy over all data sets.