Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images

Pathologists are responsible for cancer type diagnoses from histopathological cancer tissues. However, it is known that microscopic examination is tedious and time-consuming. In recent years, a long list of machine learning approaches to image classification and whole-slide segmentation has been developed to support pathologists. Although many showed exceptional performances, the majority of them are not able to rationalize their decisions. In this study, we developed an explainable classifier to support decision making for medical diagnoses. The proposed model does not provide an explanation about the causality between the input and the decisions, but offers a human-friendly explanation about the plausibility of the decision. Cumulative Fuzzy Class Membership Criterion (CFCMC) explains its decisions in three ways: through a semantical explanation about the possibilities of misclassification, showing the training sample responsible for a certain prediction and showing training samples from conflicting classes. In this paper, we explain about the mathematical structure of the classifier, which is not designed to be used as a fully automated diagnosis tool but as a support system for medical experts. We also report on the accuracy of the classifier against real world histopathological data for colorectal cancer. We also tested the acceptability of the system through clinical trials by 14 pathologists. We show that the proposed classifier is comparable to state of the art neural networks in accuracy, but more importantly it is more acceptable to be used by human experts as a diagnosis tool in the medical domain.

Artificial Intelligence and Mechanistic Modeling for Clinical Decision Making in Oncology

The amount of "big" data generated in clinical oncology, whether from molecular, imaging, pharmacological, or biological origin, brings novel challenges. To mine efficiently this source of information, mathematical models able to produce predictive algorithms and simulations are required, with applications for diagnosis, prognosis, drug development, or prediction of the response to therapy. Such mathematical and computational constructs can be subdivided into two broad classes: biologically agnostic, statistical models using artificial intelligence techniques, and physiologically based, mechanistic models. In this review, recent advances in the applications of such methods in clinical oncology are outlined. These include machine learning applied to big data (omics, imaging, or electronic health records), pharmacometrics and quantitative systems pharmacology, as well as tumor kinetics and metastasis modeling. Focus is set on studies with high potential of clinical translation, and particular attention is given to cancer immunotherapy. Perspectives are given in terms of combinations of the two approaches: "mechanistic learning."

Multiparametric MRI and Whole Slide Image-Based Pretreatment Prediction of Pathological Response to Neoadjuvant Chemoradiotherapy in Rectal Cancer: A Multicenter Radiopathomic Study

Background

The aim of this work is to combine radiological and pathological information of tumor to develop a signature for pretreatment prediction of discrepancies of pathological response at several centers and restage patients with locally advanced rectal cancer (LARC) for individualized treatment planning.

Patients and Methods

A total of 981 consecutive patients with evaluation of response according to tumor regression grade (TRG) who received nCRT were retrospectively recruited from four hospitals (primary cohort and external validation cohort 1–3); both pretreatment multiparametric MRI (mp-MRI) and whole slide image (WSI) of biopsy specimens were available for each patient. Quantitative image features were extracted from mp-MRI and WSI and used to construct a radiopathomics signature (RPS) powered by an artificial-intelligence model. Models based on mp-MRI or WSI alone were also constructed for comparison.

Results

The RPS showed overall accuracy of 79.66–87.66% in validation cohorts. The areas under the curve of RPS at specific response grades were 0.98 (TRG0), 0.93 (≤ TRG1), and 0.84 (≤ TRG2). RPS at each grade of pathological response revealed significant improvement compared with both signatures constructed without combining multiscale tumor information (P < 0.01). Moreover, RPS showed relevance to distinct probabilities of overall survival and disease-free survival in patients with LARC who underwent nCRT (P < 0.05).

Conclusions

The results of this study suggest that radiopathomics, combining both radiological information of the whole tumor and pathological information of local lesions from biopsy, could potentially predict discrepancies of pathological response prior to nCRT for better treatment planning.

Electronic supplementary material

The online version of this article (10.1245/s10434-020-08659-4) contains supplementary material, which is available to authorized users.

A hybrid unsupervised-Deep learning tandem for electrooculography time series analysis

Medical data are often tricky to get mined for patterns even by the generally demonstrated successful modern methodologies of deep learning. This paper puts forward such a medical classification task, where patient registers of two of the categories are sometimes hard to be distinguished because of samples showing characteristics of both labels in turn in several repetitions of the screening procedure. To this end, the current research appoints a pre-processing clustering step (through self-organizing maps) to group the data based on shape similarity and relabel it accordingly. Subsequently, a deep learning approach (a tandem of convolutional and long short-term memory networks) performs the training classification phase on the ‘cleaned’ samples. The dual methodology was applied for the computational diagnosis of electrooculography tests within spino-cerebral ataxia of type 2. The accuracy obtained for the discrimination into three classes was of 78.24%. The improvement that this duo brings over the deep learner alone does not stem from significantly higher accuracy results when the performance is considered for all classes. The major finding of this combination is that half of the presymptomatic cases were correctly found, in opposition to the single deep model, where this category was sacrificed by the learner in favor of a good accuracy overall. A high accuracy in general is desirable for any medical task, however the correct identification of cases before the symptoms become evident is more important.

Current Trends of Artificial Intelligence for Colorectal Cancer Pathology Image Analysis: A Systematic Review

Colorectal cancer (CRC) is one of the most common cancers requiring early pathologic diagnosis using colonoscopy biopsy samples. Recently, artificial intelligence (AI) has made significant progress and shown promising results in the field of medicine despite several limitations. We performed a systematic review of AI use in CRC pathology image analysis to visualize the state-of-the-art. Studies published between January 2000 and January 2020 were searched in major online databases including MEDLINE (PubMed, Cochrane Library, and EMBASE). Query terms included "colorectal neoplasm," "histology," and "artificial intelligence." Of 9000 identified studies, only 30 studies consisting of 40 models were selected for review. The algorithm features of the models were gland segmentation (n = 25, 62%), tumor classification (n = 8, 20%), tumor microenvironment characterization (n = 4, 10%), and prognosis prediction (n = 3, 8%). Only 20 gland segmentation models met the criteria for quantitative analysis, and the model proposed by Ding et al. (2019) performed the best. Studies with other features were in the elementary stage, although most showed impressive results. Overall, the state-of-the-art is promising for CRC pathological analysis. However, datasets in most studies had relatively limited scale and quality for clinical application of this technique. Future studies with larger datasets and high-quality annotations are required for routine practice-level validation.

A machine learning-based prognostic predictor for stage III colon cancer

Limited biomarkers have been identified as prognostic predictors for stage III colon cancer. To combat this shortfall, we developed a computer-aided approach which combing convolutional neural network with machine classifier to predict the prognosis of stage III colon cancer from routinely haematoxylin and eosin (H&E) stained tissue slides. We trained the model by using 101 cancers from West China Hospital (WCH). The predictive effectivity of the model was validated by using 67 cancers from WCH and 47 cancers from The Cancer Genome Atlas Colon Adenocarcinoma database. The selected model (Gradient Boosting-Colon) provided a hazard ratio (HR) for high- vs. low-risk recurrence of 8.976 (95% confidence interval (CI), 2.824–28.528; P, 0.000), and 10.273 (95% CI, 2.177–48.472; P, 0.003) in the two test groups, from the multivariate Cox proportional hazards analysis. It gave a HR value of 10.687(95% CI, 2.908–39.272; P, 0.001) and 5.033 (95% CI,1.792–14.132; P, 0.002) for the poor vs. good prognosis groups. Gradient Boosting-Colon is an independent machine prognostic predictor which allows stratification of stage III colon cancer into high- and low-risk recurrence groups, and poor and good prognosis groups directly from the H&E tissue slides. Our findings could provide crucial information to aid treatment planning during stage III colon cancer.

Deep learning-based survival prediction for multiple cancer types using histopathology images

Providing prognostic information at the time of cancer diagnosis has important implications for treatment and monitoring. Although cancer staging, histopathological assessment, molecular features, and clinical variables can provide useful prognostic insights, improving risk stratification remains an active research area. We developed a deep learning system (DLS) to predict disease specific survival across 10 cancer types from The Cancer Genome Atlas (TCGA). We used a weakly-supervised approach without pixel-level annotations, and tested three different survival loss functions. The DLS was developed using 9,086 slides from 3,664 cases and evaluated using 3,009 slides from 1,216 cases. In multivariable Cox regression analysis of the combined cohort including all 10 cancers, the DLS was significantly associated with disease specific survival (hazard ratio of 1.58, 95% CI 1.28–1.70, p<0.0001) after adjusting for cancer type, stage, age, and sex. In a per-cancer adjusted subanalysis, the DLS remained a significant predictor of survival in 5 of 10 cancer types. Compared to a baseline model including stage, age, and sex, the c-index of the model demonstrated an absolute 3.7% improvement (95% CI 1.0–6.5) in the combined cohort. Additionally, our models stratified patients within individual cancer stages, particularly stage II (p = 0.025) and stage III (p<0.001). By developing and evaluating prognostic models across multiple cancer types, this work represents one of the most comprehensive studies exploring the direct prediction of clinical outcomes using deep learning and histopathology images. Our analysis demonstrates the potential for this approach to provide significant prognostic information in multiple cancer types, and even within specific pathologic stages. However, given the relatively small number of cases and observed clinical events for a deep learning task of this type, we observed wide confidence intervals for model performance, thus highlighting that future work will benefit from larger datasets assembled for the purposes for survival modeling.

Deep learned tissue "fingerprints" classify breast cancers by ER/PR/Her2 status from H&E images

Because histologic types are subjective and difficult to reproduce between pathologists, tissue morphology often takes a back seat to molecular testing for the selection of breast cancer treatments. This work explores whether a deep-learning algorithm can learn objective histologic H&E features that predict the clinical subtypes of breast cancer, as assessed by immunostaining for estrogen, progesterone, and Her2 receptors (ER/PR/Her2). Translating deep learning to this and related problems in histopathology presents a challenge due to the lack of large, well-annotated data sets, which are typically required for the algorithms to learn statistically significant discriminatory patterns. To overcome this limitation, we introduce the concept of “tissue fingerprints,” which leverages large, unannotated datasets in a label-free manner to learn H&E features that can distinguish one patient from another. The hypothesis is that training the algorithm to learn the morphological differences between patients will implicitly teach it about the biologic variation between them. Following this training internship, we used the features the network learned, which we call “fingerprints,” to predict ER, PR, and Her2 status in two datasets. Despite the discovery dataset being relatively small by the standards of the machine learning community (n = 939), fingerprints enabled the determination of ER, PR, and Her2 status from whole slide H&E images with 0.89 AUC (ER), 0.81 AUC (PR), and 0.79 AUC (Her2) on a large, independent test set (n = 2531). Tissue fingerprints are concise but meaningful histopathologic image representations that capture biological information and may enable machine learning algorithms that go beyond the traditional ER/PR/Her2 clinical groupings by directly predicting theragnosis.

Deep Learning Predicts Molecular Subtype of Muscle-invasive Bladder Cancer from Conventional Histopathological Slides

BACKGROUND

Muscle-invasive bladder cancer (MIBC) is the second most common genitourinary malignancy, and is associated with high morbidity and mortality. Recently, molecular subtypes of MIBC have been identified, which have important clinical implications.

OBJECTIVE

In the current study, we tried to predict the molecular subtype of MIBC samples from conventional histomorphology alone using deep learning.

DESIGN, SETTING, AND PARTICIPANTS

Two cohorts of patients with MIBC were used: (1) The Cancer Genome Atlas Urothelial Bladder Carcinoma dataset including 407 patients and (2) our own cohort including 16 patients with treatment-naïve, primary resected MIBC. This resulted in a total of 423 digital whole slide images of tumor tissue to train, validate, and test the deep learning algorithm to predict the molecular subtype.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Various accuracy measurements including the area under the receiver operating characteristic curves were used to evaluate the deep learning model. A sliding window approach to visualize classification was used. Class activation maps were used to identify image features that are most relevant to call a specific class.

RESULTS AND LIMITATIONS

The deep learning model showed great performance in the prediction of the molecular subtype of MIBC patients from hematoxylin and eosin (HE) slides alone-similar to or better than pathology experts. Using different visualization techniques, we identified new histopathological features that were most relevant to our model.

CONCLUSIONS

Deep learning can be used to predict important molecular features in MIBC patients from HE slides alone, potentially improving the clinical management of this disease significantly.

PATIENT SUMMARY

In patients with bladder cancer, a computer program found changes in the appearance of tumor tissue under the microscope and used these to predict genetic alterations. This could potentially benefit patients.

Cellular community detection for tissue phenotyping in colorectal cancer histology images

Classification of various types of tissue in cancer histology images based on the cellular compositions is an important step towards the development of computational pathology tools for systematic digital profiling of the spatial tumor microenvironment. Most existing methods for tissue phenotyping are limited to the classification of tumor and stroma and require large amount of annotated histology images which are often not available. In the current work, we pose the problem of identifying distinct tissue phenotypes as finding communities in cellular graphs or networks. First, we train a deep neural network for cell detection and classification into five distinct cellular components. Considering the detected nuclei as nodes, potential cell-cell connections are assigned using Delaunay triangulation resulting in a cell-level graph. Based on this cell graph, a feature vector capturing potential cell-cell connection of different types of cells is computed. These feature vectors are used to construct a patch-level graph based on chi-square distance. We map patch-level nodes to the geometric space by representing each node as a vector of geodesic distances from other nodes in the network and iteratively drifting the patch nodes in the direction of positive density gradients towards maximum density regions. The proposed algorithm is evaluated on a publicly available dataset and another new large-scale dataset consisting of 280K patches of seven tissue phenotypes. The estimated communities have significant biological meanings as verified by the expert pathologists. A comparison with current state-of-the-art methods reveals significant performance improvement in tissue phenotyping.

Emerging role of deep learning-based artificial intelligence in tumor pathology

The development of digital pathology and progression of state-of-the-art algorithms for computer vision have led to increasing interest in the use of artificial intelligence (AI), especially deep learning (DL)-based AI, in tumor pathology. The DL-based algorithms have been developed to conduct all kinds of work involved in tumor pathology, including tumor diagnosis, subtyping, grading, staging, and prognostic prediction, as well as the identification of pathological features, biomarkers and genetic changes. The applications of AI in pathology not only contribute to improve diagnostic accuracy and objectivity but also reduce the workload of pathologists and subsequently enable them to spend additional time on high-level decision-making tasks. In addition, AI is useful for pathologists to meet the requirements of precision oncology. However, there are still some challenges relating to the implementation of AI, including the issues of algorithm validation and interpretability, computing systems, the unbelieving attitude of pathologists, clinicians and patients, as well as regulators and reimbursements. Herein, we present an overview on how AI-based approaches could be integrated into the workflow of pathologists and discuss the challenges and perspectives of the implementation of AI in tumor pathology.

Artificial intelligence-guided tissue analysis combined with immune infiltrate assessment predicts stage III colon cancer outcomes in PETACC08 study

OBJECTIVE

Diagnostic tests, such as Immunoscore, predict prognosis in patients with colon cancer. However, additional prognostic markers could be detected on pathological slides using artificial intelligence tools.

DESIGN

We have developed a software to detect colon tumour, healthy mucosa, stroma and immune cells on CD3 and CD8 stained slides. The lymphocyte density and surface area were quantified automatically in the tumour core (TC) and invasive margin (IM). Using a LASSO algorithm, DGMate (DiGital tuMor pArameTErs), we detected digital parameters within the tumour cells related to patient outcomes.

RESULTS

Within the dataset of 1018 patients, we observed that a poorer relapse-free survival (RFS) was associated with high IM stromal area (HR 5.65; 95% CI 2.34 to 13.67; p<0.0001) and high DGMate (HR 2.72; 95% CI 1.92 to 3.85; p<0.001). Higher CD3+ TC, CD3+ IM and CD8+ TC densities were significantly associated with a longer RFS. Analysis of variance showed that CD3+ TC yielded a similar prognostic value to the classical CD3/CD8 Immunoscore (p=0.44). A combination of the IM stromal area, DGMate and CD3, designated 'DGMuneS', outperformed Immunoscore when used in estimating patients' prognosis (C-index=0.601 vs 0.578, p=0.04) and was independently associated with patient outcomes following Cox multivariate analysis. A predictive nomogram based on DGMuneS and clinical variables identified a group of patients with less than 10% relapse risk and another group with a 50% relapse risk.

CONCLUSION

These findings suggest that artificial intelligence can potentially improve patient care by assisting pathologists in better defining stage III colon cancer patients' prognosis.

Molecular subtypes and precision treatment of triple-negative breast cancer

Triple-negative breast cancer (TNBC) is the most aggressive breast cancer subtype. Despite the progress made in precision treatment of cancer patients, targeted treatment is still at its early stage in TNBC, and chemotherapy remains the standard treatment. With the advances in next generation sequencing technology, genomic and transcriptomic analyses have provided deeper insight into the inter-tumoral heterogeneity of TNBC. Much effort has been made to classify TNBCs into different molecular subtypes according to genetic aberrations and expression signatures and to uncover novel treatment targets. In this review, we summarized the current knowledge regarding the molecular classification of TNBC and explore the future paradigm for using molecular classification to guide the development of precision treatment and clinical practice.

The Application of Deep Learning in Cancer Prognosis Prediction

Deep learning has been applied to many areas in health care, including imaging diagnosis, digital pathology, prediction of hospital admission, drug design, classification of cancer and stromal cells, doctor assistance, etc. Cancer prognosis is to estimate the fate of cancer, probabilities of cancer recurrence and progression, and to provide survival estimation to the patients. The accuracy of cancer prognosis prediction will greatly benefit clinical management of cancer patients. The improvement of biomedical translational research and the application of advanced statistical analysis and machine learning methods are the driving forces to improve cancer prognosis prediction. Recent years, there is a significant increase of computational power and rapid advancement in the technology of artificial intelligence, particularly in deep learning. In addition, the cost reduction in large scale next-generation sequencing, and the availability of such data through open source databases (e.g., TCGA and GEO databases) offer us opportunities to possibly build more powerful and accurate models to predict cancer prognosis more accurately. In this review, we reviewed the most recent published works that used deep learning to build models for cancer prognosis prediction. Deep learning has been suggested to be a more generic model, requires less data engineering, and achieves more accurate prediction when working with large amounts of data. The application of deep learning in cancer prognosis has been shown to be equivalent or better than current approaches, such as Cox-PH. With the burst of multi-omics data, including genomics data, transcriptomics data and clinical information in cancer studies, we believe that deep learning would potentially improve cancer prognosis.

Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours

Histopathological classification of gastric and colonic epithelial tumours is one of the routine pathological diagnosis tasks for pathologists. Computational pathology techniques based on Artificial intelligence (AI) would be of high benefit in easing the ever increasing workloads on pathologists, especially in regions that have shortages in access to pathological diagnosis services. In this study, we trained convolutional neural networks (CNNs) and recurrent neural networks (RNNs) on biopsy histopathology whole-slide images (WSIs) of stomach and colon. The models were trained to classify WSI into adenocarcinoma, adenoma, and non-neoplastic. We evaluated our models on three independent test sets each, achieving area under the curves (AUCs) up to 0.97 and 0.99 for gastric adenocarcinoma and adenoma, respectively, and 0.96 and 0.99 for colonic adenocarcinoma and adenoma respectively. The results demonstrate the generalisation ability of our models and the high promising potential of deployment in a practical histopathological diagnostic workflow system.

Development and Validation of a Deep Learning CT Signature to Predict Survival and Chemotherapy Benefit in Gastric Cancer: A Multicenter, Retrospective Study

OBJECTIVE

We aimed to develop a deep learning-based signature to predict prognosis and benefit from adjuvant chemotherapy using preoperative computed tomography (CT) images.

BACKGROUND

Current staging methods do not accurately predict the risk of disease relapse for patients with gastric cancer.

METHODS

We proposed a novel deep neural network (S-net) to construct a CT signature for predicting disease-free survival (DFS) and overall survival in a training cohort of 457 patients, and independently tested it in an external validation cohort of 1158 patients. An integrated nomogram was constructed to demonstrate the added value of the imaging signature to established clinicopathologic factors for individualized survival prediction. Prediction performance was assessed with respect to discrimination, calibration, and clinical usefulness.

RESULTS

The DeLIS was associated with DFS and overall survival in the overall validation cohort and among subgroups defined by clinicopathologic variables, and remained an independent prognostic factor in multivariable analysis (P< 0.001). Integrating the imaging signature and clinicopathologic factors improved prediction performance, with C-indices: 0.792-0.802 versus 0.719-0.724, and net reclassification improvement 10.1%-28.3%. Adjuvant chemotherapy was associated with improved DFS in stage II patients with high-DeLIS [hazard ratio = 0.362 (95% confidence interval 0.149-0.882)] and stage III patients with high- and intermediate-DeLIS [hazard ratio = 0.611 (0.442-0.843); 0.633 (0.433-0.925)]. On the other hand, adjuvant chemotherapy did not affect survival for patients with low-DeLIS, suggesting a predictive effect (Pinteraction = 0.048, 0.016 for DFS in stage II and III disease).

CONCLUSIONS

The proposed imaging signature improved prognostic prediction and could help identify patients most likely to benefit from adjuvant chemotherapy in gastric cancer.

Predicting primary site of secondary liver cancer with a neural estimator of metastatic origin

Purpose: Pathologists rely on relevant clinical information, visual inspection of stained tissue slide morphology, and sophisticated molecular diagnostics to accurately infer the biological origin of secondary metastatic cancer. While highly effective, this process is expensive in terms of time and clinical resources. We seek to develop and evaluate a computer vision system designed to reasonably infer metastatic origin of secondary liver cancer directly from digitized histopathological whole slide images of liver biopsy. Approach: We illustrate a two-stage deep learning approach to accomplish this task. We first train a model to identify spatially localized regions of cancerous tumor within digitized hematoxylin and eosin (H&E)-stained tissue sections of secondary liver cancer based on a pathologist's annotation of several whole slide images. Then, a second model is trained to generate predictions of the cancers' metastatic origin belonging to one of three distinct clinically relevant classes as confirmed by immunohistochemistry. Results: Our approach achieves a classification accuracy of 90.2% in determining metastatic origin of whole slide images from a held-out test set, which compares favorably to an established clinical benchmark by three board-certified pathologists whose accuracies ranged from 90.2% to 94.1% on the same prediction task. Conclusions: We illustrate the potential impact of deep learning systems to leverage morphological and structural features of H&E-stained tissue sections to guide pathological and clinical determination of the metastatic origin of secondary liver cancers.

Enhancement of Deep Learning in Image Classification Performance Using Xception with the Swish Activation Function for Colorectal Polyp Preliminary Screening

One of the leading forms of cancer is colorectal cancer (CRC), which is responsible for increasing mortality in young people. The aim of this paper is to provide an experimental modification of deep learning of Xception with Swish and assess the possibility of developing a preliminary colorectal polyp screening system by training the proposed model with a colorectal topogram dataset in two and three classes. The results indicate that the proposed model can enhance the original convolutional neural network model with evaluation classification performance by achieving accuracy of up to 98.99% for classifying into two classes and 91.48% for three classes. For testing of the model with another external image, the proposed method can also improve the prediction compared to the traditional method, with 99.63% accuracy for true prediction of two classes and 80.95% accuracy for true prediction of three classes.

Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology

In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and 'hand-crafted' feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

Connecting Histopathology Imaging and Proteomics in Kidney Cancer through Machine Learning

Proteomics data encode molecular features of diagnostic value and accurately reflect key underlying biological mechanisms in cancers. Histopathology imaging is a well-established clinical approach to cancer diagnosis. The predictive relationship between large-scale proteomics and H&E-stained histopathology images remains largely uncharacterized. Here we investigate such associations through the application of machine learning, including deep neural networks, to proteomics and histology imaging datasets generated by the Clinical Proteomic Tumor Analysis Consortium (CPTAC) from clear cell renal cell carcinoma patients. We report robust correlations between a set of diagnostic proteins and predictions generated by an imaging-based classification model. Proteins significantly correlated with the histology-based predictions are significantly implicated in immune responses, extracellular matrix reorganization, and metabolism. Moreover, we showed that the genes encoding these proteins also reliably recapitulate the biological associations with imaging-derived predictions based on strong gene–protein expression correlations. Our findings offer novel insights into the integrative modeling of histology and omics data through machine learning, as well as the methodological basis for new research opportunities in this and other cancer types.

Molecular and histological correlations in liver cancer

Hepatocellular carcinoma (HCC) is a highly heterogeneous cancer, both at the molecular and histological level. High-throughput sequencing and gene expression profiling have identified distinct transcriptomic subclasses and numerous recurrent genetic alterations; several HCC subtypes characterised by histological features have also been identified. HCC phenotype appears to be closely related to particular gene mutations, tumour subgroups and/or oncogenic pathways. Non-proliferative tumours display a well-differentiated phenotype. Among this molecular subgroup, CTNNB1-mutated HCCs constitute a homogeneous subtype, exhibiting cholestasis and microtrabecular and pseudoglandular architectural patterns. Another non-proliferative subtype has a gene expression pattern similar to that of mature hepatocytes (G4) and displays a steatohepatitic phenotype. In contrast, proliferative HCCs are most often poorly differentiated, and notably include tumours with progenitor features. A novel morphological variant of proliferative HCC - designated "macrotrabecular-massive" - was recently shown to be associated with angiogenesis activation and poor prognosis. Altogether, these findings may help to translate our knowledge of HCC biology into clinical practice, resulting in improved precision medicine for patients with this highly aggressive malignancy. This manuscript reviews the most recent data in this exciting field, discussing future directions and challenges.

Machine learning for tissue diagnostics in oncology: brave new world

Machine learning is an exciting technology with broad application in big data analysis, as well as increasingly in specialised healthcare. As a diagnostic tool in tissue workup and pathology, it has the potential for personalised and stratified approaches, but the limitations and pitfalls need to be better understood and characterised especially in this critical area of medical care.

Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer

Microsatellite instability determines whether patients with gastrointestinal cancer respond exceptionally well to immunotherapy. However, in clinical practice, not every patient is tested for MSI, because this requires additional genetic or immunohistochemical tests. Here we show that deep residual learning can predict MSI directly from H&E histology, which is ubiquitously available. This approach has the potential to provide immunotherapy to a much broader subset of patients with gastrointestinal cancer.