Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma

Artificial Intelligence Applications in Oral Cancer and Oral Dysplasia

Oral squamous cell carcinoma (OSCC) is a highly unpredictable disease with devastating mortality rates that have not changed over the past decades, in the face of advancements in treatments and biomarkers, which have improved survival for other cancers. Delays in diagnosis are frequent, leading to more disfiguring treatments and poor outcomes in patients. The clinical challenge lies in identifying those patients at highest risk for developing OSCC. Oral epithelial dysplasia (OED) is a precursor of OSCC with highly variable behavior across patients. There is no reliable clinical, pathologic, histologic or molecular biomarker to determine individual risk in OED patients. Similarly, there are no robust biomarkers to predict treatment outcomes or mortality of OSCC patients. This review aims to highlight advancements in artificial intelligence (AI)-based methods to develop predictive biomarkers of OED transformation to OSCC or predictive biomarkers of OSCC mortality and treatment response. Machine-learning based biomarkers, such as S100A7, demonstrate promising appraisal for the risk of malignant transformation of OED. Machine learning-enhanced multiplex immunohistochemistry (mIHC) workflows examine immune cell patterns and organization within the tumor immune microenvironment to generate outcome predictions in immunotherapy. Deep learning (DL) is an AI-based method using an extended neural network or related architecture with multiple "hidden" layers of simulated neurons to combine simple visual features into complex patterns. DL-based digital pathology is currently being developed to assess OED and OSCC outcomes. The integration of machine learning in epigenomics aims to examine the epigenetic modification of diseases and improve our ability to detect, classify, and predict outcomes associated with epigenetic marks. Collectively, these tools showcase promising advancements in discovery and technology, which may provide a potential solution to addressing the current limitations in predicting OED transformation and OSCC behavior, both of which are clinical challenges that must be addressed in order to improve OSCC survival.

Automated curation of large-scale cancer histopathology image datasets using deep learning

BACKGROUND

Artificial intelligence (AI) has numerous applications in pathology, supporting diagnosis and prognostication in cancer. However, most AI models are trained on highly selected data, typically one tissue slide per patient. In reality, especially for large surgical resection specimens, dozens of slides can be available for each patient. Manually sorting and labelling whole-slide images (WSIs) is a very time-consuming process, hindering the direct application of AI on the collected tissue samples from large cohorts. In this study we addressed this issue by developing a deep-learning (DL)-based method for automatic curation of large pathology datasets with several slides per patient.

METHODS

We collected multiple large multicentric datasets of colorectal cancer histopathological slides from the United Kingdom (FOXTROT, N = 21,384 slides; CR07, N = 7985 slides) and Germany (DACHS, N = 3606 slides). These datasets contained multiple types of tissue slides, including bowel resection specimens, endoscopic biopsies, lymph node resections, immunohistochemistry-stained slides, and tissue microarrays. We developed, trained, and tested a deep convolutional neural network model to predict the type of slide from the slide overview (thumbnail) image. The primary statistical endpoint was the macro-averaged area under the receiver operating curve (AUROCs) for detection of the type of slide.

RESULTS

In the primary dataset (FOXTROT), with an AUROC of 0.995 [95% confidence interval [CI]: 0.994-0.996] the algorithm achieved a high classification performance and was able to accurately predict the type of slide from the thumbnail image alone. In the two external test cohorts (CR07, DACHS) AUROCs of 0.982 [95% CI: 0.979-0.985] and 0.875 [95% CI: 0.864-0.887] were observed, which indicates the generalizability of the trained model on unseen datasets. With a confidence threshold of 0.95, the model reached an accuracy of 94.6% (7331 classified cases) in CR07 and 85.1% (2752 classified cases) for the DACHS cohort.

CONCLUSION

Our findings show that using the low-resolution thumbnail image is sufficient to accurately classify the type of slide in digital pathology. This can support researchers to make the vast resource of existing pathology archives accessible to modern AI models with only minimal manual annotations.

Histopathology Based AI Model Predicts Anti-Angiogenic Therapy Response in Renal Cancer Clinical Trial

Background

Predictive biomarkers of treatment response are lacking for metastatic clearcell renal cell carcinoma (ccRCC), a tumor type that is treated with angiogenesis inhibitors, immune checkpoint inhibitors, mTOR inhibitors and a HIF2 inhibitor. The Angioscore, an RNA-based quantification of angiogenesis, is arguably the best candidate to predict anti-angiogenic (AA) response. However, the clinical adoption of transcriptomic assays faces several challenges including standardization, time delay, and high cost. Further, ccRCC tumors are highly heterogenous, and sampling multiple areas for sequencing is impractical.

Approach

Here we present a novel deep learning (DL) approach to predict the Angioscore from ubiquitous histopathology slides. In order to overcome the lack of interpretability, one of the biggest limitations of typical DL models, our model produces a visual vascular network which is the basis of the model's prediction. To test its reliability, we applied this model to multiple cohorts including a clinical trial dataset.

Results

Our model accurately predicts the RNA-based Angioscore on multiple independent cohorts (spearman correlations of 0.77 and 0.73). Further, the predictions help unravel meaningful biology such as association of angiogenesis with grade, stage, and driver mutation status. Finally, we find our model is able to predict response to AA therapy, in both a real-world cohort and the IMmotion150 clinical trial. The predictive power of our model vastly exceeds that of CD31, a marker of vasculature, and nearly rivals the performance (c-index 0.66 vs 0.67) of the ground truth RNA-based Angioscore at a fraction of the cost.

Conclusion

By providing a robust yet interpretable prediction of the Angioscore from histopathology slides alone, our approach offers insights into angiogenesis biology and AA treatment response.

Histopathologic image-based deep learning classifier for predicting platinum-based treatment responses in high-grade serous ovarian cancer

Platinum-based chemotherapy is the cornerstone treatment for female high-grade serous ovarian carcinoma (HGSOC), but choosing an appropriate treatment for patients hinges on their responsiveness to it. Currently, no available biomarkers can promptly predict responses to platinum-based treatment. Therefore, we developed the Pathologic Risk Classifier for HGSOC (PathoRiCH), a histopathologic image-based classifier. PathoRiCH was trained on an in-house cohort (n = 394) and validated on two independent external cohorts (n = 284 and n = 136). The PathoRiCH-predicted favorable and poor response groups show significantly different platinum-free intervals in all three cohorts. Combining PathoRiCH with molecular biomarkers provides an even more powerful tool for the risk stratification of patients. The decisions of PathoRiCH are explained through visualization and a transcriptomic analysis, which bolster the reliability of our model's decisions. PathoRiCH exhibits better predictive performance than current molecular biomarkers. PathoRiCH will provide a solid foundation for developing an innovative tool to transform the current diagnostic pipeline for HGSOC.

WhARIO: whole-slide-image-based survival analysis for patients treated with immunotherapy

Purpose

Immune checkpoint inhibitors (ICIs) are now one of the standards of care for patients with lung cancer and have greatly improved both progression-free and overall survival, although < 20 % of the patients respond to the treatment, and some face acute adverse events. Although a few predictive biomarkers have integrated the clinical workflow, they require additional modalities on top of whole-slide images and lack efficiency or robustness. In this work, we propose a biomarker of immunotherapy outcome derived solely from the analysis of histology slides.

Approach

We develop a three-step framework, combining contrastive learning and nonparametric clustering to distinguish tissue patterns within the slides, before exploiting the adjacencies of previously defined regions to derive features and train a proportional hazards model for survival analysis. We test our approach on an in-house dataset of 193 patients from 5 medical centers and compare it with the gold standard tumor proportion score (TPS) biomarker.

Results

On a fivefold cross-validation (CV) of the entire dataset, the whole-slide image-based survival analysis for patients treated with immunotherapy (WhARIO) features are able to separate a low- and a high-risk group of patients with a hazard ratio (HR) of 2.29 (CI 95 = 1.48 to 3.56), whereas the TPS 1% reference threshold only reaches a HR of 1.81 (CI 95 = 1.21 to 2.69). Combining the two yields a higher HR of 2.60 (CI 95 = 1.72 to 3.94). Additional experiments on the same dataset, where one out of five centers is excluded from the CV and used as a test set, confirm these trends.

Conclusions

Our uniquely designed WhARIO features are an efficient predictor of survival for lung cancer patients who received ICI treatment. We achieve similar performance to the current gold standard biomarker, without the need to access other imaging modalities, and show that both can be used together to reach even better results.

Semi-Supervised, Attention-Based Deep Learning for Predicting TMPRSS2:ERG Fusion Status in Prostate Cancer Using Whole Slide Images

IMPLICATIONS

Our study illuminates the potential of deep learning in effectively inferring key prostate cancer genetic alterations from the tissue morphology depicted in routinely available histology slides, offering a cost-effective method that could revolutionize diagnostic strategies in oncology.

The use of weakly supervised machine learning for necrosis assessment in patients with osteosarcoma: A pilot study

Percent necrosis (PN) following chemotherapy is a prognostic factor for survival in osteosarcoma. Pathologists estimate PN by calculating tumor viability over an average of whole-slide images (WSIs). This non-standardized, labor-intensive process requires specialized training and has high interobserver variability. Therefore, we aimed to develop a machine-learning model capable of calculating PN in osteosarcoma with similar accuracy to that of a musculoskeletal pathologist. In this proof-of-concept study, we retrospectively obtained six WSIs from two patients with conventional osteosarcomas. A weakly supervised learning model was trained by using coarse and incomplete annotations of viable tumor, necrotic tumor, and nontumor tissue in WSIs. Weakly supervised learning refers to processes capable of creating predictive models on the basis of partially and imprecisely annotated data. Once "trained," the model segmented areas of tissue and determined PN of the same six WSIs. To assess model fidelity, the pathologist also estimated PN of each WSI, and we compared the estimates using Pearson's correlation and mean absolute error (MAE). MAE was 15% over the six samples, and 6.4% when an outlier was removed, for which the model inaccurately labeled cartilaginous tissue. The model and pathologist estimates were strongly, positively correlated (r = 0.85). Thus, we created and trained a weakly supervised machine learning model to segment viable tumor, necrotic tumor, and nontumor and to calculate PN with accuracy similar to that of a musculoskeletal pathologist. We expect improvement can be achieved by annotating cartilaginous and other mesenchymal tissue for better representation of the histological heterogeneity in osteosarcoma.

Artificial intelligence for predictive biomarker discovery in immuno-oncology: a systematic review

BACKGROUND

The widespread use of immune checkpoint inhibitors (ICIs) has revolutionised treatment of multiple cancer types. However, selecting patients who may benefit from ICI remains challenging. Artificial intelligence (AI) approaches allow exploitation of high-dimension oncological data in research and development of precision immuno-oncology.

MATERIALS AND METHODS

We conducted a systematic literature review of peer-reviewed original articles studying the ICI efficacy prediction in cancer patients across five data modalities: genomics (including genomics, transcriptomics, and epigenomics), radiomics, digital pathology (pathomics), and real-world and multimodality data.

RESULTS

A total of 90 studies were included in this systematic review, with 80% published in 2021-2022. Among them, 37 studies included genomic, 20 radiomic, 8 pathomic, 20 real-world, and 5 multimodal data. Standard machine learning (ML) methods were used in 72% of studies, deep learning (DL) methods in 22%, and both in 6%. The most frequently studied cancer type was non-small-cell lung cancer (36%), followed by melanoma (16%), while 25% included pan-cancer studies. No prospective study design incorporated AI-based methodologies from the outset; rather, all implemented AI as a post hoc analysis. Novel biomarkers for ICI in radiomics and pathomics were identified using AI approaches, and molecular biomarkers have expanded past genomics into transcriptomics and epigenomics. Finally, complex algorithms and new types of AI-based markers, such as meta-biomarkers, are emerging by integrating multimodal/multi-omics data.

CONCLUSION

AI-based methods have expanded the horizon for biomarker discovery, demonstrating the power of integrating multimodal data from existing datasets to discover new meta-biomarkers. While most of the included studies showed promise for AI-based prediction of benefit from immunotherapy, none provided high-level evidence for immediate practice change. A priori planned prospective trial designs are needed to cover all lifecycle steps of these software biomarkers, from development and validation to integration into clinical practice.

End-to-end deep learning method for predicting hormonal treatment response in women with atypical endometrial hyperplasia or endometrial cancer

Purpose

Endometrial cancer (EC) is the most common gynecologic malignancy in the United States, and atypical endometrial hyperplasia (AEH) is considered a high-risk precursor to EC. Hormone therapies and hysterectomy are practical treatment options for AEH and early-stage EC. Some patients prefer hormone therapies for reasons such as fertility preservation or being poor surgical candidates. However, accurate prediction of an individual patient's response to hormonal treatment would allow for personalized and potentially improved recommendations for these conditions. This study aims to explore the feasibility of using deep learning models on whole slide images (WSI) of endometrial tissue samples to predict the patient's response to hormonal treatment.

Approach

We curated a clinical WSI dataset of 112 patients from two clinical sites. An expert pathologist annotated these images by outlining AEH/EC regions. We developed an end-to-end machine learning model with mixed supervision. The model is based on image patches extracted from pathologist-annotated AEH/EC regions. Either an unsupervised deep learning architecture (Autoencoder or ResNet50), or non-deep learning (radiomics feature extraction) is used to embed the images into a low-dimensional space, followed by fully connected layers for binary prediction, which was trained with binary responder/non-responder labels established by pathologists. We used stratified sampling to partition the dataset into a development set and a test set for internal validation of the performance of our models.

Results

The autoencoder model yielded an AUROC of 0.80 with 95% CI [0.63, 0.95] on the independent test set for the task of predicting a patient with AEH/EC as a responder vs non-responder to hormonal treatment.

Conclusions

These findings demonstrate the potential of using mixed supervised machine learning models on WSIs for predicting the response to hormonal treatment in AEH/EC patients.

Improved artificial intelligence discrimination of minor histological populations by supplementing with color-adjusted images

Despite the dedicated research of artificial intelligence (AI) for pathological images, the construction of AI applicable to histopathological tissue subtypes, is limited by insufficient dataset collection owing to disease infrequency. Here, we present a solution involving the addition of supplemental tissue array (TA) images that are adjusted to the tonality of the main data using a cycle-consistent generative adversarial network (CycleGAN) to the training data for rare tissue types. F1 scores of rare tissue types that constitute < 1.2% of the training data were significantly increased by improving recall values after adding color-adjusted TA images constituting < 0.65% of total training patches. The detector also enabled the equivalent discrimination of clinical images from two distinct hospitals and the capability was more increased following color-correction of test data before AI identification (F1 score from 45.2 ± 27.1 to 77.1 ± 10.3, p < 0.01). These methods also classified intraoperative frozen sections, while excessive supplementation paradoxically decreased F1 scores. These results identify strategies for building an AI that preserves the imbalance between training data with large differences in actual disease frequencies, which is important for constructing AI for practical histopathological classification.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Histology-Based Prediction of Therapy Response to Neoadjuvant Chemotherapy for Esophageal and Esophagogastric Junction Adenocarcinomas Using Deep Learning

PURPOSE

Quantifying treatment response to gastroesophageal junction (GEJ) adenocarcinomas is crucial to provide an optimal therapeutic strategy. Routinely taken tissue samples provide an opportunity to enhance existing positron emission tomography-computed tomography (PET/CT)-based therapy response evaluation. Our objective was to investigate if deep learning (DL) algorithms are capable of predicting the therapy response of patients with GEJ adenocarcinoma to neoadjuvant chemotherapy on the basis of histologic tissue samples.

METHODS

This diagnostic study recruited 67 patients with I-III GEJ adenocarcinoma from the multicentric nonrandomized MEMORI trial including three German university hospitals TUM (University Hospital Rechts der Isar, Munich), LMU (Hospital of the Ludwig-Maximilians-University, Munich), and UME (University Hospital Essen, Essen). All patients underwent baseline PET/CT scans and esophageal biopsy before and 14-21 days after treatment initiation. Treatment response was defined as a ≥35% decrease in SUVmax from baseline. Several DL algorithms were developed to predict PET/CT-based responders and nonresponders to neoadjuvant chemotherapy using digitized histopathologic whole slide images (WSIs).

RESULTS

The resulting models were trained on TUM (n = 25 pretherapy, n = 47 on-therapy) patients and evaluated on our internal validation cohort from LMU and UME (n = 17 pretherapy, n = 15 on-therapy). Compared with multiple architectures, the best pretherapy network achieves an area under the receiver operating characteristic curve (AUROC) of 0.81 (95% CI, 0.61 to 1.00), an area under the precision-recall curve (AUPRC) of 0.82 (95% CI, 0.61 to 1.00), a balanced accuracy of 0.78 (95% CI, 0.60 to 0.94), and a Matthews correlation coefficient (MCC) of 0.55 (95% CI, 0.18 to 0.88). The best on-therapy network achieves an AUROC of 0.84 (95% CI, 0.64 to 1.00), an AUPRC of 0.82 (95% CI, 0.56 to 1.00), a balanced accuracy of 0.80 (95% CI, 0.65 to 1.00), and a MCC of 0.71 (95% CI, 0.38 to 1.00).

CONCLUSION

Our results show that DL algorithms can predict treatment response to neoadjuvant chemotherapy using WSI with high accuracy even before therapy initiation, suggesting the presence of predictive morphologic tissue biomarkers.

Current Understanding on Why Ovarian Cancer Is Resistant to Immune Checkpoint Inhibitors

The standard treatment of ovarian cancer (OC) patients, including debulking surgery and first-line chemotherapy, is unsatisfactory because of recurrent episodes in the majority (~70%) of patients with advanced OC. Clinical trials have shown only a modest (10-15%) response of OC individuals to treatment based on immune checkpoint inhibitors (ICIs). The resistance of OC to therapy is caused by various factors, including OC heterogeneity, low density of tumor-infiltrating lymphocytes (TILs), non-cellular and cellular interactions in the tumor microenvironment (TME), as well as a network of microRNA regulating immune checkpoint pathways. Moreover, ICIs are the most efficient in tumors that are marked by high microsatellite instability and high tumor mutation burden, which is rare among OC patients. The great challenge in ICI implementation is connected with distinguishing hyper-, pseudo-, and real progression of the disease. The understanding of the immunological, molecular, and genetic mechanisms of OC resistance is crucial to selecting the group of OC individuals in whom personalized treatment would be beneficial. In this review, we summarize current knowledge about the selected factors inducing OC resistance and discuss the future directions of ICI-based immunotherapy development for OC patients.

Artificial Intelligence in Clinical Oncology: From Data to Digital Pathology and Treatment

Recently, a wide spectrum of artificial intelligence (AI)-based applications in the broader categories of digital pathology, biomarker development, and treatment have been explored. In the domain of digital pathology, these have included novel analytical strategies for realizing new information derived from standard histology to guide treatment selection and biomarker development to predict treatment selection and response. In therapeutics, these have included AI-driven drug target discovery, drug design and repurposing, combination regimen optimization, modulated dosing, and beyond. Given the continued advances that are emerging, it is important to develop workflows that seamlessly combine the various segments of AI innovation to comprehensively augment the diagnostic and interventional arsenal of the clinical oncology community. To overcome challenges that remain with regard to the ideation, validation, and deployment of AI in clinical oncology, recommendations toward bringing this workflow to fruition are also provided from clinical, engineering, implementation, and health care economics considerations. Ultimately, this work proposes frameworks that can potentially integrate these domains toward the sustainable adoption of practice-changing AI by the clinical oncology community to drive improved patient outcomes.

Artificial intelligence and digital biomarker in precision pathology guiding immune therapy selection and precision oncology

BACKGROUND

The currently available immunotherapies already changed the strategy how many cancers are treated from first to last line. Understanding even the most complex heterogeneity in tumor tissue and mapping the spatial cartography of the tumor immunity allows the best and optimized selection of immune modulating agents to (re-)activate the patient's immune system and direct it against the individual cancer in the most effective way.

RECENT FINDINGS

Primary cancer and metastases maintain a high degree of plasticity to escape any immune surveillance and continue to evolve depending on many intrinsic and extrinsic factors In the field of immune-oncology (IO) immune modulating agents are recognized as practice changing therapeutic modalities. Recent studies have shown that an optimal and lasting efficacy of IO therapeutics depends on the understanding of the spatial communication network and functional context of immune and cancer cells within the tumor microenvironment. Artificial intelligence (AI) provides an insight into the immune-cancer-network through the visualization of very complex tumor and immune interactions in cancer tissue specimens and allows the computer-assisted development and clinical validation of such digital biomarker.

CONCLUSIONS

The successful implementation of AI-supported digital biomarker solutions guides the clinical selection of effective immune therapeutics based on the retrieval and visualization of spatial and contextual information from cancer tissue images and standardized data. As such, computational pathology (CP) turns into "precision pathology" delivering individual therapy response prediction. Precision Pathology does not only include digital and computational solutions but also high levels of standardized processes in the routine histopathology workflow and the use of mathematical tools to support clinical and diagnostic decisions as the basic principle of a "precision oncology".

Tumour-infiltrating lymphocytes: from prognosis to treatment selection

Tumour-infiltrating lymphocytes (TILs) are considered crucial in anti-tumour immunity. Accordingly, the presence of TILs contains prognostic and predictive value. In 2011, we performed a systematic review and meta-analysis on the prognostic value of TILs across cancer types. Since then, the advent of immune checkpoint blockade (ICB) has renewed interest in the analysis of TILs. In this review, we first describe how our understanding of the prognostic value of TIL has changed over the last decade. New insights on novel TIL subsets are discussed and give a broader view on the prognostic effect of TILs in cancer. Apart from prognostic value, evidence on the predictive significance of TILs in the immune therapy era are discussed, as well as new techniques, such as machine learning that strive to incorporate these predictive capacities within clinical trials.

Successful Targeting of CTLA-4 in a Melanoma Clinical Case: A Long-Term "One Stop Therapeutic Shop"

The anti-Cytotoxic T-Lymphocyte Antigen 4 (CTLA-4) monoclonal antibody ipilimumab was the first in-class immune-checkpoint inhibitor (ICI) approved for the treatment of melanoma patients. Initially approved for metastatic cutaneous melanoma, treatment with ipilimumab subsequently demonstrated to significantly improve recurrence free survival (RFS) in fully resected, high-risk, stage III melanoma patients. Therapeutic use of ipilimumab has also allowed the initial identification and characterization of unconventional clinical and radiological patterns of response (ie, tumor flare, pseudo-progression) that may occur during ICI therapy, unlike chemotherapy or targeted therapy. As a result, the standard Response Evaluation Criteria In Solid Tumors (RECIST) and the World Health Organization (WHO) criteria conventionally utilized to assess responses to chemo/targeted therapy have been initially replaced by the immune-related (ir) Response Criteria (irRC) and then by the irRECIST, that encompass all patterns of response typical of ICI therapy, being key for the optimal comprehensive management of treated patients. Here, we report a paradigmatic clinical case of a long-term survival in a stage III melanoma patient, experiencing tumor flares during adjuvant treatment with ipilimumab, and an untreated disease relapse several years after ending therapy.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Pan-cancer integrative histology-genomic analysis via multimodal deep learning

The rapidly emerging field of computational pathology has demonstrated promise in developing objective prognostic models from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how these data sources can be integrated to develop joint image-omic prognostic models. Additionally, identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We use multimodal deep learning to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types. Our weakly supervised, multimodal deep-learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes. We present all analyses for morphological and molecular correlates of patient prognosis across the 14 cancer types at both a disease and a patient level in an interactive open-access database to allow for further exploration, biomarker discovery, and feature assessment.

Tumor infiltrating lymphocytes (TILs) as a predictive biomarker of response to checkpoint blockers in solid tumors: a systematic review

Immunotherapy is a standard of care in many solid tumors but many patients derive limited benefit from it. There is increasing interest toward tumor infiltrating lymphocytes (TILs) since their presence may be related with good outcomes from treatment with immune checkpoint blockers. We aimed at systematically reviewing existing evidence about the role of TILs as possible predictors of response to immunotherapy in solid tumors. We reviewed 1193 records published from January 2010 until December 2021. Associations between TILs and outcomes were observed mainly in melanoma and breast cancer. Overall survival and overall response rate for advanced disease and pathological complete response for early-phase tumors were the most commonly assessed endpoints. No definitive conclusion can be drawn on the predictive role of TILs. Additional studies, exploiting data from prospective, randomized clinical trials should further evaluate TILs also with the aim of identifying standard cut-off to differentiate between high and low TILs.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Digital Pathology and Artificial Intelligence Applications in Pathology

Digital pathology is revolutionizing pathology. The introduction of digital pathology made it possible to comprehensively change the pathology diagnosis workflow, apply and develop pathological artificial intelligence (AI) models, generate pathological big data, and perform telepathology. AI algorithms, including machine learning and deep learning, are used for the detection, segmentation, registration, processing, and classification of digitized pathological images. Pathological AI algorithms can be helpfully utilized for diagnostic screening, morphometric analysis of biomarkers, the discovery of new meanings of prognosis and therapeutic response in pathological images, and improvement of diagnostic efficiency. In order to develop a successful pathological AI model, it is necessary to consider the selection of a suitable type of image for a subject, utilization of big data repositories, the setting of an effective annotation strategy, image standardization, and color normalization. This review will elaborate on the advantages and perspectives of digital pathology, AI-based approaches, the applications in pathology, and considerations and challenges in the development of pathological AI models.

Paired primary and metastatic lesions of patients with ipilimumab-treated melanoma: high variation in lymphocyte infiltration and HLA-ABC expression whereas tumor mutational load is similar and correlates with clinical outcome

BACKGROUND

Immune checkpoint inhibitors (ICI) can lead to long-term responses in patients with metastatic melanoma. Still many patients with melanoma are intrinsically resistant or acquire secondary resistance. Previous studies have used primary or metastatic tumor tissue for biomarker assessment. Especially in melanoma, metastatic lesions are often present at different anatomical sites such as skin, lymph nodes, and visceral organs. The anatomical site may directly affect the tumor microenvironment (TME). To evaluate the impact of tumor evolution on the TME and on ICI treatment outcome, we directly compared paired primary and metastatic melanoma lesions for tumor mutational burden (TMB), HLA-ABC status, and tumor infiltrating lymphocytes (TILs) of patients that received ipilimumab.

METHODS

TMB was analyzed by sequencing primary and metastatic melanoma lesions using the TruSight Oncology 500 assay. Tumor tissues were subjected to multiplex immunohistochemistry to assess HLA-ABC status and for the detection of TIL subsets (B cells, cytotoxic T cells, helper T cells, and regulatory T cells), by using a machine-learning algorithm.

RESULTS

While we observed a very good agreement between TMB of matched primary and metastatic melanoma lesions (intraclass coefficient=0.921), such association was absent for HLA-ABC status, TIL density, and subsets thereof. Interestingly, analyses of different metastatic melanoma lesions within a single patient revealed that TIL density and composition agreed remarkably well, rejecting the hypothesis that the TME of different anatomical sites affects TIL infiltration. Similarly, the HLA-ABC status between different metastatic lesions within patients was also comparable. Furthermore, high TMB, of either primary or metastatic melanoma tissue, directly correlated with response to ipilimumab, whereas lymphocyte density or composition did not. Loss of HLA-ABC in the metastatic lesion correlated to a shorter progression-free survival on ipilimumab.

CONCLUSIONS

We confirm the link between TMB and HLA-ABC status and the response to ipilimumab-based immunotherapy in melanoma, but no correlation was found for TIL density, neither in primary nor metastatic lesions. Our finding that TMB between paired primary and metastatic melanoma lesions is highly stable, demonstrates its independency of the time point and location of acquisition. TIL and HLA-ABC status in metastatic lesions of different anatomical sites are highly similar within an individual patient.

A Machine learning model trained on dual-energy CT radiomics significantly improves immunotherapy response prediction for patients with stage IV melanoma

BACKGROUND

To assess the additive value of dual-energy CT (DECT) over single-energy CT (SECT) to radiomics-based response prediction in patients with metastatic melanoma preceding immunotherapy.

MATERIAL AND METHODS

A total of 140 consecutive patients with melanoma (58 female, 63±16 years) for whom baseline DECT tumor load assessment revealed stage IV and who were subsequently treated with immunotherapy were included. Best response was determined using the clinical reports (81 responders: 27 complete response, 45 partial response, 9 stable disease). Individual lesion response was classified manually analogous to RECIST 1.1 through 1291 follow-up examinations on a total of 776 lesions (6.7±7.2 per patient). The patients were sorted chronologically into a study and a validation cohort (each n=70). The baseline DECT was examined using specialized tumor segmentation prototype software, and radiomic features were analyzed for response predictors. Significant features were selected using univariate statistics with Bonferroni correction and multiple logistic regression. The area under the receiver operating characteristic curve of the best subset was computed (AUROC). For each combination (SECT/DECT and patient response/lesion response), an individual random forest classifier with 10-fold internal cross-validation was trained on the study cohort and tested on the validation cohort to confirm the predictive performance.

RESULTS

We performed manual RECIST 1.1 response analysis on a total of 6533 lesions. Multivariate statistics selected significant features for patient response in SECT (min. brightness, R²=0.112, padj. ≤0.001) and DECT (textural coarseness, R²=0.121, padj. ≤0.001), as well as lesion response in SECT (mean absolute voxel intensity deviation, R²=0.115, padj. ≤0.001) and DECT (iodine uptake metrics, R²≥0.12, padj. ≤0.001). Applying the machine learning models to the validation cohort confirmed the additive predictive power of DECT (patient response AUROC SECT=0.5, DECT=0.75; lesion response AUROC SECT=0.61, DECT=0.85; p<0.001).

CONCLUSION

The new method of DECT-specific radiomic analysis provides a significant additive value over SECT radiomics approaches for response prediction in patients with metastatic melanoma preceding immunotherapy, especially on a lesion-based level. As mixed tumor response is not uncommon in metastatic melanoma, this lends a powerful tool for clinical decision-making and may potentially be an essential step toward individualized medicine.

Anticoagulation with Factor Xa Inhibitors Is Associated with Improved Overall Response and Progression-Free Survival in Patients with Metastatic Malignant Melanoma Receiving Immune Checkpoint Inhibitors-A Retrospective, Real-World Cohort Study

Simple Summary

The advent of immune checkpoint inhibitors (ICI) improved the prognosis for patients with advanced melanoma. However, many patients do not benefit from ICI therapy due to primary and acquired resistance. Observations in murine systems suggested that coagulation factor Xa impedes anti-tumor immunity and that the oral FXa-inhibitor (FXa-i) rivaroxaban might synergize with ICI. In this retrospective study, we could demonstrate that concomitant treatment with anticoagulants did not impact the objective response rate, progression-free survival, or overall survival of stage IV melanoma patients who were treated with ICI. Remarkably, however, patients receiving concomitant treatment with FXa-i during initial ICI therapy showed a significantly improved objective response rate and progression-free survival as compared to patients not receiving anticoagulation or patients treated with other anticoagulants, such as heparins or vitamin K antagonists. Hence, our data suggest that FXa-i may augment ICI therapy, while patients who received FXa-i were not more likely to encounter bleeding complications.

Abstract

Immune checkpoint inhibitors (ICI) significantly improved the prognosis of advanced melanoma patients. However, many patients do not derive long-term benefit from ICI therapy due to primary and acquired resistance. In this regard, it has been shown that coagulation factors contribute to cancer immune evasion and might therefore promote resistance to ICI. In particular, recent observations in murine systems demonstrated that myeloid-derived factor Xa (FXa) impedes anti-tumor immunity in the tumor microenvironment and that the oral FXa inhibitor (FXa-i) rivaroxaban synergizes with ICI. The synergistic effect of FXa inhibitors with clinical ICI therapy is unknown. We performed a retrospective study of 280 metastatic melanoma patients who were treated with ICI and stratified them for concomitant anticoagulation (AC) by medical chart review. Data on baseline patient characteristics, specific AC treatment, ICI therapy, other tumor-targeting therapies, and clinical outcomes were analyzed. Of 280 patients who received ICI, 76 received concomitant AC during initial ICI therapy. Patients on AC were treated either with heparins (n = 29), vitamin K antagonists (VKA) (n = 20), or FXa-i (n = 27). Patients requiring AC during ICI therapy showed no significantly reduced objective response rate (ORR) (p = 0.27), or progression-free (PFS; median PFS 4 vs. 4 months; p = 0.71) or overall survival (OS; median OS: 39 vs. 51 months; p = 0.31). Furthermore, patients who underwent AC did not show significantly more bleeding complications (p = 0.605) than those who were not anticoagulated. Remarkably, stratification of patients by the class of AC revealed that patients receiving FXa-i were more likely to obtain CR (26.9 vs. 12.6%, p = 0.037), and showed better ORR (69.2 vs. 36.4%, p = 0.005), PFS (median PFS: 12 months vs. 3 months; p = 0.006), and OS (median OS not reached vs. 42 months; p = 0.09) compared to patients not receiving FXa-i. Patient demographics and tumor characteristics in this patient subcohort did not significantly differ from patients not on FXa-i. In summary, our study provides first clinical evidence that the clinical application of FXa-i may enhance the efficacy of ICI therapy via the restoration of anti-tumor immunity, while patients who received FXa-i were not more likely to encounter bleeding complications.

Clinical Categorization Algorithm (CLICAL) and Machine Learning Approach (SRF-CLICAL) to Predict Clinical Benefit to Immunotherapy in Metastatic Melanoma Patients: Real-World Evidence from the Istituto Nazionale Tumori IRCCS Fondazione Pascale, Napoli, Italy

The real-life application of immune checkpoint inhibitors (ICIs) may yield different outcomes compared to the benefit presented in clinical trials. For this reason, there is a need to define the group of patients that may benefit from treatment. We retrospectively investigated 578 metastatic melanoma patients treated with ICIs at the Istituto Nazionale Tumori IRCCS Fondazione "G. Pascale" of Napoli, Italy (INT-NA). To compare patients' clinical variables (i.e., age, lactate dehydrogenase (LDH), neutrophil-lymphocyte ratio (NLR), eosinophil, BRAF status, previous treatment) and their predictive and prognostic power in a comprehensive, non-hierarchical manner, a clinical categorization algorithm (CLICAL) was defined and validated by the application of a machine learning algorithm-survival random forest (SRF-CLICAL). The comprehensive analysis of the clinical parameters by log risk-based algorithms resulted in predictive signatures that could identify groups of patients with great benefit or not, regardless of the ICI received. From a real-life retrospective analysis of metastatic melanoma patients, we generated and validated an algorithm based on machine learning that could assist with the clinical decision of whether or not to apply ICI therapy by defining five signatures of predictability with 95% accuracy.

neoDL: a novel neoantigen intrinsic feature-based deep learning model identifies IDH wild-type glioblastomas with the longest survival

Background

Neoantigen based personalized immune therapies achieve promising results in melanoma and lung cancer, but few neoantigen based models perform well in IDH wild-type GBM, and the association between neoantigen intrinsic features and prognosis remain unclear in IDH wild-type GBM. We presented a novel neoantigen intrinsic feature-based deep learning model (neoDL) to stratify IDH wild-type GBMs into subgroups with different survivals.

Results

We first derived intrinsic features for each neoantigen associated with survival, followed by applying neoDL in TCGA data cohort(AUC = 0.988, p value < 0.0001). Leave one out cross validation (LOOCV) in TCGA demonstrated that neoDL successfully classified IDH wild-type GBMs into different prognostic subgroups, which was further validated in an independent data cohort from Asian population. Long-term survival IDH wild-type GBMs identified by neoDL were found characterized by 12 protective neoantigen intrinsic features and enriched in development and cell cycle.

Conclusions

The model can be therapeutically exploited to identify IDH wild-type GBM with good prognosis who will most likely benefit from neoantigen based personalized immunetherapy. Furthermore, the prognostic intrinsic features of the neoantigens inferred from this study can be used for identifying neoantigens with high potentials of immunogenicity.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-021-04301-6.

Artificial intelligence and digital pathology: Opportunities and implications for immuno-oncology

The field of immuno-oncology has expanded rapidly over the past decade, but key questions remain. How does tumour-immune interaction regulate disease progression? How can we prospectively identify patients who will benefit from immunotherapy? Identifying measurable features of the tumour immune-microenvironment which have prognostic or predictive value will be key to making meaningful gains in these areas. Recent developments in deep learning enable big-data analysis of pathological samples. Digital approaches allow data to be acquired, integrated and analysed far beyond what is possible with conventional techniques, and to do so efficiently and at scale. This has the potential to reshape what can be achieved in terms of volume, precision and reliability of output, enabling data for large cohorts to be summarised and compared. This review examines applications of artificial intelligence (AI) to important questions in immuno-oncology (IO). We discuss general considerations that need to be taken into account before AI can be applied in any clinical setting. We describe AI methods that have been applied to the field of IO to date and present several examples of their use.

Deep learning in cancer pathology: a new generation of clinical biomarkers

Clinical workflows in oncology rely on predictive and prognostic molecular biomarkers. However, the growing number of these complex biomarkers tends to increase the cost and time for decision-making in routine daily oncology practice; furthermore, biomarkers often require tumour tissue on top of routine diagnostic material. Nevertheless, routinely available tumour tissue contains an abundance of clinically relevant information that is currently not fully exploited. Advances in deep learning (DL), an artificial intelligence (AI) technology, have enabled the extraction of previously hidden information directly from routine histology images of cancer, providing potentially clinically useful information. Here, we outline emerging concepts of how DL can extract biomarkers directly from histology images and summarise studies of basic and advanced image analysis for cancer histology. Basic image analysis tasks include detection, grading and subtyping of tumour tissue in histology images; they are aimed at automating pathology workflows and consequently do not immediately translate into clinical decisions. Exceeding such basic approaches, DL has also been used for advanced image analysis tasks, which have the potential of directly affecting clinical decision-making processes. These advanced approaches include inference of molecular features, prediction of survival and end-to-end prediction of therapy response. Predictions made by such DL systems could simplify and enrich clinical decision-making, but require rigorous external validation in clinical settings.

The Functional Crosstalk between Myeloid-Derived Suppressor Cells and Regulatory T Cells within the Immunosuppressive Tumor Microenvironment

Simple Summary

Immunotherapy improved the therapeutic landscape for patients with advanced cancer diseases. However, many patients do not benefit from immunotherapy. The bidirectional crosstalk between myeloid-derived suppressor cells (MDSC) and regulatory T cells (Treg) contributes to immune evasion, limiting the success of immunotherapy by checkpoint inhibitors. This review aims to outline the current knowledge of the role and the immunosuppressive properties of MDSC and Treg within the tumor microenvironment (TME). Furthermore, we will discuss the importance of the functional crosstalk between MDSC and Treg for immunosuppression, issuing particularly the role of cell adhesion molecules. Lastly, we will depict the impact of this interaction for cancer research and discuss several strategies aimed to target these pathways for tumor therapy.

Abstract

Immune checkpoint inhibitors (ICI) have led to profound and durable tumor regression in some patients with metastatic cancer diseases. However, many patients still do not derive benefit from immunotherapy. Here, the accumulation of immunosuppressive cell populations within the tumor microenvironment (TME), such as myeloid-derived suppressor cells (MDSC), tumor-associated macrophages (TAM), and regulatory T cells (Treg), contributes to the development of immune resistance. MDSC and Treg expand systematically in tumor patients and inhibit T cell activation and T effector cell function. Numerous studies have shown that the immunosuppressive mechanisms exerted by those inhibitory cell populations comprise soluble immunomodulatory mediators and receptor interactions. The latter are also required for the crosstalk of MDSC and Treg, raising questions about the relevance of cell–cell contacts for the establishment of their inhibitory properties. This review aims to outline the current knowledge on the crosstalk between these two cell populations, issuing particularly the potential role of cell adhesion molecules. In this regard, we further discuss the relevance of β2 integrins, which are essential for the differentiation and function of leukocytes as well as for MDSC–Treg interaction. Lastly, we aim to describe the impact of such bidirectional crosstalk for basic and applied cancer research and discuss how the targeting of these pathways might pave the way for future approaches in immunotherapy.

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

PURPOSE

Several biomarkers of response to immune checkpoint inhibitors (ICI) show potential but are not yet scalable to the clinic. We developed a pipeline that integrates deep learning on histology specimens with clinical data to predict ICI response in advanced melanoma.

EXPERIMENTAL DESIGN

We used a training cohort from New York University (New York, NY) and a validation cohort from Vanderbilt University (Nashville, TN). We built a multivariable classifier that integrates neural network predictions with clinical data. A ROC curve was generated and the optimal threshold was used to stratify patients as high versus low risk for progression. Kaplan-Meier curves compared progression-free survival (PFS) between the groups. The classifier was validated on two slide scanners (Aperio AT2 and Leica SCN400).

RESULTS

The multivariable classifier predicted response with AUC 0.800 on images from the Aperio AT2 and AUC 0.805 on images from the Leica SCN400. The classifier accurately stratified patients into high versus low risk for disease progression. Vanderbilt patients classified as high risk for progression had significantly worse PFS than those classified as low risk (P = 0.02 for the Aperio AT2; P = 0.03 for the Leica SCN400).

CONCLUSIONS

Histology slides and patients' clinicodemographic characteristics are readily available through standard of care and have the potential to predict ICI treatment outcomes. With prospective validation, we believe our approach has potential for integration into clinical practice.

Myeloid Cells as Clinical Biomarkers for Immune Checkpoint Blockade

Immune checkpoint inhibitors are becoming standard treatments in several cancer types, profoundly changing the prognosis of a fraction of patients. Currently, many efforts are being made to predict responders and to understand how to overcome resistance in non-responders. Given the crucial role of myeloid cells as modulators of T effector cell function in tumors, it is essential to understand their impact on the clinical outcome of immune checkpoint blockade and on the mechanisms of immune evasion. In this review we focus on the existing clinical evidence of the relation between the presence of myeloid cell subsets and the response to anti-PD(L)1 and anti-CTLA-4 treatment. We highlight how circulating and tumor-infiltrating myeloid populations can be used as predictive biomarkers for immune checkpoint inhibitors in different human cancers, both at baseline and on treatment. Moreover, we propose to follow the dynamics of myeloid cells during immunotherapy as pharmacodynamic biomarkers. Finally, we provide an overview of the current strategies tested in the clinic that use myeloid cell targeting together with immune checkpoint blockade with the aim of uncovering the most promising approaches for effective combinations.

Deep learning assessment of breast terminal duct lobular unit involution: Towards automated prediction of breast cancer risk

Terminal duct lobular unit (TDLU) involution is the regression of milk-producing structures in the breast. Women with less TDLU involution are more likely to develop breast cancer. A major bottleneck in studying TDLU involution in large cohort studies is the need for labor-intensive manual assessment of TDLUs. We developed a computational pathology solution to automatically capture TDLU involution measures. Whole slide images (WSIs) of benign breast biopsies were obtained from the Nurses' Health Study. A set of 92 WSIs was annotated for acini, TDLUs and adipose tissue to train deep convolutional neural network (CNN) models for detection of acini, and segmentation of TDLUs and adipose tissue. These networks were integrated into a single computational method to capture TDLU involution measures including number of TDLUs per tissue area, median TDLU span and median number of acini per TDLU. We validated our method on 40 additional WSIs by comparing with manually acquired measures. Our CNN models detected acini with an F1 score of 0.73±0.07, and segmented TDLUs and adipose tissue with Dice scores of 0.84±0.13 and 0.87±0.04, respectively. The inter-observer ICC scores for manual assessments on 40 WSIs of number of TDLUs per tissue area, median TDLU span, and median acini count per TDLU were 0.71, 0.81 and 0.73, respectively. Intra-observer reliability was evaluated on 10/40 WSIs with ICC scores of >0.8. Inter-observer ICC scores between automated results and the mean of the two observers were: 0.80 for number of TDLUs per tissue area, 0.57 for median TDLU span, and 0.80 for median acini count per TDLU. TDLU involution measures evaluated by manual and automated assessment were inversely associated with age and menopausal status. We developed a computational pathology method to measure TDLU involution. This technology eliminates the labor-intensiveness and subjectivity of manual TDLU assessment, and can be applied to future breast cancer risk studies.

Opportunities for Artificial Intelligence in Advancing Precision Medicine

PURPOSE OF REVIEW

We critically evaluate the future potential of machine learning (ML), deep learning (DL), and artificial intelligence (AI) in precision medicine. The goal of this work is to show progress in ML in digital health, to exemplify future needs and trends, and to identify any essential prerequisites of AI and ML for precision health.

RECENT FINDINGS

High-throughput technologies are delivering growing volumes of biomedical data, such as large-scale genome-wide sequencing assays; libraries of medical images; or drug perturbation screens of healthy, developing, and diseased tissue. Multi-omics data in biomedicine is deep and complex, offering an opportunity for data-driven insights and automated disease classification. Learning from these data will open our understanding and definition of healthy baselines and disease signatures. State-of-the-art applications of deep neural networks include digital image recognition, single-cell clustering, and virtual drug screens, demonstrating breadths and power of ML in biomedicine.

SUMMARY

Significantly, AI and systems biology have embraced big data challenges and may enable novel biotechnology-derived therapies to facilitate the implementation of precision medicine approaches.

Deep Learning for Whole Slide Image Analysis: An Overview

The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artifacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In this paper, we review work on the interdisciplinary attempt of training deep neural networks using whole slide images, and highlight the different ideas underlying these methodologies.