Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review

Artificial intelligence in surgical pathology - Where do we stand, where do we go?

Surgical and neuropathologists continuously search for new and disease-specific features, such as independent predictors of tumor prognosis or determinants of tumor entities and sub-entities. This is a task where artificial intelligence (AI)/machine learning (ML) systems could significantly contribute to help with tumor outcome prediction and the search for new diagnostic or treatment stratification biomarkers. AI systems are increasingly integrated into routine pathology workflows to improve accuracy, reproducibility, productivity and to reveal difficult-to-see features in complicated histological slides, including the quantification of important markers for tumor grading and staging. In this article, we review the infrastructure needed to facilitate digital and computational pathology. We address the barriers for its full deployment in the clinical setting and describe the use of AI in intraoperative or postoperative settings were frozen or formalin-fixed, paraffin-embedded materials are used. We also summarize quality assessment issues of slide digitization, new spatial biology approaches, and the determination of specific gene-expression from whole slide images. Finally, we highlight new innovative and future technologies, such as large language models, optical biopsies, and mass spectrometry imaging.

A review on federated learning in computational pathology

Training generalizable computational pathology (CPATH) algorithms is heavily dependent on large-scale, multi-institutional data. Simultaneously, healthcare data underlies strict data privacy rules, hindering the creation of large datasets. Federated Learning (FL) is a paradigm addressing this dilemma, by allowing separate institutions to collaborate in a training process while keeping each institution's data private and exchanging model parameters instead. In this study, we identify and review key developments of FL for CPATH applications. We consider 15 studies, thereby evaluating the current status of exploring and adapting this emerging technology for CPATH applications. Proof-of-concept studies have been conducted across a wide range of CPATH use cases, showcasing the performance equivalency of models trained in a federated compared to a centralized manner. Six studies focus on model aggregation or model alignment methods reporting minor ( 0 ∼ 3 % ) performance improvement compared to conventional FL techniques, while four studies explore domain alignment methods, resulting in more significant performance improvements ( 4 ∼ 20 % ). To further reduce the privacy risk posed by sharing model parameters, four studies investigated the use of privacy preservation methods, where all methods demonstrated equivalent or slightly degraded performance ( 0.2 ∼ 6 % lower). To facilitate broader, real-world environment adoption, it is imperative to establish guidelines for the setup and deployment of FL infrastructure, alongside the promotion of standardized software frameworks. These steps are crucial to 1) further democratize CPATH research by allowing smaller institutions to pool data and computational resources 2) investigating rare diseases, 3) conducting multi-institutional studies, and 4) allowing rapid prototyping on private data.

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

ROSIE: AI generation of multiplex immunofluorescence staining from histopathology images

Hematoxylin and eosin (H&E) is a common and inexpensive histopathology assay. Though widely used and information-rich, it cannot directly inform about specific molecular markers, which require additional experiments to assess. To address this gap, we present ROSIE, a deep-learning framework that computationally imputes the expression and localization of dozens of proteins from H&E images. Our model is trained on a dataset of over 1000 paired and aligned H&E and multiplex immunofluorescence (mIF) samples from 20 tissues and disease conditions, spanning over 16 million cells. Validation of our in silico mIF staining method on held-out H&E samples demonstrates that the predicted biomarkers are effective in identifying cell phenotypes, particularly distinguishing lymphocytes such as B cells and T cells, which are not readily discernible with H&E staining alone. Additionally, ROSIE facilitates the robust identification of stromal and epithelial microenvironments and immune cell subtypes like tumor-infiltrating lymphocytes (TILs), which are important for understanding tumor-immune interactions and can help inform treatment strategies in cancer research.

Deep Learning Artificial Intelligence Predicts Homologous Recombination Deficiency and Platinum Response From Histologic Slides

PURPOSE

Cancers with homologous recombination deficiency (HRD) can benefit from platinum salts and poly(ADP-ribose) polymerase inhibitors. Standard diagnostic tests for detecting HRD require molecular profiling, which is not universally available.

METHODS

We trained DeepHRD, a deep learning platform for predicting HRD from hematoxylin and eosin (H&E)-stained histopathological slides, using primary breast (n = 1,008) and ovarian (n = 459) cancers from The Cancer Genome Atlas (TCGA). DeepHRD was compared with four standard HRD molecular tests using breast (n = 349) and ovarian (n = 141) cancers from multiple independent data sets, including platinum-treated clinical cohorts with RECIST progression-free survival (PFS), complete response (CR), and overall survival (OS) endpoints.

RESULTS

DeepHRD predicted HRD from held-out H&E-stained breast cancer slides in TCGA with an AUC of 0.81 (95% CI, 0.77 to 0.85). This performance was confirmed in two independent primary breast cancer cohorts (AUC, 0.76 [95% CI, 0.71 to 0.82]). In an external platinum-treated metastatic breast cancer cohort, samples predicted as HRD had higher complete CR (AUC, 0.76 [95% CI, 0.54 to 0.93]) with 3.7-fold increase in median PFS (14.4 v 3.9 months; P = .0019) and hazard ratio (HR) of 0.45 (P = .0047). There were no significant differences in nonplatinum treatment outcome by predicted HRD status in three breast cancer cohorts, including CR (AUC, 0.39) and PFS (HR, 0.98, P = .95) in taxane-treated metastatic breast cancer. Through transfer learning to high-grade serous ovarian cancer, DeepHRD-predicted HRD samples had better OS after first-line (HR, 0.46; P = .030) and neoadjuvant (HR, 0.49; P = .015) platinum therapy in two cohorts.

CONCLUSION

DeepHRD can predict HRD in breast and ovarian cancers directly from routine H&E slides across multiple external cohorts, slide scanners, and tissue fixation variables. When compared with molecular testing, DeepHRD classified 1.8- to 3.1-fold more patients with HRD, which exhibited better OS in high-grade serous ovarian cancer and platinum-specific PFS in metastatic breast cancer.

Recommendations for reporting tissue and circulating tumour (ct)DNA next-generation sequencing results in non-small cell lung cancer

Non-small cell lung cancer is a heterogeneous disease and molecular characterisation plays an important role in its clinical management. Next-generation sequencing-based panel testing enables many molecular alterations to be interrogated simultaneously, allowing for comprehensive identification of actionable oncogenic drivers (and co-mutations) and appropriate matching of patients with targeted therapies. Despite consensus in international guidelines on the importance of broad molecular profiling, adoption of next-generation sequencing varies globally. One of the barriers to its successful implementation is a lack of accepted standards and guidelines specifically for the reporting and clinical annotation of next-generation sequencing results. Based on roundtable discussions between pathologists and oncologists, we provide best practice recommendations for the reporting of next-generation sequencing results in non-small cell lung cancer to facilitate its use and enable easy interpretation for physicians. These are intended to complement existing guidelines related to the use of next-generation sequencing (solid and liquid). Here, we discuss next-generation sequencing workflows, the structure of next-generation sequencing reports, and our recommendations for best practice thereof. The aim of these recommendations and considerations is ultimately to ensure that reports are fully interpretable, and that the most appropriate treatment options are selected based on robust molecular profiles in well-defined reports.

Training immunophenotyping deep learning models with the same-section ground truth cell label derivation method improves virtual staining accuracy

Introduction

Deep learning (DL) models predicting biomarker expression in images of hematoxylin and eosin (H&E)-stained tissues can improve access to multi-marker immunophenotyping, crucial for therapeutic monitoring, biomarker discovery, and personalized treatment development. Conventionally, these models are trained on ground truth cell labels derived from IHC-stained tissue sections adjacent to H&E-stained ones, which might be less accurate than labels from the same section. Although many such DL models have been developed, the impact of ground truth cell label derivation methods on their performance has not been studied.

Methodology

In this study, we assess the impact of cell label derivation on H&E model performance, with CD3+ T-cells in lung cancer tissues as a proof-of-concept. We compare two Pix2Pix generative adversarial network (P2P-GAN)-based virtual staining models: one trained with cell labels obtained from the same tissue section as the H&E-stained section (the 'same-section' model) and one trained on cell labels from an adjacent tissue section (the 'serial-section' model).

Results

We show that the same-section model exhibited significantly improved prediction performance compared to the 'serial-section' model. Furthermore, the same-section model outperformed the serial-section model in stratifying lung cancer patients within a public lung cancer cohort based on survival outcomes, demonstrating its potential clinical utility.

Discussion

Collectively, our findings suggest that employing ground truth cell labels obtained through the same-section approach boosts immunophenotyping DL solutions.

Deep learning-based risk stratification of preoperative breast biopsies using digital whole slide images

BACKGROUND

Nottingham histological grade (NHG) is a well established prognostic factor in breast cancer histopathology but has a high inter-assessor variability with many tumours being classified as intermediate grade, NHG2. Here, we evaluate if DeepGrade, a previously developed model for risk stratification of resected tumour specimens, could be applied to risk-stratify tumour biopsy specimens.

METHODS

A total of 11,955,755 tiles from 1169 whole slide images of preoperative biopsies from 896 patients diagnosed with breast cancer in Stockholm, Sweden, were included. DeepGrade, a deep convolutional neural network model, was applied for the prediction of low- and high-risk tumours. It was evaluated against clinically assigned grades NHG1 and NHG3 on the biopsy specimen but also against the grades assigned to the corresponding resection specimen using area under the operating curve (AUC). The prognostic value of the DeepGrade model in the biopsy setting was evaluated using time-to-event analysis.

RESULTS

Based on preoperative biopsy images, the DeepGrade model predicted resected tumour cases of clinical grades NHG1 and NHG3 with an AUC of 0.908 (95% CI: 0.88; 0.93). Furthermore, out of the 432 resected clinically-assigned NHG2 tumours, 281 (65%) were classified as DeepGrade-low and 151 (35%) as DeepGrade-high. Using a multivariable Cox proportional hazards model the hazard ratio between DeepGrade low- and high-risk groups was estimated as 2.01 (95% CI: 1.06; 3.79).

CONCLUSIONS

DeepGrade provided prediction of tumour grades NHG1 and NHG3 on the resection specimen using only the biopsy specimen. The results demonstrate that the DeepGrade model can provide decision support to identify high-risk tumours based on preoperative biopsies, thus improving early treatment decisions.

Development and deployment of a histopathology-based deep learning algorithm for patient prescreening in a clinical trial

Accurate identification of genetic alterations in tumors, such as Fibroblast Growth Factor Receptor, is crucial for treating with targeted therapies; however, molecular testing can delay patient care due to the time and tissue required. Successful development, validation, and deployment of an AI-based, biomarker-detection algorithm could reduce screening cost and accelerate patient recruitment. Here, we develop a deep-learning algorithm using >3000 H&E-stained whole slide images from patients with advanced urothelial cancers, optimized for high sensitivity to avoid ruling out trial-eligible patients. The algorithm is validated on a dataset of 350 patients, achieving an area under the curve of 0.75, specificity of 31.8% at 88.7% sensitivity, and projected 28.7% reduction in molecular testing. We successfully deploy the system in a non-interventional study comprising 89 global study clinical sites and demonstrate its potential to prioritize/deprioritize molecular testing resources and provide substantial cost savings in the drug development and clinical settings.

Identifying Patients with CSF-Venous Fistula Using Brain MRI: A Deep Learning Approach

BACKGROUND AND PURPOSE

Spontaneous intracranial hypotension is an increasingly recognized condition. Spontaneous intracranial hypotension is caused by a CSF leak, which is commonly related to a CSF-venous fistula. In patients with spontaneous intracranial hypotension, multiple intracranial abnormalities can be observed on brain MR imaging, including dural enhancement, "brain sag," and pituitary engorgement. This study seeks to create a deep learning model for the accurate diagnosis of CSF-venous fistulas via brain MR imaging.

MATERIALS AND METHODS

A review of patients with clinically suspected spontaneous intracranial hypotension who underwent digital subtraction myelogram imaging preceded by brain MR imaging was performed. The patients were categorized as having a definite CSF-venous fistula, no fistula, or indeterminate findings on a digital subtraction myelogram. The data set was split into 5 folds at the patient level and stratified by label. A 5-fold cross-validation was then used to evaluate the reliability of the model. The predictive value of the model to identify patients with a CSF leak was assessed by using the area under the receiver operating characteristic curve for each validation fold.

RESULTS

There were 129 patients were included in this study. The median age was 54 years, and 66 (51.2%) had a CSF-venous fistula. In discriminating between positive and negative cases for CSF-venous fistulas, the classifier demonstrated an average area under the receiver operating characteristic curve of 0.8668 with a standard deviation of 0.0254 across the folds.

CONCLUSIONS

This study developed a deep learning model that can predict the presence of a spinal CSF-venous fistula based on brain MR imaging in patients with suspected spontaneous intracranial hypotension. However, further model refinement and external validation are necessary before clinical adoption. This research highlights the substantial potential of deep learning in diagnosing CSF-venous fistulas by using brain MR imaging.

Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer

Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.

Immune subtyping of melanoma whole slide images using multiple instance learning

Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential predictive biomarkers. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here, we attempt to overcome this by developing deep learning models to classify gigapixel haematoxylin and eosin (H&E) stained pathology slides, which are well established in clinical workflows, into these immune subgroups. We systematically assess six different multiple instance learning (MIL) frameworks, using five different image resolutions and three different feature extraction methods. We show that pathology-specific self-supervised models using 10x resolution patches generate superior representations for the classification of immune subtypes. In addition, in a primary melanoma dataset, we achieve a mean area under the receiver operating characteristic curve (AUC) of 0.80 for classifying histopathology images into 'high' or 'low immune' subgroups and a mean AUC of 0.82 in an independent TCGA melanoma dataset. Furthermore, we show that these models are able to stratify patients into 'high' and 'low immune' subgroups with significantly different melanoma specific survival outcomes (log rank test, P< 0.005). We anticipate that MIL methods will allow us to find new biomarkers of high importance, act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests.

Computational pathology to improve biomarker testing in breast cancer: how close are we?

The recent advancements in breast cancer precision medicine have highlighted the urgency for the precise and reproducible characterization of clinically actionable biomarkers. Despite numerous standardization efforts, biomarker testing by conventional methodologies is challenged by several issues such as high inter-observer variabilities, the spatial heterogeneity of biomarkers expression, and technological heterogeneity. In this respect, artificial intelligence-based digital pathology approaches are being increasingly recognized as promising methods for biomarker testing and subsequently improved clinical management. Here, we provide an overview on the most recent advances for artificial intelligence-assisted biomarkers testing in breast cancer, with a particular focus on tumor-infiltrating lymphocytes, programmed death-ligand 1, phosphatidylinositol-3 kinase catalytic alpha, and estrogen receptor 1. Challenges and solutions for this integrative analysis in pathology laboratories are also provided.

Immune Biomarkers in Triple-Negative Breast Cancer: Improving the Predictivity of Current Testing Methods

Triple-negative breast cancer (TNBC) poses a significant challenge in terms of prognosis and disease recurrence. The limited treatment options and the development of resistance to chemotherapy make it particularly difficult to manage these patients. However, recent research has been shifting its focus towards biomarker-based approaches for TNBC, with a particular emphasis on the tumor immune landscape. Immune biomarkers in TNBC are now a subject of great interest due to the presence of tumor-infiltrating lymphocytes (TILs) in these tumors. This characteristic often coincides with the presence of PD-L1 expression on both neoplastic cells and immune cells within the tumor microenvironment. Furthermore, a subset of TNBC harbor mismatch repair deficient (dMMR) TNBC, which is frequently accompanied by microsatellite instability (MSI). All of these immune biomarkers hold actionable potential for guiding patient selection in immunotherapy. To fully capitalize on these opportunities, the identification of additional or complementary biomarkers and the implementation of highly customized testing strategies are of paramount importance in TNBC. In this regard, this article aims to provide an overview of the current state of the art in immune-related biomarkers for TNBC. Specifically, it focuses on the various testing methodologies available and sheds light on the immediate future perspectives for patient selection. By delving into the advancements made in understanding the immune landscape of TNBC, this study aims to contribute to the growing body of knowledge in the field. The ultimate goal is to pave the way for the development of more personalized testing strategies, ultimately improving outcomes for TNBC patients.