Interpretable survival prediction for colorectal cancer using deep learning

Converging deep learning and human-observed tumor-adipocyte interaction as a biomarker in colorectal cancer

BACKGROUND

Tumor-Adipose-Feature (TAF) as well as SARIFA (Stroma AReactive Invasion Front Areas) are two histologic features/biomarkers linking tumor-associated adipocytes to poor outcomes in colorectal cancer (CRC) patients. Whereas TAF was identified by deep learning (DL) algorithms, SARIFA was established as a human-observed histopathologic biomarker.

METHODS

To study the overlap between TAF and SARIFA, we performed a systematic pathological review of TAF based on all published image tiles. Additionally, we analyzed the presence/absence of TAF in SARIFA-negative CRC cases to elucidate the biologic and prognostic role of a direct tumor-adipocyte contact. TCGA-CRC gene expression data is investigated to assess the association of FABP4 (fatty-acid binding protein 4) and CD36 (fatty-acid translocase) with both TAF and CRC prognosis.

RESULTS

By investigating the TAF/SARIFA overlap, we show that many TAF patches correspond to the recently described SARIFA-phenomenon. Even though there is a pronounced morphological and biological overlap, there are differences in the concepts. The presence of TAF in SARIFA-negative CRCs is not associated with poor outcomes in this cohort, potentially highlighting the importance of a direct tumor-adipocyte interaction. Upregulation of FABP4 and CD36 gene expression seem both linked to a poor prognosis in CRC.

CONCLUSIONS

By proving the substantial overlap between human-observed SARIFA and DL-based TAF as morphologic biomarkers, we demonstrate that linking DL-based image features to independently developed histopathologic biomarkers is a promising tool in the identification of clinically and biologically meaningful biomarkers. Adipocyte-tumor-cell interactions seem to be crucial in CRC, which should be considered as biomarkers for further investigations.

Exploit Spatially Resolved Transcriptomic Data to Infer Cellular Features from Pathology Imaging Data

Digital pathology is a rapidly advancing field where deep learning methods can be employed to extract meaningful imaging features. However, the efficacy of training deep learning models is often hindered by the scarcity of annotated pathology images, particularly images with detailed annotations for small image patches or tiles. To overcome this challenge, we propose an innovative approach that leverages paired spatially resolved transcriptomic data to annotate pathology images. We demonstrate the feasibility of this approach and introduce a novel transfer-learning neural network model, STpath (Spatial Transcriptomics and pathology images), designed to predict cell type proportions or classify tumor microenvironments. Our findings reveal that the features from pre-trained deep learning models are associated with cell type identities in pathology image patches. Evaluating STpath using three distinct breast cancer datasets, we observe its promising performance despite the limited training data. STpath excels in samples with variable cell type proportions and high-resolution pathology images. As the influx of spatially resolved transcriptomic data continues, we anticipate ongoing updates to STpath, evolving it into an invaluable AI tool for assisting pathologists in various diagnostic tasks.

From Pixels to Prognosis: A Survey on AI-Driven Cancer Patient Survival Prediction Using Digital Histology Images

Survival analysis is an integral part of medical statistics that is extensively utilized to establish prognostic indices for mortality or disease recurrence, assess treatment efficacy, and tailor effective treatment plans. The identification of prognostic biomarkers capable of predicting patient survival is a primary objective in the field of cancer research. With the recent integration of digital histology images into routine clinical practice, a plethora of Artificial Intelligence (AI)-based methods for digital pathology has emerged in scholarly literature, facilitating patient survival prediction. These methods have demonstrated remarkable proficiency in analyzing and interpreting whole slide images, yielding results comparable to those of expert pathologists. The complexity of AI-driven techniques is magnified by the distinctive characteristics of digital histology images, including their gigapixel size and diverse tissue appearances. Consequently, advanced patch-based methods are employed to effectively extract features that correlate with patient survival. These computational methods significantly enhance survival prediction accuracy and augment prognostic capabilities in cancer patients. The review discusses the methodologies employed in the literature, their performance metrics, ongoing challenges, and potential solutions for future advancements. This paper explains survival analysis and feature extraction methods for analyzing cancer patients. It also compiles essential acronyms related to cancer precision medicine. Furthermore, it is noteworthy that this is the inaugural review paper in the field. The target audience for this interdisciplinary review comprises AI practitioners, medical statisticians, and progressive oncologists who are enthusiastic about translating AI-driven solutions into clinical practice. We expect this comprehensive review article to guide future research directions in the field of cancer research.

The SARIFA biomarker in the context of basic research of lipid-driven cancers

SARIFA was very recently introduced as a histomorphological biomarker with strong prognostic power for colorectal, gastric, prostate, and pancreatic cancer. It is characterized by the direct contact between tumor cells and adipocytes due to a lack of stromal reaction. This can be easily evaluated on routinely available H&E-slides with high interobserver agreement. SARIFA also reflects a specific tumor biology driven by metabolic reprogramming. Tumor cells in SARIFA-positive tumors benefit from direct interaction with adipocytes as an external source of lipids. Numerous studies have shown that lipid metabolism is crucial in carcinogenesis and cancer progression. We found that the interaction between tumor cells and adipocytes was not triggered by obesity, as previously assumed. Instead, we believe that this is due to an immunological mechanism. Knowledge about lipid metabolism in cancer from basic experiments can be transferred to develop strategies targeting this reprogramed metabolism.

The Role of Adipocytes Recruited as Part of Tumor Microenvironment in Promoting Colorectal Cancer Metastases

Adipose tissue dysfunction, which is associated with an increased risk of colorectal cancer (CRC), is a significant factor in the pathophysiology of obesity. Obesity-related inflammation and extracellular matrix (ECM) remodeling promote colorectal cancer metastasis (CRCM) by shaping the tumor microenvironment (TME). When CRC occurs, the metabolic symbiosis of tumor cells recruits adjacent adipocytes into the TME to supply energy. Meanwhile, abundant immune cells, from adipose tissue and blood, are recruited into the TME, which is stimulated by pro-inflammatory factors and triggers a chronic local pro-inflammatory TME. Dysregulated ECM proteins and cell surface adhesion molecules enhance ECM remodeling and further increase contractibility between tumor and stromal cells, which promotes epithelial-mesenchymal transition (EMT). EMT increases tumor migration and invasion into surrounding tissues or vessels and accelerates CRCM. Colorectal symbiotic microbiota also plays an important role in the promotion of CRCM. In this review, we provide adipose tissue and its contributions to CRC, with a special emphasis on the role of adipocytes, macrophages, neutrophils, T cells, ECM, and symbiotic gut microbiota in the progression of CRC and their contributions to the CRC microenvironment. We highlight the interactions between adipocytes and tumor cells, and potential therapeutic approaches to target these interactions.

Novel AI Combining CNN and SVM to Predict Colorectal Cancer Prognosis and Mutational Signatures from HE Images

Reducing recurrence following radical resection of colon cancer without over- or under-treatment remains a challenge. Postoperative adjuvant chemotherapy (Adj) is currently administered based solely on pathological tumor, node, and metastasis (TNM) stage. However, prognosis can vary significantly among patients with the same disease stage. Therefore, novel classification systems in addition to the TNM are necessary to inform decision-making regarding postoperative treatment strategies, especially stage II and III disease, and to minimize overtreatment and undertreatment with Adj. We developed a prognostic prediction system for colorectal cancer by using a combined convolutional neural network (CNN) and support vector machine (SVM) approach to extract features from hematoxyling and eosin staining (HE) images. We combined the TNM and our AI-based classification system into a TNM-AI (mTNM-AI) classification system with high discriminative power for recurrence-free survival (RFS). Furthermore, the cancer cell population recognized by this system as low risk of recurrence exhibited the mutational signature SBS87 as a genetic phenotype. The novel AI-based classification system developed here is expected to play an important role in prognostic prediction and personalized treatment selection in oncology.

The relationship between Stroma AReactive Invasion Front Areas (SARIFA), Warburg-subtype and survival: results from a large prospective series of colorectal cancer patients

BACKGROUND

Stroma AReactive Invasion Front Areas (SARIFA) is a recently identified haematoxylin & eosin (H&E)based histopathologic biomarker in gastrointestinal cancers, including colorectal cancer (CRC), defined as direct contact between tumour cells and adipocytes at the tumour invasion front. The current study aimed at validating the prognostic relevance of SARIFA in a large population-based CRC series as well as at investigating the relationship between SARIFA-status and previously established Warburg-subtypes, both surrogates of the metabolic state of the tumour cells.

METHODS

SARIFA-status (positive versus negative) was determined on H&E slides of 1,727 CRC specimens. Warburg-subtype (high versus moderate versus low) data was available from our previous study. The associations between SARIFA-status, Warburg-subtype, clinicopathological characteristics and CRC-specific as well as overall survival were investigated.

RESULTS

28.7% (n=496) CRC were SARIFA-positive. SARIFA-positivity was associated with more advanced disease stage, higher pT category, and more frequent lymph node involvement (all p<0.001). SARIFA-positivity was more common in Warburg-high CRC. 44.2% (n=219) of SARIFA-positive CRCs were Warburg-high compared to 22.8% (n=113) being Warburg-low and 33.1% (n=164) being Warburg-moderate (p<0.001). In multivariable-adjusted analysis, patients with SARIFA-positive CRCs had significantly poorer CRC-specific (HRCRC-specific 1.65; 95% CI 1.41-1.93) and overall survival (HRoverall survival 1.46; 95% CI 1.28-1.67) independent of clinically known risk factors and independent of Warburg-subtype. Combining the SARIFA-status and the Warburg-subtype to a combination score (SARIFA-negative/Warburg-high versus SARIFA-positive/Warburg-low versus SARIFA-positive/Warburg-high, and so on) did not improve the survival prediction compared to the use of SARIFA-status alone (SARIFA-negative + Warburg-high: HRCRC-specific 1.08; 95% CI 0.84-1.38; SARIFA-positive + Warburg-low: HRCRC-specific 1.79; 95% CI 1.32-2.41; SARIFA-positive + Warburg-high: HRCRC-specific 1.58; 95% CI 1.23-2.04).

CONCLUSIONS

Our current study is the by far largest external validation of SARIFA-positivity as a novel independent negative prognostic H&E-based biomarker in CRC. In addition, our study shows that SARIFA-positivity is associated with the Warburg-high subtype. Further research is warranted to provide a more mechanistic understanding of the underlying tumour biology. Based on our data, we conclude SARIFA-status should be implemented in pathologic routine practice to stratify CRC patients.

Prediction of recurrence risk in endometrial cancer with multimodal deep learning

Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. Here we developed HECTOR (histopathology-based endometrial cancer tailored outcome risk), a multimodal deep learning prognostic model using hematoxylin and eosin-stained, whole-slide images and tumor stage as input, on 2,072 patients from eight EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n = 353) and two external (n = 160 and n = 151) test sets of 0.789, 0.828 and 0.815, respectively, outperforming the current gold standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low-, intermediate- and high-risk groups, respectively, by Kaplan-Meier analysis). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold standard and may help delivery of personalized treatment in EC.

The Concept of Stroma AReactive Invasion Front Areas (SARIFA) as a new prognostic biomarker for lipid-driven cancers holds true in pancreatic ductal adenocarcinoma

BACKGROUND

Pancreatic ductal adenocarcinoma (PDAC) is a 'difficult-to-treat' entity. To forecast its prognosis, we introduced a new biomarker, SARIFA (stroma areactive invasion front areas), which are areas at the tumour invasion front lacking desmoplastic stroma reaction upon malignant invasion in the surrounding tissue, leading to direct contact between tumour cells and adipocytes. SARIFA showed its significance in gastric and colorectal carcinoma, revealing lipid metabolism alternations that promote tumour progression.

METHODS

We reviewed the SARIFA status of 166 PDAC cases on all available H&E-stained tumour slides from archival Whipple-resection specimens. SARIFA positivity was defined as SARIFA detection in at least 66% of the available slides. To investigate alterations in tumour metabolism and microenvironment, we performed immunohistochemical staining for FABP4, CD36 and CD68. To verify and quantify a supposed delipidation of adipocytes, adipose tissue was digitally morphometrised.

RESULTS

In total, 53 cases (32%) were classified as SARIFA positive and 113 (68%) as SARIFA negative. Patients with SARIFA-positive PDAC showed a significantly worse overall survival compared with SARIFA-negative cases (median overall survival: 11.0 months vs. 22.0 months, HR: 1.570 (1.082-2.278), 95% CI, p = 0.018), which was independent from other prognostic markers (p = 0.014). At the invasion front of SARIFA-positive PDAC, we observed significantly higher expression of FABP4 (p < 0.0001) and higher concentrations of CD68+ macrophages (p = 0.031) related to a higher risk of tumour progression. CD36 staining showed no significant expression differences. The adipocyte areas at the invasion front were significantly smaller, with mean values of 4021 ± 1058 µm2 and 1812 ± 1008 µm2 for the SARIFA-negative and -positive cases, respectively (p < 0.001).

CONCLUSIONS

SARIFA is a promising prognostic biomarker for PDAC. Its assessment is characterised by simplicity and low effort. The mechanisms behind SARIFA suggest a tumour-promoting increased lipid metabolism and altered immune background, both showing new therapeutic avenues.

Prognostic Analysis Combining Histopathological Features and Clinical Information to Predict Colorectal Cancer Survival from Whole-Slide Images

BACKGROUND

Colorectal cancer (CRC) is a malignant tumor within the digestive tract with both a high incidence rate and mortality. Early detection and intervention could improve patient clinical outcomes and survival.

METHODS

This study computationally investigates a set of prognostic tissue and cell features from diagnostic tissue slides. With the combination of clinical prognostic variables, the pathological image features could predict the prognosis in CRC patients. Our CRC prognosis prediction pipeline sequentially consisted of three modules: (1) A MultiTissue Net to delineate outlines of different tissue types within the WSI of CRC for further ROI selection by pathologists. (2) Development of three-level quantitative image metrics related to tissue compositions, cell shape, and hidden features from a deep network. (3) Fusion of multi-level features to build a prognostic CRC model for predicting survival for CRC.

RESULTS

Experimental results suggest that each group of features has a particular relationship with the prognosis of patients in the independent test set. In the fusion features combination experiment, the accuracy rate of predicting patients' prognosis and survival status is 81.52%, and the AUC value is 0.77.

CONCLUSION

This paper constructs a model that can predict the postoperative survival of patients by using image features and clinical information. Some features were found to be associated with the prognosis and survival of patients.

A whole-slide foundation model for digital pathology from real-world data

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles1-3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data6. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision-language pretraining for pathology7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

Stroma AReactive Invasion Front Areas (SARIFA): a novel histopathologic biomarker in colorectal cancer patients and its association with the luminal tumour proportion

BACKGROUND

Stroma AReactive Invasion Front Areas (SARIFA) is a novel prognostic histopathologic biomarker measured at the invasive front in haematoxylin & eosin (H&E) stained colon and gastric cancer resection specimens. The aim of the current study was to validate the prognostic relevance of SARIFA-status in colorectal cancer (CRC) patients and investigate its association with the luminal proportion of tumour (PoT).

METHODS

We established the SARIFA-status in 164 CRC resection specimens. The relationship between SARIFA-status, clinicopathological characteristics, recurrence-free survival (RFS), cancer-specific survival (CSS), and PoT was investigated.

RESULTS

SARIFA-status was positive in 22.6% of all CRCs. SARIFA-positivity was related to higher pT, pN, pTNM stage and high grade of differentiation. SARIFA-positivity was associated with shorter RFS independent of known prognostic factors analysing all CRCs (RFS: hazard ratio (HR) 2.6, p = 0.032, CSS: HR 2.4, p = 0.05) and shorter RFS and CSS analysing only rectal cancers. SARIFA-positivity, which was measured at the invasive front, was associated with PoT-low (p = 0.009), e.g., higher stroma content, and lower vessel density (p = 0.0059) measured at the luminal tumour surface.

CONCLUSION

Here, we validated the relationship between SARIFA-status and prognosis in CRC patients and provided first evidence for a potential prognostic relevance in the subgroup of rectal cancer patients. Interestingly, CRCs with different SARIFA-status also showed histological differences measurable at the luminal tumour surface. Further studies to better understand the relationship between high luminal intratumoural stroma content and absence of a stroma reaction at the invasive front (SARIFA-positivity) are warranted and may inform future treatment decisions in CRC patients.

Use of Deep Learning to Evaluate Tumor Microenvironmental Features for Prediction of Colon Cancer Recurrence

Deep learning may detect biologically important signals embedded in tumor morphologic features that confer distinct prognoses. Tumor morphologic features were quantified to enhance patient risk stratification within DNA mismatch repair (MMR) groups using deep learning. Using a quantitative segmentation algorithm (QuantCRC) that identifies 15 distinct morphologic features, we analyzed 402 resected stage III colon carcinomas [191 deficient (d)-MMR; 189 proficient (p)-MMR] from participants in a phase III trial of FOLFOX-based adjuvant chemotherapy. Results were validated in an independent cohort (176 d-MMR; 1,094 p-MMR). Association of morphologic features with clinicopathologic variables, MMR, KRAS, BRAFV600E, and time-to-recurrence (TTR) was determined. Multivariable Cox proportional hazards models were developed to predict TTR. Tumor morphologic features differed significantly by MMR status. Cancers with p-MMR had more immature desmoplastic stroma. Tumors with d-MMR had increased inflammatory stroma, epithelial tumor-infiltrating lymphocytes (TIL), high-grade histology, mucin, and signet ring cells. Stromal subtype did not differ by BRAFV600E or KRAS status. In p-MMR tumors, multivariable analysis identified tumor-stroma ratio (TSR) as the strongest feature associated with TTR [HRadj 2.02; 95% confidence interval (CI), 1.14-3.57; P = 0.018; 3-year recurrence: 40.2% vs. 20.4%; Q1 vs. Q2-4]. Among d-MMR tumors, extent of inflammatory stroma (continuous HRadj 0.98; 95% CI, 0.96-0.99; P = 0.028; 3-year recurrence: 13.3% vs. 33.4%, Q4 vs. Q1) and N stage were the most robust prognostically. Association of TSR with TTR was independently validated. In conclusion, QuantCRC can quantify morphologic differences within MMR groups in routine tumor sections to determine their relative contributions to patient prognosis, and may elucidate relevant pathophysiologic mechanisms driving prognosis.

SIGNIFICANCE

A deep learning algorithm can quantify tumor morphologic features that may reflect underlying mechanisms driving prognosis within MMR groups. TSR was the most robust morphologic feature associated with TTR in p-MMR colon cancers. Extent of inflammatory stroma and N stage were the strongest prognostic features in d-MMR tumors. TIL density was not independently prognostic in either MMR group.

Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy

Ensuring diagnostic performance of artificial intelligence (AI) before introduction into clinical practice is essential. Growing numbers of studies using AI for digital pathology have been reported over recent years. The aim of this work is to examine the diagnostic accuracy of AI in digital pathology images for any disease. This systematic review and meta-analysis included diagnostic accuracy studies using any type of AI applied to whole slide images (WSIs) for any disease. The reference standard was diagnosis by histopathological assessment and/or immunohistochemistry. Searches were conducted in PubMed, EMBASE and CENTRAL in June 2022. Risk of bias and concerns of applicability were assessed using the QUADAS-2 tool. Data extraction was conducted by two investigators and meta-analysis was performed using a bivariate random effects model, with additional subgroup analyses also performed. Of 2976 identified studies, 100 were included in the review and 48 in the meta-analysis. Studies were from a range of countries, including over 152,000 whole slide images (WSIs), representing many diseases. These studies reported a mean sensitivity of 96.3% (CI 94.1-97.7) and mean specificity of 93.3% (CI 90.5-95.4). There was heterogeneity in study design and 99% of studies identified for inclusion had at least one area at high or unclear risk of bias or applicability concerns. Details on selection of cases, division of model development and validation data and raw performance data were frequently ambiguous or missing. AI is reported as having high diagnostic accuracy in the reported areas but requires more rigorous evaluation of its performance.

Advancing Artificial Intelligence Integration Into the Pathology Workflow: Exploring Opportunities in Gastrointestinal Tract Biopsies

This review aims to present a comprehensive overview of the current landscape of artificial intelligence (AI) applications in the analysis of tubular gastrointestinal biopsies. These publications cover a spectrum of conditions, ranging from inflammatory ailments to malignancies. Moving beyond the conventional diagnosis based on hematoxylin and eosin-stained whole-slide images, the review explores additional implications of AI, including its involvement in interpreting immunohistochemical results, molecular subtyping, and the identification of cellular spatial biomarkers. Furthermore, the review examines how AI can contribute to enhancing the quality and control of diagnostic processes, introducing new workflow options, and addressing the limitations and caveats associated with current AI platforms in this context.

Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions

Artificial intelligence (AI) in oncology is advancing beyond algorithm development to integration into clinical practice. This review describes the current state of the field, with a specific focus on clinical integration. AI applications are structured according to cancer type and clinical domain, focusing on the four most common cancers and tasks of detection, diagnosis, and treatment. These applications encompass various data modalities, including imaging, genomics, and medical records. We conclude with a summary of existing challenges, evolving solutions, and potential future directions for the field.

SIGNIFICANCE

AI is increasingly being applied to all aspects of oncology, where several applications are maturing beyond research and development to direct clinical integration. This review summarizes the current state of the field through the lens of clinical translation along the clinical care continuum. Emerging areas are also highlighted, along with common challenges, evolving solutions, and potential future directions for the field.

Membrane Trafficking-Related Genes Predict Tumor Immune Microenvironment and Prognosis in Colorectal Cancer

Colorectal cancer (CRC) is a heterogeneous disease with varying clinical outcomes. The identification of distinct subgroups of CRC patients based on molecular profiling can aid in better understanding the disease and improving patient outcomes. This study aimed to investigate the potential of membrane trafficking-related genes (MTRGs) in sub-grouping colorectal cancer patients based on their overall survival and immune microenvironments. Consensus clustering analysis identified two distinct clusters with different expression profiles of membrane trafficking-related genes. The patients in cluster 1 had a significantly better overall survival than those in cluster 2. Furthermore, the immune microenvironments in the two clusters were also found to be significantly different, with cluster 1 having a higher immune score and more immune cells present. Functional analysis of differentially expressed genes between the two clusters revealed that MTRGs were involved in immune response and metabolic processes, and a risk signature model based on MTRGs was established to predict the prognosis of CRC patients. These findings suggest that MTRGs play a crucial role in the immune microenvironment and overall survival of CRC patients and may provide a potential target for personalized therapy.

Post-COVID highlights: Challenges and solutions of artificial intelligence techniques for swift identification of COVID-19

Since the onset of the COVID-19 pandemic in 2019, there has been a concerted effort to develop cost-effective, non-invasive, and rapid AI-based tools. These tools were intended to alleviate the burden on healthcare systems, control the rapid spread of the virus, and enhance intervention outcomes, all in response to this unprecedented global crisis. As we transition into a post-COVID era, we retrospectively evaluate these proposed studies and offer a review of the techniques employed in AI diagnostic models, with a focus on the solutions proposed for different challenges. This review endeavors to provide insights into the diverse solutions designed to address the multifaceted challenges that arose during the pandemic. By doing so, we aim to prepare the AI community for the development of AI tools tailored to address public health emergencies effectively.

Self-Supervised Learning Reveals Clinically Relevant Histomorphological Patterns for Therapeutic Strategies in Colon Cancer

Self-supervised learning (SSL) automates the extraction and interpretation of histopathology features on unannotated hematoxylin-and-eosin-stained whole-slide images (WSIs). We trained an SSL Barlow Twins-encoder on 435 TCGA colon adenocarcinoma WSIs to extract features from small image patches. Leiden community detection then grouped tiles into histomorphological phenotype clusters (HPCs). HPC reproducibility and predictive ability for overall survival was confirmed in an independent clinical trial cohort (N=1213 WSIs). This unbiased atlas resulted in 47 HPCs displaying unique and sharing clinically significant histomorphological traits, highlighting tissue type, quantity, and architecture, especially in the context of tumor stroma. Through in-depth analysis of these HPCs, including immune landscape and gene set enrichment analysis, and association to clinical outcomes, we shed light on the factors influencing survival and responses to treatments like standard adjuvant chemotherapy and experimental therapies. Further exploration of HPCs may unveil new insights and aid decision-making and personalized treatments for colon cancer patients.

Novel biomarker SARIFA in colorectal cancer: highly prognostic, not genetically driven and histologic indicator of a distinct tumor biology

SARIFA (Stroma AReactive Invasion Front Areas) has recently emerged as a promising histopathological biomarker for colon and gastric cancer. To elucidate the underlying tumor biology, we assessed SARIFA-status in tissue specimens from The-Cancer-Genome-Atlas (TCGA) cohorts COAD (colonic adenocarcinoma) and READ (rectal adenocarcinoma). For the final analysis, 207 CRC patients could be included, consisting of 69 SARIFA-positive and 138 SARIFA-negative cases. In this external validation cohort, H&E-based SARIFA-positivity was strongly correlated with unfavorable overall, disease-specific, and progression-free survival, partly outperforming conventional prognostic factors. SARIFA-positivity was not associated with known high-risk genetic profiles, such as BRAF V600E mutations or microsatellite-stable status. Transcriptionally, SARIFA-positive CRCs exhibited an overlap with CRC consensus molecular subtypes CMS1 and CMS4, along with distinct differential gene expression patterns, linked to lipid metabolism and increased stromal cell infiltration scores (SIIS). Gene-expression-based drug sensitivity prediction revealed a differential treatment response in SARIFA-positive CRCs. In conclusion, SARIFA represents the H&E-based counterpart of an aggressive tumor biology, demonstrating a partial overlap with CMS1/4 and also adding a further biological layer related to lipid metabolism. Our findings underscore SARIFA-status as an ideal biomarker for refined patient stratification and novel drug developments, particularly given its cost-effective assessment based on routinely available H&E slides.

SARIFA as a new histopathological biomarker is associated with adverse clinicopathological characteristics, tumor-promoting fatty-acid metabolism, and might predict a metastatic pattern in pT3a prostate cancer

BACKGROUND

Recently, we introduced Stroma-AReactive-Invasion-Front-Areas (SARIFA) as a novel hematoxylin-eosin (H&E)-based histopathologic prognostic biomarker for various gastrointestinal cancers, closely related to lipid metabolism. To date, no studies on SARIFA, which is defined as direct tumor-adipocyte-interaction, beyond the alimentary tract exist. Hence, the objective of our current investigation was to study the significance of SARIFA in pT3a prostate cancer (PCa) and explore its association with lipid metabolism in PCa as lipid metabolism plays a key role in PCa development and progression.

METHODS

To this end, we evaluated SARIFA-status in 301 radical prostatectomy specimens and examined the relationship between SARIFA-status, clinicopathological characteristics, overall survival, and immunohistochemical expression of FABP4 and CD36 (proteins closely involved in fatty-acid metabolism). Additionally, we investigated the correlation between SARIFA and biochemical recurrence-free survival (BRFS) and PSMA-positive recurrences in PET/CT imaging in a patient subgroup. Moreover, a quantitative SARIFA cut-off was established to further understand the underlying tumor biology.

RESULTS

SARIFA positivity occurred in 59.1% (n = 178) of pT3a PCas. Our analysis demonstrated that SARIFA positivity is strongly associated with established high-risk features, such as R1 status, extraprostatic extension, and higher initial PSA values. Additionally, we observed an upregulation of immunohistochemical CD36 expression specifically at SARIFAs (p = 0.00014). Kaplan-Meier analyses revealed a trend toward poorer outcomes, particularly in terms of BRFS (p = 0.1). More extensive tumor-adipocyte interaction, assessed as quantity-dependent SARIFA-status on H&E slides, is also significantly associated with high-risk features, such as lymph node metastasis, and seems to be associated with worse survival outcomes (p = 0.16). Moreover, SARIFA positivity appeared to be linked to more distant lymph node and bone metastasis, although statistical significance was slightly not achieved (both p > 0.05).

CONCLUSIONS

This is the first study to introduce SARIFA as easy-and-fast-to-assess H&E-based biomarker in locally advanced PCa. SARIFA as the histopathologic correlate of a distinct tumor biology, closely related to lipid metabolism, could pave the way to a more detailed patient stratification and to the development of novel drugs targeting lipid metabolism in pT3a PCa. On the basis of this biomarker discovery study, further research efforts on the prognostic and predictive role of SARIFA in PCa can be designed.

AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images

The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.

End-to-end prognostication in colorectal cancer by deep learning: a retrospective, multicentre study

BACKGROUND

Precise prognosis prediction in patients with colorectal cancer (ie, forecasting survival) is pivotal for individualised treatment and care. Histopathological tissue slides of colorectal cancer specimens contain rich prognostically relevant information. However, existing studies do not have multicentre external validation with real-world sample processing protocols, and algorithms are not yet widely used in clinical routine.

METHODS

In this retrospective, multicentre study, we collected tissue samples from four groups of patients with resected colorectal cancer from Australia, Germany, and the USA. We developed and externally validated a deep learning-based prognostic-stratification system for automatic prediction of overall and cancer-specific survival in patients with resected colorectal cancer. We used the model-predicted risk scores to stratify patients into different risk groups and compared survival outcomes between these groups. Additionally, we evaluated the prognostic value of these risk groups after adjusting for established prognostic variables.

FINDINGS

We trained and validated our model on a total of 4428 patients. We found that patients could be divided into high-risk and low-risk groups on the basis of the deep learning-based risk score. On the internal test set, the group with a high-risk score had a worse prognosis than the group with a low-risk score, as reflected by a hazard ratio (HR) of 4·50 (95% CI 3·33-6·09) for overall survival and 8·35 (5·06-13·78) for disease-specific survival (DSS). We found consistent performance across three large external test sets. In a test set of 1395 patients, the high-risk group had a lower DSS than the low-risk group, with an HR of 3·08 (2·44-3·89). In two additional test sets, the HRs for DSS were 2·23 (1·23-4·04) and 3·07 (1·78-5·3). We showed that the prognostic value of the deep learning-based risk score is independent of established clinical risk factors.

INTERPRETATION

Our findings indicate that attention-based self-supervised deep learning can robustly offer a prognosis on clinical outcomes in patients with colorectal cancer, generalising across different populations and serving as a potentially new prognostic tool in clinical decision making for colorectal cancer management. We release all source codes and trained models under an open-source licence, allowing other researchers to reuse and build upon our work.

FUNDING

The German Federal Ministry of Health, the Max-Eder-Programme of German Cancer Aid, the German Federal Ministry of Education and Research, the German Academic Exchange Service, and the EU.

Personalizing adjuvant therapy for patients with colorectal cancer

The current standard-of-care adjuvant treatment for patients with colorectal cancer (CRC) comprises a fluoropyrimidine (5-fluorouracil or capecitabine) as a single agent or in combination with oxaliplatin, for either 3 or 6 months. Selection of therapy depends on conventional histopathological staging procedures, which constitute a blunt tool for patient stratification. Given the relatively marginal survival benefits that patients can derive from adjuvant treatment, improving the safety of chemotherapy regimens and identifying patients most likely to benefit from them is an area of unmet need. Patient stratification should enable distinguishing those at low risk of recurrence and a high chance of cure by surgery from those at higher risk of recurrence who would derive greater absolute benefits from chemotherapy. To this end, genetic analyses have led to the discovery of germline determinants of toxicity from fluoropyrimidines, the identification of patients at high risk of life-threatening toxicity, and enabling dose modulation to improve safety. Thus far, results from analyses of resected tissue to identify mutational or transcriptomic signatures with value as prognostic biomarkers have been rather disappointing. In the past few years, the application of artificial intelligence-driven models to digital images of resected tissue has identified potentially useful algorithms that stratify patients into distinct prognostic groups. Similarly, liquid biopsy approaches involving measurements of circulating tumour DNA after surgery are additionally useful tools to identify patients at high and low risk of tumour recurrence. In this Perspective, we provide an overview of the current landscape of adjuvant therapy for patients with CRC and discuss how new technologies will enable better personalization of therapy in this setting.

Stroma AReactive Invasion Front Areas (SARIFA) proves prognostic relevance in gastric carcinoma and is based on a tumor-adipocyte interaction indicating an altered immune response

BACKGROUND

Recently, we presented Stroma AReactive Invasion Front Areas (SARIFA) as a new histomorphologic negative prognostic biomarker in gastric cancer. It is defined as direct contact between tumor cells and fat cells. The aim of this study was to further elucidate the underlying genomic, transcriptional, and immunological mechanisms of the SARIFA phenomenon.

METHODS

To address these questions, SARIFA was classified on H&E-stained tissue sections of three cohorts: an external cohort (n = 489, prognostic validation), the TCGA-STAD cohort (n = 194, genomic and transcriptomic analysis), and a local cohort (n = 60, digital spatial profiling (whole transcriptome) and double RNA in situ hybridization/immunostaining of cytokines).

RESULTS

SARIFA status proved to be an independent negative prognostic factor for overall survival in an external cohort of gastric carcinomas. In TCGA-STAD cohort, SARIFA is not driven by distinct genomic alterations, whereas the gene expression analyses showed an upregulation of FABP4 in SARIFA-positive tumors. In addition, the transcriptional regulations of white adipocyte differentiation, triglyceride metabolism, and catabolism were upregulated in pathway analyses. In the DSP analysis of SARIFA-positive tumors, FABP4 and the transcriptional regulation of white adipocyte differentiation were upregulated in macrophages. Additionally, a significantly lower expression of the cytokines IL6 and TNFα was observed at the invasion front.

CONCLUSIONS

SARIFA proves to be a strong negative prognostic biomarker in advanced gastric cancer, implicating an interaction of tumor cells with tumor-promoting adipocytes with crucial changes in tumor cell metabolism. SARIFA is not driven by tumor genetics but is very likely driven by an altered immune response as a causative mechanism.

Interpretable Inference and Classification of Tissue Types in Histological Colorectal Cancer Slides Based on Ensembles Adaptive Boosting Prototype Tree

Digital pathology images are treated as the "gold standard" for the diagnosis of colorectal lesions, especially colon cancer. Real-time, objective and accurate inspection results will assist clinicians to choose symptomatic treatment in a timely manner, which is of great significance in clinical medicine. However, Manual methods suffers from long inspection cycle and serious reliance on subjective interpretation. It is also a challenging task for existing computer-aided diagnosis methods to obtain models that are both accurate and interpretable. Models that exhibit high accuracy are always more complex and opaque, while interpretable models may lack the necessary accuracy. Therefore, the framework of ensemble adaptive boosting prototype tree is proposed to predict the colorectal pathology images and provide interpretable inference by visualizing the decision-making process in each base learner. The results showed that the proposed method could effectively address the "accuracy-interpretability trade-off" issue by ensemble of m adaptive boosting neural prototype trees. The superior performance of the framework provides a novel paradigm for interpretable inference and high-precision prediction of pathology image patches in computational pathology.

Deep Learning and Colon Cancer Interpretation: Rise of the Machine

The rapidly evolving development of artificial intelligence (AI) has spurred the development of numerous algorithms that augment information obtained from routine pathologic review of hematoxylin and eosin-stained slides. AI tools that predict prognosis and underlying molecular alterations have been the focus of much of the research to date. The results of these studies highlight the tremendous potential of AI to enhance our pathology reports by providing rapid predictions of key features that influence therapy and outcomes.

Deep learning trained on lymph node status predicts outcome from gastric cancer histopathology: a retrospective multicentric study

AIM

Gastric cancer (GC) is a tumour entity with highly variant outcomes. Lymph node metastasis is a prognostically adverse biomarker. We hypothesised that GC primary tissue contains information that is predictive of lymph node status and patient prognosis and that this information can be extracted using deep learning (DL).

METHODS

Using three patient cohorts comprising 1146 patients, we trained and validated a DL system to predict lymph node status directly from haematoxylin and eosin-stained GC tissue sections. We investigated the concordance between the DL-based prediction from the primary tumour slides (aiN score) and the histopathological lymph node status (pN). Furthermore, we assessed the prognostic value of the aiN score alone and when combined with the pN status.

RESULTS

The aiN score predicted the pN status reaching area under the receiver operating characteristic curves of 0.71 in the training cohort and 0.69 and 0.65 in the two test cohorts. In a multivariate Cox analysis, the aiN score was an independent predictor of patient survival with hazard ratios of 1.5 in the training cohort and of 1.3 and 2.2 in the two test cohorts. A combination of the aiN score and the pN status prognostically stratified patients by survival with p-values <0.05 in logrank tests.

CONCLUSION

GC primary tumour tissue contains additional prognostic information that is accessible using the aiN score. In combination with the pN status, this can be used for personalised management of GC patients after prospective validation.

Deep learning-derived spatial organization features on histology images predicts prognosis in colorectal liver metastasis patients after hepatectomy

Histopathological images of colorectal liver metastases (CRLM) contain rich morphometric information that may predict patients' outcomes. However, to our knowledge, no study has reported any practical deep learning framework based on the histology images of CRLM, and their direct association with prognosis remains largely unknown. In this study, we developed a deep learning-based framework for fully automated tissue classification and quantification of clinically relevant spatial organization features (SOFs) in H&E-stained images of CRLM. The SOFs based risk-scoring system demonstrated a strong and robust prognostic value that is independent of the current clinical risk score (CRS) system in independent clinical cohorts. Our framework enables fully automated tissue classification of H&E images of CRLM, which could significantly reduce assessment subjectivity and the workload of pathologists. The risk-scoring system provides a time- and cost-efficient tool to assist clinical decision-making for patients with CRLM, which could potentially be implemented in clinical practice.

A comprehensive survey of intestine histopathological image analysis using machine vision approaches

Colorectal Cancer (CRC) is currently one of the most common and deadly cancers. CRC is the third most common malignancy and the fourth leading cause of cancer death worldwide. It ranks as the second most frequent cause of cancer-related deaths in the United States and other developed countries. Histopathological images contain sufficient phenotypic information, they play an indispensable role in the diagnosis and treatment of CRC. In order to improve the objectivity and diagnostic efficiency for image analysis of intestinal histopathology, Computer-aided Diagnosis (CAD) methods based on machine learning (ML) are widely applied in image analysis of intestinal histopathology. In this investigation, we conduct a comprehensive study on recent ML-based methods for image analysis of intestinal histopathology. First, we discuss commonly used datasets from basic research studies with knowledge of intestinal histopathology relevant to medicine. Second, we introduce traditional ML methods commonly used in intestinal histopathology, as well as deep learning (DL) methods. Then, we provide a comprehensive review of the recent developments in ML methods for segmentation, classification, detection, and recognition, among others, for histopathological images of the intestine. Finally, the existing methods have been studied, and the application prospects of these methods in this field are given.

Colorectal cancer risk stratification on histological slides based on survival curves predicted by deep learning

Studies have shown that colorectal cancer prognosis can be predicted by deep learning-based analysis of histological tissue sections of the primary tumor. So far, this has been achieved using a binary prediction. Survival curves might contain more detailed information and thus enable a more fine-grained risk prediction. Therefore, we established survival curve-based CRC survival predictors and benchmarked them against standard binary survival predictors, comparing their performance extensively on the clinical high and low risk subsets of one internal and three external cohorts. Survival curve-based risk prediction achieved a very similar risk stratification to binary risk prediction for this task. Exchanging other components of the pipeline, namely input tissue and feature extractor, had largely identical effects on model performance independently of the type of risk prediction. An ensemble of all survival curve-based models exhibited a more robust performance, as did a similar ensemble based on binary risk prediction. Patients could be further stratified within clinical risk groups. However, performance still varied across cohorts, indicating limited generalization of all investigated image analysis pipelines, whereas models using clinical data performed robustly on all cohorts.

Prediction of Five-Year Survival Rate for Rectal Cancer Using Markov Models of Convolutional Features of RhoB Expression on Tissue Microarray

The ability to predict survival in cancer is clinically important because the finding can help patients and physicians make optimal treatment decisions. Artificial intelligence in the context of deep learning has been increasingly realized by the informatics-oriented medical community as a powerful machine-learning technology for cancer research, diagnosis, prediction, and treatment. This paper presents the combination of deep learning, data coding, and probabilistic modeling for predicting five-year survival in a cohort of patients with rectal cancer using images of RhoB expression on biopsies. Using about one-third of the patients' data for testing, the proposed approach achieved 90% prediction accuracy, which is much higher than the direct use of the best pretrained convolutional neural network (70%) and the best coupling of a pretrained model and support vector machines (70%).

A self-supervised vision transformer to predict survival from histopathology in renal cell carcinoma

PURPOSE

To develop and validate an interpretable deep learning model to predict overall and disease-specific survival (OS/DSS) in clear cell renal cell carcinoma (ccRCC).

METHODS

Digitised haematoxylin and eosin-stained slides from The Cancer Genome Atlas were used as a training set for a vision transformer (ViT) to extract image features with a self-supervised model called DINO (self-distillation with no labels). Extracted features were used in Cox regression models to prognosticate OS and DSS. Kaplan-Meier for univariable evaluation and Cox regression analyses for multivariable evaluation of the DINO-ViT risk groups were performed for prediction of OS and DSS. For validation, a cohort from a tertiary care centre was used.

RESULTS

A significant risk stratification was achieved in univariable analysis for OS and DSS in the training (n = 443, log rank test, p < 0.01) and validation set (n = 266, p < 0.01). In multivariable analysis, including age, metastatic status, tumour size and grading, the DINO-ViT risk stratification was a significant predictor for OS (hazard ratio [HR] 3.03; 95%-confidence interval [95%-CI] 2.11-4.35; p < 0.01) and DSS (HR 4.90; 95%-CI 2.78-8.64; p < 0.01) in the training set but only for DSS in the validation set (HR 2.31; 95%-CI 1.15-4.65; p = 0.02). DINO-ViT visualisation showed that features were mainly extracted from nuclei, cytoplasm, and peritumoural stroma, demonstrating good interpretability.

CONCLUSION

The DINO-ViT can identify high-risk patients using histological images of ccRCC. This model might improve individual risk-adapted renal cancer therapy in the future.

Integrative deep learning analysis improves colon adenocarcinoma patient stratification at risk for mortality

BACKGROUND

Colorectal cancers are the fourth most diagnosed cancer and the second leading cancer in number of deaths. Many clinical variables, pathological features, and genomic signatures are associated with patient risk, but reliable patient stratification in the clinic remains a challenging task. Here we assess how image, clinical, and genomic features can be combined to predict risk.

METHODS

We developed and evaluated integrative deep learning models combining formalin-fixed, paraffin-embedded (FFPE) whole slide images (WSIs), clinical variables, and mutation signatures to stratify colon adenocarcinoma (COAD) patients based on their risk of mortality. Our models were trained using a dataset of 108 patients from The Cancer Genome Atlas (TCGA), and were externally validated on newly generated dataset from Wayne State University (WSU) of 123 COAD patients and rectal adenocarcinoma (READ) patients in TCGA (N = 52).

FINDINGS

We first observe that deep learning models trained on FFPE WSIs of TCGA-COAD separate high-risk (OS < 3 years, N = 38) and low-risk (OS > 5 years, N = 25) patients (AUC = 0.81 ± 0.08, 5 year survival p < 0.0001, 5 year relative risk = 1.83 ± 0.04) though such models are less effective at predicting overall survival (OS) for moderate-risk (3 years < OS < 5 years, N = 45) patients (5 year survival p-value = 0.5, 5 year relative risk = 1.05 ± 0.09). We find that our integrative models combining WSIs, clinical variables, and mutation signatures can improve patient stratification for moderate-risk patients (5 year survival p < 0.0001, 5 year relative risk = 1.87 ± 0.07). Our integrative model combining image and clinical variables is also effective on an independent pathology dataset (WSU-COAD, N = 123) generated by our team (5 year survival p < 0.0001, 5 year relative risk = 1.52 ± 0.08), and the TCGA-READ data (5 year survival p < 0.0001, 5 year relative risk = 1.18 ± 0.17). Our multicenter integrative image and clinical model trained on combined TCGA-COAD and WSU-COAD is effective in predicting risk on TCGA-READ (5 year survival p < 0.0001, 5 year relative risk = 1.82 ± 0.13) data. Pathologist review of image-based heatmaps suggests that nuclear size pleomorphism, intense cellularity, and abnormal structures are associated with high-risk, while low-risk regions have more regular and small cells. Quantitative analysis shows high cellularity, high ratios of tumor cells, large tumor nuclei, and low immune infiltration are indicators of high-risk tiles.

INTERPRETATION

The improved stratification of colorectal cancer patients from our computational methods can be beneficial for treatment plans and enrollment of patients in clinical trials.

FUNDING

This study was supported by the National Cancer Institutes (Grant No. R01CA230031 and P30CA034196). The funders had no roles in study design, data collection and analysis or preparation of the manuscript.

Recent advances of pathomics in colorectal cancer diagnosis and prognosis

Colorectal cancer (CRC) is one of the most common malignancies, with the third highest incidence and the second highest mortality in the world. To improve the therapeutic outcome, the risk stratification and prognosis predictions would help guide clinical treatment decisions. Achieving these goals have been facilitated by the fast development of artificial intelligence (AI) -based algorithms using radiological and pathological data, in combination with genomic information. Among them, features extracted from pathological images, termed pathomics, are able to reflect sub-visual characteristics linking to better stratification and prediction of therapeutic responses. In this paper, we review recent advances in pathological image-based algorithms in CRC, focusing on diagnosis of benign and malignant lesions, micro-satellite instability, as well as prediction of neoadjuvant chemoradiotherapy and the prognosis of CRC patients.

The role of deep learning in diagnosing colorectal cancer

Colon cancer is a major public health issue, affecting a growing number of individuals worldwide. Proper and early diagnosis of colon cancer is the necessary first step toward effective treatment and/or prevention of future disease relapse. Artificial intelligence and its subtypes, deep learning in particular, tend nowadays to have an expanding role in all fields of medicine, and diagnosing colon cancer is no exception. This report aims to summarize the entire application spectrum of deep learning in all diagnostic tests regarding colon cancer, from endoscopy and histologic examination to medical imaging and screening serologic tests.

Interpretable deep learning for improving cancer patient survival based on personal transcriptomes

Precision medicine chooses the optimal drug for a patient by considering individual differences. With the tremendous amount of data accumulated for cancers, we develop an interpretable neural network to predict cancer patient survival based on drug prescriptions and personal transcriptomes (CancerIDP). The deep learning model achieves 96% classification accuracy in distinguishing short-lived from long-lived patients. The Pearson correlation between predicted and actual months-to-death values is as high as 0.937. About 27.4% of patients may survive longer with an alternative medicine chosen by our deep learning model. The median survival time of all patients can increase by 3.9 months. Our interpretable neural network model reveals the most discriminating pathways in the decision-making process, which will further facilitate mechanistic studies of drug development for cancers.

Al-Biruni Earth Radius Optimization with Transfer Learning Based Histopathological Image Analysis for Lung and Colon Cancer Detection

An early diagnosis of lung and colon cancer (LCC) is critical for improved patient outcomes and effective treatment. Histopathological image (HSI) analysis has emerged as a robust tool for cancer diagnosis. HSI analysis for a LCC diagnosis includes the analysis and examination of tissue samples attained from the LCC to recognize lesions or cancerous cells. It has a significant role in the staging and diagnosis of this tumor, which aids in the prognosis and treatment planning, but a manual analysis of the image is subject to human error and is also time-consuming. Therefore, a computer-aided approach is needed for the detection of LCC using HSI. Transfer learning (TL) leverages pretrained deep learning (DL) algorithms that have been trained on a larger dataset for extracting related features from the HIS, which are then used for training a classifier for a tumor diagnosis. This manuscript offers the design of the Al-Biruni Earth Radius Optimization with Transfer Learning-based Histopathological Image Analysis for Lung and Colon Cancer Detection (BERTL-HIALCCD) technique. The purpose of the study is to detect LCC effectually in histopathological images. To execute this, the BERTL-HIALCCD method follows the concepts of computer vision (CV) and transfer learning for accurate LCC detection. When using the BERTL-HIALCCD technique, an improved ShuffleNet model is applied for the feature extraction process, and its hyperparameters are chosen by the BER system. For the effectual recognition of LCC, a deep convolutional recurrent neural network (DCRNN) model is applied. Finally, the coati optimization algorithm (COA) is exploited for the parameter choice of the DCRNN approach. For examining the efficacy of the BERTL-HIALCCD technique, a comprehensive group of experiments was conducted on a large dataset of histopathological images. The experimental outcomes demonstrate that the combination of AER and COA algorithms attain an improved performance in cancer detection over the compared models.

Bayesian risk prediction model for colorectal cancer mortality through integration of clinicopathologic and genomic data

Routine tumor-node-metastasis (TNM) staging of colorectal cancer is imperfect in predicting survival due to tumor pathobiological heterogeneity and imprecise assessment of tumor spread. We leveraged Bayesian additive regression trees (BART), a statistical learning technique, to comprehensively analyze patient-specific tumor characteristics for the improvement of prognostic prediction. Of 75 clinicopathologic, immune, microbial, and genomic variables in 815 stage II-III patients within two U.S.-wide prospective cohort studies, the BART risk model identified seven stable survival predictors. Risk stratifications (low risk, intermediate risk, and high risk) based on model-predicted survival were statistically significant (hazard ratios 0.19-0.45, vs. higher risk; P < 0.0001) and could be externally validated using The Cancer Genome Atlas (TCGA) data (P = 0.0004). BART demonstrated model flexibility, interpretability, and comparable or superior performance to other machine-learning models. Integrated bioinformatic analyses using BART with tumor-specific factors can robustly stratify colorectal cancer patients into prognostic groups and be readily applied to clinical oncology practice.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Privacy risks of whole-slide image sharing in digital pathology

Access to large volumes of so-called whole-slide images-high-resolution scans of complete pathological slides-has become a cornerstone of the development of novel artificial intelligence methods in pathology for diagnostic use, education/training of pathologists, and research. Nevertheless, a methodology based on risk analysis for evaluating the privacy risks associated with sharing such imaging data and applying the principle "as open as possible and as closed as necessary" is still lacking. In this article, we develop a model for privacy risk analysis for whole-slide images which focuses primarily on identity disclosure attacks, as these are the most important from a regulatory perspective. We introduce a taxonomy of whole-slide images with respect to privacy risks and mathematical model for risk assessment and design . Based on this risk assessment model and the taxonomy, we conduct a series of experiments to demonstrate the risks using real-world imaging data. Finally, we develop guidelines for risk assessment and recommendations for low-risk sharing of whole-slide image data.

MHAttnSurv: Multi-head attention for survival prediction using whole-slide pathology images

Whole slide images (WSI) based survival prediction has attracted increasing interest in pathology. Despite this, extracting prognostic information from WSIs remains a challenging task due to their enormous size and the scarcity of pathologist annotations. Previous studies have utilized multiple instance learning approach to combine information from several randomly sampled patches, but this approach may not be adequate as different visual patterns may contribute unequally to prognosis prediction. In this study, we introduce a multi-head attention mechanism that allows each attention head to independently explore the utility of various visual patterns on a tumor slide, thereby enabling more comprehensive information extraction from WSIs. We evaluated our approach on four cancer types from The Cancer Genome Atlas database. Our model achieved an average c-index of 0.640, outperforming three existing state-of-the-art approaches for WSI-based survival prediction on these datasets. Visualization of attention maps reveals that the attention heads synergistically focus on different morphological patterns, providing additional evidence for the effectiveness of multi-head attention in survival prediction.

Predicting lymph node metastasis from primary tumor histology and clinicopathologic factors in colorectal cancer using deep learning

BACKGROUND

Presence of lymph node metastasis (LNM) influences prognosis and clinical decision-making in colorectal cancer. However, detection of LNM is variable and depends on a number of external factors. Deep learning has shown success in computational pathology, but has struggled to boost performance when combined with known predictors.

METHODS

Machine-learned features are created by clustering deep learning embeddings of small patches of tumor in colorectal cancer via k-means, and then selecting the top clusters that add predictive value to a logistic regression model when combined with known baseline clinicopathological variables. We then analyze performance of logistic regression models trained with and without these machine-learned features in combination with the baseline variables.

RESULTS

The machine-learned extracted features provide independent signal for the presence of LNM (AUROC: 0.638, 95% CI: [0.590, 0.683]). Furthermore, the machine-learned features add predictive value to the set of 6 clinicopathologic variables in an external validation set (likelihood ratio test, p < 0.00032; AUROC: 0.740, 95% CI: [0.701, 0.780]). A model incorporating these features can also further risk-stratify patients with and without identified metastasis (p < 0.001 for both stage II and stage III).

CONCLUSION

This work demonstrates an effective approach to combine deep learning with established clinicopathologic factors in order to identify independently informative features associated with LNM. Further work building on these specific results may have important impact in prognostication and therapeutic decision making for LNM. Additionally, this general computational approach may prove useful in other contexts.

Histopathology images predict multi-omics aberrations and prognoses in colorectal cancer patients

Histopathologic assessment is indispensable for diagnosing colorectal cancer (CRC). However, manual evaluation of the diseased tissues under the microscope cannot reliably inform patient prognosis or genomic variations crucial for treatment selections. To address these challenges, we develop the Multi-omics Multi-cohort Assessment (MOMA) platform, an explainable machine learning approach, to systematically identify and interpret the relationship between patients' histologic patterns, multi-omics, and clinical profiles in three large patient cohorts (n = 1888). MOMA successfully predicts the overall survival, disease-free survival (log-rank test P-value<0.05), and copy number alterations of CRC patients. In addition, our approaches identify interpretable pathology patterns predictive of gene expression profiles, microsatellite instability status, and clinically actionable genetic alterations. We show that MOMA models are generalizable to multiple patient populations with different demographic compositions and pathology images collected from distinctive digitization methods. Our machine learning approaches provide clinically actionable predictions that could inform treatments for colorectal cancer patients.

Explainability and causability in digital pathology

The current move towards digital pathology enables pathologists to use artificial intelligence (AI)-based computer programmes for the advanced analysis of whole slide images. However, currently, the best-performing AI algorithms for image analysis are deemed black boxes since it remains - even to their developers - often unclear why the algorithm delivered a particular result. Especially in medicine, a better understanding of algorithmic decisions is essential to avoid mistakes and adverse effects on patients. This review article aims to provide medical experts with insights on the issue of explainability in digital pathology. A short introduction to the relevant underlying core concepts of machine learning shall nurture the reader's understanding of why explainability is a specific issue in this field. Addressing this issue of explainability, the rapidly evolving research field of explainable AI (XAI) has developed many techniques and methods to make black-box machine-learning systems more transparent. These XAI methods are a first step towards making black-box AI systems understandable by humans. However, we argue that an explanation interface must complement these explainable models to make their results useful to human stakeholders and achieve a high level of causability, i.e. a high level of causal understanding by the user. This is especially relevant in the medical field since explainability and causability play a crucial role also for compliance with regulatory requirements. We conclude by promoting the need for novel user interfaces for AI applications in pathology, which enable contextual understanding and allow the medical expert to ask interactive 'what-if'-questions. In pathology, such user interfaces will not only be important to achieve a high level of causability. They will also be crucial for keeping the human-in-the-loop and bringing medical experts' experience and conceptual knowledge to AI processes.

Artificial intelligence based personalized predictive survival among colorectal cancer patients

BACKGROUND AND OBJECTIVE

Colorectal cancer is a major health concern. It is now the third most common cancer and the fourth leading cause of cancer mortality worldwide. The aim of this study was to evaluate the performance of machine learning algorithms for predicting survival of colorectal cancer patients 1 to 5 years after diagnosis, and identify the most important variables.

METHODS

A sample of 1236 patients diagnosed with colorectal cancer and 118 predictor variables has been used. The outcome of interest was a binary variable indicating whether the patient survived the number of years in question or not. 20 predictor variables were selected using mutual information score with the outcome. We implemented 11 machine learning algorithms and evaluated their performance with a 5 by 2-fold cross-validation with stratified folds and with paired Student's t-tests. We compared the results with the Kaplan-Meier estimator and Cox's proportional hazard regression.

RESULTS

Using the 20 most important predictor variables for each of the survival years, the logistic regression algorithm achieved an area under the receiver operating characteristic curve of 0.850 (0.014 SD, 0.840-0.860 95 % CI) for the 1-year, and 0.872 (0.014 SD, 0.861-0.882 95% CI) for the 5-year survival prediction. Using only the 5 most important predictor variables, the corresponding values are 0.793 (0.020 SD, 0.778-0.807 95% CI) and 0.794 (0.011 SD, 0.785-0.802 95% CI). The most important variables for 1-year prediction were number of R residual, M distant metastasis, overall stage, probable recurrence within 5 years, and tumour length, whereas for 5-year prediction the most important were probable recurrence within 5 years, R residual, M distant metastasis, number of positive lymph nodes, and palliative chemotherapy. Biomarkers do not appear among the top 20 most important ones. For all survival intervals, the probability of the top model agrees with the Kaplan-Meier estimate, both in the interval of one standard deviation and in the 95% confidence interval.

CONCLUSIONS

The findings suggest that machine learning algorithms can predict the survival probability of colorectal cancer patients and can be used to inform the patients and assist decision-making in clinical care management. In addition, this study unveils the most essential variables for estimating survival short- and long-term among patients with Colorectal cancer.

From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment

Machine learning (ML) is increasingly used in clinical oncology to diagnose cancers, predict patient outcomes, and inform treatment planning. Here, we review recent applications of ML across the clinical oncology workflow. We review how these techniques are applied to medical imaging and to molecular data obtained from liquid and solid tumor biopsies for cancer diagnosis, prognosis, and treatment design. We discuss key considerations in developing ML for the distinct challenges posed by imaging and molecular data. Finally, we examine ML models approved for cancer-related patient usage by regulatory agencies and discuss approaches to improve the clinical usefulness of ML.

Pathologist Validation of a Machine Learning-Derived Feature for Colon Cancer Risk Stratification

IMPORTANCE

Identifying new prognostic features in colon cancer has the potential to refine histopathologic review and inform patient care. Although prognostic artificial intelligence systems have recently demonstrated significant risk stratification for several cancer types, studies have not yet shown that the machine learning-derived features associated with these prognostic artificial intelligence systems are both interpretable and usable by pathologists.

OBJECTIVE

To evaluate whether pathologist scoring of a histopathologic feature previously identified by machine learning is associated with survival among patients with colon cancer.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study used deidentified, archived colorectal cancer cases from January 2013 to December 2015 from the University of Milano-Bicocca. All available histologic slides from 258 consecutive colon adenocarcinoma cases were reviewed from December 2021 to February 2022 by 2 pathologists, who conducted semiquantitative scoring for tumor adipose feature (TAF), which was previously identified via a prognostic deep learning model developed with an independent colorectal cancer cohort.

MAIN OUTCOMES AND MEASURES

Prognostic value of TAF for overall survival and disease-specific survival as measured by univariable and multivariable regression analyses. Interpathologist agreement in TAF scoring was also evaluated.

RESULTS

A total of 258 colon adenocarcinoma histopathologic cases from 258 patients (138 men [53%]; median age, 67 years [IQR, 65-81 years]) with stage II (n = 119) or stage III (n = 139) cancer were included. Tumor adipose feature was identified in 120 cases (widespread in 63 cases, multifocal in 31, and unifocal in 26). For overall survival analysis after adjustment for tumor stage, TAF was independently prognostic in 2 ways: TAF as a binary feature (presence vs absence: hazard ratio [HR] for presence of TAF, 1.55 [95% CI, 1.07-2.25]; P = .02) and TAF as a semiquantitative categorical feature (HR for widespread TAF, 1.87 [95% CI, 1.23-2.85]; P = .004). Interpathologist agreement for widespread TAF vs lower categories (absent, unifocal, or multifocal) was 90%, corresponding to a κ metric at this threshold of 0.69 (95% CI, 0.58-0.80).

CONCLUSIONS AND RELEVANCE

In this prognostic study, pathologists were able to learn and reproducibly score for TAF, providing significant risk stratification on this independent data set. Although additional work is warranted to understand the biological significance of this feature and to establish broadly reproducible TAF scoring, this work represents the first validation to date of human expert learning from machine learning in pathology. Specifically, this validation demonstrates that a computationally identified histologic feature can represent a human-identifiable, prognostic feature with the potential for integration into pathology practice.

Alterations in Natural Killer Cells in Colorectal Cancer Patients with Stroma AReactive Invasion Front Areas (SARIFA)

BACKGROUND

Recently, our group introduced Stroma AReactive Invasion Front Areas (SARIFA) as an independent prognostic predictor for a poorer outcome in colon cancer patients, which is probably based on immunologic alterations combined with a direct tumor-adipocyte interaction: the two together reflecting a distinct tumor biology. Considering it is already known that peripheral immune cells are altered in colorectal cancer (CRC) patients, this study aims to investigate the changes in lymphocyte subsets in SARIFA-positive cases and correlate these changes with the local immune response.

METHODS

Flow cytometry was performed to analyze B, T, and natural killer (NK) cells in the peripheral blood (PB) of 45 CRC patients. Consecutively, lymphocytes in PB, tumor-infiltrating lymphocytes (TILs), and CD56+ and CD57+ lymphocytes at the invasion front and the tumor center were compared between patients with SARIFA-positive and SARIFA-negative CRCs.

RESULTS

Whereas no differences could be observed regarding most PB lymphocyte populations as well as TILs, NK cells were dramatically reduced in the PB of SARIFA-positive cases. Moreover, CD56 and CD57 immunohistochemistry suggested SARIFA-status-dependent changes regarding NK cells and NK-like lymphocytes in the tumor microenvironment.

CONCLUSION

This study proves that our newly introduced biomarker, SARIFA, comes along with distinct immunologic alterations, especially regarding NK cells.

Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer

Although it has long been known that the immune cell composition has a strong prognostic and predictive value in colorectal cancer (CRC), scoring systems such as the immunoscore (IS) or quantification of intraepithelial lymphocytes are only slowly being adopted into clinical routine use and have their limitations. To address this we established and evaluated a multistain deep learning model (MSDLM) utilizing artificial intelligence (AI) to determine the AImmunoscore (AIS) in more than 1,000 patients with CRC. Our model had high prognostic capabilities and outperformed other clinical, molecular and immune cell-based parameters. It could also be used to predict the response to neoadjuvant therapy in patients with rectal cancer. Using an explainable AI approach, we confirmed that the MSDLM's decisions were based on established cellular patterns of anti-tumor immunity. Hence, the AIS could provide clinicians with a valuable decision-making tool based on the tumor immune microenvironment.