Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer

Weakly-supervised deep learning models enable HER2-low prediction from H &E stained slides

BACKGROUND

Human epidermal growth factor receptor 2 (HER2)-low breast cancer has emerged as a new subtype of tumor, for which novel antibody-drug conjugates have shown beneficial effects. Assessment of HER2 requires several immunohistochemistry tests with an additional in situ hybridization test if a case is classified as HER2 2+. Therefore, novel cost-effective methods to speed up the HER2 assessment are highly desirable.

METHODS

We used a self-supervised attention-based weakly supervised method to predict HER2-low directly from 1437 histopathological images from 1351 breast cancer patients. We built six distinct models to explore the ability of classifiers to distinguish between the HER2-negative, HER2-low, and HER2-high classes in different scenarios. The attention-based model was used to comprehend the decision-making process aimed at relevant tissue regions.

RESULTS

Our results indicate that the effectiveness of classification models hinges on the consistency and dependability of assay-based tests for HER2, as the outcomes from these tests are utilized as the baseline truth for training our models. Through the use of explainable AI, we reveal histologic patterns associated with the HER2 subtypes.

CONCLUSION

Our findings offer a demonstration of how deep learning technologies can be applied to identify HER2 subgroup statuses, potentially enriching the toolkit available for clinical decision-making in oncology.

Converging deep learning and human-observed tumor-adipocyte interaction as a biomarker in colorectal cancer

BACKGROUND

Tumor-Adipose-Feature (TAF) as well as SARIFA (Stroma AReactive Invasion Front Areas) are two histologic features/biomarkers linking tumor-associated adipocytes to poor outcomes in colorectal cancer (CRC) patients. Whereas TAF was identified by deep learning (DL) algorithms, SARIFA was established as a human-observed histopathologic biomarker.

METHODS

To study the overlap between TAF and SARIFA, we performed a systematic pathological review of TAF based on all published image tiles. Additionally, we analyzed the presence/absence of TAF in SARIFA-negative CRC cases to elucidate the biologic and prognostic role of a direct tumor-adipocyte contact. TCGA-CRC gene expression data is investigated to assess the association of FABP4 (fatty-acid binding protein 4) and CD36 (fatty-acid translocase) with both TAF and CRC prognosis.

RESULTS

By investigating the TAF/SARIFA overlap, we show that many TAF patches correspond to the recently described SARIFA-phenomenon. Even though there is a pronounced morphological and biological overlap, there are differences in the concepts. The presence of TAF in SARIFA-negative CRCs is not associated with poor outcomes in this cohort, potentially highlighting the importance of a direct tumor-adipocyte interaction. Upregulation of FABP4 and CD36 gene expression seem both linked to a poor prognosis in CRC.

CONCLUSIONS

By proving the substantial overlap between human-observed SARIFA and DL-based TAF as morphologic biomarkers, we demonstrate that linking DL-based image features to independently developed histopathologic biomarkers is a promising tool in the identification of clinically and biologically meaningful biomarkers. Adipocyte-tumor-cell interactions seem to be crucial in CRC, which should be considered as biomarkers for further investigations.

Exploit Spatially Resolved Transcriptomic Data to Infer Cellular Features from Pathology Imaging Data

Digital pathology is a rapidly advancing field where deep learning methods can be employed to extract meaningful imaging features. However, the efficacy of training deep learning models is often hindered by the scarcity of annotated pathology images, particularly images with detailed annotations for small image patches or tiles. To overcome this challenge, we propose an innovative approach that leverages paired spatially resolved transcriptomic data to annotate pathology images. We demonstrate the feasibility of this approach and introduce a novel transfer-learning neural network model, STpath (Spatial Transcriptomics and pathology images), designed to predict cell type proportions or classify tumor microenvironments. Our findings reveal that the features from pre-trained deep learning models are associated with cell type identities in pathology image patches. Evaluating STpath using three distinct breast cancer datasets, we observe its promising performance despite the limited training data. STpath excels in samples with variable cell type proportions and high-resolution pathology images. As the influx of spatially resolved transcriptomic data continues, we anticipate ongoing updates to STpath, evolving it into an invaluable AI tool for assisting pathologists in various diagnostic tasks.

Prognostic Analysis Combining Histopathological Features and Clinical Information to Predict Colorectal Cancer Survival from Whole-Slide Images

BACKGROUND

Colorectal cancer (CRC) is a malignant tumor within the digestive tract with both a high incidence rate and mortality. Early detection and intervention could improve patient clinical outcomes and survival.

METHODS

This study computationally investigates a set of prognostic tissue and cell features from diagnostic tissue slides. With the combination of clinical prognostic variables, the pathological image features could predict the prognosis in CRC patients. Our CRC prognosis prediction pipeline sequentially consisted of three modules: (1) A MultiTissue Net to delineate outlines of different tissue types within the WSI of CRC for further ROI selection by pathologists. (2) Development of three-level quantitative image metrics related to tissue compositions, cell shape, and hidden features from a deep network. (3) Fusion of multi-level features to build a prognostic CRC model for predicting survival for CRC.

RESULTS

Experimental results suggest that each group of features has a particular relationship with the prognosis of patients in the independent test set. In the fusion features combination experiment, the accuracy rate of predicting patients' prognosis and survival status is 81.52%, and the AUC value is 0.77.

CONCLUSION

This paper constructs a model that can predict the postoperative survival of patients by using image features and clinical information. Some features were found to be associated with the prognosis and survival of patients.

Unsupervised mutual transformer learning for multi-gigapixel Whole Slide Image classification

The classification of gigapixel Whole Slide Images (WSIs) is an important task in the emerging area of computational pathology. There has been a surge of interest in deep learning models for WSI classification with clinical applications such as cancer detection or prediction of cellular mutations. Most supervised methods require expensive and labor-intensive manual annotations by expert pathologists. Weakly supervised Multiple Instance Learning (MIL) methods have recently demonstrated excellent performance; however, they still require large-scale slide-level labeled training datasets that require a careful inspection of each slide by an expert pathologist. In this work, we propose a fully unsupervised WSI classification algorithm based on mutual transformer learning. The instances (i.e., patches) from gigapixel WSIs are transformed into a latent space and then inverse-transformed to the original space. Using the transformation loss, pseudo labels are generated and cleaned using a transformer label cleaner. The proposed transformer-based pseudo-label generator and cleaner modules mutually train each other iteratively in an unsupervised manner. A discriminative learning mechanism is introduced to improve normal versus cancerous instance labeling. In addition to the unsupervised learning, we demonstrate the effectiveness of the proposed framework for weakly supervised learning and cancer subtype classification as downstream analysis. Extensive experiments on four publicly available datasets show better performance of the proposed algorithm compared to the existing state-of-the-art methods.

A mutual reconstruction network model for few-shot classification of histological images: addressing interclass similarity and intraclass diversity

Background

The automated classification of histological images is crucial for the diagnosis of cancer. The limited availability of well-annotated datasets, especially for rare cancers, poses a significant challenge for deep learning methods due to the small number of relevant images. This has led to the development of few-shot learning approaches, which bear considerable clinical importance, as they are designed to overcome the challenges of data scarcity in deep learning for histological image classification. Traditional methods often ignore the challenges of intraclass diversity and interclass similarities in histological images. To address this, we propose a novel mutual reconstruction network model, aimed at meeting these challenges and improving the few-shot classification performance of histological images.

Methods

The key to our approach is the extraction of subtle and discriminative features. We introduce a feature enhancement module (FEM) and a mutual reconstruction module to increase differences between classes while reducing variance within classes. First, we extract features of support and query images using a feature extractor. These features are then processed by the FEM, which uses a self-attention mechanism for self-reconstruction of features, enhancing the learning of detailed features. These enhanced features are then input into the mutual reconstruction module. This module uses enhanced support features to reconstruct enhanced query features and vice versa. The classification of query samples is based on weighted calculations of the distances between query features and reconstructed query features and between support features and reconstructed support features.

Results

We extensively evaluated our model using a specially created few-shot histological image dataset. The results showed that in a 5-way 10-shot setup, our model achieved an impressive accuracy of 92.09%. This is a 23.59% improvement in accuracy compared to the model-agnostic meta-learning (MAML) method, which does not focus on fine-grained attributes. In the more challenging, 5-way 1-shot setting, our model also performed well, demonstrating a 18.52% improvement over the ProtoNet, which does not address this challenge. Additional ablation studies indicated the effectiveness and complementary nature of each module and confirmed our method's ability to parse small differences between classes and large variations within classes in histological images. These findings strongly support the superiority of our proposed method in the few-shot classification of histological images.

Conclusions

The mutual reconstruction network provides outstanding performance in the few-shot classification of histological images, successfully overcoming the challenges of similarities between classes and diversity within classes. This marks a significant advancement in the automated classification of histological images.

Radiology and multi-scale data integration for precision oncology

In this Perspective paper we explore the potential of integrating radiological imaging with other data types, a critical yet underdeveloped area in comparison to the fusion of other multi-omic data. Radiological images provide a comprehensive, three-dimensional view of cancer, capturing features that would be missed by biopsies or other data modalities. This paper explores the complexities and challenges of incorporating medical imaging into data integration models, in the context of precision oncology. We present the different categories of imaging-omics integration and discuss recent progress, highlighting the opportunities that arise from bringing together spatial data on different scales.

Artificial Intelligence Models for the Detection of Microsatellite Instability from Whole-Slide Imaging of Colorectal Cancer

With the advent of whole-slide imaging (WSI), a technology that can digitally scan whole slides in high resolution, pathology is undergoing a digital revolution. Detecting microsatellite instability (MSI) in colorectal cancer is crucial for proper treatment, as it identifies patients responsible for immunotherapy. Even though universal testing for MSI is recommended, particularly in patients affected by colorectal cancer (CRC), many patients remain untested, and they reside mainly in low-income countries. A critical need exists for accessible, low-cost tools to perform MSI pre-screening. Here, the potential predictive role of the most relevant artificial intelligence-driven models in predicting microsatellite instability directly from histology alone is discussed, focusing on CRC. The role of deep learning (DL) models in identifying the MSI status is here analyzed in the most relevant studies reporting the development of algorithms trained to this end. The most important performance and the most relevant deficiencies are discussed for every AI method. The models proposed for algorithm sharing among multiple research and clinical centers, including federal learning (FL) and swarm learning (SL), are reported. According to all the studies reported here, AI models are valuable tools for predicting MSI status on WSI alone in CRC. The use of digitized H&E-stained sections and a trained algorithm allow the extraction of relevant molecular information, such as MSI status, in a short time and at a low cost. The possible advantages related to introducing DL methods in routine surgical pathology are underlined here, and the acceleration of the digital transformation of pathology departments and services is recommended.

AI-driven Discovery of Morphomolecular Signatures in Toxicology

Early identification of drug toxicity is essential yet challenging in drug development. At the preclinical stage, toxicity is assessed with histopathological examination of tissue sections from animal models to detect morphological lesions. To complement this analysis, toxicogenomics is increasingly employed to understand the mechanism of action of the compound and ultimately identify lesion-specific safety biomarkers for which in vitro assays can be designed. However, existing works that aim to identify morphological correlates of expression changes rely on qualitative or semi-quantitative morphological characterization and remain limited in scale or morphological diversity. Artificial intelligence (AI) offers a promising approach for quantitatively modeling this relationship at an unprecedented scale. Here, we introduce GEESE, an AI model designed to impute morphomolecular signatures in toxicology data. Our model was trained to predict 1,536 gene targets on a cohort of 8,231 hematoxylin and eosin-stained liver sections from Rattus norvegicus across 127 preclinical toxicity studies. The model, evaluated on 2,002 tissue sections from 29 held-out studies, can yield pseudo-spatially resolved gene expression maps, which we correlate with six key drug-induced liver injuries (DILI). From the resulting 25 million lesion-expression pairs, we established quantitative relations between up and downregulated genes and lesions. Validation of these signatures against toxicogenomic databases, pathway enrichment analyses, and human hepatocyte cell lines asserted their relevance. Overall, our study introduces new methods for characterizing toxicity at an unprecedented scale and granularity, paving the way for AI-driven discovery of toxicity biomarkers.

Gut microbiota-based machine-learning signature for the diagnosis of alcohol-associated and metabolic dysfunction-associated steatotic liver disease

Alcoholic-associated liver disease (ALD) and metabolic dysfunction-associated steatotic liver disease (MASLD) show a high prevalence rate worldwide. As gut microbiota represents current state of ALD and MASLD via gut-liver axis, typical characteristics of gut microbiota can be used as a potential diagnostic marker in ALD and MASLD. Machine learning (ML) algorithms improve diagnostic performance in various diseases. Using gut microbiota-based ML algorithms, we evaluated the diagnostic index for ALD and MASLD. Fecal 16S rRNA sequencing data of 263 ALD (control, elevated liver enzyme [ELE], cirrhosis, and hepatocellular carcinoma [HCC]) and 201 MASLD (control and ELE) subjects were collected. For external validation, 126 ALD and 84 MASLD subjects were recruited. Four supervised ML algorithms (support vector machine, random forest, multilevel perceptron, and convolutional neural network) were used for classification with 20, 40, 60, and 80 features, in which three nonsupervised ML algorithms (independent component analysis, principal component analysis, linear discriminant analysis, and random projection) were used for feature reduction. A total of 52 combinations of ML algorithms for each pair of subgroups were performed with 60 hyperparameter variations and Stratified ShuffleSplit tenfold cross validation. The ML models of the convolutional neural network combined with principal component analysis achieved areas under the receiver operating characteristic curve (AUCs) > 0.90. In ALD, the diagnostic AUC values of the ML strategy (vs. control) were 0.94, 0.97, and 0.96 for ELE, cirrhosis, and liver cancer, respectively. The AUC value (vs. control) for MASLD (ELE) was 0.93. In the external validation, the AUC values of ALD and MASLD (vs control) were > 0.90 and 0.88, respectively. The gut microbiota-based ML strategy can be used for the diagnosis of ALD and MASLD.ClinicalTrials.gov NCT04339725.

Mining the interpretable prognostic features from pathological image of intrahepatic cholangiocarcinoma using multi-modal deep learning

BACKGROUND

The advances in deep learning-based pathological image analysis have invoked tremendous insights into cancer prognostication. Still, lack of interpretability remains a significant barrier to clinical application.

METHODS

We established an integrative prognostic neural network for intrahepatic cholangiocarcinoma (iCCA), towards a comprehensive evaluation of both architectural and fine-grained information from whole-slide images. Then, leveraging on multi-modal data, we conducted extensive interrogative approaches to the models, to extract and visualize the morphological features that most correlated with clinical outcome and underlying molecular alterations.

RESULTS

The models were developed and optimized on 373 iCCA patients from our center and demonstrated consistent accuracy and robustness on both internal (n = 213) and external (n = 168) cohorts. The occlusion sensitivity map revealed that the distribution of tertiary lymphoid structures, the geometric traits of the invasive margin, the relative composition of tumor parenchyma and stroma, the extent of necrosis, the presence of the disseminated foci, and the tumor-adjacent micro-vessels were the determining architectural features that impacted on prognosis. Quantifiable morphological vector extracted by CellProfiler demonstrated that tumor nuclei from high-risk patients exhibited significant larger size, more distorted shape, with less prominent nuclear envelope and textural contrast. The multi-omics data (n = 187) further revealed key molecular alterations left morphological imprints that could be attended by the network, including glycolysis, hypoxia, apical junction, mTORC1 signaling, and immune infiltration.

CONCLUSIONS

We proposed an interpretable deep-learning framework to gain insights into the biological behavior of iCCA. Most of the significant morphological prognosticators perceived by the network are comprehensible to human minds.

Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology

Artificial Intelligence (AI)-based image analysis has immense potential to support diagnostic histopathology, including cancer diagnostics. However, developing supervised AI methods requires large-scale annotated datasets. A potentially powerful solution is to augment training data with synthetic data. Latent diffusion models, which can generate high-quality, diverse synthetic images, are promising. However, the most common implementations rely on detailed textual descriptions, which are not generally available in this domain. This work proposes a method that constructs structured textual prompts from automatically extracted image features. We experiment with the PCam dataset, composed of tissue patches only loosely annotated as healthy or cancerous. We show that including image-derived features in the prompt, as opposed to only healthy and cancerous labels, improves the Fréchet Inception Distance (FID) by 88.6. We also show that pathologists find it challenging to detect synthetic images, with a median sensitivity/specificity of 0.55/0.55. Finally, we show that synthetic data effectively train AI models.

Deep Learning Histology for Prediction of Lymph Node Metastases and Tumor Regression after Neoadjuvant FLOT Therapy of Gastroesophageal Adenocarcinoma

BACKGROUND

The aim of this study was to establish a deep learning prediction model for neoadjuvant FLOT chemotherapy response. The neural network utilized clinical data and visual information from whole-slide images (WSIs) of therapy-naïve gastroesophageal cancer biopsies.

METHODS

This study included 78 patients from the University Hospital of Cologne and 59 patients from the University Hospital of Heidelberg used as external validation.

RESULTS

After surgical resection, 33 patients from Cologne (42.3%) were ypN0 and 45 patients (57.7%) were ypN+, while 23 patients from Heidelberg (39.0%) were ypN0 and 36 patients (61.0%) were ypN+ (p = 0.695). The neural network had an accuracy of 92.1% to predict lymph node metastasis and the area under the curve (AUC) was 0.726. A total of 43 patients from Cologne (55.1%) had less than 50% residual vital tumor (RVT) compared to 34 patients from Heidelberg (57.6%, p = 0.955). The model was able to predict tumor regression with an error of ±14.1% and an AUC of 0.648.

CONCLUSIONS

This study demonstrates that visual features extracted by deep learning from therapy-naïve biopsies of gastroesophageal adenocarcinomas correlate with positive lymph nodes and tumor regression. The results will be confirmed in prospective studies to achieve early allocation of patients to the most promising treatment.

Matrix metalloproteinase 9 expression and glioblastoma survival prediction using machine learning on digital pathological images

This study aimed to apply pathomics to predict Matrix metalloproteinase 9 (MMP9) expression in glioblastoma (GBM) and investigate the underlying molecular mechanisms associated with pathomics. Here, we included 127 GBM patients, 78 of whom were randomly allocated to the training and test cohorts for pathomics modeling. The prognostic significance of MMP9 was assessed using Kaplan-Meier and Cox regression analyses. PyRadiomics was used to extract the features of H&E-stained whole slide images. Feature selection was performed using the maximum relevance and minimum redundancy (mRMR) and recursive feature elimination (RFE) algorithms. Prediction models were created using support vector machines (SVM) and logistic regression (LR). The performance was assessed using ROC analysis, calibration curve assessment, and decision curve analysis. MMP9 expression was elevated in patients with GBM. This was an independent prognostic factor for GBM. Six features were selected for the pathomics model. The area under the curves (AUCs) of the training and test subsets were 0.828 and 0.808, respectively, for the SVM model and 0.778 and 0.754, respectively, for the LR model. The C-index and calibration plots exhibited effective estimation abilities. The pathomics score calculated using the SVM model was highly correlated with overall survival time. These findings indicate that MMP9 plays a crucial role in GBM development and prognosis. Our pathomics model demonstrated high efficacy for predicting MMP9 expression levels and prognosis of patients with GBM.

Machine learning evaluation of immune infiltrate through digital tumour score allows prediction of survival outcome in a pooled analysis of three international stage III colon cancer cohorts

BACKGROUND

T-cell immune infiltrates are robust prognostic variables in localised colon cancer. Evaluation of prognosis using artificial intelligence is an emerging field. We evaluated whether machine learning analysis improved prediction of patient outcome in comparison with analysis of T cell infiltrate only or in association with clinical variables.

METHODS

We used data from two phase III clinical trials (Prodige-13 and PETACC08) and one retrospective Italian cohort (HARMONY). Cohorts were split into training (N = 692), internal validation (N = 297) and external validation (N = 672) sets. Tumour slides were stained with CD3mAb. CD3 Machine Learning (CD3ML) score was computed using graphical parameters within the tumour tiles obtained from CD3 slides. CD3 infiltrates in tumour core and invasive margin were automatically detected. Associations of CD3 infiltrates and CD3ML with 5-year Disease-Free Survival (DFS) were examined using univariate and multivariable survival models by Cox regression.

FINDINGS

CD3 density both in the invasive margin and the tumour core were significantly associated with DFS in the different sets. Similarly, CD3ML score was significantly associated with DFS in all sets. CD3 assessment did not provide added value on top of CD3ML assessment (Likelihood Ratio Test (LRT), p = 0.13). In contrast, CD3ML improved prediction of DFS when combined with a clinical risk stage (LRT, p = 0.001). Stratified by clinical risk score (High or Low), patients with low CD3ML score had better DFS.

INTERPRETATION

In all tested sets, machine learning analysis of tumour cells improved prediction of prognosis compared to clinical parameters. Adding tumour-infiltrating lymphocytes assessment did not improve prognostic determination.

FUNDING

This research received no external funding.

Opening the Black Box: Spatial Transcriptomics and the Relevance of Artificial Intelligence-Detected Prognostic Regions in High-Grade Serous Carcinoma

Image-based deep learning models are used to extract new information from standard hematoxylin and eosin pathology slides; however, biological interpretation of the features detected by artificial intelligence (AI) remains a challenge. High-grade serous carcinoma of the ovary (HGSC) is characterized by aggressive behavior and chemotherapy resistance, but also exhibits striking variability in outcome. Our understanding of this disease is limited, partly due to considerable tumor heterogeneity. We previously trained an AI model to identify HGSC tumor regions that are highly associated with outcome status but are indistinguishable by conventional morphologic methods. Here, we applied spatially resolved transcriptomics to further profile the AI-identified tumor regions in 16 patients (8 per outcome group) and identify molecular features related to disease outcome in patients who underwent primary debulking surgery and platinum-based chemotherapy. We examined formalin-fixed paraffin-embedded tissue from (1) regions identified by the AI model as highly associated with short or extended chemotherapy response, and (2) background tumor regions (not identified by the AI model as highly associated with outcome status) from the same tumors. We show that the transcriptomic profiles of AI-identified regions are more distinct than background regions from the same tumors, are superior in predicting outcome, and differ in several pathways including those associated with chemoresistance in HGSC. Further, we find that poor outcome and good outcome regions are enriched by different tumor subpopulations, suggesting distinctive interaction patterns. In summary, our work presents proof of concept that AI-guided spatial transcriptomic analysis improves recognition of biologic features relevant to patient outcomes.

Deceptive learning in histopathology

AIMS

Deep learning holds immense potential for histopathology, automating tasks that are simple for expert pathologists and revealing novel biology for tasks that were previously considered difficult or impossible to solve by eye alone. However, the extent to which the visual strategies learned by deep learning models in histopathological analysis are trustworthy or not has yet to be systematically analysed. Here, we systematically evaluate deep neural networks (DNNs) trained for histopathological analysis in order to understand if their learned strategies are trustworthy or deceptive.

METHODS AND RESULTS

We trained a variety of DNNs on a novel data set of 221 whole-slide images (WSIs) from lung adenocarcinoma patients, and evaluated their effectiveness at (1) molecular profiling of KRAS versus EGFR mutations, (2) determining the primary tissue of a tumour and (3) tumour detection. While DNNs achieved above-chance performance on molecular profiling, they did so by exploiting correlations between histological subtypes and mutations, and failed to generalise to a challenging test set obtained through laser capture microdissection (LCM). In contrast, DNNs learned robust and trustworthy strategies for determining the primary tissue of a tumour as well as detecting and localising tumours in tissue.

CONCLUSIONS

Our work demonstrates that DNNs hold immense promise for aiding pathologists in analysing tissue. However, they are also capable of achieving seemingly strong performance by learning deceptive strategies that leverage spurious correlations, and are ultimately unsuitable for research or clinical work. The framework we propose for model evaluation and interpretation is an important step towards developing reliable automated systems for histopathological analysis.

Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning

Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.

Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images

BACKGROUND AND OBJECTIVE

Cell segmentation in bright-field histological slides is a crucial topic in medical image analysis. Having access to accurate segmentation allows researchers to examine the relationship between cellular morphology and clinical observations. Unfortunately, most segmentation methods known today are limited to nuclei and cannot segment the cytoplasm.

METHODS

We present a new network architecture Cyto R-CNN that is able to accurately segment whole cells (with both the nucleus and the cytoplasm) in bright-field images. We also present a new dataset CytoNuke, consisting of multiple thousand manual annotations of head and neck squamous cell carcinoma cells. Utilizing this dataset, we compared the performance of Cyto R-CNN to other popular cell segmentation algorithms, including QuPath's built-in algorithm, StarDist, Cellpose and a multi-scale Attention Deeplabv3+. To evaluate segmentation performance, we calculated AP50, AP75 and measured 17 morphological and staining-related features for all detected cells. We compared these measurements to the gold standard of manual segmentation using the Kolmogorov-Smirnov test.

RESULTS

Cyto R-CNN achieved an AP50 of 58.65% and an AP75 of 11.56% in whole-cell segmentation, outperforming all other methods (QuPath 19.46/0.91%; StarDist 45.33/2.32%; Cellpose 31.85/5.61%, Deeplabv3+ 3.97/1.01%). Cell features derived from Cyto R-CNN showed the best agreement to the gold standard (D¯=0.15) outperforming QuPath (D¯=0.22), StarDist (D¯=0.25), Cellpose (D¯=0.23) and Deeplabv3+ (D¯=0.33).

CONCLUSION

Our newly proposed Cyto R-CNN architecture outperforms current algorithms in whole-cell segmentation while providing more reliable cell measurements than any other model. This could improve digital pathology workflows, potentially leading to improved diagnosis. Moreover, our published dataset can be used to develop further models in the future.

Overcoming data scarcity in biomedical imaging with a foundational multi-task model

Foundational models, pretrained on a large scale, have demonstrated substantial success across non-medical domains. However, training these models typically requires large, comprehensive datasets, which contrasts with the smaller and more specialized datasets common in biomedical imaging. Here we propose a multi-task learning strategy that decouples the number of training tasks from memory requirements. We trained a universal biomedical pretrained model (UMedPT) on a multi-task database including tomographic, microscopic and X-ray images, with various labeling strategies such as classification, segmentation and object detection. The UMedPT foundational model outperformed ImageNet pretraining and previous state-of-the-art models. For classification tasks related to the pretraining database, it maintained its performance with only 1% of the original training data and without fine-tuning. For out-of-domain tasks it required only 50% of the original training data. In an external independent validation, imaging features extracted using UMedPT proved to set a new standard for cross-center transferability.

[Artificial intelligence in kidney transplant pathology]

BACKGROUND

Artificial intelligence (AI) systems have showed promising results in digital pathology, including digital nephropathology and specifically also kidney transplant pathology.

AIM

Summarize the current state of research and limitations in the field of AI in kidney transplant pathology diagnostics and provide a future outlook.

MATERIALS AND METHODS

Literature search in PubMed and Web of Science using the search terms "deep learning", "transplant", and "kidney". Based on these results and studies cited in the identified literature, a selection was made of studies that have a histopathological focus and use AI to improve kidney transplant diagnostics.

RESULTS AND CONCLUSION

Many studies have already made important contributions, particularly to the automation of the quantification of some histopathological lesions in nephropathology. This likely can be extended to automatically quantify all relevant lesions for a kidney transplant, such as Banff lesions. Important limitations and challenges exist in the collection of representative data sets and the updates of Banff classification, making large-scale studies challenging. The already positive study results make future AI support in kidney transplant pathology appear likely.

Contrastive Pre-Training and Multiple Instance Learning for Predicting Tumor Microsatellite Instability

Accurate classification between tumor MicroSatellite Stability (MSS) and Instability (MSI) is crucial in gastrointestinal (GI) cancer prognosis and treatment. In this paper, we present a novel two-stage weakly supervised methodology, leveraging the synergy of Multiple Instance Learning (MIL) and a unique Contrastive Clustering Network (CCNet), aimed at enhancing MSI prediction in Whole Slide Image (WSI) analysis of GI cancers. In our framework, we utilize a contrastive learning-based feature extractor, coupled with MIL's efficient labeling. Our approach shows notable improvement in MSI classification, outperforming existing methods. Experiments using colorectal cancer and stomach adenocarcinoma datasets demonstrate the model's efficacy and generalizability, marking an advance in computational pathology and cancer diagnostics. Furthermore, we explored the efficacy of transfer learning using our model, examining the performance of pretrained feature extractors from ImageNet and STAD datasets. Our framework outperforms existing methods when pretrained on STAD and transferred to CRC data.Clinical relevance- The potential to enhance the accuracy of gastrointestinal cancer diagnosis and prognosis through advanced machine learning techniques. By leveraging transfer learning and weakly supervised frameworks clinicians can benefit from improved MSI prediction in histopathological images, aiding in personalized treatment strategies and patient outcomes.

A fully automated and explainable algorithm for predicting malignant transformation in oral epithelial dysplasia

Oral epithelial dysplasia (OED) is a premalignant histopathological diagnosis given to lesions of the oral cavity. Its grading suffers from significant inter-/intra-observer variability, and does not reliably predict malignancy progression, potentially leading to suboptimal treatment decisions. To address this, we developed an artificial intelligence (AI) algorithm, that assigns an Oral Malignant Transformation (OMT) risk score based on the Haematoxylin and Eosin (H&E) stained whole slide images (WSIs). Our AI pipeline leverages an in-house segmentation model to detect and segment both nuclei and epithelium. Subsequently, a shallow neural network utilises interpretable morphological and spatial features, emulating histological markers, to predict progression. We conducted internal cross-validation on our development cohort (Sheffield; n = 193 cases) and independent validation on two external cohorts (Birmingham and Belfast; n = 89 cases). On external validation, the proposed OMTscore achieved an AUROC = 0.75 (Recall = 0.92) in predicting OED progression, outperforming other grading systems (Binary: AUROC = 0.72, Recall = 0.85). Survival analyses showed the prognostic value of our OMTscore (C-index = 0.60, p = 0.02), compared to WHO (C-index = 0.64, p = 0.003) and binary grades (C-index = 0.65, p < 0.001). Nuclear analyses elucidated the presence of peri-epithelial and intra-epithelial lymphocytes in highly predictive patches of transforming cases (p < 0.001). This is the first study to propose a completely automated, explainable, and externally validated algorithm for predicting OED transformation. Our algorithm shows comparable-to-human-level performance, offering a promising solution to the challenges of grading OED in routine clinical practice.

AI-based histopathology image analysis reveals a distinct subset of endometrial cancers

Endometrial cancer (EC) has four molecular subtypes with strong prognostic value and therapeutic implications. The most common subtype (NSMP; No Specific Molecular Profile) is assigned after exclusion of the defining features of the other three molecular subtypes and includes patients with heterogeneous clinical outcomes. In this study, we employ artificial intelligence (AI)-powered histopathology image analysis to differentiate between p53abn and NSMP EC subtypes and consequently identify a sub-group of NSMP EC patients that has markedly inferior progression-free and disease-specific survival (termed 'p53abn-like NSMP'), in a discovery cohort of 368 patients and two independent validation cohorts of 290 and 614 from other centers. Shallow whole genome sequencing reveals a higher burden of copy number abnormalities in the 'p53abn-like NSMP' group compared to NSMP, suggesting that this group is biologically distinct compared to other NSMP ECs. Our work demonstrates the power of AI to detect prognostically different and otherwise unrecognizable subsets of EC where conventional and standard molecular or pathologic criteria fall short, refining image-based tumor classification. This study's findings are applicable exclusively to females.

PET/CT deep learning prognosis for treatment decision support in esophageal squamous cell carcinoma

OBJECTIVES

The clinical decision-making regarding choosing surgery alone (SA) or surgery followed by postoperative adjuvant chemotherapy (SPOCT) in esophageal squamous cell carcinoma (ESCC) remains controversial. We aim to propose a pre-therapy PET/CT image-based deep learning approach to improve the survival benefit and clinical management of ESCC patients.

METHODS

This retrospective multicenter study included 837 ESCC patients from three institutions. Prognostic biomarkers integrating six networks were developed to build an ESCC prognosis (ESCCPro) model and predict the survival probability of ESCC patients treated with SA and SPOCT. Patients who did not undergo surgical resection were in a control group. Overall survival (OS) was the primary end-point event. The expected improvement in survival prognosis with the application of ESCCPro to assign treatment protocols was estimated by comparing the survival of patients in each subgroup. Seven clinicians with varying experience evaluated how ESCCPro performed in assisting clinical decision-making.

RESULTS

In this retrospective multicenter study, patients receiving SA had a median OS 9.2 months longer than controls. No significant differences in survival were found between SA patients with predicted poor outcomes and the controls (p > 0.05). It was estimated that if ESCCPro was used to determine SA and SPOCT eligibility, the median OS in the ESCCPro-recommended SA group and SPOCT group would have been 15.3 months and 24.9 months longer, respectively. In addition, ESCCPro also significantly improved prognosis accuracy, certainty, and the efficiency of clinical experts.

CONCLUSION

ESCCPro assistance improved the survival benefit of ESCC patients and the clinical decision-making among the two treatment approaches.

CRITICAL RELEVANCE STATEMENT

The ESCCPro model for treatment decision-making is promising to improve overall survival in ESCC patients undergoing surgical resection and patients undergoing surgery followed by postoperative adjuvant chemotherapy.

KEY POINTS

ESCC is associated with a poor prognosis and unclear ideal treatments. ESCCPro predicts the survival of patients with ESCC and the expected benefit from SA. ESCCPro improves clinicians' stratification of patients' prognoses.

Diagnosis to dissection: AI's role in early detection and surgical intervention for gastric cancer

Gastric cancer remains a formidable health challenge worldwide; early detection and effective surgical intervention are critical for improving patient outcomes. This comprehensive review explores the evolving landscape of gastric cancer management, emphasizing the significant contributions of artificial intelligence (AI) in revolutionizing both diagnostic and therapeutic approaches. Despite advancements in the medical field, the subtle nature of early gastric cancer symptoms often leads to late-stage diagnoses, where survival rates are notably decreased. Historically, the treatment of gastric cancer has transitioned from palliative care to surgical resection, evolving further with the introduction of minimally invasive surgical (MIS) techniques. In the current era, AI has emerged as a transformative force, enhancing the precision of early gastric cancer detection through sophisticated image analysis, and supporting surgical decision-making with predictive modeling and real-time preop-, intraop-, and postoperative guidance. However, the deployment of AI in healthcare raises significant ethical, legal, and practical challenges, including the necessity for ongoing professional education and the development of standardized protocols to ensure patient safety and the effective use of AI technologies. Future directions point toward a synergistic integration of AI with clinical best practices, promising a new era of personalized, efficient, and safer gastric cancer management.

Progress and prospects of biomarker-based targeted therapy and immune checkpoint inhibitors in advanced gastric cancer

Gastric cancer and gastroesophageal junction cancer represent the leading cause of tumor-related death worldwide. Although advances in immunotherapy and molecular targeted therapy have expanded treatment options, they have not significantly altered the prognosis for patients with unresectable or metastatic gastric cancer. A minority of patients, particularly those with PD-L1-positive, HER-2-positive, or MSI-high tumors, may benefit more from immune checkpoint inhibitors and/or HER-2-directed therapies in advanced stages. However, for those lacking specific targets and unique molecular features, conventional chemotherapy remains the only recommended effective and durable regimen. In this review, we summarize the roles of various signaling pathways and further investigate the available targets. Then, the current results of phase II/III clinical trials in advanced gastric cancer, along with the superiorities and limitations of the existing biomarkers, are specifically discussed. Finally, we will offer our insights in precision treatment pattern when encountering the substantial challenges.

Artificial Intelligence in Colorectal Cancer: From Patient Screening over Tailoring Treatment Decisions to Identification of Novel Biomarkers

BACKGROUND

Artificial intelligence (AI) is increasingly entering and transforming not only medical research but also clinical practice. In the last 10 years, new AI methods have enabled computers to perform visual tasks, reaching high performance and thereby potentially supporting and even outperforming human experts. This is in particular relevant for colorectal cancer (CRC), which is the 3rd most common cancer type in general, as along the CRC patient journey many complex visual tasks need to be performed: from endoscopy over imaging to histopathology; the screening, diagnosis, and treatment of CRC involve visual image analysis tasks.

SUMMARY

In all these clinical areas, AI models have shown promising results by supporting physicians, improving accuracy, and providing new biological insights and biomarkers. By predicting prognostic and predictive biomarkers from routine images/slides, AI models could lead to an improved patient stratification for precision oncology approaches in the near future. Moreover, it is conceivable that AI models, in particular together with innovative techniques such as single-cell or spatial profiling, could help identify novel clinically as well as biologically meaningful biomarkers that could pave the way to new therapeutic approaches.

KEY MESSAGES

Here, we give a comprehensive overview of AI in colorectal cancer, describing and discussing these developments as well as the next steps which need to be taken to incorporate AI methods more broadly into the clinical care of CRC.

Automated classification of cellular expression in multiplexed imaging data with Nimbus

Multiplexed imaging offers a powerful approach to characterize the spatial topography of tissues in both health and disease. To analyze such data, the specific combination of markers that are present in each cell must be enumerated to enable accurate phenotyping, a process that often relies on unsupervised clustering. We constructed the Pan-Multiplex (Pan-M) dataset containing 197 million distinct annotations of marker expression across 15 different cell types. We used Pan-M to create Nimbus, a deep learning model to predict marker positivity from multiplexed image data. Nimbus is a pre-trained model that uses the underlying images to classify marker expression across distinct cell types, from different tissues, acquired using different microscope platforms, without requiring any retraining. We demonstrate that Nimbus predictions capture the underlying staining patterns of the full diversity of markers present in Pan-M. We then show how Nimbus predictions can be integrated with downstream clustering algorithms to robustly identify cell subtypes in image data. We have open-sourced Nimbus and Pan-M to enable community use at https://github.com/angelolab/Nimbus-Inference.

3-Dimensional Reconstruction From Histopathological Sections: A Systematic Review

Although pathological tissue analysis is typically performed on single 2-dimensional (2D) histologic reference slides, 3-dimensional (3D) reconstruction from a sequence of histologic sections could provide novel opportunities for spatial analysis of the extracted tissue. In this review, we analyze recent works published after 2018 and report information on the extracted tissue types, the section thickness, and the number of sections used for reconstruction. By analyzing the technological requirements for 3D reconstruction, we observe that software tools exist, both free and commercial, which include the functionality to perform 3D reconstruction from a sequence of histologic images. Through the analysis of the most recent works, we provide an overview of the workflows and tools that are currently used for 3D reconstruction from histologic sections and address points for future work, such as a missing common file format or computer-aided analysis of the reconstructed model.

A guide to artificial intelligence for cancer researchers

Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.

Precision medicine in colorectal cancer: Leveraging multi-omics, spatial omics, and artificial intelligence

Colorectal cancer (CRC) is a leading cause of cancer-related deaths. Recent advancements in genomic technologies and analytical approaches have revolutionized CRC research, enabling precision medicine. This review highlights the integration of multi-omics, spatial omics, and artificial intelligence (AI) in advancing precision medicine for CRC. Multi-omics approaches have uncovered molecular mechanisms driving CRC progression, while spatial omics have provided insights into the spatial heterogeneity of gene expression in CRC tissues. AI techniques have been utilized to analyze complex datasets, identify new treatment targets, and enhance diagnosis and prognosis. Despite the tumor's heterogeneity and genetic and epigenetic complexity, the fusion of multi-omics, spatial omics, and AI shows the potential to overcome these challenges and advance precision medicine in CRC. The future lies in integrating these technologies to provide deeper insights and enable personalized therapies for CRC patients.

Automated curation of large-scale cancer histopathology image datasets using deep learning

BACKGROUND

Artificial intelligence (AI) has numerous applications in pathology, supporting diagnosis and prognostication in cancer. However, most AI models are trained on highly selected data, typically one tissue slide per patient. In reality, especially for large surgical resection specimens, dozens of slides can be available for each patient. Manually sorting and labelling whole-slide images (WSIs) is a very time-consuming process, hindering the direct application of AI on the collected tissue samples from large cohorts. In this study we addressed this issue by developing a deep-learning (DL)-based method for automatic curation of large pathology datasets with several slides per patient.

METHODS

We collected multiple large multicentric datasets of colorectal cancer histopathological slides from the United Kingdom (FOXTROT, N = 21,384 slides; CR07, N = 7985 slides) and Germany (DACHS, N = 3606 slides). These datasets contained multiple types of tissue slides, including bowel resection specimens, endoscopic biopsies, lymph node resections, immunohistochemistry-stained slides, and tissue microarrays. We developed, trained, and tested a deep convolutional neural network model to predict the type of slide from the slide overview (thumbnail) image. The primary statistical endpoint was the macro-averaged area under the receiver operating curve (AUROCs) for detection of the type of slide.

RESULTS

In the primary dataset (FOXTROT), with an AUROC of 0.995 [95% confidence interval [CI]: 0.994-0.996] the algorithm achieved a high classification performance and was able to accurately predict the type of slide from the thumbnail image alone. In the two external test cohorts (CR07, DACHS) AUROCs of 0.982 [95% CI: 0.979-0.985] and 0.875 [95% CI: 0.864-0.887] were observed, which indicates the generalizability of the trained model on unseen datasets. With a confidence threshold of 0.95, the model reached an accuracy of 94.6% (7331 classified cases) in CR07 and 85.1% (2752 classified cases) for the DACHS cohort.

CONCLUSION

Our findings show that using the low-resolution thumbnail image is sufficient to accurately classify the type of slide in digital pathology. This can support researchers to make the vast resource of existing pathology archives accessible to modern AI models with only minimal manual annotations.

Challenges and perspectives in use of artificial intelligence to support treatment recommendations in clinical oncology

Artificial intelligence (AI) promises to be the next revolutionary step in modern society. Yet, its role in all fields of industry and science need to be determined. One very promising field is represented by AI-based decision-making tools in clinical oncology leading to more comprehensive, personalized therapy approaches. In this review, the authors provide an overview on all relevant technical applications of AI in oncology, which are required to understand the future challenges and realistic perspectives for decision-making tools. In recent years, various applications of AI in medicine have been developed focusing on the analysis of radiological and pathological images. AI applications encompass large amounts of complex data supporting clinical decision-making and reducing errors by objectively quantifying all aspects of the data collected. In clinical oncology, almost all patients receive a treatment recommendation in a multidisciplinary cancer conference at the beginning and during their treatment periods. These highly complex decisions are based on a large amount of information (of the patients and of the various treatment options), which need to be analyzed and correctly classified in a short time. In this review, the authors describe the technical and medical requirements of AI to address these scientific challenges in a multidisciplinary manner. Major challenges in the use of AI in oncology and decision-making tools are data security, data representation, and explainability of AI-based outcome predictions, in particular for decision-making processes in multidisciplinary cancer conferences. Finally, limitations and potential solutions are described and compared for current and future research attempts.

Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathology

In the spectrum of colorectal tumors, microsatellite-stable (MSS) tumors with DNA polymerase ε (POLE) mutations exhibit a hypermutated profile, holding the potential to respond to immunotherapy similarly to their microsatellite-instable (MSI) counterparts. Yet, due to their rarity and the associated testing costs, systematic screening for these mutations is not commonly pursued. Notably, the histopathological phenotype resulting from POLE mutations is theorized to resemble that of MSI. This resemblance not only could facilitate their detection by a transformer-based Deep Learning (DL) system trained on MSI pathology slides, but also indicates the possibility for MSS patients with POLE mutations to access enhanced treatment options, which might otherwise be overlooked. To harness this potential, we trained a Deep Learning classifier on a large dataset with the ground truth for microsatellite status and subsequently validated its capabilities for MSI and POLE detection across three external cohorts. Our model accurately identified MSI status in both the internal and external resection cohorts using pathology images alone. Notably, with a classification threshold of 0.5, over 75% of POLE driver mutant patients in the external resection cohorts were flagged as "positive" by a DL system trained on MSI status. In a clinical setting, deploying this DL model as a preliminary screening tool could facilitate the efficient identification of clinically relevant MSI and POLE mutations in colorectal tumors, in one go.

Histopathologic image-based deep learning classifier for predicting platinum-based treatment responses in high-grade serous ovarian cancer

Platinum-based chemotherapy is the cornerstone treatment for female high-grade serous ovarian carcinoma (HGSOC), but choosing an appropriate treatment for patients hinges on their responsiveness to it. Currently, no available biomarkers can promptly predict responses to platinum-based treatment. Therefore, we developed the Pathologic Risk Classifier for HGSOC (PathoRiCH), a histopathologic image-based classifier. PathoRiCH was trained on an in-house cohort (n = 394) and validated on two independent external cohorts (n = 284 and n = 136). The PathoRiCH-predicted favorable and poor response groups show significantly different platinum-free intervals in all three cohorts. Combining PathoRiCH with molecular biomarkers provides an even more powerful tool for the risk stratification of patients. The decisions of PathoRiCH are explained through visualization and a transcriptomic analysis, which bolster the reliability of our model's decisions. PathoRiCH exhibits better predictive performance than current molecular biomarkers. PathoRiCH will provide a solid foundation for developing an innovative tool to transform the current diagnostic pipeline for HGSOC.

Development and interpretation of a pathomics-driven ensemble model for predicting the response to immunotherapy in gastric cancer

BACKGROUND

Only a subset of patients with gastric cancer experience long-term benefits from immune checkpoint inhibitors (ICIs). Currently, there is a deficiency in precise predictive biomarkers for ICI efficacy. The aim of this study was to develop and validate a pathomics-driven ensemble model for predicting the response to ICIs in gastric cancer, using H&E-stained whole slide images (WSI).

METHODS

This multicenter study retrospectively collected and analyzed H&E-stained WSIs and clinical data from 584 patients with gastric cancer. An ensemble model, integrating four classifiers: least absolute shrinkage and selection operator, k-nearest neighbors, decision trees, and random forests, was developed and validated using pathomics features, with the objective of predicting the therapeutic efficacy of immune checkpoint inhibition. Model performance was evaluated using metrics including the area under the curve (AUC), sensitivity, and specificity. Additionally, SHAP (SHapley Additive exPlanations) analysis was used to explain the model's predicted values as the sum of the attribution values for each input feature. Pathogenomics analysis was employed to explain the molecular mechanisms underlying the model's predictions.

RESULTS

Our pathomics-driven ensemble model effectively stratified the response to ICIs in training cohort (AUC 0.985 (95% CI 0.971 to 0.999)), which was further validated in internal validation cohort (AUC 0.921 (95% CI 0.839 to 0.999)), as well as in external validation cohort 1 (AUC 0.914 (95% CI 0.837 to 0.990)), and external validation cohort 2 (0.927 (95% CI 0.802 to 0.999)). The univariate Cox regression analysis revealed that the prediction signature of pathomics-driven ensemble model was a prognostic factor for progression-free survival in patients with gastric cancer who underwent immunotherapy (p<0.001, HR 0.35 (95% CI 0.24 to 0.50)), and remained an independent predictor after multivariable Cox regression adjusted for clinicopathological variables, (including sex, age, carcinoembryonic antigen, carbohydrate antigen 19-9, therapy regime, line of therapy, differentiation, location and programmed death ligand 1 (PD-L1) expression in all patients (p<0.001, HR 0.34 (95% CI 0.24 to 0.50)). Pathogenomics analysis suggested that the ensemble model is driven by molecular-level immune, cancer, metabolism-related pathways, and was correlated with the immune-related characteristics, including immune score, Estimation of STromal and Immune cells in MAlignant Tumor tissues using Expression data score, and tumor purity.

CONCLUSIONS

Our pathomics-driven ensemble model exhibited high accuracy and robustness in predicting the response to ICIs using WSIs. Therefore, it could serve as a novel and valuable tool to facilitate precision immunotherapy.

Tissue Elasticity as a Diagnostic Marker of Molecular Mutations in Morphologically Heterogeneous Colorectal Cancer

The presence of molecular mutations in colorectal cancer (CRC) is a decisive factor in selecting the most effective first-line therapy. However, molecular analysis is routinely performed only in a limited number of patients with remote metastases. We propose to use tissue stiffness as a marker of the presence of molecular mutations in CRC samples. For this purpose, we applied compression optical coherence elastography (C-OCE) to calculate stiffness values in regions corresponding to specific CRC morphological patterns (n = 54). In parallel to estimating stiffness, molecular analysis from the same zones was performed to establish their relationships. As a result, a high correlation between the presence of KRAS/NRAS/BRAF driver mutations and high stiffness values was revealed regardless of CRC morphological pattern type. Further, we proposed threshold stiffness values for label-free targeted detection of molecular alterations in CRC tissues: for KRAS, NRAS, or BRAF driver mutation-above 803 kPa (sensitivity-91%; specificity-80%; diagnostic accuracy-85%), and only for KRAS driver mutation-above 850 kPa (sensitivity-90%; specificity-88%; diagnostic accuracy-89%). To conclude, C-OCE estimation of tissue stiffness can be used as a clinical diagnostic tool for preliminary screening of genetic burden in CRC tissues.

Regulatory T cells as crucial trigger and potential target for hyperprogressive disease subsequent to PD-1/PD-L1 blockade for cancer treatment

PD-1/PD-L1 blockade therapy has brought great success to cancer treatment. Nevertheless, limited beneficiary populations and even hyperprogressive disease (HPD) greatly constrain the application of PD-1/PD-L1 inhibitors in clinical treatment. HPD is a special pattern of disease progression with rapid tumor growth and even serious consequences of patient death, which requires urgent attention. Among the many predisposing causes of HPD, regulatory T cells (Tregs) are suspected because they are amplified in cases of HPD. Tregs express PD-1 thus PD-1/PD-L1 blockade therapy may have an impact on Tregs which leads to HPD. Tregs are a subset of CD4+ T cells expressing FoxP3 and play critical roles in suppressing immunity. Tregs migrate toward tumors in the presence of chemokines to suppress antitumor immune responses, causing cancer cells to grow and proliferate. Studies have shown that deleting Tregs could enhance the efficacy of PD-1/PD-L1 blockade therapy and reduce the occurrence of HPD. This suggests that immunotherapy combined with Treg depletion may be an effective means of avoiding HPD. In this review, we summarized the immunosuppressive-related functions of Tregs in antitumor therapy and focused on advances in therapy combining Tregs depletion with PD-1/PD-L1 blockade in clinical studies. Moreover, we provided an outlook on Treg-targeted HPD early warning for PD-1/PD-L1 blockade therapy.

Prediction of Mismatch Repair Status in Endometrial Cancer from Histological Slide Images Using Various Deep Learning-Based Algorithms

The application of deep learning algorithms to predict the molecular profiles of various cancers from digital images of hematoxylin and eosin (H&E)-stained slides has been reported in recent years, mainly for gastric and colon cancers. In this study, we investigated the potential use of H&E-stained endometrial cancer slide images to predict the associated mismatch repair (MMR) status. H&E-stained slide images were collected from 127 cases of the primary lesion of endometrial cancer. After digitization using a Nanozoomer virtual slide scanner (Hamamatsu Photonics), we segmented the scanned images into 5397 tiles of 512 × 512 pixels. The MMR proteins (PMS2, MSH6) were immunohistochemically stained, classified into MMR proficient/deficient, and annotated for each case and tile. We trained several neural networks, including convolutional and attention-based networks, using tiles annotated with the MMR status. Among the tested networks, ResNet50 exhibited the highest area under the receiver operating characteristic curve (AUROC) of 0.91 for predicting the MMR status. The constructed prediction algorithm may be applicable to other molecular profiles and useful for pre-screening before implementing other, more costly genetic profiling tests.

A multicenter proof-of-concept study on deep learning-based intraoperative discrimination of primary central nervous system lymphoma

Accurate intraoperative differentiation of primary central nervous system lymphoma (PCNSL) remains pivotal in guiding neurosurgical decisions. However, distinguishing PCNSL from other lesions, notably glioma, through frozen sections challenges pathologists. Here we sought to develop and validate a deep learning model capable of precisely distinguishing PCNSL from non-PCNSL lesions, especially glioma, using hematoxylin and eosin (H&E)-stained frozen whole-slide images. Also, we compared its performance against pathologists of varying expertise. Additionally, a human-machine fusion approach integrated both model and pathologic diagnostics. In external cohorts, LGNet achieved AUROCs of 0.965 and 0.972 in distinguishing PCNSL from glioma and AUROCs of 0.981 and 0.993 in differentiating PCNSL from non-PCNSL lesions. Outperforming several pathologists, LGNet significantly improved diagnostic performance, further augmented to some extent by fusion approach. LGNet's proficiency in frozen section analysis and its synergy with pathologists indicate its valuable role in intraoperative diagnosis, particularly in discriminating PCNSL from glioma, alongside other lesions.

A diagnostic strategy for pulmonary fat embolism based on routine H&E staining using computational pathology

Pulmonary fat embolism (PFE) as a cause of death often occurs in trauma cases such as fractures and soft tissue contusions. Traditional PFE diagnosis relies on subjective methods and special stains like oil red O. This study utilizes computational pathology, combining digital pathology and deep learning algorithms, to precisely quantify fat emboli in whole slide images using conventional hematoxylin-eosin (H&E) staining. The results demonstrate deep learning's ability to identify fat droplet morphology in lung microvessels, achieving an area under the receiver operating characteristic (ROC) curve (AUC) of 0.98. The AI-quantified fat globules generally matched the Falzi scoring system with oil red O staining. The relative quantity of fat emboli against lung area was calculated by the algorithm, determining a diagnostic threshold of 8.275% for fatal PFE. A diagnostic strategy based on this threshold achieved a high AUC of 0.984, similar to manual identification with special stains but surpassing H&E staining. This demonstrates computational pathology's potential as an affordable, rapid, and precise method for fatal PFE diagnosis in forensic practice.

A systematic review of machine learning-based tumor-infiltrating lymphocytes analysis in colorectal cancer: Overview of techniques, performance metrics, and clinical outcomes

The incidence of colorectal cancer (CRC), one of the deadliest cancers around the world, is increasing. Tissue microenvironment (TME) features such as tumor-infiltrating lymphocytes (TILs) can have a crucial impact on diagnosis or decision-making for treating patients with CRC. While clinical studies showed that TILs improve the host immune response, leading to a better prognosis, inter-observer agreement for quantifying TILs is not perfect. Incorporating machine learning (ML) based applications in clinical routine may promote diagnosis reliability. Recently, ML has shown potential for making progress in routine clinical procedures. We aim to systematically review the TILs analysis based on ML in CRC histological images. Deep learning (DL) and non-DL techniques can aid pathologists in identifying TILs, and automated TILs are associated with patient outcomes. However, a large multi-institutional CRC dataset with a diverse and multi-ethnic population is necessary to generalize ML methods.

Advancing Artificial Intelligence Integration Into the Pathology Workflow: Exploring Opportunities in Gastrointestinal Tract Biopsies

This review aims to present a comprehensive overview of the current landscape of artificial intelligence (AI) applications in the analysis of tubular gastrointestinal biopsies. These publications cover a spectrum of conditions, ranging from inflammatory ailments to malignancies. Moving beyond the conventional diagnosis based on hematoxylin and eosin-stained whole-slide images, the review explores additional implications of AI, including its involvement in interpreting immunohistochemical results, molecular subtyping, and the identification of cellular spatial biomarkers. Furthermore, the review examines how AI can contribute to enhancing the quality and control of diagnostic processes, introducing new workflow options, and addressing the limitations and caveats associated with current AI platforms in this context.

[Artificial intelligence for pathology-how, where, and why?]

Artificial intelligence promises many innovations and simplifications in pathology, but also raises just as many questions and uncertainties. In this article, we provide a brief overview of the current status, the goals already achieved by existing algorithms, and the remaining challenges.

Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions

Artificial intelligence (AI) in oncology is advancing beyond algorithm development to integration into clinical practice. This review describes the current state of the field, with a specific focus on clinical integration. AI applications are structured according to cancer type and clinical domain, focusing on the four most common cancers and tasks of detection, diagnosis, and treatment. These applications encompass various data modalities, including imaging, genomics, and medical records. We conclude with a summary of existing challenges, evolving solutions, and potential future directions for the field.

SIGNIFICANCE

AI is increasingly being applied to all aspects of oncology, where several applications are maturing beyond research and development to direct clinical integration. This review summarizes the current state of the field through the lens of clinical translation along the clinical care continuum. Emerging areas are also highlighted, along with common challenges, evolving solutions, and potential future directions for the field.

Computational Pathology for Prediction of Isocitrate Dehydrogenase Gene Mutation from Whole Slide Images in Adult Patients with Diffuse Glioma

Isocitrate dehydrogenase gene (IDH) mutation is one of the most important molecular markers of glioma. Accurate detection of IDH status is a crucial step for integrated diagnosis of adult-type diffuse gliomas. Herein, a clustering-based hybrid of a convolutional neural network and a vision transformer deep learning model was developed to detect IDH mutation status from annotation-free hematoxylin and eosin-stained whole slide pathologic images of 2275 adult patients with diffuse gliomas. For comparison, a pure convolutional neural network, a pure vision transformer, and a classic multiple-instance learning model were also assessed. The hybrid model achieved an area under the receiver operating characteristic curve of 0.973 in the validation set and 0.953 in the external test set, outperforming the other models. The hybrid model's ability in IDH detection between difficult subgroups with different IDH status but shared histologic features, achieving areas under the receiver operating characteristic curve ranging from 0.850 to 0.985 in validation and test sets. These data suggest that the proposed hybrid model has a potential to be used as a computational pathology tool for preliminary rapid detection of IDH mutation from whole slide images in adult patients with diffuse gliomas.

Multi-scale feature fusion for prediction of IDH1 mutations in glioma histopathological images

BACKGROUND AND OBJECTIVE

Mutations in isocitrate dehydrogenase 1 (IDH1) play a crucial role in the prognosis, diagnosis, and treatment of gliomas. However, current methods for determining its mutation status, such as immunohistochemistry and gene sequencing, are difficult to implement widely in routine clinical diagnosis. Recent studies have shown that using deep learning methods based on pathological images of glioma can predict the mutation status of the IDH1 gene. However, our research focuses on utilizing multi-scale information in pathological images to improve the accuracy of predicting IDH1 gene mutations, thereby providing an accurate and cost-effective prediction method for routine clinical diagnosis.

METHODS

In this paper, we propose a multi-scale fusion gene identification network (MultiGeneNet). The network first uses two feature extractors to obtain feature maps at different scale images, and then by employing a bilinear pooling layer based on Hadamard product to realize the fusion of multi-scale features. Through fully exploiting the complementarity among features at different scales, we are able to obtain a more comprehensive and rich representation of multi-scale features.

RESULTS

Based on the Hematoxylin and Eosin stained pathological section dataset of 296 patients, our method achieved an accuracy of 83.575 % and an AUC of 0.886, thus significantly outperforming other single-scale methods.

CONCLUSIONS

Our method can be deployed in medical aid systems at very low cost, serving as a diagnostic or prognostic tool for glioma patients in medically underserved areas.