Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Decoding pathology: the role of computational pathology in research and diagnostics

Traditional histopathology, characterized by manual quantifications and assessments, faces challenges such as low-throughput and inter-observer variability that hinder the introduction of precision medicine in pathology diagnostics and research. The advent of digital pathology allowed the introduction of computational pathology, a discipline that leverages computational methods, especially based on deep learning (DL) techniques, to analyze histopathology specimens. A growing body of research shows impressive performances of DL-based models in pathology for a multitude of tasks, such as mutation prediction, large-scale pathomics analyses, or prognosis prediction. New approaches integrate multimodal data sources and increasingly rely on multi-purpose foundation models. This review provides an introductory overview of advancements in computational pathology and discusses their implications for the future of histopathology in research and diagnostics.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

Combining Artificial Intelligence and Simplified Image Processing for the Automatic Detection of Mycobacterium tuberculosis in Acid-fast Stain : A Cross-institute Training and Validation Study

Tuberculosis (TB) poses a significant health threat in Taiwan, necessitating efficient detection methods. Traditional screening for acid-fast positive bacilli in acid-fast stain is time-consuming and prone to human error due to staining artifacts. To address this, we present an automated TB detection platform leveraging deep learning and image processing. Whole slide images from 2 hospitals were collected and processed on a high-performance system. The system utilizes an image processing technique to highlight red, rod-like regions and a modified EfficientNet model for binary classification of TB-positive regions. Our approach achieves a 97% accuracy in tile-based TB image classification, with minimal loss during the image processing step. By setting a 0.99 threshold, false positives are significantly reduced, resulting in a 94% detection rate when assisting pathologists, compared with 68% without artificial intelligence assistance. Notably, our system efficiently identifies artifacts and contaminants, addressing challenges in digital slide interpretation. Cross-hospital validation demonstrates the system's adaptability. The proposed artificial intelligence-assisted pipeline improves both detection rates and time efficiency, making it a promising tool for routine pathology work in TB detection.

Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning

Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.

Computational Pathology for Prediction of Isocitrate Dehydrogenase Gene Mutation from Whole Slide Images in Adult Patients with Diffuse Glioma

Isocitrate dehydrogenase gene (IDH) mutation is one of the most important molecular markers of glioma. Accurate detection of IDH status is a crucial step for integrated diagnosis of adult-type diffuse gliomas. Herein, a clustering-based hybrid of a convolutional neural network and a vision transformer deep learning model was developed to detect IDH mutation status from annotation-free hematoxylin and eosin-stained whole slide pathologic images of 2275 adult patients with diffuse gliomas. For comparison, a pure convolutional neural network, a pure vision transformer, and a classic multiple-instance learning model were also assessed. The hybrid model achieved an area under the receiver operating characteristic curve of 0.973 in the validation set and 0.953 in the external test set, outperforming the other models. The hybrid model's ability in IDH detection between difficult subgroups with different IDH status but shared histologic features, achieving areas under the receiver operating characteristic curve ranging from 0.850 to 0.985 in validation and test sets. These data suggest that the proposed hybrid model has a potential to be used as a computational pathology tool for preliminary rapid detection of IDH mutation from whole slide images in adult patients with diffuse gliomas.

A novel transformer-based aggregation model for predicting gene mutations in lung adenocarcinoma

In recent years, predicting gene mutations on whole slide imaging (WSI) has gained prominence. The primary challenge is extracting global information and achieving unbiased semantic aggregation. To address this challenge, we propose a novel Transformer-based aggregation model, employing a self-learning weight aggregation mechanism to mitigate semantic bias caused by the abundance of features in WSI. Additionally, we adopt a random patch training method, which enhances model learning richness by randomly extracting feature vectors from WSI, thus addressing the issue of limited data. To demonstrate the model's effectiveness in predicting gene mutations, we leverage the lung adenocarcinoma dataset from Shandong Provincial Hospital for prior knowledge learning. Subsequently, we assess TP53, CSMD3, LRP1B, and TTN gene mutations using lung adenocarcinoma tissue pathology images and clinical data from The Cancer Genome Atlas (TCGA). The results indicate a notable increase in the AUC (Area Under the ROC Curve) value, averaging 4%, attesting to the model's performance improvement. Our research offers an efficient model to explore the correlation between pathological image features and molecular characteristics in lung adenocarcinoma patients. This model introduces a novel approach to clinical genetic testing, expected to enhance the efficiency of identifying molecular features and genetic testing in lung adenocarcinoma patients, ultimately providing more accurate and reliable results for related studies.

Deep learning framework for comprehensive molecular and prognostic stratifications of triple-negative breast cancer

Triple-negative breast cancer (TNBC) is the most challenging breast cancer subtype. Molecular stratification and target therapy bring clinical benefit for TNBC patients, but it is difficult to implement comprehensive molecular testing in clinical practice. Here, using our multi-omics TNBC cohort (N = 425), a deep learning-based framework was devised and validated for comprehensive predictions of molecular features, subtypes and prognosis from pathological whole slide images. The framework first incorporated a neural network to decompose the tissue on WSIs, followed by a second one which was trained based on certain tissue types for predicting different targets. Multi-omics molecular features were analyzed including somatic mutations, copy number alterations, germline mutations, biological pathway activities, metabolomics features and immunotherapy biomarkers. It was shown that the molecular features with therapeutic implications can be predicted including the somatic PIK3CA mutation, germline BRCA2 mutation and PD-L1 protein expression (area under the curve [AUC]: 0.78, 0.79 and 0.74 respectively). The molecular subtypes of TNBC can be identified (AUC: 0.84, 0.85, 0.93 and 0.73 for the basal-like immune-suppressed, immunomodulatory, luminal androgen receptor, and mesenchymal-like subtypes respectively) and their distinctive morphological patterns were revealed, which provided novel insights into the heterogeneity of TNBC. A neural network integrating image features and clinical covariates stratified patients into groups with different survival outcomes (log-rank P < 0.001). Our prediction framework and neural network models were externally validated on the TNBC cases from TCGA (N = 143) and appeared robust to the changes in patient population. For potential clinical translation, we built a novel online platform, where we modularized and deployed our framework along with the validated models. It can realize real-time one-stop prediction for new cases. In summary, using only pathological WSIs, our proposed framework can enable comprehensive stratifications of TNBC patients and provide valuable information for therapeutic decision-making. It had the potential to be clinically implemented and promote the personalized management of TNBC.

Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers

Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists' tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.

Deep learning-based identification of esophageal cancer subtypes through analysis of high-resolution histopathology images

Esophageal cancer (EC) remains a significant health challenge globally, with increasing incidence and high mortality rates. Despite advances in treatment, there remains a need for improved diagnostic methods and understanding of disease progression. This study addresses the significant challenges in the automatic classification of EC, particularly in distinguishing its primary subtypes: adenocarcinoma and squamous cell carcinoma, using histopathology images. Traditional histopathological diagnosis, while being the gold standard, is subject to subjectivity and human error and imposes a substantial burden on pathologists. This study proposes a binary class classification system for detecting EC subtypes in response to these challenges. The system leverages deep learning techniques and tissue-level labels for enhanced accuracy. We utilized 59 high-resolution histopathological images from The Cancer Genome Atlas (TCGA) Esophageal Carcinoma dataset (TCGA-ESCA). These images were preprocessed, segmented into patches, and analyzed using a pre-trained ResNet101 model for feature extraction. For classification, we employed five machine learning classifiers: Support Vector Classifier (SVC), Logistic Regression (LR), Decision Tree (DT), AdaBoost (AD), Random Forest (RF), and a Feed-Forward Neural Network (FFNN). The classifiers were evaluated based on their prediction accuracy on the test dataset, yielding results of 0.88 (SVC and LR), 0.64 (DT and AD), 0.82 (RF), and 0.94 (FFNN). Notably, the FFNN classifier achieved the highest Area Under the Curve (AUC) score of 0.92, indicating its superior performance, followed closely by SVC and LR, with a score of 0.87. This suggested approach holds promising potential as a decision-support tool for pathologists, particularly in regions with limited resources and expertise. The timely and precise detection of EC subtypes through this system can substantially enhance the likelihood of successful treatment, ultimately leading to reduced mortality rates in patients with this aggressive cancer.

A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images

BACKGROUND

The objective of this comprehensive pan-cancer study is to evaluate the potential of deep learning (DL) for molecular profiling of multi-omic biomarkers directly from hematoxylin and eosin (H&E)-stained whole slide images.

METHODS

A total of 12,093 DL models predicting 4031 multi-omic biomarkers across 32 cancer types were trained and validated. The study included a broad range of genetic, transcriptomic, and proteomic biomarkers, as well as established prognostic markers, molecular subtypes, and clinical outcomes.

RESULTS

Here we show that 50% of the models achieve an area under the curve (AUC) of 0.644 or higher. The observed AUC for 25% of the models is at least 0.719 and exceeds 0.834 for the top 5%. Molecular profiling with image-based histomorphological features is generally considered feasible for most of the investigated biomarkers and across different cancer types. The performance appears to be independent of tumor purity, sample size, and class ratio (prevalence), suggesting a degree of inherent predictability in histomorphology.

CONCLUSIONS

The results demonstrate that DL holds promise to predict a wide range of biomarkers across the omics spectrum using only H&E-stained histological slides of solid tumors. This paves the way for accelerating diagnosis and developing more precise treatments for cancer patients.

One label is all you need: Interpretable AI-enhanced histopathology for oncology

Artificial Intelligence (AI)-enhanced histopathology presents unprecedented opportunities to benefit oncology through interpretable methods that require only one overall label per hematoxylin and eosin (H&E) slide with no tissue-level annotations. We present a structured review of these methods organized by their degree of verifiability and by commonly recurring application areas in oncological characterization. First, we discuss morphological markers (tumor presence/absence, metastases, subtypes, grades) in which AI-identified regions of interest (ROIs) within whole slide images (WSIs) verifiably overlap with pathologist-identified ROIs. Second, we discuss molecular markers (gene expression, molecular subtyping) that are not verified via H&E but rather based on overlap with positive regions on adjacent tissue. Third, we discuss genetic markers (mutations, mutational burden, microsatellite instability, chromosomal instability) that current technologies cannot verify if AI methods spatially resolve specific genetic alterations. Fourth, we discuss the direct prediction of survival to which AI-identified histopathological features quantitatively correlate but are nonetheless not mechanistically verifiable. Finally, we discuss in detail several opportunities and challenges for these one-label-per-slide methods within oncology. Opportunities include reducing the cost of research and clinical care, reducing the workload of clinicians, personalized medicine, and unlocking the full potential of histopathology through new imaging-based biomarkers. Current challenges include explainability and interpretability, validation via adjacent tissue sections, reproducibility, data availability, computational needs, data requirements, domain adaptability, external validation, dataset imbalances, and finally commercialization and clinical potential. Ultimately, the relative ease and minimum upfront cost with which relevant data can be collected in addition to the plethora of available AI methods for outcome-driven analysis will surmount these current limitations and achieve the innumerable opportunities associated with AI-driven histopathology for the benefit of oncology.

Deep Learning-Based Stratification of Gastric Cancer Patients from Hematoxylin and Eosin-Stained Whole Slide Images by Predicting Molecular Features for Immunotherapy Response

Determining the molecular characteristics of cancer patients is crucial for optimal immunotherapy decisions. The aim of this study was to screen immunotherapy beneficiaries by predicting key molecular features from hematoxylin and eosin-stained images based on deep learning models. An independent data set from Asian gastric cancer patients was included for external validation. In addition, a segmentation model (Horizontal-Vertical Network) was used to quantify the cellular composition of tumor stroma. The model performance was evaluated by measuring the area under the curve (AUC). The tumor extraction model achieved an AUC of 0.9386 and 0.9062 in the internal and external test sets, respectively. The stratification model could predict the immunotherapy-sensitive subtypes (AUC range, 0.8685 to 0.9461), the genetic mutations (AUC range, 0.8283 to 0.9225), and the pathway activity (AUC range, 0.7568 to 0.8612) fairly accurately. In external validation, the prediction performance of Epstein-Barr virus and programmed cell death ligand 1 expression status achieved AUCs of 0.7906 and 0.6384, respectively. The segmentation model identified a relatively high proportion of inflammatory cells and connective cells in some immunotherapy-sensitive subtypes. The deep learning-based models potentially may serve as a valuable tool to screen for the beneficiaries of immunotherapy in gastric cancer patients.

Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin (H&E)-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an approach for predicting response to multiple targeted and immunotherapies from H&E-slides. In difference from existing approaches that aim to predict treatment response directly from the slides, ENLIGHT-DeepPT is an indirect two-step approach consisting of (1) DeepPT, a new deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response based on the DeepPT inferred expression values. DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Our key contribution is showing that ENLIGHT-DeepPT successfully predicts true responders in five independent patients' cohorts involving four different treatments spanning six cancer types with an overall odds ratio of 2.44, increasing the baseline response rate by 43.47% among predicted responders, without the need for any treatment data for training. Furthermore, its prediction accuracy on these datasets is comparable to a supervised approach predicting the response directly from the images, which needs to be trained and tested on the same cohort. ENLIGHT-DeepPT future application could provide clinicians with rapid treatment recommendations to an array of different therapies and importantly, may contribute to advancing precision oncology in developing countries.

Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey

Digital pathology technologies, including whole slide imaging (WSI), have significantly improved modern clinical practices by facilitating storing, viewing, processing, and sharing digital scans of tissue glass slides. Researchers have proposed various artificial intelligence (AI) solutions for digital pathology applications, such as automated image analysis, to extract diagnostic information from WSI for improving pathology productivity, accuracy, and reproducibility. Feature extraction methods play a crucial role in transforming raw image data into meaningful representations for analysis, facilitating the characterization of tissue structures, cellular properties, and pathological patterns. These features have diverse applications in several digital pathology applications, such as cancer prognosis and diagnosis. Deep learning-based feature extraction methods have emerged as a promising approach to accurately represent WSI contents and have demonstrated superior performance in histology-related tasks. In this survey, we provide a comprehensive overview of feature extraction methods, including both manual and deep learning-based techniques, for the analysis of WSIs. We review relevant literature, analyze the discriminative and geometric features of WSIs (i.e., features suited to support the diagnostic process and extracted by "engineered" methods as opposed to AI), and explore predictive modeling techniques using AI and deep learning. This survey examines the advances, challenges, and opportunities in this rapidly evolving field, emphasizing the potential for accurate diagnosis, prognosis, and decision-making in digital pathology.

Artificial Intelligence to Predict the BRAF V595E Mutation in Canine Urinary Bladder Urothelial Carcinomas

In dogs, the BRAF mutation (V595E) is common in bladder and prostate cancer and represents a specific diagnostic marker. Recent advantages in artificial intelligence (AI) offer new opportunities in the field of tumour marker detection. While AI histology studies have been conducted in humans to detect BRAF mutation in cancer, comparable studies in animals are lacking. In this study, we used commercially available AI histology software to predict BRAF mutation in whole slide images (WSI) of bladder urothelial carcinomas (UC) stained with haematoxylin and eosin (HE), based on a training (n = 81) and a validation set (n = 96). Among 96 WSI, 57 showed identical PCR and AI-based BRAF predictions, resulting in a sensitivity of 58% and a specificity of 63%. The sensitivity increased substantially to 89% when excluding small or poor-quality tissue sections. Test reliability depended on tumour differentiation (p < 0.01), presence of inflammation (p < 0.01), slide quality (p < 0.02) and sample size (p < 0.02). Based on a small subset of cases with available adjacent non-neoplastic urothelium, AI was able to distinguish malignant from benign epithelium. This is the first study to demonstrate the use of AI histology to predict BRAF mutation status in canine UC. Despite certain limitations, the results highlight the potential of AI in predicting molecular alterations in routine tissue sections.

hist2RNA: An Efficient Deep Learning Architecture to Predict Gene Expression from Breast Cancer Histopathology Images

Gene expression can be used to subtype breast cancer with improved prediction of risk of recurrence and treatment responsiveness over that obtained using routine immunohistochemistry (IHC). However, in the clinic, molecular profiling is primarily used for ER+ breast cancer, which is costly, tissue destructive, requires specialised platforms, and takes several weeks to obtain a result. Deep learning algorithms can effectively extract morphological patterns in digital histopathology images to predict molecular phenotypes quickly and cost-effectively. We propose a new, computationally efficient approach called hist2RNA inspired by bulk RNA sequencing techniques to predict the expression of 138 genes (incorporated from 6 commercially available molecular profiling tests), including luminal PAM50 subtype, from hematoxylin and eosin (H&E)-stained whole slide images (WSIs). The training phase involves the aggregation of extracted features for each patient from a pretrained model to predict gene expression at the patient level using annotated H&E images from The Cancer Genome Atlas (TCGA, n = 335). We demonstrate successful gene prediction on a held-out test set (n = 160, corr = 0.82 across patients, corr = 0.29 across genes) and perform exploratory analysis on an external tissue microarray (TMA) dataset (n = 498) with known IHC and survival information. Our model is able to predict gene expression and luminal PAM50 subtype (Luminal A versus Luminal B) on the TMA dataset with prognostic significance for overall survival in univariate analysis (c-index = 0.56, hazard ratio = 2.16 (95% CI 1.12-3.06), p < 5 × 10^-3), and independent significance in multivariate analysis incorporating standard clinicopathological variables (c-index = 0.65, hazard ratio = 1.87 (95% CI 1.30-2.68), p < 5 × 10^-3). The proposed strategy achieves superior performance while requiring less training time, resulting in less energy consumption and computational cost compared to patch-based models. Additionally, hist2RNA predicts gene expression that has potential to determine luminal molecular subtypes which correlates with overall survival, without the need for expensive molecular testing.

A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer

The success of training computer-vision models heavily relies on the support of large-scale, real-world images with annotations. Yet such an annotation-ready dataset is difficult to curate in pathology due to the privacy protection and excessive annotation burden. To aid in computational pathology, synthetic data generation, curation, and annotation present a cost-effective means to quickly enable data diversity that is required to boost model performance at different stages. In this study, we introduce a large-scale synthetic pathological image dataset paired with the annotation for nuclei semantic segmentation, termed as Synthetic Nuclei and annOtation Wizard (SNOW). The proposed SNOW is developed via a standardized workflow by applying the off-the-shelf image generator and nuclei annotator. The dataset contains overall 20k image tiles and 1,448,522 annotated nuclei with the CC-BY license. We show that SNOW can be used in both supervised and semi-supervised training scenarios. Extensive results suggest that synthetic-data-trained models are competitive under a variety of model training settings, expanding the scope of better using synthetic images for enhancing downstream data-driven clinical tasks.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

Targeting multifunctional magnetic nanowires for drug delivery in cancer cell death: an emerging paradigm

Apoptosis, often known as programmed cell death is a mechanism used by numerous species to maintain tissue homeostasis. The process leading to cell death is complicated because it requires the stimulation of caspases. According to several studies, nanowires have important medical benefits, can kill cells by adhering to cancer cells, destroying them, and killing the entire cell using a triple attack that integrates vibration, heat, and drug delivery to trigger apoptosis. The sewage effluents and industrial, fertilizer and organic wastes decomposition can produce elevated levels of chemicals in the environment which may interrupt the cell cycle and activate apoptosis. The purpose of this review is to give a thorough summary of the evidence that is currently available on apoptosis. Current review discussed topics like the morphological and biochemical alterations that occur during apoptosis, as well as the various mechanisms that cause cell death, including the intrinsic (or mitochondrial), extrinsic (or death receptor), and intrinsic endoplasmic reticulum pathway. The apoptosis reduction in cancer development is mediated by (i) an imbalance between pro- and anti-apoptotic proteins, such as members of the B-cell lymphoma-2 (BCL2) family of proteins, tumour protein 53 and inhibitor of apoptosis proteins, (ii) a reduction in caspase activity, and (iii) impaired death receptor signalling. This review does an excellent task of outlining the function of nanowires in both apoptosis induction and targeted drug delivery for cancer cells. A comprehensive summary of the relevance of nanowires synthesised for the purpose of inducing apoptosis in cancer cells has been compiled collectively.

A robust and lightweight deep attention multiple instance learning algorithm for predicting genetic alterations

Self-attention mechanism-based algorithms are attractive in digital pathology due to their interpretability, but suffer from computation complexity. This paper presents a novel, lightweight Attention-based Multiple Instance Mutation Learning (AMIML) model to allow small-scale attention operations for predicting gene mutations. Compared to the standard self-attention model, AMIML reduces the number of model parameters by approximately 70%. Using data for 24 clinically relevant genes from four cancer cohorts in TCGA studies (UCEC, BRCA, GBM, and KIRC), we compare AMIML with a standard self-attention model, five other deep learning models, and four traditional machine learning models. The results show that AMIML has excellent robustness and outperforms all the baseline algorithms in the vast majority of the tested genes. Conversely, the performance of the reference deep learning and machine learning models vary across different genes, and produce suboptimal prediction for certain genes. Furthermore, with the flexible and interpretable attention-based pooling mechanism, AMIML can further zero in and detect predictive image patches.

Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology

Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.

Interpretable deep learning model to predict the molecular classification of endometrial cancer from haematoxylin and eosin-stained whole-slide images: a combined analysis of the PORTEC randomised trials and clinical cohorts

BACKGROUND

Endometrial cancer can be molecularly classified into POLE^mut, mismatch repair deficient (MMRd), p53 abnormal (p53abn), and no specific molecular profile (NSMP) subgroups. We aimed to develop an interpretable deep learning pipeline for whole-slide-image-based prediction of the four molecular classes in endometrial cancer (im4MEC), to identify morpho-molecular correlates, and to refine prognostication.

METHODS

This combined analysis included diagnostic haematoxylin and eosin-stained slides and molecular and clinicopathological data from 2028 patients with intermediate-to-high-risk endometrial cancer from the PORTEC-1 (n=466), PORTEC-2 (n=375), and PORTEC-3 (n=393) randomised trials and the TransPORTEC pilot study (n=110), the Medisch Spectrum Twente cohort (n=242), a case series of patients with POLE^mut endometrial cancer in the Leiden Endometrial Cancer Repository (n=47), and The Cancer Genome Atlas-Uterine Corpus Endometrial Carcinoma cohort (n=395). PORTEC-3 was held out as an independent test set and a four-fold cross validation was performed. Performance was measured with the macro and class-wise area under the receiver operating characteristic curve (AUROC). Whole-slide images were segmented into tiles of 360 μm resized to 224 × 224 pixels. im4MEC was trained to learn tile-level morphological features with self-supervised learning and to molecularly classify whole-slide images with an attention mechanism. The top 20 tiles with the highest attention scores were reviewed to identify morpho-molecular correlates. Predictions of a nuclear classification deep learning model serve to derive interpretable morphological features. We analysed 5-year recurrence-free survival and explored prognostic refinement by molecular class using the Kaplan-Meier method.

FINDINGS

im4MEC attained macro-average AUROCs of 0·874 (95% CI 0·856-0·893) on four-fold cross-validation and 0·876 on the independent test set. The class-wise AUROCs were 0·849 for POLE^mut (n=51), 0·844 for MMRd (n=134), 0·883 for NSMP (n=120), and 0·928 for p53abn (n=88). POLE^mut and MMRd tiles had a high density of lymphocytes, p53abn tiles had strong nuclear atypia, and the morphology of POLE^mut and MMRd endometrial cancer overlapped. im4MEC highlighted a low tumour-to-stroma ratio as a potentially novel characteristic feature of the NSMP class. 5-year recurrence-free survival was significantly different between im4MEC predicted molecular classes in PORTEC-3 (log-rank p<0·0001). The ten patients with aggressive p53abn endometrial cancer that was predicted as MMRd showed inflammatory morphology and appeared to have a better prognosis than patients with correctly predicted p53abn endometrial cancer (p=0·30). The four patients with NSMP endometrial cancer that was predicted as p53abn showed higher nuclear atypia and appeared to have a worse prognosis than patients with correctly predicted NSMP (p=0·13). Patients with MMRd endometrial cancer predicted as POLE^mut had an excellent prognosis, as do those with true POLE^mut endometrial cancer.

INTERPRETATION

We present the first interpretable deep learning model, im4MEC, for haematoxylin and eosin-based prediction of molecular endometrial cancer classification. im4MEC robustly identified morpho-molecular correlates and could enable further prognostic refinement of patients with endometrial cancer.

FUNDING

The Hanarth Foundation, the Promedica Foundation, and the Swiss Federal Institutes of Technology.

Application of Deep Learning in Histopathology Images of Breast Cancer: A Review

With the development of artificial intelligence technology and computer hardware functions, deep learning algorithms have become a powerful auxiliary tool for medical image analysis. This study was an attempt to use statistical methods to analyze studies related to the detection, segmentation, and classification of breast cancer in pathological images. After an analysis of 107 articles on the application of deep learning to pathological images of breast cancer, this study is divided into three directions based on the types of results they report: detection, segmentation, and classification. We introduced and analyzed models that performed well in these three directions and summarized the related work from recent years. Based on the results obtained, the significant ability of deep learning in the application of breast cancer pathological images can be recognized. Furthermore, in the classification and detection of pathological images of breast cancer, the accuracy of deep learning algorithms has surpassed that of pathologists in certain circumstances. Our study provides a comprehensive review of the development of breast cancer pathological imaging-related research and provides reliable recommendations for the structure of deep learning network models in different application scenarios.

Optimization of deep learning models for the prediction of gene mutations using unsupervised clustering

Deep learning models are increasingly being used to interpret whole-slide images (WSIs) in digital pathology and to predict genetic mutations. Currently, it is commonly assumed that tumor regions have most of the predictive power. However, it is reasonable to assume that other tissues from the tumor microenvironment may also provide important predictive information. In this paper, we propose an unsupervised clustering-based multiple-instance deep learning model for the prediction of genetic mutations using WSIs of three cancer types obtained from The Cancer Genome Atlas. Our proposed model facilitates the identification of spatial regions related to specific gene mutations and exclusion of patches that lack predictive information through the use of unsupervised clustering. This results in a more accurate prediction of gene mutations when compared with models using all image patches on WSIs and two recently published algorithms for all three different cancer types evaluated in this study. In addition, our study validates the hypothesis that the prediction of gene mutations solely based on tumor regions on WSI slides may not always provide the best performance. Other tissue types in the tumor microenvironment could provide a better prediction ability than tumor tissues alone. These results highlight the heterogeneity in the tumor microenvironment and the importance of identification of predictive image patches in digital pathology prediction tasks.

Multi-modality artificial intelligence in digital pathology

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

The evolving role of morphology in endometrial cancer diagnostics: From histopathology and molecular testing towards integrative data analysis by deep learning

Endometrial cancer (EC) diagnostics is evolving into a system in which molecular aspects are increasingly important. The traditional histological subtype-driven classification has shifted to a molecular-based classification that stratifies EC into DNA polymerase epsilon mutated (POLEmut), mismatch repair deficient (MMRd), and p53 abnormal (p53abn), and the remaining EC as no specific molecular profile (NSMP). The molecular EC classification has been implemented in the World Health Organization 2020 classification and the 2021 European treatment guidelines, as it serves as a better basis for patient management. As a result, the integration of the molecular class with histopathological variables has become a critical focus of recent EC research. Pathologists have observed and described several morphological characteristics in association with specific genomic alterations, but these appear insufficient to accurately classify patients according to molecular subgroups. This requires pathologists to rely on molecular ancillary tests in routine workup. In this new era, it has become increasingly challenging to assign clinically relevant weights to histological and molecular features on an individual patient basis. Deep learning (DL) technology opens new options for the integrative analysis of multi-modal image and molecular datasets with clinical outcomes. Proof-of-concept studies in other cancers showed promising accuracy in predicting molecular alterations from H&E-stained tumor slide images. This suggests that some morphological characteristics that are associated with molecular alterations could be identified in EC, too, expanding the current understanding of the molecular-driven EC classification. Here in this review, we report the morphological characteristics of the molecular EC classification currently identified in the literature. Given the new challenges in EC diagnostics, this review discusses, therefore, the potential supportive role that DL could have, by providing an outlook on all relevant studies using DL on histopathology images in various cancer types with a focus on EC. Finally, we touch upon how DL might shape the management of future EC patients.

Multiparametric MR radiomics in brain glioma: models comparation to predict biomarker status

Background

Genotype status of glioma have important significance to clinical treatment and prognosis. At present, there are few studies on the prediction of multiple genotype status in glioma by method of multi-sequence radiomics. The purpose of the study is to compare the performance of clinical features (age, sex, WHO grade, MRI morphological features etc.), radiomics features from multi MR sequence (T2WI, T1WI, DWI, ADC, CE-MRI (contrast enhancement)), and a combined multiple features model in predicting biomarker status (IDH, MGMT, TERT, 1p/19q of glioma.

Methods

In this retrospective analysis, 81 glioma patients confirmed by histology were enrolled in this study. Five MRI sequences were used for radiomic feature extraction. Finally, 107 features were extracted from each sequence on Pyradiomics software, separately. These included 18 first-order metrics, such as the mean, standard deviation, skewness, and kurtosis etc., 14 shape features and second-order metrics including 24 grey level run length matrix (GLCM), 16 grey level run length matrix (GLRLM), 16 grey level size zone matrix (GLSZM), 5 neighboring gray tone difference matrix (NGTDM), and 14 grey level dependence matrix (GLDM). Then, Univariate analysis and LASSO (Least absolute shrinkage and selection operator regression model were used to data dimension reduction, feature selection, and radiomics signature building. Significant features (p < 0.05 by multivariate logistic regression were retained to establish clinical model, T1WI model, T2WI model, T1 + C (T1WI contrast enhancement model, DWI model and ADC model, multi sequence model. Clinical features were combined with multi sequence model to establish a combined model. The predictive performance was validated by receiver operating characteristic curve (ROC analysis and decision curve analysis (DCA).

Results

The combined model showed the better performance in some groups of genotype status among some models (IDH AUC = 0.93, MGMT AUC = 0.88, TERT AUC = 0.76). Multi sequence model performed better than single sequence model in IDH, MGMT, TERT. There was no significant difference among the models in predicting 1p/19q status. Decision curve analysis showed combined model has higher clinical benefit than multi sequence model.

Conclusion

Multi sequence model is an effective method to identify the genotype status of cerebral glioma. Combined with clinical models can better distinguish genotype status of glioma.

Key Points

The combined model showed the higher performance compare with other models in predicting genotype status of IDH, MGMT, TERT.

Multi sequence model showed a better predictive model than that of a single sequence model.

Compared with other models, the combined model and multi sequence model show no advantage in prediction of 1p/19q status.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12880-022-00865-8.

Predicting oncogene mutations of lung cancer using deep learning and histopathologic features on whole-slide images

•

Developed a deep learning model for predicting somatic mutations of LUAD patients.

•

Utilized LUAD subtype-related histological features to predict five major genetic mutations.

•

Investigated multiple transfer learning scenarios to characterize morphological features.

•

Achieved strong performance in predicting EGFR genetic mutations across multiple datasets.

Lung cancer is a leading cause of death in both men and women globally. The recent development of tumor molecular profiling has opened opportunities for targeted therapies for lung adenocarcinoma (LUAD) patients. However, the lack of access to molecular profiling or cost and turnaround time associated with it could hinder oncologists' willingness to order frequent molecular tests, limiting potential benefits from precision medicine. In this study, we developed a weakly supervised deep learning model for predicting somatic mutations of LUAD patients based on formalin-fixed paraffin-embedded (FFPE) whole-slide images (WSIs) using LUAD subtypes-related histological features and recent advances in computer vision. Our study was performed on a total of 747 hematoxylin and eosin (H&E) stained FFPE LUAD WSIs and the genetic mutation data of 232 patients who were treated at Dartmouth-Hitchcock Medical Center (DHMC). We developed our convolutional neural network-based models to analyze whole slides and predict five major genetic mutations, i.e., BRAF, EGFR, KRAS, STK11, and TP53. We additionally used 111 cases from the LUAD dataset of the CPTAC-3 study for external validation. Our model achieved an AUROC of 0.799 (95% CI: 0.686–0.904) and 0.686 (95% CI: 0.620–0.752) for predicting EGFR genetic mutations on the DHMC and CPTAC-3 test sets, respectively. Predicting TP53 genetic mutations also showed promising outcomes. Our results demonstrated that H&E stained FFPE LUAD whole slides could be utilized to predict oncogene mutations, such as EGFR, indicating that somatic mutations could present subtle morphological characteristics in histology slides, where deep learning-based feature extractors can learn such latent information.

Artificial intelligence to identify genetic alterations in conventional histopathology

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold-standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53 and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows. This article is protected by copyright. All rights reserved.

Deep Learning With Radiomics for Disease Diagnosis and Treatment: Challenges and Potential

The high-throughput extraction of quantitative imaging features from medical images for the purpose of radiomic analysis, i.e., radiomics in a broad sense, is a rapidly developing and emerging research field that has been attracting increasing interest, particularly in multimodality and multi-omics studies. In this context, the quantitative analysis of multidimensional data plays an essential role in assessing the spatio-temporal characteristics of different tissues and organs and their microenvironment. Herein, recent developments in this method, including manually defined features, data acquisition and preprocessing, lesion segmentation, feature extraction, feature selection and dimension reduction, statistical analysis, and model construction, are reviewed. In addition, deep learning-based techniques for automatic segmentation and radiomic analysis are being analyzed to address limitations such as rigorous workflow, manual/semi-automatic lesion annotation, and inadequate feature criteria, and multicenter validation. Furthermore, a summary of the current state-of-the-art applications of this technology in disease diagnosis, treatment response, and prognosis prediction from the perspective of radiology images, multimodality images, histopathology images, and three-dimensional dose distribution data, particularly in oncology, is presented. The potential and value of radiomics in diagnostic and therapeutic strategies are also further analyzed, and for the first time, the advances and challenges associated with dosiomics in radiotherapy are summarized, highlighting the latest progress in radiomics. Finally, a robust framework for radiomic analysis is presented and challenges and recommendations for future development are discussed, including but not limited to the factors that affect model stability (medical big data and multitype data and expert knowledge in medical), limitations of data-driven processes (reproducibility and interpretability of studies, different treatment alternatives for various institutions, and prospective researches and clinical trials), and thoughts on future directions (the capability to achieve clinical applications and open platform for radiomics analysis).