Artificial Intelligence Algorithms to Assess Hormonal Status From Tissue Microarrays in Patients With Breast Cancer

Deep learning applications in breast cancer histopathological imaging: diagnosis, treatment, and prognosis

Breast cancer is the most common malignant tumor among women worldwide and remains one of the leading causes of death among women. Its incidence and mortality rates are continuously rising. In recent years, with the rapid advancement of deep learning (DL) technology, DL has demonstrated significant potential in breast cancer diagnosis, prognosis evaluation, and treatment response prediction. This paper reviews relevant research progress and applies DL models to image enhancement, segmentation, and classification based on large-scale datasets from TCGA and multiple centers. We employed foundational models such as ResNet50, Transformer, and Hover-net to investigate the performance of DL models in breast cancer diagnosis, treatment, and prognosis prediction. The results indicate that DL techniques have significantly improved diagnostic accuracy and efficiency, particularly in predicting breast cancer metastasis and clinical prognosis. Furthermore, the study emphasizes the crucial role of robust databases in developing highly generalizable models. Future research will focus on addressing challenges related to data management, model interpretability, and regulatory compliance, ultimately aiming to provide more precise clinical treatment and prognostic evaluation programs for breast cancer patients.

Explainable breast cancer molecular expression prediction using multi-task deep-learning based on 3D whole breast ultrasound

OBJECTIVES

To noninvasively estimate three breast cancer biomarkers, estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) and enhance performance and interpretability via multi-task deep learning.

METHODS

The study included 388 breast cancer patients who received the 3D whole breast ultrasound system (3DWBUS) examinations at Xijing Hospital between October 2020 and September 2021. Two predictive models, a single-task and a multi-task, were developed; the former predicts biomarker expression, while the latter combines tumor segmentation with biomarker prediction to enhance interpretability. Performance evaluation included individual and overall prediction metrics, and Delong's test was used for performance comparison. The models' attention regions were visualized using Grad-CAM + + technology.

RESULTS

All patients were randomly split into a training set (n = 240, 62%), a validation set (n = 60, 15%), and a test set (n = 88, 23%). In the individual evaluation of ER, PR, and HER2 expression prediction, the single-task and multi-task models achieved respective AUCs of 0.809 and 0.735 for ER, 0.688 and 0.767 for PR, and 0.626 and 0.697 for HER2, as observed in the test set. In the overall evaluation, the multi-task model demonstrated superior performance in the test set, achieving a higher macro AUC of 0.733, in contrast to 0.708 for the single-task model. The Grad-CAM + + method revealed that the multi-task model exhibited a stronger focus on diseased tissue areas, improving the interpretability of how the model worked.

CONCLUSION

Both models demonstrated impressive performance, with the multi-task model excelling in accuracy and offering improved interpretability on noninvasive 3DWBUS images using Grad-CAM + + technology.

CRITICAL RELEVANCE STATEMENT

The multi-task deep learning model exhibits effective prediction for breast cancer biomarkers, offering direct biomarker identification and improved clinical interpretability, potentially boosting the efficiency of targeted drug screening.

KEY POINTS

Tumoral biomarkers are paramount for determining breast cancer treatment. The multi-task model can improve prediction performance, and improve interpretability in clinical practice. The 3D whole breast ultrasound system-based deep learning models excelled in predicting breast cancer biomarkers.

Weakly-supervised deep learning models enable HER2-low prediction from H &E stained slides

BACKGROUND

Human epidermal growth factor receptor 2 (HER2)-low breast cancer has emerged as a new subtype of tumor, for which novel antibody-drug conjugates have shown beneficial effects. Assessment of HER2 requires several immunohistochemistry tests with an additional in situ hybridization test if a case is classified as HER2 2+. Therefore, novel cost-effective methods to speed up the HER2 assessment are highly desirable.

METHODS

We used a self-supervised attention-based weakly supervised method to predict HER2-low directly from 1437 histopathological images from 1351 breast cancer patients. We built six distinct models to explore the ability of classifiers to distinguish between the HER2-negative, HER2-low, and HER2-high classes in different scenarios. The attention-based model was used to comprehend the decision-making process aimed at relevant tissue regions.

RESULTS

Our results indicate that the effectiveness of classification models hinges on the consistency and dependability of assay-based tests for HER2, as the outcomes from these tests are utilized as the baseline truth for training our models. Through the use of explainable AI, we reveal histologic patterns associated with the HER2 subtypes.

CONCLUSION

Our findings offer a demonstration of how deep learning technologies can be applied to identify HER2 subgroup statuses, potentially enriching the toolkit available for clinical decision-making in oncology.

Pathomic model based on histopathological features and machine learning to predict IDO1 status and its association with breast cancer prognosis

PURPOSE

To establish a pathomic model using histopathological image features for predicting indoleamine 2,3-dioxygenase 1 (IDO1) status and its relationship with overall survival (OS) in breast cancer.

METHODS

A pathomic model was constructed using machine learning and histopathological images obtained from The Cancer Genome Atlas database to predict IDO1 expression. The model performance was evaluated based on the area under the curve, calibration curve, and decision curve analysis (DCA). Prediction scores (PSes) were generated from the model and applied to divide the patients into two groups. Survival outcomes, gene set enrichment, immune microenvironment, and tumor mutations were assessed between the two groups.

RESULTS

Survival analysis followed by multivariate correction revealed that high IDO1 is a protective factor for OS. Further, the model was calibrated, and it exhibited good discrimination. Additionally, the DCA showed that the proposed model provided a good clinical net benefit. The Kaplan-Meier analysis revealed a positive correlation between high PS and improved OS. Univariate and multivariate Cox regression analyses demonstrated that PS is an independent protective factor for OS. Moreover, differentially expressed genes were enriched in various essential biological processes, including extracellular matrix receptor interaction, angiogenesis, transforming growth factor β signaling, epithelial mesenchymal transition, cell junction, tryptophan metabolism, and heme metabolic processes. PS was positively correlated with M1 macrophages, CD8 + T cells, T follicular helper cells, and tumor mutational burden.

CONCLUSION

These results indicate the potential ability of the proposed pathomic model to predict IDO1 status and the OS of breast cancer patients.

A guide to artificial intelligence for cancer researchers

Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.

Using Vision Transformer for high robustness and generalization in predicting EGFR mutation status in lung adenocarcinoma

BACKGROUND

Lung adenocarcinoma is a common cause of cancer-related deaths worldwide, and accurate EGFR genotyping is crucial for optimal treatment outcomes. Conventional methods for identifying the EGFR genotype have several limitations. Therefore, we proposed a deep learning model using non-invasive CT images to predict EGFR mutation status with robustness and generalizability.

METHODS

A total of 525 patients were enrolled at the local hospital to serve as the internal data set for model training and validation. In addition, a cohort of 30 patients from the publicly available Cancer Imaging Archive Data Set was selected for external testing. All patients underwent plain chest CT, and their EGFR mutation status labels were categorized as either mutant or wild type. The CT images were analyzed using a self-attention-based ViT-B/16 model to predict the EGFR mutation status, and the model's performance was evaluated. To produce an attention map indicating the suspicious locations of EGFR mutations, Grad-CAM was utilized.

RESULTS

The ViT deep learning model achieved impressive results, with an accuracy of 0.848, an AUC of 0.868, a sensitivity of 0.924, and a specificity of 0.718 on the validation cohort. Furthermore, in the external test cohort, the model achieved comparable performances, with an accuracy of 0.833, an AUC of 0.885, a sensitivity of 0.900, and a specificity of 0.800.

CONCLUSIONS

The ViT model demonstrates a high level of accuracy in predicting the EGFR mutation status of lung adenocarcinoma patients. Moreover, with the aid of attention maps, the model can assist clinicians in making informed clinical decisions.

Early Breast Cancer Risk Assessment: Integrating Histopathology with Artificial Intelligence

Effective risk assessment in early breast cancer is essential for informed clinical decision-making, yet consensus on defining risk categories remains challenging. This paper explores evolving approaches in risk stratification, encompassing histopathological, immunohistochemical, and molecular biomarkers alongside cutting-edge artificial intelligence (AI) techniques. Leveraging machine learning, deep learning, and convolutional neural networks, AI is reshaping predictive algorithms for recurrence risk, thereby revolutionizing diagnostic accuracy and treatment planning. Beyond detection, AI applications extend to histological subtyping, grading, lymph node assessment, and molecular feature identification, fostering personalized therapy decisions. With rising cancer rates, it is crucial to implement AI to accelerate breakthroughs in clinical practice, benefiting both patients and healthcare providers. However, it is important to recognize that while AI offers powerful automation and analysis tools, it lacks the nuanced understanding, clinical context, and ethical considerations inherent to human pathologists in patient care. Hence, the successful integration of AI into clinical practice demands collaborative efforts between medical experts and computational pathologists to optimize patient outcomes.

Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers

Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists' tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.

Deep learning on tertiary lymphoid structures in hematoxylin-eosin predicts cancer prognosis and immunotherapy response

Tertiary lymphoid structures (TLSs) have been associated with favorable immunotherapy responses and prognosis in various cancers. Despite their significance, their quantification using multiplex immunohistochemistry (mIHC) staining of T and B lymphocytes remains labor-intensive, limiting its clinical utility. To address this challenge, we curated a dataset from matched mIHC and H&E whole-slide images (WSIs) and developed a deep learning model for automated segmentation of TLSs. The model achieved Dice coefficients of 0.91 on the internal test set and 0.866 on the external validation set, along with intersection over union (IoU) scores of 0.819 and 0.787, respectively. The TLS ratio, defined as the segmented TLS area over the total tissue area, correlated with B lymphocyte levels and the expression of CXCL13, a chemokine associated with TLS formation, in 6140 patients spanning 16 tumor types from The Cancer Genome Atlas (TCGA). The prognostic models for overall survival indicated that the inclusion of the TLS ratio with TNM staging significantly enhanced the models' discriminative ability, outperforming the traditional models that solely incorporated TNM staging, in 10 out of 15 TCGA tumor types. Furthermore, when applied to biopsied treatment-naïve tumor samples, higher TLS ratios predicted a positive immunotherapy response across multiple cohorts, including specific therapies for esophageal squamous cell carcinoma, non-small cell lung cancer, and stomach adenocarcinoma. In conclusion, our deep learning-based approach offers an automated and reproducible method for TLS segmentation and quantification, highlighting its potential in predicting immunotherapy response and informing cancer prognosis.

Vision Transformers for Breast Cancer Human Epidermal Growth Factor Receptor 2 Expression Staging without Immunohistochemical Staining

Accurate staging of human epidermal growth factor receptor 2 (HER2) expression is vital for evaluating breast cancer treatment efficacy. However, it typically involves costly and complex immunohistochemical staining, along with hematoxylin and eosin staining. This work presents customized vision transformers for staging HER2 expression in breast cancer using only hematoxylin and eosin-stained images. The proposed algorithm comprised three modules: a localization module for weakly localizing critical image features using spatial transformers, an attention module for global learning via vision transformers, and a loss module to determine proximity to a HER2 expression level based on input images by calculating ordinal loss. Results, reported with 95% CIs, reveal the proposed approach's success in HER2 expression staging: area under the receiver operating characteristic curve, 0.9202 ± 0.01; precision, 0.922 ± 0.01; sensitivity, 0.876 ± 0.01; and specificity, 0.959 ± 0.02 over fivefold cross-validation. Comparatively, this approach significantly outperformed conventional vision transformer models and state-of-the-art convolutional neural network models (P < 0.001). Furthermore, it surpassed existing methods when evaluated on an independent test data set. This work holds great importance, aiding HER2 expression staging in breast cancer treatment while circumventing the costly and time-consuming immunohistochemical staining procedure, thereby addressing diagnostic disparities in low-resource settings and low-income countries.

Prediction of prognosis and treatment response in ovarian cancer patients from histopathology images using graph deep learning: a multicenter retrospective study

BACKGROUND

Ovarian cancer (OV) is a prevalent and deadly disease with high mortality rates. The development of accurate prognostic tools and personalized therapeutic strategies is crucial for improving patient outcomes.

METHODS

A graph-based deep learning model, the Ovarian Cancer Digital Pathology Index (OCDPI), was introduced to predict prognosis and response to adjuvant therapy using hematoxylin and eosin (H&E)-stained whole-slide images (WSIs). The OCDPI was developed using formalin-fixed, paraffin-embedded (FFPE) WSIs from the TCGA-OV cohort, and was externally validated in two independent cohorts from the Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial (PLCO) and Harbin Medical University Cancer Hospital (HMUCH).

RESULTS

The OCDPI showed prognostic ability for overall survival prediction in the PLCO (HR, 1.916; 95% CI, 1.380-2.660; log-rank test, P < 0.001) and HMUCH (HR, 2.796; 95% CI, 1.404-5.568; log-rank test, P = 0.0022) cohorts. Patients with low OCDPI experienced better survival benefits and lower recurrence rates following adjuvant therapy compared to those with high OCDPI. Multivariable analyses, adjusting for clinicopathological factors, consistently identified OCDPI as an independent prognostic factor across all cohorts (all P < 0.05). Furthermore, OCDPI performed well in patients with low-grade tumors or fresh-frozen slides, and could differentiate between HRD-deficient or HRD-intact patients with and without sensitivity to adjuvant therapy.

CONCLUSION

The results from this multicenter cohort study indicate that the OCDPI may serve as a valuable and labor-saving tool to improve prognostic and predictive clinical decision-making in patients with OV.

Future Practices of Breast Pathology Using Digital and Computational Pathology

Pathology clinical practice has evolved by adopting technological advancements initially regarded as potentially disruptive, such as electron microscopy, immunohistochemistry, and genomic sequencing. Breast pathology has a critical role as a medical domain, where the patient's pathology diagnosis has significant implications for prognostication and treatment of diseases. The advent of digital and computational pathology has brought about significant advancements in the field, offering new possibilities for enhancing diagnostic accuracy and improving patient care. Digital slide scanning enables to conversion of glass slides into high-fidelity digital images, supporting the review of cases in a digital workflow. Digitization offers the capability to render specimen diagnoses, digital archival of patient specimens, collaboration, and telepathology. Integration of image analysis and machine learning-based systems layered atop the high-resolution digital images offers novel workflows to assist breast pathologists in their clinical, educational, and research endeavors. Decision support tools may improve the detection and classification of breast lesions and the quantification of immunohistochemical studies. Computational biomarkers may help to contribute to patient management or outcomes. Furthermore, using digital and computational pathology may increase standardization and quality assurance, especially in areas with high interobserver variability. This review explores the current landscape and possible future applications of digital and computational techniques in the field of breast pathology.

Artificial Intelligence in Breast Cancer Diagnosis and Personalized Medicine

Breast cancer is a significant cause of cancer-related mortality in women worldwide. Early and precise diagnosis is crucial, and clinical outcomes can be markedly enhanced. The rise of artificial intelligence (AI) has ushered in a new era, notably in image analysis, paving the way for major advancements in breast cancer diagnosis and individualized treatment regimens. In the diagnostic workflow for patients with breast cancer, the role of AI encompasses screening, diagnosis, staging, biomarker evaluation, prognostication, and therapeutic response prediction. Although its potential is immense, its complete integration into clinical practice is challenging. Particularly, these challenges include the imperatives for extensive clinical validation, model generalizability, navigating the "black-box" conundrum, and pragmatic considerations of embedding AI into everyday clinical environments. In this review, we comprehensively explored the diverse applications of AI in breast cancer care, underlining its transformative promise and existing impediments. In radiology, we specifically address AI in mammography, tomosynthesis, risk prediction models, and supplementary imaging methods, including magnetic resonance imaging and ultrasound. In pathology, our focus is on AI applications for pathologic diagnosis, evaluation of biomarkers, and predictions related to genetic alterations, treatment response, and prognosis in the context of breast cancer diagnosis and treatment. Our discussion underscores the transformative potential of AI in breast cancer management and emphasizes the importance of focused research to realize the full spectrum of benefits of AI in patient care.

HAHNet: a convolutional neural network for HER2 status classification of breast cancer

OBJECTIVE

Breast cancer is a significant health issue for women, and human epidermal growth factor receptor-2 (HER2) plays a crucial role as a vital prognostic and predictive factor. The HER2 status is essential for formulating effective treatment plans for breast cancer. However, the assessment of HER2 status using immunohistochemistry (IHC) is time-consuming and costly. Existing computational methods for evaluating HER2 status have limitations and lack sufficient accuracy. Therefore, there is an urgent need for an improved computational method to better assess HER2 status, which holds significant importance in saving lives and alleviating the burden on pathologists.

RESULTS

This paper analyzes the characteristics of histological images of breast cancer and proposes a neural network model named HAHNet that combines multi-scale features with attention mechanisms for HER2 status classification. HAHNet directly classifies the HER2 status from hematoxylin and eosin (H&E) stained histological images, reducing additional costs. It achieves superior performance compared to other computational methods.

CONCLUSIONS

According to our experimental results, the proposed HAHNet achieved high performance in classifying the HER2 status of breast cancer using only H&E stained samples. It can be applied in case classification, benefiting the work of pathologists and potentially helping more breast cancer patients.

Data-Efficient Computational Pathology Platform for Faster and Cheaper Breast Cancer Subtype Identifications: Development of a Deep Learning Model

BACKGROUND

Breast cancer subtyping is a crucial step in determining therapeutic options, but the molecular examination based on immunohistochemical staining is expensive and time-consuming. Deep learning opens up the possibility to predict the subtypes based on the morphological information from hematoxylin and eosin staining, a much cheaper and faster alternative. However, training the predictive model conventionally requires a large number of histology images, which is challenging to collect by a single institute.

OBJECTIVE

We aimed to develop a data-efficient computational pathology platform, 3DHistoNet, which is capable of learning from z-stacked histology images to accurately predict breast cancer subtypes with a small sample size.

METHODS

We retrospectively examined 401 cases of patients with primary breast carcinoma diagnosed between 2018 and 2020 at the Department of Pathology, National Cancer Center, South Korea. Pathology slides of the patients with breast carcinoma were prepared according to the standard protocols. Age, gender, histologic grade, hormone receptor (estrogen receptor [ER], progesterone receptor [PR], and androgen receptor [AR]) status, erb-B2 receptor tyrosine kinase 2 (HER2) status, and Ki-67 index were evaluated by reviewing medical charts and pathological records.

RESULTS

The area under the receiver operating characteristic curve and decision curve were analyzed to evaluate the performance of our 3DHistoNet platform for predicting the ER, PR, AR, HER2, and Ki67 subtype biomarkers with 5-fold cross-validation. We demonstrated that 3DHistoNet can predict all clinically important biomarkers (ER, PR, AR, HER2, and Ki67) with performance exceeding the conventional multiple instance learning models by a considerable margin (area under the receiver operating characteristic curve: 0.75-0.91 vs 0.67-0.8). We further showed that our z-stack histology scanning method can make up for insufficient training data sets without any additional cost incurred. Finally, 3DHistoNet offered an additional capability to generate attention maps that reveal correlations between Ki67 and histomorphological features, which renders the hematoxylin and eosin image in higher fidelity to the pathologist.

CONCLUSIONS

Our stand-alone, data-efficient pathology platform that can both generate z-stacked images and predict key biomarkers is an appealing tool for breast cancer diagnosis. Its development would encourage morphology-based diagnosis, which is faster, cheaper, and less error-prone compared to the protein quantification method based on immunohistochemical staining.

Heterophile carbohydrate antigen N-glycolylneuraminic acid as a potential biomarker in patients with hepatocellular carcinoma

BACKGROUND AND OBJECTIVES

Hepatocellular carcinoma (HCC) has a high recurrence rate even after radical hepatectomy. More optimal biomarkers may help improve recurrence and prognosis.

METHODS

We investigated whether the oncological properties of N-glycolylneuraminic acid (NeuGc) can participate in the prognosis of HCC. We evaluated the NeuGc antigen (Ag) expression in the HCC tissues and measured the preoperative anti-NeuGc IgG antibodies (Abs) in the sera of the patients with HCC. We compared the clinical characteristics and survival rate in the hepatectomized patients (initial; n = 66, recurrent; n = 34) with and without the NeuGc Ag or Abs.

RESULTS

Multivariate analyses showed positive expression of NeuGc Ag in HCC tissues (Odds ratio; initial = 6.3, recurrent = 14.0) and higher titers of preoperative anti-NeuGc Ab (Odds ratio; initial = 4.9; recurrent = 3.8), which could be the predictive factors related to early recurrence. Both the NeuGc Ag-positive and Ab-positive groups in the initial hepatectomized patients exhibited significantly shorter recurrent free survival compared to those in the negative groups.

CONCLUSIONS

Our findings suggested that anti-NeuGc Ab titers and NeuGc Ag expression in the HCC tissues can be used as the predictive factors for the postoperative recurrence and prognosis of HCC.

Deep learning radiopathomics based on preoperative US images and biopsy whole slide images can distinguish between luminal and non-luminal tumors in early-stage breast cancers

BACKGROUND

For patients with early-stage breast cancers, neoadjuvant treatment is recommended for non-luminal tumors instead of luminal tumors. Preoperative distinguish between luminal and non-luminal cancers at early stages will facilitate treatment decisions making. However, the molecular immunohistochemical subtypes based on biopsy specimens are not always consistent with final results based on surgical specimens due to the high intra-tumoral heterogeneity. Given that, we aimed to develop and validate a deep learning radiopathomics (DLRP) model to preoperatively distinguish between luminal and non-luminal breast cancers at early stages based on preoperative ultrasound (US) images, and hematoxylin and eosin (H&E)-stained biopsy slides.

METHODS

This multicentre study included three cohorts from a prospective study conducted by our team and registered on the Chinese Clinical Trial Registry (ChiCTR1900027497). Between January 2019 and August 2021, 1809 US images and 603 H&E-stained whole slide images (WSIs) from 603 patients with early-stage breast cancers were obtained. A Resnet18 model pre-trained on ImageNet and a multi-instance learning based attention model were used to extract the features of US and WSIs, respectively. An US-guided Co-Attention module (UCA) was designed for feature fusion of US and WSIs. The DLRP model was constructed based on these three feature sets including deep learning US feature, deep learning WSIs feature and UCA-fused feature from a training cohort (1467 US images and 489 WSIs from 489 patients). The DLRP model's diagnostic performance was validated in an internal validation cohort (342 US images and 114 WSIs from 114 patients) and an external test cohort (270 US images and 90 WSIs from 90 patients). We also compared diagnostic efficacy of the DLRP model with that of deep learning radiomics model and deep learning pathomics model in the external test cohort.

FINDINGS

The DLRP yielded high performance with area under the curve (AUC) values of 0.929 (95% CI 0.865-0.968) in the internal validation cohort, and 0.900 (95% CI 0.819-0.953) in the external test cohort. The DLRP also outperformed deep learning radiomics model based on US images only (AUC 0.815 [0.719-0.889], p = 0.027) and deep learning pathomics model based on WSIs only (AUC 0.802 [0.704-0.878], p = 0.013) in the external test cohort.

INTERPRETATION

The DLRP can effectively distinguish between luminal and non-luminal breast cancers at early stages before surgery based on pretherapeutic US images and biopsy H&E-stained WSIs, providing a tool to facilitate treatment decision making in early-stage breast cancers.

FUNDING

Natural Science Foundation of Guangdong Province (No. 2023A1515011564), and National Natural Science Foundation of China (No. 91959127; No. 81971631).

A review on deep learning applications in highly multiplexed tissue imaging data analysis

Since its introduction into the field of oncology, deep learning (DL) has impacted clinical discoveries and biomarker predictions. DL-driven discoveries and predictions in oncology are based on a variety of biological data such as genomics, proteomics, and imaging data. DL-based computational frameworks can predict genetic variant effects on gene expression, as well as protein structures based on amino acid sequences. Furthermore, DL algorithms can capture valuable mechanistic biological information from several spatial "omics" technologies, such as spatial transcriptomics and spatial proteomics. Here, we review the impact that the combination of artificial intelligence (AI) with spatial omics technologies has had on oncology, focusing on DL and its applications in biomedical image analysis, encompassing cell segmentation, cell phenotype identification, cancer prognostication, and therapy prediction. We highlight the advantages of using highly multiplexed images (spatial proteomics data) compared to single-stained, conventional histopathological ("simple") images, as the former can provide deep mechanistic insights that cannot be obtained by the latter, even with the aid of explainable AI. Furthermore, we provide the reader with the advantages/disadvantages of DL-based pipelines used in preprocessing highly multiplexed images (cell segmentation, cell type annotation). Therefore, this review also guides the reader to choose the DL-based pipeline that best fits their data. In conclusion, DL continues to be established as an essential tool in discovering novel biological mechanisms when combined with technologies such as highly multiplexed tissue imaging data. In balance with conventional medical data, its role in clinical routine will become more important, supporting diagnosis and prognosis in oncology, enhancing clinical decision-making, and improving the quality of care for patients. Since its introduction into the field of oncology, deep learning (DL) has impacted clinical discoveries and biomarker predictions. DL-driven discoveries and predictions in oncology are based on a variety of biological data such as genomics, proteomics, and imaging data. DL-based computational frameworks can predict genetic variant effects on gene expression, as well as protein structures based on amino acid sequences. Furthermore, DL algorithms can capture valuable mechanistic biological information from several spatial "omics" technologies, such as spatial transcriptomics and spatial proteomics. Here, we review the impact that the combination of artificial intelligence (AI) with spatial omics technologies has had on oncology, focusing on DL and its applications in biomedical image analysis, encompassing cell segmentation, cell phenotype identification, cancer prognostication, and therapy prediction. We highlight the advantages of using highly multiplexed images (spatial proteomics data) compared to single-stained, conventional histopathological ("simple") images, as the former can provide deep mechanistic insights that cannot be obtained by the latter, even with the aid of explainable AI. Furthermore, we provide the reader with the advantages/disadvantages of the DL-based pipelines used in preprocessing the highly multiplexed images (cell segmentation, cell type annotation). Therefore, this review also guides the reader to choose the DL-based pipeline that best fits their data. In conclusion, DL continues to be established as an essential tool in discovering novel biological mechanisms when combined with technologies such as highly multiplexed tissue imaging data. In balance with conventional medical data, its role in clinical routine will become more important, supporting diagnosis and prognosis in oncology, enhancing clinical decision-making, and improving the quality of care for patients.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

An open source pipeline for quantitative immunohistochemistry image analysis of inflammatory skin disease using artificial intelligence

BACKGROUND

The application of artificial intelligence (AI) to whole slide images has the potential to improve research reliability and ultimately diagnostic efficiency and service capacity. Image annotation plays a key role in AI and digital pathology. However, the work-streams required for tissue-specific (skin) and immunostain-specific annotation has not been extensively studied compared with the development of AI algorithms.

OBJECTIVES

The objective of this study is to develop a common workflow for annotating whole slide images of biopsies from inflammatory skin disease immunostained with a variety of epidermal and dermal markers prior to the development of the AI-assisted analysis pipeline.

METHODS

A total of 45 slides containing 3-5 sections each were scanned using Aperio AT2 slide scanner (Leica Biosystems). These slides were annotated by hand using a commonly used image analysis tool which resulted in more than 4000 images blocks. We used deep learning (DL) methodology to first sequentially segment (epidermis and upper dermis), with the exclusion of common artefacts and second to quantify the immunostained signal in those two compartments of skin biopsies and the ratio of positive cells.

RESULTS

We validated two DL models using 10-fold validation runs and by comparing to ground truth manually annotated data. The models achieved an average (global) accuracy of 95.0% for the segmentation of epidermis and dermis and 86.1% for the segmentation of positive/negative cells.

CONCLUSIONS

The application of two DL models in sequence facilitates accurate segmentation of epidermal and dermal structures, exclusion of common artefacts and enables the quantitative analysis of the immunostained signal. However, inaccurate annotation of the slides for training the DL model can decrease the accuracy of the output. Our open source code will facilitate further external validation across different immunostaining platforms and slide scanners.

Determining HER2 Status by Artificial Intelligence: An Investigation of Primary, Metastatic, and HER2 Low Breast Tumors

The expression of human epidermal growth factor receptor 2 (HER2) protein or gene transcripts is critical for therapeutic decision making in breast cancer. We examined the performance of a digitalized and artificial intelligence (AI)-assisted workflow for HER2 status determination in accordance with the American Society of Clinical Oncology (ASCO)/College of Pathologists (CAP) guidelines. Our preliminary cohort consisted of 495 primary breast carcinomas, and our study cohort included 67 primary breast carcinomas and 30 metastatic deposits, which were evaluated for HER2 status by immunohistochemistry (IHC) and in situ hybridization (ISH). Three practicing breast pathologists independently assessed and scored slides, building the ground truth. Following a washout period, pathologists were provided with the results of the AI digital image analysis (DIA) and asked to reassess the slides. Both rounds of assessment from the pathologists were compared to the AI results and ground truth for each slide. We observed an overall HER2 positivity rate of 15% in our study cohort. Moderate agreement (Cohen's κ 0.59) was observed between the ground truth and AI on IHC, with most discrepancies occurring between 0 and 1+ scores. Inter-observer agreement amongst pathologists was substantial (Fleiss´ κ 0.77) and pathologists' agreement with AI scores was 80.6%. Substantial agreement of the AI with the ground truth (Cohen´s κ 0.80) was detected on ISH-stained slides, and the accuracy of AI was similar for the primary and metastatic tumors. We demonstrated the feasibility of a combined HER2 IHC and ISH AI workflow, with a Cohen's κ of 0.94 when assessed in accordance with the ASCO/CAP recommendations.

Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review

Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.

Breast Cancer Dataset, Classification and Detection Using Deep Learning

Incorporating scientific research into clinical practice via clinical informatics, which includes genomics, proteomics, bioinformatics, and biostatistics, improves patients' treatment. Computational pathology is a growing subspecialty with the potential to integrate whole slide images, multi-omics data, and health informatics. Pathology and laboratory medicine are critical to diagnosing cancer. This work will review existing computational and digital pathology methods for breast cancer diagnosis with a special focus on deep learning. The paper starts by reviewing public datasets related to breast cancer diagnosis. Additionally, existing deep learning methods for breast cancer diagnosis are reviewed. The publicly available code repositories are introduced as well. The paper is closed by highlighting challenges and future works for deep learning-based diagnosis.

Clinical implication of low estrogen receptor (ER-low) expression in breast cancer

Breast cancer is a heterogeneous disease, and the estrogen receptor (ER) remains the most important biomarker in breast oncology. Most guidelines set a positive expression threshold of 1% staining in immunohistochemistry (IHC) to define ER positivity. However, different expression levels may be associated with diverse degrees of sensitivity to endocrine therapy as ER expression may impact breast cancer molecular biology as a continuous variable. ER-lo tumors, defined as those with 1-10% ER expression, represent a relatively small subgroup of breast cancer patients, with an estimated prevalence of 2-7%. These tumors are similar to ERneg disease in their molecular landscape, clinicopathological characteristics, prognosis, and response to therapy. Nevertheless, a proportion may retain some degree of ER signaling dependency, and the possibility of responding to some degree to endocrine therapy cannot be completely ruled out. This review article discusses the most important considerations regarding the definition of ER positivity, pathology assessment, prognosis, and therapeutic implication of ERlo breast cancer from the medical oncology perspective.

Strategies for Enhancing the Multi-Stage Classification Performances of HER2 Breast Cancer from Hematoxylin and Eosin Images

Breast cancer is a significant health concern among women. Prompt diagnosis can diminish the mortality rate and direct patients to take steps for cancer treatment. Recently, deep learning has been employed to diagnose breast cancer in the context of digital pathology. To help in this area, a transfer learning-based model called 'HE-HER2Net' has been proposed to diagnose multiple stages of HER2 breast cancer (HER2-0, HER2-1+, HER2-2+, HER2-3+) on H&E (hematoxylin & eosin) images from the BCI dataset. HE-HER2Net is the modified version of the Xception model, which is additionally comprised of global average pooling, several batch normalization layers, dropout layers, and dense layers with a swish activation function. This proposed model exceeds all existing models in terms of accuracy (0.87), precision (0.88), recall (0.86), and AUC score (0.98) immensely. In addition, our proposed model has been explained through a class-discriminative localization technique using Grad-CAM to build trust and to make the model more transparent. Finally, nuclei segmentation has been performed through the StarDist method.

Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer

Programmed death ligand-1 (PD-L1) has been recently adopted for breast cancer as a predictive biomarker for immunotherapies. The cost, time, and variability of PD-L1 quantification by immunohistochemistry (IHC) are a challenge. In contrast, hematoxylin and eosin (H&E) is a robust staining used routinely for cancer diagnosis. Here, we show that PD-L1 expression can be predicted from H&E-stained images by employing state-of-the-art deep learning techniques. With the help of two expert pathologists and a designed annotation software, we construct a dataset to assess the feasibility of PD-L1 prediction from H&E in breast cancer. In a cohort of 3,376 patients, our system predicts the PD-L1 status in a high area under the curve (AUC) of 0.91 - 0.93. Our system is validated on two external datasets, including an independent clinical trial cohort, showing consistent prediction performance. Furthermore, the proposed system predicts which cases are prone to pathologists miss-interpretation, showing it can serve as a decision support and quality assurance system in clinical practice.

Label-Free Virtual HER2 Immunohistochemical Staining of Breast Tissue using Deep Learning

The immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) biomarker is widely practiced in breast tissue analysis, preclinical studies, and diagnostic decisions, guiding cancer treatment and investigation of pathogenesis. HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist, which typically takes one day to prepare in a laboratory, increasing analysis time and associated costs. Here, we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images, matching the standard HER2 IHC staining that is chemically performed on the same tissue sections. The efficacy of this virtual HER2 staining framework was demonstrated by quantitative analysis, in which three board-certified breast pathologists blindly graded the HER2 scores of virtually stained and immunohistochemically stained HER2 whole slide images (WSIs) to reveal that the HER2 scores determined by inspecting virtual IHC images are as accurate as their immunohistochemically stained counterparts. A second quantitative blinded study performed by the same diagnosticians further revealed that the virtually stained HER2 images exhibit a comparable staining quality in the level of nuclear detail, membrane clearness, and absence of staining artifacts with respect to their immunohistochemically stained counterparts. This virtual HER2 staining framework bypasses the costly, laborious, and time-consuming IHC staining procedures in laboratory and can be extended to other types of biomarkers to accelerate the IHC tissue staining used in life sciences and biomedical workflow.

HEROHE Challenge: Predicting HER2 Status in Breast Cancer from Hematoxylin–Eosin Whole-Slide Imaging

Breast cancer is the most common malignancy in women worldwide, and is responsible for more than half a million deaths each year. The appropriate therapy depends on the evaluation of the expression of various biomarkers, such as the human epidermal growth factor receptor 2 (HER2) transmembrane protein, through specialized techniques, such as immunohistochemistry or in situ hybridization. In this work, we present the HER2 on hematoxylin and eosin (HEROHE) challenge, a parallel event of the 16th European Congress on Digital Pathology, which aimed to predict the HER2 status in breast cancer based only on hematoxylin–eosin-stained tissue samples, thus avoiding specialized techniques. The challenge consisted of a large, annotated, whole-slide images dataset (509), specifically collected for the challenge. Models for predicting HER2 status were presented by 21 teams worldwide. The best-performing models are presented by detailing the network architectures and key parameters. Methods are compared and approaches, core methodologies, and software choices contrasted. Different evaluation metrics are discussed, as well as the performance of the presented models for each of these metrics. Potential differences in ranking that would result from different choices of evaluation metrics highlight the need for careful consideration at the time of their selection, as the results show that some metrics may misrepresent the true potential of a model to solve the problem for which it was developed. The HEROHE dataset remains publicly available to promote advances in the field of computational pathology.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Pediatric Sarcomas: The Next Generation of Molecular Studies

Pediatric sarcomas constitute one of the largest groups of childhood cancers, following hematopoietic, neural, and renal lesions. Partly because of their diversity, they continue to offer challenges in diagnosis and treatment. In spite of the diagnostic, nosologic, and therapeutic gains made with genetic technology, newer means for investigation are needed. This article reviews emerging technology being used to study human neoplasia and how these methods might be applicable to pediatric sarcomas. Methods reviewed include single cell RNA sequencing (scRNAseq), spatial multi-omics, high-throughput functional genomics, and clustered regularly interspersed short palindromic sequence-Cas9 (CRISPR-Cas9) technology. In spite of these advances, the field continues to be challenged by a dearth of properly annotated materials, particularly from recurrences and metastases and pre- and post-treatment samples.

Breast Cancer Molecular Subtype Prediction on Pathological Images with Discriminative Patch Selection and Multi-Instance Learning

Molecular subtypes of breast cancer are important references to personalized clinical treatment. For cost and labor savings, only one of the patient’s paraffin blocks is usually selected for subsequent immunohistochemistry (IHC) to obtain molecular subtypes. Inevitable block sampling error is risky due to the tumor heterogeneity and could result in a delay in treatment. Molecular subtype prediction from conventional H&amp;E pathological whole slide images (WSI) using the AI method is useful and critical to assist pathologists to pre-screen proper paraffin block for IHC. It is a challenging task since only WSI-level labels of molecular subtypes from IHC can be obtained without detailed local region information. Gigapixel WSIs are divided into a huge amount of patches to be computationally feasible for deep learning, while with coarse slide-level labels, patch-based methods may suffer from abundant noise patches, such as folds, overstained regions, or non-tumor tissues. A weakly supervised learning framework based on discriminative patch selection and multi-instance learning was proposed for breast cancer molecular subtype prediction from H&amp;E WSIs. Firstly, co-teaching strategy using two networks was adopted to learn molecular subtype representations and filter out some noise patches. Then, a balanced sampling strategy was used to handle the imbalance in subtypes in the dataset. In addition, a noise patch filtering algorithm that used local outlier factor based on cluster centers was proposed to further select discriminative patches. Finally, a loss function integrating local patch with global slide constraint information was used to fine-tune MIL framework on obtained discriminative patches and further improve the prediction performance of molecular subtyping. The experimental results confirmed the effectiveness of the proposed AI method and our models outperformed even senior pathologists, which has the potential to assist pathologists to pre-screen paraffin blocks for IHC in clinic.

Progress on deep learning in digital pathology of breast cancer: a narrative review

BACKGROUND AND OBJECTIVE

Pathology is the gold standard criteria for breast cancer diagnosis and has important guiding value in formulating the clinical treatment plan and predicting the prognosis. However, traditional microscopic examinations of tissue sections are time consuming and labor intensive, with unavoidable subjective variations. Deep learning (DL) can evaluate and extract the most important information from images with less need for human instruction, providing a promising approach to assist in the pathological diagnosis of breast cancer. To provide an informative and up-to-date summary on the topic of DL-based diagnostic systems for breast cancer pathology image analysis and discuss the advantages and challenges to the routine clinical application of digital pathology.

METHODS

A PubMed search with keywords ("breast neoplasm" or "breast cancer") and ("pathology" or "histopathology") and ("artificial intelligence" or "deep learning") was conducted. Relevant publications in English published from January 2000 to October 2021 were screened manually for their title, abstract, and even full text to determine their true relevance. References from the searched articles and other supplementary articles were also studied.

KEY CONTENT AND FINDINGS

DL-based computerized image analysis has obtained impressive achievements in breast cancer pathology diagnosis, classification, grading, staging, and prognostic prediction, providing powerful methods for faster, more reproducible, and more precise diagnoses. However, all artificial intelligence (AI)-assisted pathology diagnostic models are still in the experimental stage. Improving their economic efficiency and clinical adaptability are still required to be developed as the focus of further researches.

CONCLUSIONS

Having searched PubMed and other databases and summarized the application of DL-based AI models in breast cancer pathology, we conclude that DL is undoubtedly a promising tool for assisting pathologists in routines, but further studies are needed to realize the digitization and automation of clinical pathology.

Artificial intelligence to identify genetic alterations in conventional histopathology

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold-standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53 and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows. This article is protected by copyright. All rights reserved.

A whole-slide image (WSI)-based immunohistochemical feature prediction system improves the subtyping of lung cancer

BACKGROUND

Clinically, accurate pathological diagnosis is often challenged by insufficient tissue amounts and the unaffordability of additional immunohistochemical or genetic tests; thus, there is an urgent need for a universal approach to improve the subtyping of lung cancer without the above limitations. Here we aimed to develop a deep learning system to predict the immunohistochemistry (IHC) phenotype directly from whole-slide images (WSIs) to improve the subtyping of lung cancer from surgical resection and biopsy specimens.

METHODS

A total of 1914 patients with lung cancer from three independent hospitals in China were enrolled for WSI-based immunohistochemical feature prediction system (WIFPS) development and validation.

RESULTS

The WIFPS could directly predict the IHC status of nine subtype-specific biomarkers, including CK7, TTF-1, Napsin A, CK5/6, P63, P40, CD56, Synaptophysin, and Chromogranin A, achieving average areas under the curve (AUCs) of 0.912, 0.906, and 0.888 and overall diagnostic accuracies of 0.925, 0.941, and 0.887 in the validation datasets of total, external surgical resection specimens and biopsy specimens, respectively. The histological subtyping performance of the WIFPS remained comparable with that of general pathologists (GPs), with Cohen's kappa values ranging from 0.7646 to 0.8282. Furthermore, the WIFPS could be trained to not only predict the IHC status of anaplastic lymphoma kinase (ALK), programmed death-1 (PD-1), and programmed death ligand 1 (PD-L1), but also predict EGFR and KRAS mutation status, with AUCs from 0.525 to 0.917, as detected in separate populations.

CONCLUSIONS

In this study, the WIFPS showed its proficiency as a useful complement to traditional histologic subtyping for integrated immunohistochemical spectrum prediction as well as potential in the detection of gene mutations.

Outcome and Biomarker Supervised Deep Learning for Survival Prediction in Two Multicenter Breast Cancer Series

BACKGROUND

Prediction of clinical outcomes for individual cancer patients is an important step in the disease diagnosis and subsequently guides the treatment and patient counseling. In this work, we develop and evaluate a joint outcome and biomarker supervised (estrogen receptor expression and ERBB2 expression and gene amplification) multitask deep learning model for prediction of outcome in breast cancer patients in two nation-wide multicenter studies in Finland (the FinProg and FinHer studies). Our approach combines deep learning with expert knowledge to provide more accurate, robust, and integrated prediction of breast cancer outcomes.

MATERIALS AND METHODS

Using deep learning, we trained convolutional neural networks (CNNs) with digitized tissue microarray (TMA) samples of primary hematoxylin-eosin-stained breast cancer specimens from 693 patients in the FinProg series as input and breast cancer-specific survival as the endpoint. The trained algorithms were tested on 354 TMA patient samples in the same series. An independent set of whole-slide (WS) tumor samples from 674 patients in another multicenter study (FinHer) was used to validate and verify the generalization of the outcome prediction based on CNN models by Cox survival regression and concordance index (c-index). Visual cancer tissue characterization, i.e., number of mitoses, tubules, nuclear pleomorphism, tumor-infiltrating lymphocytes, and necrosis was performed on TMA samples in the FinProg test set by a pathologist and combined with deep learning-based outcome prediction in a multitask algorithm.

RESULTS

The multitask algorithm achieved a hazard ratio (HR) of 2.0 (95% confidence interval [CI] 1.30-3.00), P < 0.001, c-index of 0.59 on the 354 test set of FinProg patients, and an HR of 1.7 (95% CI 1.2-2.6), P = 0.003, c-index 0.57 on the WS tumor samples from 674 patients in the independent FinHer series. The multitask CNN remained a statistically independent predictor of survival in both test sets when adjusted for histological grade, tumor size, and axillary lymph node status in a multivariate Cox analyses. An improved accuracy (c-index 0.66) was achieved when deep learning was combined with the tissue characteristics assessed visually by a pathologist.

CONCLUSIONS

A multitask deep learning algorithm supervised by both patient outcome and biomarker status learned features in basic tissue morphology predictive of survival in a nationwide, multicenter series of patients with breast cancer. The algorithms generalized to another independent multicenter patient series and whole-slide breast cancer samples and provide prognostic information complementary to that of a comprehensive series of established prognostic factors.

L’intelligence artificielle pourrait-elle intervenir dans l’aide au diagnostic des cancers du sein ? – L’exemple de HER2

HER2 is an important prognostic and predictive biomarker in breast cancer. Its detection makes it possible to define which patients will benefit from a targeted treatment. While assessment of HER2 status by immunohistochemistry in positive vs negative categories is well implemented and reproducible, the introduction of a new "HER2-low" category could raise some concerns about its scoring and reproducibility. We herein described the current HER2 testing methods and the application of innovative machine learning techniques to improve these determinations, as well as the main challenges and opportunities related to the implementation of digital pathology in the up-and-coming AI era.

Digital pathology and artificial intelligence in translational medicine and clinical practice

Traditional pathology approaches have played an integral role in the delivery of diagnosis, semi-quantitative or qualitative assessment of protein expression, and classification of disease. Technological advances and the increased focus on precision medicine have recently paved the way for the development of digital pathology-based approaches for quantitative pathologic assessments, namely whole slide imaging and artificial intelligence (AI)-based solutions, allowing us to explore and extract information beyond human visual perception. Within the field of immuno-oncology, the application of such methodologies in drug development and translational research have created invaluable opportunities for deciphering complex pathophysiology and the discovery of novel biomarkers and drug targets. With an increasing number of treatment options available for any given disease, practitioners face the growing challenge of selecting the most appropriate treatment for each patient. The ever-increasing utilization of AI-based approaches substantially expands our understanding of the tumor microenvironment, with digital approaches to patient stratification and selection for diagnostic assays supporting the identification of the optimal treatment regimen based on patient profiles. This review provides an overview of the opportunities and limitations around implementing AI-based methods in biomarker discovery and patient selection and discusses how advances in digital pathology and AI should be considered in the current landscape of translational medicine, touching on challenges this technology may face if adopted in clinical settings. The traditional role of pathologists in delivering accurate diagnoses or assessing biomarkers for companion diagnostics may be enhanced in precision, reproducibility, and scale by AI-powered analysis tools.

Artificial intelligence applied to breast pathology

The convergence of digital pathology and computer vision is increasingly enabling computers to perform tasks performed by humans. As a result, artificial intelligence (AI) is having an astoundingly positive effect on the field of pathology, including breast pathology. Research using machine learning and the development of algorithms that learn patterns from labeled digital data based on "deep learning" neural networks and feature-engineered approaches to analyze histology images have recently provided promising results. Thus far, image analysis and more complex AI-based tools have demonstrated excellent success performing tasks such as the quantification of breast biomarkers and Ki67, mitosis detection, lymph node metastasis recognition, tissue segmentation for diagnosing breast carcinoma, prognostication, computational assessment of tumor-infiltrating lymphocytes, and prediction of molecular expression as well as treatment response and benefit of therapy from routine H&E images. This review critically examines the literature regarding these applications of AI in the area of breast pathology.

Molecular Biomarkers Predict Pathological Complete Response of Neoadjuvant Chemotherapy in Breast Cancer Patients: Review

Simple Summary

Breast cancer is the most common cancer in women worldwide. Although many studies have aimed to understand the genetic basis of breast cancer, leading to increasingly accurate diagnoses, only a few molecular biomarkers are used in clinical practice to predict response to therapy. Current studies aim to develop more personalized therapies to decrease the adverse effects of chemotherapy. Personalized medicine not only requires clinical, but also molecular characterization of tumors, which allows the use of more effective drugs for each patient. The aim of this study was to identify potential molecular biomarkers that can predict the response to therapy after neoadjuvant chemotherapy in patients with breast cancer. In this review, we summarize genomic, transcriptomic, and proteomic biomarkers that can help predict the response to therapy.

Abstract

Neoadjuvant chemotherapy (NAC) is often used to treat locally advanced disease for tumor downstaging, thus improving the chances of breast-conserving surgery. From the NAC response, it is possible to obtain prognostic information as patients may reach a pathological complete response (pCR). Those who do might have significant advantages in terms of survival rates. Breast cancer (BC) is a heterogeneous disease that requires personalized treatment strategies. The development of targeted therapies depends on identifying biomarkers that can be used to assess treatment efficacy as well as the discovery of new and more accurate therapeutic agents. With the development of new “OMICS” technologies, i.e., genomics, transcriptomics, and proteomics, among others, the discovery of new biomarkers is increasingly being used in the context of clinical practice, bringing us closer to personalized management of BC treatment. The aim of this review is to compile the main biomarkers that predict pCR in BC after NAC.

A Pyramid Architecture-Based Deep Learning Framework for Breast Cancer Detection

Breast cancer diagnosis is a critical step in clinical decision making, and this is achieved by making a pathological slide and gives a decision by the doctors, which is the method of final decision making for cancer diagnosis. Traditionally, the doctors usually check the pathological images by visual inspection under the microscope. Whole-slide images (WSIs) have supported the state-of-the-art diagnosis results and have been admitted as the gold standard clinically. However, this task is time-consuming and labour-intensive, and all of these limitations make low efficiency in decision making. Medical image processing protocols have been used for this task during the last decades and have obtained satisfactory results under some conditions; especially in the deep learning era, it has exhibited the advantages than those in the shallow learning period. In this paper, we proposed a novel breast cancer region mining framework based on deep pyramid architecture from multilevel and multiscale breast pathological WSIs. We incorporate the tissue- and cell-level information together and integrate these into a LSTM model for the final sequence modelling, which successfully keeps the WSIs' integration and is not mentioned by the prevalence frameworks. The experiment results demonstrated that our proposed framework greatly improved the detection accuracy than that only using tissue-level information.

Deep learning-based predictive biomarker of pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer

Background

Pathological complete response (pCR) is considered a surrogate endpoint for favorable survival in breast cancer patients treated with neoadjuvant chemotherapy (NAC). Predictive biomarkers of treatment response are crucial for guiding treatment decisions. With the hypothesis that histological information on tumor biopsy images could predict NAC response in breast cancer, we proposed a novel deep learning (DL)-based biomarker that predicts pCR from images of hematoxylin and eosin (H&E)-stained tissue and evaluated its predictive performance.

Methods

In total, 540 breast cancer patients receiving standard NAC were enrolled. Based on H&E-stained images, DL methods were employed to automatically identify tumor epithelium and predict pCR by scoring the identified tumor epithelium to produce a histopathological biomarker, the pCR-score. The predictive performance of the pCR-score was assessed and compared with that of conventional biomarkers including stromal tumor-infiltrating lymphocytes (sTILs) and subtype.

Results

The pCR-score derived from H&E staining achieved an area under the curve (AUC) of 0.847 in predicting pCR directly, and achieved accuracy, F1 score, and AUC of 0.853, 0.503, and 0.822 processed by the logistic regression method, respectively, higher than either sTILs or subtype; a prediction model of pCR constructed by integrating sTILs, subtype and pCR-score yielded a mean AUC of 0.890, outperforming the baseline sTIL-subtype model by 0.051 (0.839, P  =  0.001).

Conclusion

The DL-based pCR-score from histological images is predictive of pCR better than sTILs and subtype, and holds the great potentials for a more accurate stratification of patients for NAC.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12967-021-03020-z.

Deep learning-based classification of breast cancer cells using transmembrane receptor dynamics

MOTIVATION

Motions of transmembrane receptors on cancer cell surfaces can reveal biophysical features of the cancer cells, thus providing a method for characterizing cancer cell phenotypes. While conventional analysis of receptor motions in the cell membrane mostly relies on the mean-squared displacement plots, much information is lost when producing these plots from the trajectories. Here we employ deep learning to classify breast cancer cell types based on the trajectories of epidermal growth factor receptor (EGFR). Our model is an artificial neural network trained on the EGFR motions acquired from six breast cancer cell lines of varying invasiveness and receptor status: MCF7 (hormone receptor positive), BT474 (HER2-positive), SKBR3 (HER2-positive), MDA-MB-468 (triple negative, TN), MDA-MB-231 (TN) and BT549 (TN).

RESULTS

The model successfully classified the trajectories within individual cell lines with 83% accuracy and predicted receptor status with 85% accuracy. To further validate the method, epithelial-mesenchymal transition (EMT) was induced in benign MCF10A cells, noninvasive MCF7 cancer cells and highly invasive MDA-MB-231 cancer cells, and EGFR trajectories from these cells were tested. As expected, after EMT induction, both MCF10A and MCF7 cells showed higher rates of classification as TN cells, but not the MDA-MB-231 cells. Whereas deep learning-based cancer cell classifications are primarily based on the optical transmission images of cell morphology and the fluorescence images of cell organelles or cytoskeletal structures, here we demonstrated an alternative way to classify cancer cells using a dynamic, biophysical feature that is readily accessible.

AVAILABILITY AND IMPLEMENTATION

A python implementation of deep learning-based classification can be found at https://github.com/soonwoohong/Deep-learning-for-EGFR-trajectory-classification.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma

Deep learning has emerged as the technique of choice for identifying hidden patterns in cell imaging data but is often criticized as "black box." Here, we employ a generative neural network in combination with supervised machine learning to classify patient-derived melanoma xenografts as "efficient" or "inefficient" metastatic, validate predictions regarding melanoma cell lines with unknown metastatic efficiency in mouse xenografts, and use the network to generate in silico cell images that amplify the critical predictive cell properties. These exaggerated images unveiled pseudopodial extensions and increased light scattering as hallmark properties of metastatic cells. We validated this interpretation using live cells spontaneously transitioning between states indicative of low and high metastatic efficiency. This study illustrates how the application of artificial intelligence can support the identification of cellular properties that are predictive of complex phenotypes and integrated cell functions but are too subtle to be identified in the raw imagery by a human expert. A record of this paper's transparent peer review process is included in the supplemental information. VIDEO ABSTRACT.

Prediction of BRCA Gene Mutation in Breast Cancer Based on Deep Learning and Histopathology Images

BACKGROUND

Breast cancer is one of the most common cancers and the leading cause of death from cancer among women worldwide. The genetic predisposition to breast cancer may be associated with a mutation in particular genes such as gene BRCA1/2. Patients who carry a germline pathogenic mutation in BRCA1/2 genes have a significantly increased risk of developing breast cancer and might benefit from targeted therapy. However, genetic testing is time consuming and costly. This study aims to predict the risk of gBRCA mutation by using the whole-slide pathology features of breast cancer H&E stains and the patients' gBRCA mutation status.

METHODS

In this study, we trained a deep convolutional neural network (CNN) of ResNet on whole-slide images (WSIs) to predict the gBRCA mutation in breast cancer. Since the dimensions are too large for slide-based training, we divided WSI into smaller tiles with the original resolution. The tile-based classification was then combined by adding the positive classification result to generate the combined slide-based accuracy. Models were trained based on the annotated tumor location and gBRCA mutation status labeled by a designated breast cancer pathologist. Four models were trained on tiles cropped at 5×, 10×, 20×, and 40× magnification, assuming that low magnification and high magnification may provide different levels of information for classification.

RESULTS

A trained model was validated through an external dataset that contains 17 mutants and 47 wilds. In the external validation dataset, AUCs (95% CI) of DL models that used 40×, 20×, 10×, and 5× magnification tiles among all cases were 0.766 (0.763-0.769), 0.763 (0.758-0.769), 0.750 (0.738-0.761), and 0.551 (0.526-0.575), respectively, while the corresponding magnification slides among all cases were 0.774 (0.642-0.905), 0.804 (0.676-0.931), 0.828 (0.691-0.966), and 0.635 (0.471-0.798), respectively. The study also identified the influence of histological grade to the accuracy of the prediction.

CONCLUSION

In this paper, the combination of pathology and molecular omics was used to establish the gBRCA mutation risk prediction model, revealing the correlation between the whole-slide histopathological images and gRCA mutation risk. The results indicated that the prediction accuracy is likely to improve as the training data expand. The findings demonstrated that deep CNNs could be used to assist pathologists in the detection of gene mutation in breast cancer.

The impact of site-specific digital histology signatures on deep learning model accuracy and bias

The Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. Site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the image characteristics constituting this site-specific digital histology signature. We demonstrate that these site-specific signatures lead to biased accuracy for prediction of features including survival, genomic mutations, and tumor stage. Furthermore, ethnicity can also be inferred from site-specific signatures, which must be accounted for to ensure equitable application of DL. These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.

Determining breast cancer biomarker status and associated morphological features using deep learning

Background

Breast cancer management depends on biomarkers including estrogen receptor, progesterone receptor, and human epidermal growth factor receptor 2 (ER/PR/HER2). Though existing scoring systems are widely used and well-validated, they can involve costly preparation and variable interpretation. Additionally, discordances between histology and expected biomarker findings can prompt repeat testing to address biological, interpretative, or technical reasons for unexpected results.

Methods

We developed three independent deep learning systems (DLS) to directly predict ER/PR/HER2 status for both focal tissue regions (patches) and slides using hematoxylin-and-eosin-stained (H&E) images as input. Models were trained and evaluated using pathologist annotated slides from three data sources. Areas under the receiver operator characteristic curve (AUCs) were calculated for test sets at both a patch-level (>135 million patches, 181 slides) and slide-level (n = 3274 slides, 1249 cases, 37 sites). Interpretability analyses were performed using Testing with Concept Activation Vectors (TCAV), saliency analysis, and pathologist review of clustered patches.

Results

The patch-level AUCs are 0.939 (95%CI 0.936–0.941), 0.938 (0.936–0.940), and 0.808 (0.802–0.813) for ER/PR/HER2, respectively. At the slide level, AUCs are 0.86 (95%CI 0.84–0.87), 0.75 (0.73–0.77), and 0.60 (0.56–0.64) for ER/PR/HER2, respectively. Interpretability analyses show known biomarker-histomorphology associations including associations of low-grade and lobular histology with ER/PR positivity, and increased inflammatory infiltrates with triple-negative staining.

Conclusions

This study presents rapid breast cancer biomarker estimation from routine H&E slides and builds on prior advances by prioritizing interpretability of computationally learned features in the context of existing pathological knowledge.

Artificial intelligence grading of breast cancer: a promising method to refine prognostic classification for management precision

AIM

Artificial intelligence (AI)-based breast cancer grading may help to overcome perceived limitations of human assessment. Here, the potential value of AI grade was evaluated at the molecular level and in predicting patient outcome.

METHODS AND RESULTS

A supervised convolutional neural network (CNN) model was trained on images of 612 breast cancers from The Cancer Genome Atlas (TCGA). The test set, obtained from the Cooperative Human Tissue Network (CHTN), comprised 1058 cancers with corresponding survival data. Upon reversal, a CNN was trained from images of 1537 CHTN cancers and tested on 397 TCGA cancers. In TCGA, mRNA models were trained using AI grade and Nottingham grade (NG) as labels. Performance of mRNA models in predicting patient outcome was evaluated using data from 1807 cancers from the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) cohort. In selecting images for training, nucleolar prominence determined high- versus low-grade cancer cells. In CHTN, NG corresponded to significant survival stratification in stages 1, 2 and 3 cancers, while AI grade showed significance in stages 1 and 2 and borderline in stage 3 tumours. In METABRIC, the mRNA model trained from AI grade was not significantly different to the NG-based model. The gene which best described AI grade was TRIP13, a gene involved with mitotic spindle assembly.

CONCLUSION

An AI grade trained from the morphologically distinctive feature of nucleolar prominence could transmit significant patient outcome information across three independent patient cohorts. AI grade shows promise in gene discovery and for second opinions.

Machine learning-based image analysis for accelerating the diagnosis of complicated preneoplastic and neoplastic ductal lesions in breast biopsy tissues

PURPOSE

Diagnosis of breast preneoplastic and neoplastic lesions is difficult due to their similar morphology in breast biopsy specimens. To diagnose these lesions, pathologists perform immunohistochemical analysis and consult with expert breast pathologists. These additional examinations are time-consuming and expensive. Artificial intelligence (AI)-based image analysis has recently improved, and may help in ordinal pathological diagnosis. Here, we showed the significance of machine learning-based image analysis of breast preneoplastic and neoplastic lesions for facilitating high-throughput diagnosis.

METHODS

Images were obtained from normal mammary glands, hyperplastic lesions, preneoplastic lesions and neoplastic lesions, such as usual ductal hyperplasia (UDH), columnar cell lesion (CCL), ductal carcinoma in situ (DCIS), and DCIS with comedo necrosis (comedo DCIS) in breast biopsy specimens. The original enhanced convoluted neural network (CNN) system was used for analyzing the pathological images.

RESULTS

The AI-based image analysis provided the following area under the curve values (AUC): normal lesion versus DCIS, 0.9902; DCIS versus comedo DCIS, 0.9942; normal lesion versus CCL, 0.9786; and UDH versus DCIS, 1.000. Multiple comparison analysis showed precision and recall scores similar to those of single comparison analysis. Based on the gradient-weighted class activation mapping (Grad-CAM) used to visualize the important regions reflecting the result of CNN analysis, the ratio of stromal tissue in the whole weighted area was significantly higher in UDH and CCL than that in DCIS.

CONCLUSIONS

These analyses may provide a more accurate and rapid pathological diagnosis of patients. Moreover, Grad-CAM identifies uncharted important histological characteristics for newer pathological findings and targets of research for understanding diseases.

Nanogenomics and Artificial Intelligence: A Dynamic Duo for the Fight Against Breast Cancer

Application software is utilized to aid in the diagnosis of breast cancer. Yet, recent advances in artificial intelligence (AI) are addressing challenges related to the detection, classification, and monitoring of different types of tumors. AI can apply deep learning algorithms to perform automated analysis on mammographic or histologic examinations. Large volume of data generated by digitalized mammogram or whole-slide images can be interoperated through advanced machine learning. This enables fast evaluation of every tissue patch on an image, resulting in a quicker more sensitivity, and more reproducible diagnoses compared to human performance. On the other hand, cancer cell-exosomes which are extracellular vesicles released by cancer cells into the blood circulation, are being explored as cancer biomarker. Recent studies on cancer-exosome-content revealed that the encapsulated miRNA and other biomolecules are indicative of tumor sub-type, possible metastasis and prognosis. Thus, theoretically, through nanogenomicas, a profile of each breast tumor sub-type, estrogen receptor status, and potential metastasis site can be constructed. Then, a laboratory instrument, fitted with an AI program, can be used to diagnose suspected patients by matching their sera miRNA and biomolecules composition with the available template profiles. In this paper, we discuss the advantages of establishing a nanogenomics-AI-based breast cancer diagnostic approach, compared to the gold standard radiology or histology based approaches that are currently being adapted to AI. Also, we discuss the advantages of building the diagnostic and prognostic biomolecular profiles for breast cancers based on the exosome encapsulated content, rather than the free circulating miRNA and other biomolecules.

Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology

Tissue Biomarkers are information written in the tissue and used in Pathology to recognize specific subsets of patients with diagnostic, prognostic or predictive purposes, thus representing the key elements of Personalized Medicine. The advent of Artificial Intelligence (AI) promises to further reinforce the role of Pathology in the scenario of Personalized Medicine: AI-based devices are expected to standardize the evaluation of tissue biomarkers and also to discover novel information, which would otherwise be ignored by human review, and use them to make specific predictions. In this review we will present how AI has been used to support Tissue Biomarkers evaluation in the specific field of Pathology, give an insight to the intriguing field of AI-based biomarkers and discuss possible advantages, risk and perspectives for Pathology.