Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer

Deep Learning of radiology-genomics integration for computational oncology: A mini review

In the field of computational oncology, patient status is often assessed using radiology-genomics, which includes two key technologies and data, such as radiology and genomics. Recent advances in deep learning have facilitated the integration of radiology-genomics data, and even new omics data, significantly improving the robustness and accuracy of clinical predictions. These factors are driving artificial intelligence (AI) closer to practical clinical applications. In particular, deep learning models are crucial in identifying new radiology-genomics biomarkers and therapeutic targets, supported by explainable AI (xAI) methods. This review focuses on recent developments in deep learning for radiology-genomics integration, highlights current challenges, and outlines some research directions for multimodal integration and biomarker discovery of radiology-genomics or radiology-omics that are urgently needed in computational oncology.

Potential application of artificial intelligence in cancer therapy

PURPOSE OF REVIEW

This review underscores the critical role and challenges associated with the widespread adoption of artificial intelligence in cancer care to enhance disease management, streamline clinical processes, optimize data retrieval of health information, and generate and synthesize evidence.

RECENT FINDINGS

Advancements in artificial intelligence models and the development of digital biomarkers and diagnostics are applicable across the cancer continuum from early detection to survivorship care. Additionally, generative artificial intelligence has promised to streamline clinical documentation and patient communications, generate structured data for clinical trial matching, automate cancer registries, and facilitate advanced clinical decision support. Widespread adoption of artificial intelligence has been slow because of concerns about data diversity and data shift, model reliability and algorithm bias, legal oversight, and high information technology and infrastructure costs.

SUMMARY

Artificial intelligence models have significant potential to transform cancer care. Efforts are underway to deploy artificial intelligence models in the cancer practice, evaluate their clinical impact, and enhance their fairness and explainability. Standardized guidelines for the ethical integration of artificial intelligence models in cancer care pathways and clinical operations are needed. Clear governance and oversight will be necessary to gain trust in artificial intelligence-assisted cancer care by clinicians, scientists, and patients.

Correlation and predictive value of pathological complete response and ultrasound characteristic parameters in neoadjuvant chemotherapy for breast

BACKGROUND

Breast cancer ranks as one of the most prevalent malignant tumors among women, significantly endangering their health and lives. While radical surgery has been a pivotal method for halting disease progression, it alone is insufficient for enhancing the quality of life for patients.

AIM

To investigate the correlation between ultrasound characteristic parameters of breast cancer lesions and clinical efficacy in patients undergoing neoadjuvant chemotherapy (NAC).

METHODS

Employing a case-control study design, this research involved 178 breast cancer patients treated with NAC at our hospital from July 2019 to June 2022. According to the Miller-Payne grading system, the pathological response, i.e. efficacy, of the NAC in the initial breast lesion after NAC was evaluated. Of these, 59 patients achieved a pathological complete response (PCR), while 119 did not (non-PCR group). Ultrasound characteristics prior to NAC were compared between these groups, and the association of various factors with NAC efficacy was analyzed using univariate and multivariate approaches.

RESULTS

In the PCR group, the incidence of posterior echo attenuation, lesion diameter ≥ 2.0 cm, and Alder blood flow grade ≥ II were significantly lower compared to the non-PCR group (P < 0.05). The area under the curve values for predicting NAC efficacy using posterior echo attenuation, lesion diameter, and Alder grade were 0.604, 0.603, and 0.583, respectively. Also, rates of pathological stage II, lymph node metastasis, vascular invasion, and positive Ki-67 expression were significantly lower in the PCR group (P < 0.05). Logistic regression analysis identified posterior echo attenuation, lesion diameter ≥ 2.0 cm, Alder blood flow grade ≥ II, pathological stage III, vascular invasion, and positive Ki-67 expression as independent predictors of poor response to NAC in breast cancer patients (P < 0.05).

CONCLUSION

While ultrasound characteristics such as posterior echo attenuation, lesion diameter ≥ 2.0 cm, and Alder blood flow grade ≥ II exhibit limited predictive value for NAC efficacy, they are significantly associated with poor response to NAC in breast cancer patients.

Bimodal machine learning model for unstable hips in infants: integration of radiographic images with automatically-generated clinical measurements

Bimodal convolutional neural networks (CNNs) are frequently combined with patient information or several medical images to enhance the diagnostic performance. However, the technologies that integrate automatically generated clinical measurements within the images are scarce. Hence, we developed a bimodal model that produced automatic algorithm for clinical measurement (aaCM) from radiographic images and integrated the model with CNNs. In this multicenter research project, the diagnostic performance of the model was investigated with 813 radiographic hip images of infants at risk of developmental dysplasia of the hips (232 and 581 images of unstable and stable hips, respectively), with the ground truth defined by provocative examinations. The results indicated that the accuracy of aaCM was equal or higher than that of specialists, and the bimodal model showed better diagnostic performance than LightGBM, XGBoost, SVM, and single CNN models. aaCM can provide expert's knowledge in a high level, and our proposed bimodal model has better performance than the state-of-art models.

Multimodal data integration for oncology in the era of deep neural networks: a review

Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.

Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging

Explainable artificial intelligence (XAI) has experienced a vast increase in recognition over the last few years. While the technical developments are manifold, less focus has been placed on the clinical applicability and usability of systems. Moreover, not much attention has been given to XAI systems that can handle multimodal and longitudinal data, which we postulate are important features in many clinical workflows. In this study, we review, from a clinical perspective, the current state of XAI for multimodal and longitudinal datasets and highlight the challenges thereof. Additionally, we propose the XAI orchestrator, an instance that aims to help clinicians with the synopsis of multimodal and longitudinal data, the resulting AI predictions, and the corresponding explainability output. We propose several desirable properties of the XAI orchestrator, such as being adaptive, hierarchical, interactive, and uncertainty-aware.

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Ultrasound and diffuse optical tomography-transformer model for assessing pathological complete response to neoadjuvant chemotherapy in breast cancer

Significance

We evaluate the efficiency of integrating ultrasound (US) and diffuse optical tomography (DOT) images for predicting pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) in breast cancer patients. The ultrasound-diffuse optical tomography (USDOT)-Transformer model represents a significant step toward accurate prediction of pCR, which is critical for personalized treatment planning.

Aim

We aim to develop and assess the performance of the USDOT-Transformer model, which combines US and DOT images with tumor receptor biomarkers to predict the pCR of breast cancer patients under NAC.

Approach

We developed the USDOT-Transformer model using a dual-input transformer to process co-registered US and DOT images along with tumor receptor biomarkers. Our dataset comprised imaging data from 60 patients at multiple time points during their chemotherapy treatment. We used fivefold cross-validation to assess the model's performance, comparing its results against a single modality of US or DOT.

Results

The USDOT-Transformer model demonstrated excellent predictive performance, with a mean area under the receiving characteristic curve of 0.96 (95%CI: 0.93 to 0.99) across the fivefold cross-validation. The integration of US and DOT images significantly enhanced the model's ability to predict pCR, outperforming models that relied on a single imaging modality (0.87 for US and 0.82 for DOT). This performance indicates the potential of advanced deep learning techniques and multimodal imaging data for improving the accuracy (ACC) of pCR prediction.

Conclusion

The USDOT-Transformer model offers a promising non-invasive approach for predicting pCR to NAC in breast cancer patients. By leveraging the structural and functional information from US and DOT images, the model offers a faster and more reliable tool for personalized treatment planning. Future work will focus on expanding the dataset and refining the model to further improve its accuracy and generalizability.

AI Applications to Breast MRI: Today and Tomorrow

In breast imaging, there is an unrelenting increase in the demand for breast imaging services, partly explained by continuous expanding imaging indications in breast diagnosis and treatment. As the human workforce providing these services is not growing at the same rate, the implementation of artificial intelligence (AI) in breast imaging has gained significant momentum to maximize workflow efficiency and increase productivity while concurrently improving diagnostic accuracy and patient outcomes. Thus far, the implementation of AI in breast imaging is at the most advanced stage with mammography and digital breast tomosynthesis techniques, followed by ultrasound, whereas the implementation of AI in breast magnetic resonance imaging (MRI) is not moving along as rapidly due to the complexity of MRI examinations and fewer available dataset. Nevertheless, there is persisting interest in AI-enhanced breast MRI applications, even as the use of and indications of breast MRI continue to expand. This review presents an overview of the basic concepts of AI imaging analysis and subsequently reviews the use cases for AI-enhanced MRI interpretation, that is, breast MRI triaging and lesion detection, lesion classification, prediction of treatment response, risk assessment, and image quality. Finally, it provides an outlook on the barriers and facilitators for the adoption of AI in breast MRI. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 6.

Explainable 3D CNN based on baseline breast DCE-MRI to give an early prediction of pathological complete response to neoadjuvant chemotherapy

BACKGROUND

So far, baseline Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) has played a key role for the application of sophisticated artificial intelligence-based models using Convolutional Neural Networks (CNNs) to extract quantitative imaging information as earlier indicators of pathological Complete Response (pCR) achievement in breast cancer patients treated with neoadjuvant chemotherapy (NAC). However, these models did not exploit the DCE-MRI exams in their full geometry as 3D volume but analysed only few individual slices independently, thus neglecting the depth information.

METHOD

This study aimed to develop an explainable 3D CNN, which fulfilled the task of pCR prediction before the beginning of NAC, by leveraging the 3D information of post-contrast baseline breast DCE-MRI exams. Specifically, for each patient, the network took in input a 3D sequence containing the tumor region, which was previously automatically identified along the DCE-MRI exam. A visual explanation of the decision-making process of the network was also provided.

RESULTS

To the best of our knowledge, our proposal is competitive than other models in the field, which made use of imaging data alone, reaching a median AUC value of 81.8%, 95%CI [75.3%; 88.3%], a median accuracy value of 78.7%, 95%CI [74.8%; 82.5%], a median sensitivity value of 69.8%, 95%CI [59.6%; 79.9%] and a median specificity value of 83.3%, 95%CI [82.6%; 84.0%], respectively. The median and CIs were computed according to a 10-fold cross-validation scheme for 5 rounds.

CONCLUSION

Finally, this proposal holds high potential to support clinicians on non-invasively early pursuing or changing patient-centric NAC pathways.

Artificial Intelligence-Enhanced Breast MRI: Applications in Breast Cancer Primary Treatment Response Assessment and Prediction

ABSTRACT

Primary systemic therapy (PST) is the treatment of choice in patients with locally advanced breast cancer and is nowadays also often used in patients with early-stage breast cancer. Although imaging remains pivotal to assess response to PST accurately, the use of imaging to predict response to PST has the potential to not only better prognostication but also allow the de-escalation or omission of potentially toxic treatment with undesirable adverse effects, the accelerated implementation of new targeted therapies, and the mitigation of surgical delays in selected patients. In response to the limited ability of radiologists to predict response to PST via qualitative, subjective assessments of tumors on magnetic resonance imaging (MRI), artificial intelligence-enhanced MRI with classical machine learning, and in more recent times, deep learning, have been used with promising results to predict response, both before the start of PST and in the early stages of treatment. This review provides an overview of the current applications of artificial intelligence to MRI in assessing and predicting response to PST, and discusses the challenges and limitations of their clinical implementation.

Radiomic model based on magnetic resonance imaging for predicting pathological complete response after neoadjuvant chemotherapy in breast cancer patients

Purpose

To establish a model combining radiomic and clinicopathological factors based on magnetic resonance imaging to predict pathological complete response (pCR) after neoadjuvant chemotherapy in breast cancer patients.

Method

MRI images and clinicopathologic data of 329 eligible breast cancer patients from the Affiliated Hospital of Qingdao University from August 2018 to August 2022 were included in this study. All patients received neoadjuvant chemotherapy (NAC), and imaging examinations were performed before and after NAC. A total of 329 patients were randomly allocated to a training set and a test set at a ratio of 7:3. We mainly studied the following three types of prediction models: radiomic models, clinical models, and clinical-radiomic models. All models were evaluated using subject operating characteristic curve analysis and area under the curve (AUC), decision curve analysis (DCA) and calibration curves.

Results

The AUCs of the clinical prediction model, independent imaging model and clinical combined imaging model in the training set were 0.864 0.968 and 0.984, and those in the test set were 0.724, 0.754 and 0.877, respectively. According to DCA and calibration curves, the clinical-radiomic model showed good predictive performance in both the training set and the test set, and we found that we had developed a more concise clinical-radiomic nomogram.

Conclusion

We have developed a clinical-radiomic model by integrating radiomic features and clinical factors to predict pCR after NAC in breast cancer patients, thereby contributing to the personalized treatment of patients.

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

There is a wide application of deep learning technique to unimodal medical image analysis with significant classification accuracy performance observed. However, real-world diagnosis of some chronic diseases such as breast cancer often require multimodal data streams with different modalities of visual and textual content. Mammography, magnetic resonance imaging (MRI) and image-guided breast biopsy represent a few of multimodal visual streams considered by physicians in isolating cases of breast cancer. Unfortunately, most studies applying deep learning techniques to solving classification problems in digital breast images have often narrowed their study to unimodal samples. This is understood considering the challenging nature of multimodal image abnormality classification where the fusion of high dimension heterogeneous features learned needs to be projected into a common representation space. This paper presents a novel deep learning approach combining a dual/twin convolutional neural network (TwinCNN) framework to address the challenge of breast cancer image classification from multi-modalities. First, modality-based feature learning was achieved by extracting both low and high levels features using the networks embedded with TwinCNN. Secondly, to address the notorious problem of high dimensionality associated with the extracted features, binary optimization method is adapted to effectively eliminate non-discriminant features in the search space. Furthermore, a novel method for feature fusion is applied to computationally leverage the ground-truth and predicted labels for each sample to enable multimodality classification. To evaluate the proposed method, digital mammography images and digital histopathology breast biopsy samples from benchmark datasets namely MIAS and BreakHis respectively. Experimental results obtained showed that the classification accuracy and area under the curve (AUC) for the single modalities yielded 0.755 and 0.861871 for histology, and 0.791 and 0.638 for mammography. Furthermore, the study investigated classification accuracy resulting from the fused feature method, and the result obtained showed that 0.977, 0.913, and 0.667 for histology, mammography, and multimodality respectively. The findings from the study confirmed that multimodal image classification based on combination of image features and predicted label improves performance. In addition, the contribution of the study shows that feature dimensionality reduction based on binary optimizer supports the elimination of non-discriminant features capable of bottle-necking the classifier.

Deep Learning-Based Segmentation of Locally Advanced Breast Cancer on MRI in Relation to Residual Cancer Burden: A Multi-Institutional Cohort Study

BACKGROUND

While several methods have been proposed for automated assessment of breast-cancer response to neoadjuvant chemotherapy on breast MRI, limited information is available about their performance across multiple institutions.

PURPOSE

To assess the value and robustness of deep learning-derived volumes of locally advanced breast cancer (LABC) on MRI to infer the presence of residual disease after neoadjuvant chemotherapy.

STUDY TYPE

Retrospective.

SUBJECTS

Training cohort: 102 consecutive female patients with LABC scheduled for neoadjuvant chemotherapy (NAC) from a single institution (age: 25-73 years). Independent testing cohort: 55 consecutive female patients with LABC from four institutions (age: 25-72 years).

FIELD STRENGTH/SEQUENCE

Training cohort: single vendor 1.5 T or 3.0 T. Testing cohort: multivendor 3.0 T. Gradient echo dynamic contrast-enhanced sequences.

ASSESSMENT

A convolutional neural network (nnU-Net) was trained to segment LABC. Based on resulting tumor volumes, an extremely randomized tree model was trained to assess residual cancer burden (RCB)-0/I vs. RCB-II/III. An independent model was developed using functional tumor volume (FTV). Models were tested on an independent testing cohort and response assessment performance and robustness across multiple institutions were assessed.

STATISTICAL TESTS

The receiver operating characteristic (ROC) was used to calculate the area under the ROC curve (AUC). DeLong's method was used to compare AUCs. Correlations were calculated using Pearson's method. P values <0.05 were considered significant.

RESULTS

Automated segmentation resulted in a median (interquartile range [IQR]) Dice score of 0.87 (0.62-0.93), with similar volumetric measurements (R = 0.95, P < 0.05). Automated volumetric measurements were significantly correlated with FTV (R = 0.80). Tumor volume-derived from deep learning of DCE-MRI was associated with RCB, yielding an AUC of 0.76 to discriminate between RCB-0/I and RCB-II/III, performing similar to the FTV-based model (AUC = 0.77, P = 0.66). Performance was comparable across institutions (IQR AUC: 0.71-0.84).

DATA CONCLUSION

Deep learning-based segmentation estimates changes in tumor load on DCE-MRI that are associated with RCB after NAC and is robust against variations between institutions.

EVIDENCE LEVEL

2.

TECHNICAL EFFICACY

Stage 4.

Exploring Neoadjuvant Chemotherapy, Predictive Models, Radiomic, and Pathological Markers in Breast Cancer: A Comprehensive Review

Breast cancer retains its position as the most prevalent form of malignancy among females on a global scale. The careful selection of appropriate treatment for each patient holds paramount importance in effectively managing breast cancer. Neoadjuvant chemotherapy (NACT) plays a pivotal role in the comprehensive treatment of this disease. Administering chemotherapy before surgery, NACT becomes a powerful tool in reducing tumor size, potentially enabling fewer invasive surgical procedures and even rendering initially inoperable tumors amenable to surgery. However, a significant challenge lies in the varying responses exhibited by different patients towards NACT. To address this challenge, researchers have focused on developing prediction models that can identify those who would benefit from NACT and those who would not. Such models have the potential to reduce treatment costs and contribute to a more efficient and accurate management of breast cancer. Therefore, this review has two objectives: first, to identify the most effective radiomic markers correlated with NACT response, and second, to explore whether integrating radiomic markers extracted from radiological images with pathological markers can enhance the predictive accuracy of NACT response. This review will delve into addressing these research questions and also shed light on the emerging research direction of leveraging artificial intelligence techniques for predicting NACT response, thereby shaping the future landscape of breast cancer treatment.

Dendritic Cell Subpopulations Are Associated with Prognostic Characteristics of Breast Cancer after Neoadjuvant Chemotherapy-An Observational Study

Breast cancer (BC) is the most prevalent malignancy in women and researchers have strived to develop optimal strategies for its diagnosis and management. Neoadjuvant chemotherapy (NAC), which reduces tumor size, risk of metastasis and patient mortality, often also allows for a de-escalation of breast and axillary surgery. Nonetheless, complete pathological response (pCR) is achieved in no more than 40% of patients who underwent NAC. Dendritic cells (DCs) are professional antigen-presenting cells present in the tumor microenvironment. The multitude of their subtypes was shown to be associated with the pathological and clinical characteristics of BC, but it was not evaluated in BC tissue after NAC. We found that highe r densities of CD123+ plasmacytoid DCs (pDCs) were present in tumors that did not show pCR and had a higher residual cancer burden (RCB) score and class. They were of higher stage and grade and more frequently HER2-negative. The density of CD123+ pCDs was an independent predictor of pCR in the studied group. DC-LAMP+ mature DCs (mDCs) were also related to characteristics of clinical relevance (i.e., pCR, RCB, and nuclear grade), although no clear trends were identified. We conclude that CD123+ pDCs are candidates for a novel biomarker of BC response to NAC.

Bimodal artificial intelligence using TabNet for differentiating spinal cord tumors-Integration of patient background information and images

We proposed a bimodal artificial intelligence that integrates patient information with images to diagnose spinal cord tumors. Our model combines TabNet, a state-of-the-art deep learning model for tabular data for patient information, and a convolutional neural network for images. As training data, we collected 259 spinal tumor patients (158 for schwannoma and 101 for meningioma). We compared the performance of the image-only unimodal model, table-only unimodal model, bimodal model using a gradient-boosting decision tree, and bimodal model using TabNet. Our proposed bimodal model using TabNet performed best (area under the receiver-operating characteristic curve [AUROC]: 0.91) in the training data and significantly outperformed the physicians' performance. In the external validation using 62 cases from the other two facilities, our bimodal model showed an AUROC of 0.92, proving the robustness of the model. The bimodal analysis using TabNet was effective for differentiating spinal tumors.

Advances in the application of computational pathology in diagnosis, immunomicroenvironment recognition, and immunotherapy evaluation of breast cancer: a narrative review

BACKGROUND

Breast cancer (BC) is a prevalent and highly lethal malignancy affecting women worldwide. Immunotherapy has emerged as a promising therapeutic strategy for BC, offering potential improvements in patient survival. Neoadjuvant therapy (NAT) has also gained significant clinical traction. With the advancement of computer technology, Artificial Intelligence (AI) has been increasingly applied in pathology research, expanding and redefining the scope of the field. This narrative review aims to provide a comprehensive overview of the current literature on the application of computational pathology in BC, specifically focusing on diagnosis, immune microenvironment recognition, and the evaluation of immunotherapy and NAT response.

METHODS

A thorough examination of relevant literature was conducted, focusing on studies investigating the role of computational pathology in BC diagnosis, immune microenvironment recognition, and immunotherapy and NAT assessment.

RESULTS

The application of computational pathology has shown significant potential in BC management. AI-based techniques enable improved diagnosis and classification of BC subtypes, enhance the identification and characterization of the immune microenvironment, and facilitate the evaluation of immunotherapy and NAT response. However, challenges related to data quality, standardization, and algorithm development still need to be addressed.

CONCLUSION

The integration of computational pathology and AI has transformative implications for BC patient care. By leveraging AI-based technologies, clinicians can make more informed decisions in diagnosis, treatment planning, and therapeutic response assessment. Future research should focus on refining AI algorithms, addressing technical challenges, and conducting large-scale clinical validation studies to facilitate the translation of computational pathology into routine clinical practice for BC patients.

Application of Machine Learning Algorithm in Predicting Axillary Lymph Node Metastasis from Breast Cancer on Preoperative Chest CT

Axillary lymph node (ALN) status is one of the most critical prognostic factors in patients with breast cancer. However, ALN evaluation with contrast-enhanced CT (CECT) has been challenging. Machine learning (ML) is known to show excellent performance in image recognition tasks. The purpose of our study was to evaluate the performance of the ML algorithm for predicting ALN metastasis by combining preoperative CECT features of both ALN and primary tumor. This was a retrospective single-institutional study of a total of 266 patients with breast cancer who underwent preoperative chest CECT. Random forest (RF), extreme gradient boosting (XGBoost), and neural network (NN) algorithms were used. Statistical analysis and recursive feature elimination (RFE) were adopted as feature selection for ML. The best ML-based ALN prediction model for breast cancer was NN with RFE, which achieved an AUROC of 0.76 ± 0.11 and an accuracy of 0.74 ± 0.12. By comparing NN with RFE model performance with and without ALN features from CECT, NN with RFE model with ALN features showed better performance at all performance evaluations, which indicated the effect of ALN features. Through our study, we were able to demonstrate that the ML algorithm could effectively predict the final diagnosis of ALN metastases from CECT images of the primary tumor and ALN. This suggests that ML has the potential to differentiate between benign and malignant ALNs.

Multimodal Spatiotemporal Deep Learning Framework to Predict Response of Breast Cancer to Neoadjuvant Systemic Therapy

Current approaches to breast cancer therapy include neoadjuvant systemic therapy (NST). The efficacy of NST is measured by pathologic complete response (pCR). A patient who attains pCR has significantly enhanced disease-free survival progress. The accurate prediction of pCR in response to a given treatment regimen could increase the likelihood of achieving pCR and prevent toxicities caused by treatments that are not effective. Th early prediction of response to NST can increase the likelihood of survival and help with decisions regarding breast-conserving surgery. An automated NST prediction framework that is able to precisely predict which patient undergoing NST will achieve a pathological complete response (pCR) at an early stage of treatment is needed. Here, we propose an end-to-end efficient multimodal spatiotemporal deep learning framework (deep-NST) framework to predict the outcome of NST prior or at an early stage of treatment. The deep-NST model incorporates imaging data captured at different timestamps of NST regimens, a tumor's molecular data, and a patient's demographic data. The efficacy of the proposed work is validated on the publicly available ISPY-1 dataset, in terms of accuracy, area under the curve (AUC), and computational complexity. In addition, seven ablation experiments were carried out to evaluate the impact of each design module in the proposed work. The experimental results show that the proposed framework performs significantly better than other recent methods.

Predicting pathological complete response to neoadjuvant systemic therapy for triple-negative breast cancers using deep learning on multiparametric MRIs

We trained and validated a deep learning model that can predict the treatment response to neoadjuvant systemic therapy (NAST) for patients with triple negative breast cancer (TNBC). Dynamic contrast enhanced (DCE) MRI and diffusion-weighted imaging (DWI) of the pre-treatment (baseline) and after four cycles (C4) of doxorubicin/cyclophosphamide treatment were used as inputs to the model for prediction of pathologic complete response (pCR). Based on the standard pCR definition that includes disease status in either breast or axilla, the model achieved areas under the receiver operating characteristic curves (AUCs) of 0.96 ± 0.05, 0.78 ± 0.09, 0.88 ± 0.02, and 0.76 ± 0.03, for the training, validation, testing, and prospective testing groups, respectively. For the pCR status of breast only, the retrained model achieved prediction AUCs of 0.97 ± 0.04, 0.82 ± 0.10, 0.86 ± 0.03, and 0.83 ± 0.02, for the training, validation, testing, and prospective testing groups, respectively. Thus, the developed deep learning model is highly promising for predicting the treatment response to NAST of TNBC.Clinical Relevance- Deep learning based on serial and multiparametric MRIs can potentially distinguish TNBC patients with pCR from non-pCR at the early stage of neoadjuvant systemic therapy, potentially enabling more personalized treatment of TNBC patients.

Predicting pathological complete response based on weakly and semi-supervised joint learning from breast cancer MRI

Neoadjuvant chemotherapy (NAC) is the standard treatment for breast cancer patients. Patients achieving complete pathological response (pCR) after NAC usually have a good prognosis. However, automatic pCR prediction has been a challenging problem due to lacking well annotations in 3D MRI. Thus far, unifying different annotation information to predict the tumor's early response to NAC has not been systematically addressed. This paper proposes a weakly and semi-supervised joint learning method that integrates attentional features from multi-parametric MRI with radiomic features for predicting pCR to NAC in breast cancer patients. The attention-based multi-instance learning (MIL) is first developed to generate informative MRI bag-level features and mine key instances. The mean-teacher framework is then employed to segment tumor regions in a semi-supervised setting for extracting radiomic features. We perform experiments on 442 patients' data and show that our method achieves an AUC value of 0.85 in pCR prediction, which is superior to comparative methods. It is also shown that learning from multi-parametric MRI outperforms that of single-parameter MRI in pCR prediction.

Novel computational biology modeling system can accurately forecast response to neoadjuvant therapy in early breast cancer

BACKGROUND

Generalizable population-based studies are unable to account for individual tumor heterogeneity that contributes to variability in a patient's response to physician-chosen therapy. Although molecular characterization of tumors has advanced precision medicine, in early-stage and locally advanced breast cancer patients, predicting a patient's response to neoadjuvant therapy (NAT) remains a gap in current clinical practice. Here, we perform a study in an independent cohort of early-stage and locally advanced breast cancer patients to forecast tumor response to NAT and assess the stability of a previously validated biophysical simulation platform.

METHODS

A single-blinded study was performed using a retrospective database from a single institution (9/2014-12/2020). Patients included: ≥ 18 years with breast cancer who completed NAT, with pre-treatment dynamic contrast enhanced magnetic resonance imaging. Demographics, chemotherapy, baseline (pre-treatment) MRI and pathologic data were input into the TumorScope Predict (TS) biophysical simulation platform to generate predictions. Primary outcomes included predictions of pathological complete response (pCR) versus residual disease (RD) and final volume for each tumor. For validation, post-NAT predicted pCR and tumor volumes were compared to actual pathological assessment and MRI-assessed volumes. Predicted pCR was pre-defined as residual tumor volume ≤ 0.01 cm³ (≥ 99.9% reduction).

RESULTS

The cohort consisted of eighty patients; 36 Caucasian and 40 African American. Most tumors were high-grade (54.4% grade 3) invasive ductal carcinomas (90.0%). Receptor subtypes included hormone receptor positive (HR+)/human epidermal growth factor receptor 2 positive (HER2+, 30%), HR+/HER2- (35%), HR-/HER2+ (12.5%) and triple negative breast cancer (TNBC, 22.5%). Simulated tumor volume was significantly correlated with post-treatment radiographic MRI calculated volumes (r = 0.53, p = 1.3 × 10^-7, mean absolute error of 6.57%). TS prediction of pCR compared favorably to pathological assessment (pCR: TS n = 28; Path n = 27; RD: TS n = 52; Path n = 53), for an overall accuracy of 91.2% (95% CI: 82.8% - 96.4%; Clopper-Pearson interval). Five-year risk of recurrence demonstrated similar prognostic performance between TS predictions (Hazard ratio (HR): - 1.99; 95% CI [- 3.96, - 0.02]; p = 0.043) and clinically assessed pCR (HR: - 1.76; 95% CI [- 3.75, 0.23]; p = 0.054).

CONCLUSION

We demonstrated TS ability to simulate and model tumor in vivo conditions in silico and forecast volume response to NAT across breast tumor subtypes.

Deep Learning Network with Spatial Attention Module for Detecting Acute Bilirubin Encephalopathy in Newborns Based on Multimodal MRI

BACKGROUND

Acute bilirubin encephalopathy (ABE) is a significant cause of neonatal mortality and disability. Early detection and treatment of ABE can prevent the further development of ABE and its long-term complications. Due to the limited classification ability of single-modal magnetic resonance imaging (MRI), this study aimed to validate the classification performance of a new deep learning model based on multimodal MRI images. Additionally, the study evaluated the effect of a spatial attention module (SAM) on improving the model's diagnostic performance in distinguishing ABE.

METHODS

This study enrolled a total of 97 neonates diagnosed with ABE and 80 neonates diagnosed with hyperbilirubinemia (HB, non-ABE). Each patient underwent three types of multimodal imaging, which included T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and an apparent diffusion coefficient (ADC) map. A multimodal MRI classification model based on the ResNet18 network with spatial attention modules was built to distinguish ABE from non-ABE. All combinations of the three types of images were used as inputs to test the model's classification performance, and we also analyzed the prediction performance of models with SAMs through comparative experiments.

RESULTS

The results indicated that the diagnostic performance of the multimodal image combination was better than any single-modal image, and the combination of T1WI and T2WI achieved the best classification performance (accuracy = 0.808 ± 0.069, area under the curve = 0.808 ± 0.057). The ADC images performed the worst among the three modalities' images. Adding spatial attention modules significantly improved the model's classification performance.

CONCLUSION

Our experiment showed that a multimodal image classification network with spatial attention modules significantly improved the accuracy of ABE classification.

Longitudinal MRI-based fusion novel model predicts pathological complete response in breast cancer treated with neoadjuvant chemotherapy: a multicenter, retrospective study

Background

Accurate identification of pCR to neoadjuvant chemotherapy (NAC) is essential for determining appropriate surgery strategy and guiding resection extent in breast cancer. However, a non-invasive tool to predict pCR accurately is lacking. Our study aims to develop ensemble learning models using longitudinal multiparametric MRI to predict pCR in breast cancer.

Methods

From July 2015 to December 2021, we collected pre-NAC and post-NAC multiparametric MRI sequences per patient. We then extracted 14,676 radiomics and 4096 deep learning features and calculated additional delta-value features. In the primary cohort (n = 409), the inter-class correlation coefficient test, U-test, Boruta and the least absolute shrinkage and selection operator regression were used to select the most significant features for each subtype of breast cancer. Five machine learning classifiers were then developed to predict pCR accurately for each subtype. The ensemble learning strategy was used to integrate the single-modality models. The diagnostic performances of models were evaluated in the three external cohorts (n = 343, 170 and 340, respectively).

Findings

A total of 1262 patients with breast cancer from four centers were enrolled in this study, and pCR rates were 10.6% (52/491), 54.3% (323/595) and 37.5% (66/176) in HR+/HER2-, HER2+ and TNBC subtype, respectively. Finally, 20, 15 and 13 features were selected to construct the machine learning models in HR+/HER2-, HER2+ and TNBC subtypes, respectively. The multi-Layer Perception (MLP) yields the best diagnostic performances in all subtypes. For the three subtypes, the stacking model integrating pre-, post- and delta-models yielded the highest AUCs of 0.959, 0.974 and 0.958 in the primary cohort, and AUCs of 0.882-0.908, 0.896-0.929 and 0.837-0.901 in the external validation cohorts, respectively. The stacking model had accuracies of 85.0%-88.9%, sensitivities of 80.0%-86.3%, and specificities of 87.4%-91.5% in the external validation cohorts.

Interpretation

Our study established a novel tool to predict the responses of breast cancer to NAC and achieve excellent performance. The models could help to determine post-NAC surgery strategy for breast cancer.

Funding

This study is supported by grants from the National Natural Science Foundation of China (82171898, 82103093), the Deng Feng project of high-level hospital construction (DFJHBF202109), the Guangdong Basic and Applied Basic Research Foundation (grant number, 2020A1515010346, 2022A1515012277), the Science and Technology Planning Project of Guangzhou City (202002030236), the Beijing Medical Award Foundation (YXJL-2020-0941-0758), and the Beijing Science and Technology Innovation Medical Development Foundation (KC2022-ZZ-0091-5). Funding sources were not involved in the study design, data collection, analysis and interpretation, writing of the report, or decision to submit the article for publication.

From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment

Machine learning (ML) is increasingly used in clinical oncology to diagnose cancers, predict patient outcomes, and inform treatment planning. Here, we review recent applications of ML across the clinical oncology workflow. We review how these techniques are applied to medical imaging and to molecular data obtained from liquid and solid tumor biopsies for cancer diagnosis, prognosis, and treatment design. We discuss key considerations in developing ML for the distinct challenges posed by imaging and molecular data. Finally, we examine ML models approved for cancer-related patient usage by regulatory agencies and discuss approaches to improve the clinical usefulness of ML.

Prediction of pathologic complete response to neoadjuvant systemic therapy in triple negative breast cancer using deep learning on multiparametric MRI

Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer. Neoadjuvant systemic therapy (NAST) followed by surgery are currently standard of care for TNBC with 50-60% of patients achieving pathologic complete response (pCR). We investigated ability of deep learning (DL) on dynamic contrast enhanced (DCE) MRI and diffusion weighted imaging acquired early during NAST to predict TNBC patients' pCR status in the breast. During the development phase using the images of 130 TNBC patients, the DL model achieved areas under the receiver operating characteristic curves (AUCs) of 0.97 ± 0.04 and 0.82 ± 0.10 for the training and the validation, respectively. The model achieved an AUC of 0.86 ± 0.03 when evaluated in the independent testing group of 32 patients. In an additional prospective blinded testing group of 48 patients, the model achieved an AUC of 0.83 ± 0.02. These results demonstrated that DL based on multiparametric MRI can potentially differentiate TNBC patients with pCR or non-pCR in the breast early during NAST.

Evaluation of treatment responses among subgroups of breast cancer patients receiving neoadjuvant chemotherapy

BACKGROUND

Breast MRIs are helpful for determining treatment plans, responses, and prospective survival analyses. In this retrospective cross-sectional study, we compared the preoperative MRI treatment response to neoadjuvant chemotherapy (NAC) administration with the postoperative pathological response in breast cancer patients.

MATERIALS AND METHODS

We analyzed data from 108 hospitalized patients receiving NAC between 2020 and 2022. We used MRI to evaluate the treatment response to NAC in patients with locally advanced breast cancers who had not received any prior treatment. We recorded the longest diameter of the primary tumor and the numbers of secondary tumors and axillary lymph nodes. In addition, we examined the correlation between the MRI response rate and pathological specimen results.

RESULTS

In our subgroup analyses, we found the best pathological response in patients with luminal B (Ki-67 index >14%) breast cancer and positivity for both hormone receptor and HER-2 markers. After comparing the pathological and radiological treatment responses in tumors and lymph nodes, the sensitivities were 90.3% for the pathological assessment and 42.8% for the radiological assessment, while the accuracies were 84.2% for the pathological assessment and 61.1% for the radiological assessment.

CONCLUSION

Using MRI techniques and sequence intervals and examining the histopathological characteristics of tumors may help increase the accuracy of the pathological complete response.

Deep Learning Prediction of Pathologic Complete Response in Breast Cancer Using MRI and Other Clinical Data: A Systematic Review

Breast cancer patients who have pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) are more likely to have better clinical outcomes. The ability to predict which patient will respond to NAC early in the treatment course is important because it could help to minimize unnecessary toxic NAC and to modify regimens mid-treatment to achieve better efficacy. Machine learning (ML) is increasingly being used in radiology and medicine because it can identify relationships amongst complex data elements to inform outcomes without the need to specify such relationships a priori. One of the most popular deep learning methods that applies to medical images is the Convolutional Neural Networks (CNN). In contrast to supervised ML, deep learning CNN can operate on the whole images without requiring radiologists to manually contour the tumor on images. Although there have been many review papers on supervised ML prediction of pCR, review papers on deep learning prediction of pCR are sparse. Deep learning CNN could also incorporate multiple image types, clinical data such as demographics and molecular subtypes, as well as data from multiple treatment time points to predict pCR. The goal of this study is to perform a systematic review of deep learning methods that use whole-breast MRI images without annotation or tumor segmentation to predict pCR in breast cancer.

A Review of Radiomics in Predicting Therapeutic Response in Colorectal Liver Metastases: From Traditional to Artificial Intelligence Techniques

An early evaluation of colorectal cancer liver metastasis (CRCLM) is crucial in determining treatment options that ultimately affect patient survival rates and outcomes. Radiomics (quantitative imaging features) have recently gained popularity in diagnostic and therapeutic strategies. Despite this, radiomics faces many challenges and limitations. This study sheds light on these limitations by reviewing the studies that used radiomics to predict therapeutic response in CRCLM. Despite radiomics’ potential to enhance clinical decision-making, it lacks standardization. According to the results of this study, the instability of radiomics quantification is caused by changes in CT scan parameters used to obtain CT scans, lesion segmentation methods used for contouring liver metastases, feature extraction methods, and dataset size used for experimentation and validation. Accordingly, the study recommends combining radiomics with deep learning to improve prediction accuracy.

Artificial intelligence for multimodal data integration in oncology

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, histology, and genomics to electronic health records. Current artificial intelligence (AI) models operate mainly in the realm of a single modality, neglecting the broader clinical context, which inevitably diminishes their potential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. AI models are also capable of discovering novel patterns within and across modalities suitable for explaining differences in patient outcomes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion and association discovery. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges in clinical adoption and discuss emerging solutions.

Multimodal Prediction of Five-Year Breast Cancer Recurrence in Women Who Receive Neoadjuvant Chemotherapy

In current clinical practice, it is difficult to predict whether a patient receiving neoadjuvant chemotherapy (NAC) for breast cancer is likely to encounter recurrence after treatment and have the cancer recur locally in the breast or in other areas of the body. We explore the use of clinical history, immunohistochemical markers, and multiparametric magnetic resonance imaging (DCE, ADC, Dixon) to predict the risk of post-treatment recurrence within five years. We performed a retrospective study on a cohort of 1738 patients from Institut Curie and analyzed the data using classical machine learning, image processing, and deep learning. Our results demonstrate the ability to predict recurrence prior to NAC treatment initiation using each modality alone, and the possible improvement achieved by combining the modalities. When evaluated on holdout data, the multimodal model achieved an AUC of 0.75 (CI: 0.70, 0.80) and 0.57 specificity at 0.90 sensitivity. We then stratified the data based on known prognostic biomarkers. We found that our models can provide accurate recurrence predictions (AUC > 0.89) for specific groups of women under 50 years old with poor prognoses. A version of our method won second place at the BMMR2 Challenge, with a very small margin from being first, and was a standout from the other challenge entries.

Breast Magnetic Resonance Imaging for Patients With Newly Diagnosed Breast Cancer: A Review

Despite the high sensitivity and widespread use of preoperative magnetic resonance imaging (MRI), the American Cancer Society and the National Comprehensive Cancer Network guidelines do not recommend the routine use of preoperative MRI owing to the conflicting results and lack of clear benefit to the surgical outcome (reoperation and mastectomy) and long-term clinical outcomes (local recurrence and metachronous contralateral breast cancer). Preoperative MRI detects additional cancers that are occult at mammography and ultrasound but increases the rate of mastectomy. Concerns about overdiagnosis and overtreatment of preoperative MRI might be mitigated by adjusting the confounding factors when conducting studies, using the state-of-the-art image-guided biopsy technique, applying the radiologists’ cumulative experiences in interpreting MRI findings, and performing multiple lumpectomies in patients with multicentric cancer. Among the various imaging methods, dynamic contrast-enhanced MRI has the highest accuracy in predicting pathologic complete response after neoadjuvant chemotherapy. Prospective trials aimed at applying the MRI information to the de-escalation of surgical or radiation treatments are underway. In this review, current studies on the clinical outcomes of preoperative breast MRI are updated, and circumstances in which MRI may be useful for surgical planning are discussed.

Deep Learning With Radiomics for Disease Diagnosis and Treatment: Challenges and Potential

The high-throughput extraction of quantitative imaging features from medical images for the purpose of radiomic analysis, i.e., radiomics in a broad sense, is a rapidly developing and emerging research field that has been attracting increasing interest, particularly in multimodality and multi-omics studies. In this context, the quantitative analysis of multidimensional data plays an essential role in assessing the spatio-temporal characteristics of different tissues and organs and their microenvironment. Herein, recent developments in this method, including manually defined features, data acquisition and preprocessing, lesion segmentation, feature extraction, feature selection and dimension reduction, statistical analysis, and model construction, are reviewed. In addition, deep learning-based techniques for automatic segmentation and radiomic analysis are being analyzed to address limitations such as rigorous workflow, manual/semi-automatic lesion annotation, and inadequate feature criteria, and multicenter validation. Furthermore, a summary of the current state-of-the-art applications of this technology in disease diagnosis, treatment response, and prognosis prediction from the perspective of radiology images, multimodality images, histopathology images, and three-dimensional dose distribution data, particularly in oncology, is presented. The potential and value of radiomics in diagnostic and therapeutic strategies are also further analyzed, and for the first time, the advances and challenges associated with dosiomics in radiotherapy are summarized, highlighting the latest progress in radiomics. Finally, a robust framework for radiomic analysis is presented and challenges and recommendations for future development are discussed, including but not limited to the factors that affect model stability (medical big data and multitype data and expert knowledge in medical), limitations of data-driven processes (reproducibility and interpretability of studies, different treatment alternatives for various institutions, and prospective researches and clinical trials), and thoughts on future directions (the capability to achieve clinical applications and open platform for radiomics analysis).