Modality-correlation embedding model for breast tumor diagnosis with mammography and ultrasound images

Optimizing breast lesions diagnosis and decision-making with a deep learning fusion model integrating ultrasound and mammography: a dual-center retrospective study

BACKGROUND

This study aimed to develop a BI-RADS network (DL-UM) via integrating ultrasound (US) and mammography (MG) images and explore its performance in improving breast lesion diagnosis and management when collaborating with radiologists, particularly in cases with discordant US and MG Breast Imaging Reporting and Data System (BI-RADS) classifications.

METHODS

We retrospectively collected image data from 1283 women with breast lesions who underwent both US and MG within one month at two medical centres and categorised them into concordant and discordant BI-RADS classification subgroups. We developed a DL-UM network via integrating US and MG images, and DL networks using US (DL-U) or MG (DL-M) alone, respectively. The performance of DL-UM network for breast lesion diagnosis was evaluated using ROC curves and compared to DL-U and DL-M networks in the external testing dataset. The diagnostic performance of radiologists with different levels of experience under the assistance of DL-UM network was also evaluated.

RESULTS

In the external testing dataset, DL-UM outperformed DL-M in sensitivity (0.962 vs. 0.833, P = 0.016) and DL-U in specificity (0.667 vs. 0.526, P = 0.030), respectively. In the discordant BI-RADS classification subgroup, DL-UM achieved an AUC of 0.910. The diagnostic performance of four radiologists improved when collaborating with the DL-UM network, with AUCs increased from 0.674-0.772 to 0.889-0.910, specificities from 52.1%-75.0 to 81.3-87.5% and reducing unnecessary biopsies by 16.1%-24.6%, particularly for junior radiologists. Meanwhile, DL-UM outputs and heatmaps enhanced radiologists' trust and improved interobserver agreement between US and MG, with weighted kappa increased from 0.048 to 0.713 (P < 0.05).

CONCLUSIONS

The DL-UM network, integrating complementary US and MG features, assisted radiologists in improving breast lesion diagnosis and management, potentially reducing unnecessary biopsies.

Artificial intelligence-assisted multimodal imaging for the clinical applications of breast cancer: a bibliometric analysis

BACKGROUND

Breast cancer (BC) remains a leading cause of cancer-related mortality among women globally, with increasing incidence rates posing significant public health challenges. Recent advancements in artificial intelligence (AI) have revolutionized medical imaging, particularly in enhancing diagnostic accuracy and prognostic capabilities for BC. While multimodal imaging combined with AI has shown remarkable potential, a comprehensive analysis is needed to synthesize current research and identify emerging trends and hotspots in AI-assisted multimodal imaging for BC.

METHODS

This study analyzed literature on AI-assisted multimodal imaging in BC from January 2010 to November 2024 in Web of Science Core Collection (WoSCC). Bibliometric and visualization tools, including VOSviewer, CiteSpace, and the Bibliometrix R package, were employed to assess countries, institutions, authors, journals, and keywords.

RESULTS

A total of 80 publications were included, revealing a steady increase in annual publications and citations, with a notable surge post-2021. China led in productivity and citations, while Germany exhibited the highest citation average. The United States demonstrated the strongest international collaboration. The most productive institution and author are Radboud University Nijmegen and Xi, Xiaoming. Publications were predominantly published in Computerized Medical Imaging and Graphics, with Qian, XJ's 2021 study on BC risk prediction under deep learning frameworks being the most influential. Keyword analysis highlighted themes such as "breast cancer", "classification", and "deep learning".

CONCLUSIONS

AI-assisted multimodal imaging has significantly advanced BC diagnosis and management, with promising future developments. This study offers researchers a comprehensive overview of current frameworks and emerging research directions. Future efforts are expected to focus on improving diagnostic precision and refining therapeutic strategies through optimized imaging techniques and AI algorithms, emphasizing international collaboration to drive innovation and clinical translation.

DIFLF: A domain-invariant features learning framework for single-source domain generalization in mammogram classification

BACKGROUND AND OBJECTIVE

Single-source domain generalization (SSDG) aims to generalize a deep learning (DL) model trained on one source dataset to multiple unseen datasets. This is important for the clinical applications of DL-based models to breast cancer screening, wherein a DL-based model is commonly developed in an institute and then tested in other institutes. One challenge of SSDG is to alleviate the domain shifts using only one domain dataset.

METHODS

The present study proposed a domain-invariant features learning framework (DIFLF) for single-source domain. Specifically, a style-augmentation module (SAM) and a content-style disentanglement module (CSDM) are proposed in DIFLF. SAM includes two different color jitter transforms, which transforms each mammogram in the source domain into two synthesized mammograms with new styles. Thus, it can greatly increase the feature diversity of the source domain, reducing the overfitting of the trained model. CSDM includes three feature disentanglement units, which extracts domain-invariant content (DIC) features by disentangling them from domain-specific style (DSS) features, reducing the influence of the domain shifts resulting from different feature distributions. Our code is available for open access on Github (https://github.com/85675/DIFLF).

RESULTS

DIFLF is trained in a private dataset (PRI1), and tested first in another private dataset (PRI2) with similar feature distribution to PRI1 and then tested in two public datasets (INbreast and MIAS) with greatly different feature distributions from PRI1. As revealed by the experiment results, DIFLF presents excellent performance for classifying mammograms in the unseen target datasets of PRI2, INbreast, and MIAS. The accuracy and AUC of DIFLF are 0.917 and 0.928 in PRI2, 0.882 and 0.893 in INbreast, 0.767 and 0.710 in MIAS, respectively.

CONCLUSIONS

DIFLF can alleviate the influence of domain shifts only using one source dataset. Moreover, DIFLF can achieve an excellent mammogram classification performance even in the unseen datasets with great feature distribution differences from the training dataset.

Interactively Fusing Global and Local Features for Benign and Malignant Classification of Breast Ultrasound Images

OBJECTIVE

Breast ultrasound (BUS) is used to classify benign and malignant breast tumors, and its automatic classification can reduce subjectivity. However, current convolutional neural networks (CNNs) face challenges in capturing global features, while vision transformer (ViT) networks have limitations in effectively extracting local features. Therefore, this study aimed to develop a deep learning method that enables the interaction and updating of intermediate features between CNN and ViT to achieve high-accuracy BUS image classification.

METHODS

This study introduced the CNN and transformer multi-stage fusion network (CTMF-Net) consisting of two branches: a CNN branch and a transformer branch. The CNN branch employs visual geometry group as its backbone, while the transformer branch utilizes ViT as its base network. Both branches were divided into four stages. At the end of each stage, a proposed feature interaction module facilitated feature interaction and fusion between the two branches. Additionally, the convolutional block attention module was employed to enhance relevant features after each stage of the CNN branch. Extensive experiments were conducted using various state-of-the-art deep-learning classification methods on three public breast ultrasound datasets (SYSU, UDIAT and BUSI).

RESULTS

For the internal validation on SYSU and UDIAT, our proposed method CTMF-Net achieved the highest accuracy of 90.14 ± 0.58% on SYSU and 92.04 ± 4.90% on UDIAT, which showed superior classification performance over other state-of-art networks (p < 0.05). Additionally, for external validation on BUSI, CTMF-Net showed outstanding performance, achieving the highest area under the curve score of 0.8704 when trained on SYSU, marking a 0.0126 improvement over the second-best visual geometry group attention ViT method. Similarly, when applied to UDIAT, CTMF-Net achieved an area under the curve score of 0.8505, surpassing the second-best global context ViT method by 0.0130.

CONCLUSION

Our proposed method, CTMF-Net, outperforms all existing methods and can effectively assist doctors in achieving more accurate classification performance of breast tumors.

Deep learning-based multimodal spatial transcriptomics analysis for cancer

The advent of deep learning (DL) and multimodal spatial transcriptomics (ST) has revolutionized cancer research, offering unprecedented insights into tumor biology. This book chapter explores the integration of DL with ST to advance cancer diagnostics, treatment planning, and precision medicine. DL, a subset of artificial intelligence, employs neural networks to model complex patterns in vast datasets, significantly enhancing diagnostic and treatment applications. In oncology, convolutional neural networks excel in image classification, segmentation, and tumor volume analysis, essential for identifying tumors and optimizing radiotherapy. The chapter also delves into multimodal data analysis, which integrates genomic, proteomic, imaging, and clinical data to offer a holistic understanding of cancer biology. Leveraging diverse data sources, researchers can uncover intricate details of tumor heterogeneity, microenvironment interactions, and treatment responses. Examples include integrating MRI data with genomic profiles for accurate glioma grading and combining proteomic and clinical data to uncover drug resistance mechanisms. DL's integration with multimodal data enables comprehensive and actionable insights for cancer diagnosis and treatment. The synergy between DL models and multimodal data analysis enhances diagnostic accuracy, personalized treatment planning, and prognostic modeling. Notable applications include ST, which maps gene expression patterns within tissue contexts, providing critical insights into tumor heterogeneity and potential therapeutic targets. In summary, the integration of DL and multimodal ST represents a paradigm shift towards more precise and personalized oncology. This chapter elucidates the methodologies and applications of these advanced technologies, highlighting their transformative potential in cancer research and clinical practice.