Modality-correlation embedding model for breast tumor diagnosis with mammography and ultrasound images

Interactively Fusing Global and Local Features for Benign and Malignant Classification of Breast Ultrasound Images

OBJECTIVE

Breast ultrasound (BUS) is used to classify benign and malignant breast tumors, and its automatic classification can reduce subjectivity. However, current convolutional neural networks (CNNs) face challenges in capturing global features, while vision transformer (ViT) networks have limitations in effectively extracting local features. Therefore, this study aimed to develop a deep learning method that enables the interaction and updating of intermediate features between CNN and ViT to achieve high-accuracy BUS image classification.

METHODS

This study introduced the CNN and transformer multi-stage fusion network (CTMF-Net) consisting of two branches: a CNN branch and a transformer branch. The CNN branch employs visual geometry group as its backbone, while the transformer branch utilizes ViT as its base network. Both branches were divided into four stages. At the end of each stage, a proposed feature interaction module facilitated feature interaction and fusion between the two branches. Additionally, the convolutional block attention module was employed to enhance relevant features after each stage of the CNN branch. Extensive experiments were conducted using various state-of-the-art deep-learning classification methods on three public breast ultrasound datasets (SYSU, UDIAT and BUSI).

RESULTS

For the internal validation on SYSU and UDIAT, our proposed method CTMF-Net achieved the highest accuracy of 90.14 ± 0.58% on SYSU and 92.04 ± 4.90% on UDIAT, which showed superior classification performance over other state-of-art networks (p < 0.05). Additionally, for external validation on BUSI, CTMF-Net showed outstanding performance, achieving the highest area under the curve score of 0.8704 when trained on SYSU, marking a 0.0126 improvement over the second-best visual geometry group attention ViT method. Similarly, when applied to UDIAT, CTMF-Net achieved an area under the curve score of 0.8505, surpassing the second-best global context ViT method by 0.0130.

CONCLUSION

Our proposed method, CTMF-Net, outperforms all existing methods and can effectively assist doctors in achieving more accurate classification performance of breast tumors.

DIFLF: A domain-invariant features learning framework for single-source domain generalization in mammogram classification

BACKGROUND AND OBJECTIVE

Single-source domain generalization (SSDG) aims to generalize a deep learning (DL) model trained on one source dataset to multiple unseen datasets. This is important for the clinical applications of DL-based models to breast cancer screening, wherein a DL-based model is commonly developed in an institute and then tested in other institutes. One challenge of SSDG is to alleviate the domain shifts using only one domain dataset.

METHODS

The present study proposed a domain-invariant features learning framework (DIFLF) for single-source domain. Specifically, a style-augmentation module (SAM) and a content-style disentanglement module (CSDM) are proposed in DIFLF. SAM includes two different color jitter transforms, which transforms each mammogram in the source domain into two synthesized mammograms with new styles. Thus, it can greatly increase the feature diversity of the source domain, reducing the overfitting of the trained model. CSDM includes three feature disentanglement units, which extracts domain-invariant content (DIC) features by disentangling them from domain-specific style (DSS) features, reducing the influence of the domain shifts resulting from different feature distributions. Our code is available for open access on Github (https://github.com/85675/DIFLF).

RESULTS

DIFLF is trained in a private dataset (PRI1), and tested first in another private dataset (PRI2) with similar feature distribution to PRI1 and then tested in two public datasets (INbreast and MIAS) with greatly different feature distributions from PRI1. As revealed by the experiment results, DIFLF presents excellent performance for classifying mammograms in the unseen target datasets of PRI2, INbreast, and MIAS. The accuracy and AUC of DIFLF are 0.917 and 0.928 in PRI2, 0.882 and 0.893 in INbreast, 0.767 and 0.710 in MIAS, respectively.

CONCLUSIONS

DIFLF can alleviate the influence of domain shifts only using one source dataset. Moreover, DIFLF can achieve an excellent mammogram classification performance even in the unseen datasets with great feature distribution differences from the training dataset.

Deep learning-based multimodal spatial transcriptomics analysis for cancer

The advent of deep learning (DL) and multimodal spatial transcriptomics (ST) has revolutionized cancer research, offering unprecedented insights into tumor biology. This book chapter explores the integration of DL with ST to advance cancer diagnostics, treatment planning, and precision medicine. DL, a subset of artificial intelligence, employs neural networks to model complex patterns in vast datasets, significantly enhancing diagnostic and treatment applications. In oncology, convolutional neural networks excel in image classification, segmentation, and tumor volume analysis, essential for identifying tumors and optimizing radiotherapy. The chapter also delves into multimodal data analysis, which integrates genomic, proteomic, imaging, and clinical data to offer a holistic understanding of cancer biology. Leveraging diverse data sources, researchers can uncover intricate details of tumor heterogeneity, microenvironment interactions, and treatment responses. Examples include integrating MRI data with genomic profiles for accurate glioma grading and combining proteomic and clinical data to uncover drug resistance mechanisms. DL's integration with multimodal data enables comprehensive and actionable insights for cancer diagnosis and treatment. The synergy between DL models and multimodal data analysis enhances diagnostic accuracy, personalized treatment planning, and prognostic modeling. Notable applications include ST, which maps gene expression patterns within tissue contexts, providing critical insights into tumor heterogeneity and potential therapeutic targets. In summary, the integration of DL and multimodal ST represents a paradigm shift towards more precise and personalized oncology. This chapter elucidates the methodologies and applications of these advanced technologies, highlighting their transformative potential in cancer research and clinical practice.