The Added Value of a Computer-Aided Diagnosis System in Differential Diagnosis of Breast Lesions by Radiologists With Different Experience

Artificial Intelligence for Breast Ultrasound: AJR Expert Panel Narrative Review

Breast ultrasound is used in a wide variety of clinical scenarios, including both diagnostic and screening applications. Limitations of ultrasound, however, include its low specificity and, for automated breast ultrasound screening, the time necessary to review whole-breast ultrasound images. As of this writing, four AI tools that are approved or cleared by the FDA address these limitations. Current tools, which are intended to provide decision support for lesion classification and/or detection, have been shown to increase specificity among nonspecialists and to decrease interpretation times. Potential future applications include triage of patients with palpable masses in low-resource settings, preoperative prediction of axillary lymph node metastasis, and preoperative prediction of neoadjuvant chemotherapy response. Challenges in the development and clinical deployment of AI for ultrasound include the limited availability of curated training datasets compared with mammography, the high variability in ultrasound image acquisition due to equipment- and operator-related factors (which may limit algorithm generalizability), and the lack of postimplementation evaluation studies. Furthermore, current AI tools for lesion classification were developed based on 2D data, but diagnostic accuracy could potentially be improved if multimodal ultrasound data were used, such as color Doppler, elastography, cine clips, and 3D imaging.

Improving the Diagnostic Performance and Breast Imaging Reporting and Data System Category Agreement of Less Experienced Radiologists by Utilizing Computer-Aided Diagnosis Software for Breast Ultrasound

ABSTRACT

This study aimed to assess the effectiveness of intelligence-based computer-aided diagnosis (CAD) software in ultrasound (US) and its potential to improve the diagnostic performance of less experienced radiologists, as well as the agreement on Breast Imaging Reporting and Data System (BI-RADS) categories with the experienced radiologist. Images of 385 breast lesions in 351 female taken from January 2019 to December 2020 were included. Two less experienced radiologists independently reviewed US images with and without CAD assistance, recording final assessments using the BI-RADS category. The diagnostic performance of CAD and radiologists were calculated and compared. Kappa statistics were used to determine agreement between the experienced radiologist and the less experienced radiologists, based on BI-RADS category before and after using CAD software. The sensitivity, specificity, accuracy, positive predictive value, and negative predictive value of CAD software were 95.5%, 71.5%, 81.3%, 69.8%, and 95.9%, respectively, and those were improved in junior radiologist and intermediate-level radiologist after the addition of CAD. Additionally, with the assistance of CAD, the area under the curve was improved for both the junior radiologist and radiologist (0.704 vs 0.847 and 0.876 vs 0.900, P = 0.009, 0.005), although it remained lower than the senior radiologist. The agreement of BI-RADS category between the less experienced and the experienced radiologists showed a significant improvement (P = 0.04, 0.000). The CAD on US could improve less experienced radiologists' diagnostic performance and agreement on BI-RADS categories, making it an effective decision-making tool in clinical practice.

Comparative Analysis of the Diagnostic Value of S-Detect Technology in Different Planes Versus the BI-RADS Classification for Breast Lesions

RATIONALE AND OBJECTIVES

S-Detect, a deep learning-based Computer-Aided Detection system, is recognized as an important tool for diagnosing breast lesions using ultrasound imaging. However, it may exhibit inconsistent findings across multiple imaging planes. This study aims to evaluate the diagnostic performance of S-Detect in different planes and identify factors contributing to these inconsistencies.

MATERIALS AND METHODS

A retrospective cohort study was conducted on 711 patients with 756 breast lesions between January 2019 and January 2022. S-Detect was utilized to assess lesions in radial and anti-radial planes. BI-RADS classifications were employed for comparative analysis. The diagnostic performance was compared within each group, and p-values were computed for intergroup comparisons. Univariable and multivariable analyses were conducted to identify factors contributing to diagnostic inconsistency in S-Detect across planes.

RESULTS

Among 756 breast lesions, 668 (88.4%) exhibited consistent S-Detect outcomes across planes while 88 (11.6%) were inconsistent. In the consistent group, the diagnostic accuracy and area under the curve (AUC) of S-Detect were significantly higher than those of BI-RADS (accuracy: 91.2% vs. 84.9%, p = 0.045; AUC: 0.916 vs. 0.859, p = 0.036). In the inconsistent group, the diagnostic accuracy and AUC of S-Detect in radial and anti-radial planes were lower than those of BI-RADS (accuracy: 47.7% for radial, 52.2% for anti-radial vs. 69.3% for BI-RADS, p = 0.014, p-anti = 0.039; AUC: 0.503 for radial, 0.497 for anti-radial vs. 0.739 for BI-RADS, p = 0.042, p-anti <0.001). Diagnostic inconsistency in S-Detect across planes was significantly associated with lesion size, indistinct or angular margins, and enhancement posterior acoustic features (p < 0.05).

CONCLUSION

S-Detect has outperformed BI-RADS in diagnostic precision under conditions of inter-planar concordance. However, its diagnostic efficacy is compromised in scenarios of inter-planar discordance. Under these circumstances, the results of S-Detect should be carefully referenced.

Prospective study of AI-assisted prediction of breast malignancies in physical health examinations: role of off-the-shelf AI software and comparison to radiologist performance

Objective

In physical health examinations, breast sonography is a commonly used imaging method, but it can lead to repeated exams and unnecessary biopsy due to discrepancies among radiologists and health centers. This study explores the role of off-the-shelf artificial intelligence (AI) software in assisting radiologists to classify incidentally found breast masses in two health centers.

Methods

Female patients undergoing breast ultrasound examinations with incidentally discovered breast masses were categorized according to the 5th edition of the Breast Imaging Reporting and Data System (BI-RADS), with categories 3 to 5 included in this study. The examinations were conducted at two municipal health centers from May 2021 to May 2023.The final pathological results from surgical resection or biopsy served as the gold standard for comparison. Ultrasonographic images were obtained in longitudinal and transverse sections, and two junior radiologists and one senior radiologist independently assessed the images without knowing the pathological findings. The BI-RADS classification was adjusted following AI assistance, and diagnostic performance was compared using receiver operating characteristic curves.

Results

A total of 196 patients with 202 breast masses were included in the study, with pathological results confirming 107 benign and 95 malignant masses. The receiver operating characteristic curve showed that experienced breast radiologists had higher diagnostic performance in BI-RADS classification than junior radiologists, similar to AI classification (AUC = 0.936, 0.806, 0.896, and 0.950, p < 0.05). The AI software improved the accuracy, sensitivity, and negative predictive value of the adjusted BI-RADS classification for the junior radiologists' group (p< 0.05), while no difference was observed in the senior radiologist group. Furthermore, AI increased the negative predictive value for BI-RADS 4a masses and the positive predictive value for 4b masses among radiologists (p < 0.05). AI enhances the sensitivity of invasive breast cancer detection more effectively than ductal carcinoma in situ and rare subtypes of breast cancer.

Conclusions

The AI software enhances diagnostic efficiency for breast masses, reducing the performance gap between junior and senior radiologists, particularly for BI-RADS 4a and 4b masses. This improvement reduces unnecessary repeat examinations and biopsies, optimizing medical resource utilization and enhancing overall diagnostic effectiveness.

Diagnostic performance of deep learning in ultrasound diagnosis of breast cancer: a systematic review

Deep learning (DL) has been widely investigated in breast ultrasound (US) for distinguishing between benign and malignant breast masses. This systematic review of test diagnosis aims to examine the accuracy of DL, compared to human readers, for the diagnosis of breast cancer in the US under clinical settings. Our literature search included records from databases including PubMed, Embase, Scopus, and Cochrane Library. Test accuracy outcomes were synthesized to compare the diagnostic performance of DL and human readers as well as to evaluate the assistive role of DL to human readers. A total of 16 studies involving 9238 female participants were included. There were no prospective studies comparing the test accuracy of DL versus human readers in clinical workflows. Diagnostic test results varied across the included studies. In 14 studies employing standalone DL systems, DL showed significantly lower sensitivities in 5 studies with comparable specificities and outperformed human readers at higher specificities in another 4 studies; in the remaining studies, DL models and human readers showed equivalent test outcomes. In 12 studies that assessed assistive DL systems, no studies proved the assistive role of DL in the overall diagnostic performance of human readers. Current evidence is insufficient to conclude that DL outperforms human readers or enhances the accuracy of diagnostic breast US in a clinical setting. Standardization of study methodologies is required to improve the reproducibility and generalizability of DL research, which will aid in clinical translation and application.

Artificial intelligence in breast ultrasound: application in clinical practice

Ultrasound (US) is a widely accessible and extensively used tool for breast imaging. It is commonly used as an additional screening tool, especially for women with dense breast tissue. Advances in artificial intelligence (AI) have led to the development of various AI systems that assist radiologists in identifying and diagnosing breast lesions using US. This article provides an overview of the background and supporting evidence for the use of AI in hand held breast US. It discusses the impact of AI on clinical workflow, covering breast cancer detection, diagnosis, prediction of molecular subtypes, evaluation of axillary lymph node status, and response to neoadjuvant chemotherapy. Additionally, the article highlights the potential significance of AI in breast US for low and middle income countries.

Diagnostic performance of ultrasound with computer-aided diagnostic system in detecting breast cancer

Purpose

This study aims to examine the performance of breast ultrasound with a computer-aided diagnostic (CAD) system in detecting malignant breast cancer compared to conventional ultrasound and investigate the effects on smaller tumor sizes (≤20 mm).

Methods

This retrospective analysis included 123 patients with breast masses between March 2021 and July 2023. By using pathology results from biopsies or surgeries as the gold standard, we calculated and compared the diagnostic performances of conventional ultrasound and CAD, including sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and area under the receiver operating characteristic curve (AUC). A subgroup analysis of masses ≤20 mm in size was performed.

Results

Twenty-seven patients were pathologically diagnosed with malignant breast cancer. CAD had a higher specificity (92.71 % vs. 62.5 %) and accuracy (93.5 % vs. 69.92 %) than conventional ultrasound. The AUC of CAD was significantly greater than that of conventional ultrasonography (0.9450 vs. 0.7940, p < 0.0001). The agreement between the CAD and pathology results was almost perfect (kappa = 0.82, p < 0.0001). In patients with masses ≤20 mm, the effect was consistent: CAD had higher specificity (91.43 % vs. 51.43 %), higher accuracy (90.70 % vs. 58.14 %), and a higher AUC (0.8946 vs. 0.6946, p < 0.0001) than conventional ultrasound. Thirty-one downgrades were observed in BI-RADS 4A and 4B based on CAD, all of which were proven to be benign.

Conclusion

Compared to conventional breast ultrasound, CAD had better diagnostic performance, with higher specificity, accuracy, and AUC. CAD can help recognize benign lesions, especially in patients with BI-RADS 4A, and avoid unnecessary invasive procedures.

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system

Background

For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective

The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods

The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results

The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion

The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

The Role of a Deep Learning-Based Computer-Aided Diagnosis System and Elastography in Reducing Unnecessary Breast Lesion Biopsies

OBJECTIVES

Ultrasound examination has inter-observer and intra-observer variability and a high false-positive rate. The aim of this study was to evaluate the value of the combined use of a deep learning-based computer-aided diagnosis (CAD) system and ultrasound elastography with conventional ultrasound (US) in increasing specificity and reducing unnecessary breast lesions biopsies.

MATERIALS AND METHODS

Conventional US, CAD system, and strain elastography (SE) were retrospectively performed on 216 breast lesions before biopsy or surgery. The area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and biopsy rate were compared between conventional US and the combination of conventional US, SE, and CAD system.

RESULTS

Of 216 lesions, 54 were malignant and 162 were benign. The addition of CAD system and SE to conventional US increased the AUC from 0.716 to 0.910 and specificity from 46.9% to 85.8% without a loss in sensitivity while 89.2% (66 of 74) of benign lesions in Breast Imaging Reporting and Data System (BI-RADS) category 4A lesions would avoid unnecessary biopsies.

CONCLUSION

The addition of CAD system and SE to conventional US improved specificity and AUC without loss of sensitivity, and reduced unnecessary biopsies.

The feasibility of a novel computer-aided classification system for the characterisation and diagnosis of breast masses on ultrasound: a single-centre preliminary test study

AIM

To introduce a novel computer-aided classification (CAC) system and investigate the feasibility of characterising and diagnosing breast masses on ultrasound (US).

MATERIALS AND METHODS

A total of 246 breast masses were included. US features and the final assessment categories of the breast masses were analysed by a radiologist and the CAC system according to the Breast Imaging Reporting and Data System (BI-RADS) lexicon. The CAC system evaluated the BI-RADS assessment from the fusion of multi-view and colour Doppler US images without (SmartBreast) or with combining clinical variables (m-CAC system). The diagnostic performance and agreement of US characteristics between the radiologist and the CAC system were compared.

RESULTS

The agreement between the radiologist and the CAC system was substantial for mass shape (κ = 0.673), orientation (κ = 0.682), margin (κ = 0.622), posterior features (κ = 0.629), calcifications in a mass (κ = 0.709) and vascularity (κ = 0.745), fair for echo pattern (κ = 0.379), and moderate for BI-RADS assessment (κ = 0.575). With BI-RADS 4a as the cut-off value, the specificity (52.5% versus 25%, p<0.0001) and accuracy (73.98% versus 62.6%, p=0.0002) of the m-CAC system were improved without significant loss of sensitivity (94.44% versus 98.41%, p=0.1250) compared with the SmartBreast. The m-CAC system showed similar specificity (52.5% versus 45.83%, p=0.2430) and accuracy (73.98% versus 73.58%, p=1.0000) as the radiologist, but a lower sensitivity (94.44% versus 100%, p=0.0156).

CONCLUSION

The CAC system showed an acceptable agreement with the radiologist for characterisation of breast lesions. It has the potential to mimic the decision-making behaviour of radiologists for the classification of breast lesions.

Artificial Intelligence for Breast US

US is a widely available, commonly used, and indispensable imaging modality for breast evaluation. It is often the primary imaging modality for the detection and diagnosis of breast cancer in low-resource settings. In addition, it is frequently employed as a supplemental screening tool via either whole breast handheld US or automated breast US among women with dense breasts. In recent years, a variety of artificial intelligence systems have been developed to assist radiologists with the detection and diagnosis of breast lesions on US. This article reviews the background and evidence supporting the use of artificial intelligence tools for breast US, describes implementation strategies and impact on clinical workflow, and discusses potential emerging roles and future directions.