1
|
Ma S, Li Y, Yin J, Niu Q, An Z, Du L, Li F, Gu J. Prospective study of AI-assisted prediction of breast malignancies in physical health examinations: role of off-the-shelf AI software and comparison to radiologist performance. Front Oncol 2024; 14:1374278. [PMID: 38756651 PMCID: PMC11096442 DOI: 10.3389/fonc.2024.1374278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2024] [Accepted: 04/19/2024] [Indexed: 05/18/2024] Open
Abstract
Objective In physical health examinations, breast sonography is a commonly used imaging method, but it can lead to repeated exams and unnecessary biopsy due to discrepancies among radiologists and health centers. This study explores the role of off-the-shelf artificial intelligence (AI) software in assisting radiologists to classify incidentally found breast masses in two health centers. Methods Female patients undergoing breast ultrasound examinations with incidentally discovered breast masses were categorized according to the 5th edition of the Breast Imaging Reporting and Data System (BI-RADS), with categories 3 to 5 included in this study. The examinations were conducted at two municipal health centers from May 2021 to May 2023.The final pathological results from surgical resection or biopsy served as the gold standard for comparison. Ultrasonographic images were obtained in longitudinal and transverse sections, and two junior radiologists and one senior radiologist independently assessed the images without knowing the pathological findings. The BI-RADS classification was adjusted following AI assistance, and diagnostic performance was compared using receiver operating characteristic curves. Results A total of 196 patients with 202 breast masses were included in the study, with pathological results confirming 107 benign and 95 malignant masses. The receiver operating characteristic curve showed that experienced breast radiologists had higher diagnostic performance in BI-RADS classification than junior radiologists, similar to AI classification (AUC = 0.936, 0.806, 0.896, and 0.950, p < 0.05). The AI software improved the accuracy, sensitivity, and negative predictive value of the adjusted BI-RADS classification for the junior radiologists' group (p< 0.05), while no difference was observed in the senior radiologist group. Furthermore, AI increased the negative predictive value for BI-RADS 4a masses and the positive predictive value for 4b masses among radiologists (p < 0.05). AI enhances the sensitivity of invasive breast cancer detection more effectively than ductal carcinoma in situ and rare subtypes of breast cancer. Conclusions The AI software enhances diagnostic efficiency for breast masses, reducing the performance gap between junior and senior radiologists, particularly for BI-RADS 4a and 4b masses. This improvement reduces unnecessary repeat examinations and biopsies, optimizing medical resource utilization and enhancing overall diagnostic effectiveness.
Collapse
Affiliation(s)
- Sai Ma
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yanfang Li
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jun Yin
- Department of Ultrasound, Shanghai Fourth People’s Hospital, Shanghai, China
| | - Qinghua Niu
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Zichen An
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Lianfang Du
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Fan Li
- Department of Ultrasound, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jiying Gu
- Department of Ultrasound, Shidong Hospital, Yangpu District, Shanghai, China
| |
Collapse
|
2
|
Dan Q, Xu Z, Burrows H, Bissram J, Stringer JSA, Li Y. Diagnostic performance of deep learning in ultrasound diagnosis of breast cancer: a systematic review. NPJ Precis Oncol 2024; 8:21. [PMID: 38280946 PMCID: PMC10821881 DOI: 10.1038/s41698-024-00514-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 12/08/2023] [Indexed: 01/29/2024] Open
Abstract
Deep learning (DL) has been widely investigated in breast ultrasound (US) for distinguishing between benign and malignant breast masses. This systematic review of test diagnosis aims to examine the accuracy of DL, compared to human readers, for the diagnosis of breast cancer in the US under clinical settings. Our literature search included records from databases including PubMed, Embase, Scopus, and Cochrane Library. Test accuracy outcomes were synthesized to compare the diagnostic performance of DL and human readers as well as to evaluate the assistive role of DL to human readers. A total of 16 studies involving 9238 female participants were included. There were no prospective studies comparing the test accuracy of DL versus human readers in clinical workflows. Diagnostic test results varied across the included studies. In 14 studies employing standalone DL systems, DL showed significantly lower sensitivities in 5 studies with comparable specificities and outperformed human readers at higher specificities in another 4 studies; in the remaining studies, DL models and human readers showed equivalent test outcomes. In 12 studies that assessed assistive DL systems, no studies proved the assistive role of DL in the overall diagnostic performance of human readers. Current evidence is insufficient to conclude that DL outperforms human readers or enhances the accuracy of diagnostic breast US in a clinical setting. Standardization of study methodologies is required to improve the reproducibility and generalizability of DL research, which will aid in clinical translation and application.
Collapse
Affiliation(s)
- Qing Dan
- Department of Ultrasound, Nanfang Hospital, Southern Medical University, 510515, Guangzhou, China
- Global Women's Health, The University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA
| | - Ziting Xu
- Department of Ultrasound, Nanfang Hospital, Southern Medical University, 510515, Guangzhou, China
| | - Hannah Burrows
- Health Sciences Library, The University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA
| | - Jennifer Bissram
- Health Sciences Library, The University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA
| | - Jeffrey S A Stringer
- Global Women's Health, The University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA.
| | - Yingjia Li
- Department of Ultrasound, Nanfang Hospital, Southern Medical University, 510515, Guangzhou, China.
| |
Collapse
|
3
|
Fruchtman Brot H, Mango VL. Artificial intelligence in breast ultrasound: application in clinical practice. Ultrasonography 2024; 43:3-14. [PMID: 38109894 PMCID: PMC10766882 DOI: 10.14366/usg.23116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Revised: 08/14/2023] [Accepted: 08/29/2023] [Indexed: 12/20/2023] Open
Abstract
Ultrasound (US) is a widely accessible and extensively used tool for breast imaging. It is commonly used as an additional screening tool, especially for women with dense breast tissue. Advances in artificial intelligence (AI) have led to the development of various AI systems that assist radiologists in identifying and diagnosing breast lesions using US. This article provides an overview of the background and supporting evidence for the use of AI in hand held breast US. It discusses the impact of AI on clinical workflow, covering breast cancer detection, diagnosis, prediction of molecular subtypes, evaluation of axillary lymph node status, and response to neoadjuvant chemotherapy. Additionally, the article highlights the potential significance of AI in breast US for low and middle income countries.
Collapse
|
4
|
Song P, Zhang L, Bai L, Wang Q, Wang Y. Diagnostic performance of ultrasound with computer-aided diagnostic system in detecting breast cancer. Heliyon 2023; 9:e20712. [PMID: 37860526 PMCID: PMC10582378 DOI: 10.1016/j.heliyon.2023.e20712] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 09/23/2023] [Accepted: 10/04/2023] [Indexed: 10/21/2023] Open
Abstract
Purpose This study aims to examine the performance of breast ultrasound with a computer-aided diagnostic (CAD) system in detecting malignant breast cancer compared to conventional ultrasound and investigate the effects on smaller tumor sizes (≤20 mm). Methods This retrospective analysis included 123 patients with breast masses between March 2021 and July 2023. By using pathology results from biopsies or surgeries as the gold standard, we calculated and compared the diagnostic performances of conventional ultrasound and CAD, including sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and area under the receiver operating characteristic curve (AUC). A subgroup analysis of masses ≤20 mm in size was performed. Results Twenty-seven patients were pathologically diagnosed with malignant breast cancer. CAD had a higher specificity (92.71 % vs. 62.5 %) and accuracy (93.5 % vs. 69.92 %) than conventional ultrasound. The AUC of CAD was significantly greater than that of conventional ultrasonography (0.9450 vs. 0.7940, p < 0.0001). The agreement between the CAD and pathology results was almost perfect (kappa = 0.82, p < 0.0001). In patients with masses ≤20 mm, the effect was consistent: CAD had higher specificity (91.43 % vs. 51.43 %), higher accuracy (90.70 % vs. 58.14 %), and a higher AUC (0.8946 vs. 0.6946, p < 0.0001) than conventional ultrasound. Thirty-one downgrades were observed in BI-RADS 4A and 4B based on CAD, all of which were proven to be benign. Conclusion Compared to conventional breast ultrasound, CAD had better diagnostic performance, with higher specificity, accuracy, and AUC. CAD can help recognize benign lesions, especially in patients with BI-RADS 4A, and avoid unnecessary invasive procedures.
Collapse
Affiliation(s)
- Pengjie Song
- Department of Ultrasound, Gangkou Hospital, Qinhuangdao, Hebei, 066000, China
| | - Li Zhang
- Department of Ultrasound, Gangkou Hospital, Qinhuangdao, Hebei, 066000, China
| | - Longmei Bai
- Department of Ultrasound, Gangkou Hospital, Qinhuangdao, Hebei, 066000, China
| | - Qing Wang
- Department of Ultrasound, Gangkou Hospital, Qinhuangdao, Hebei, 066000, China
| | - Yanlei Wang
- Department of Ultrasound, Peking University Third Hospital Qinhuangdao Hospital, Qinhuangdao, Hebei, 066000, China
| |
Collapse
|
5
|
Nuutinen M, Leskelä RL. Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system. HEALTH AND TECHNOLOGY 2023; 13:1-14. [PMID: 37363342 PMCID: PMC10262137 DOI: 10.1007/s12553-023-00763-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 06/01/2023] [Indexed: 06/28/2023]
Abstract
Background For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required. Objective The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS. Methods The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered. Results The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment. Conclusion The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices. Supplementary Information The online version contains supplementary material available at 10.1007/s12553-023-00763-1.
Collapse
Affiliation(s)
- Mikko Nuutinen
- Nordic Healthcare Group, Helsinki, Finland
- Haartman Institute, University of Helsinki, Helsinki, Finland
| | | |
Collapse
|
6
|
Wang Y, Tang L, Chen P, Chen M. The Role of a Deep Learning-Based Computer-Aided Diagnosis System and Elastography in Reducing Unnecessary Breast Lesion Biopsies. Clin Breast Cancer 2023; 23:e112-e121. [PMID: 36653206 DOI: 10.1016/j.clbc.2022.12.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2022] [Revised: 11/27/2022] [Accepted: 12/20/2022] [Indexed: 12/24/2022]
Abstract
OBJECTIVES Ultrasound examination has inter-observer and intra-observer variability and a high false-positive rate. The aim of this study was to evaluate the value of the combined use of a deep learning-based computer-aided diagnosis (CAD) system and ultrasound elastography with conventional ultrasound (US) in increasing specificity and reducing unnecessary breast lesions biopsies. MATERIALS AND METHODS Conventional US, CAD system, and strain elastography (SE) were retrospectively performed on 216 breast lesions before biopsy or surgery. The area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and biopsy rate were compared between conventional US and the combination of conventional US, SE, and CAD system. RESULTS Of 216 lesions, 54 were malignant and 162 were benign. The addition of CAD system and SE to conventional US increased the AUC from 0.716 to 0.910 and specificity from 46.9% to 85.8% without a loss in sensitivity while 89.2% (66 of 74) of benign lesions in Breast Imaging Reporting and Data System (BI-RADS) category 4A lesions would avoid unnecessary biopsies. CONCLUSION The addition of CAD system and SE to conventional US improved specificity and AUC without loss of sensitivity, and reduced unnecessary biopsies.
Collapse
Affiliation(s)
- Yuqun Wang
- Department of Ultrasound Medicine, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai China
| | - Lei Tang
- Department of Ultrasound Medicine, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai China
| | - Pingping Chen
- Department of Ultrasound Medicine, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai China
| | - Man Chen
- Department of Ultrasound Medicine, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai China.
| |
Collapse
|
7
|
Gu Y, Xu W, Liu Y, An X, Li J, Cong L, Zhu L, He X, Wang H, Jiang Y. The feasibility of a novel computer-aided classification system for the characterisation and diagnosis of breast masses on ultrasound: a single-centre preliminary test study. Clin Radiol 2023:S0009-9260(23)00130-7. [PMID: 37069025 DOI: 10.1016/j.crad.2023.03.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2022] [Revised: 03/07/2023] [Accepted: 03/15/2023] [Indexed: 04/19/2023]
Abstract
AIM To introduce a novel computer-aided classification (CAC) system and investigate the feasibility of characterising and diagnosing breast masses on ultrasound (US). MATERIALS AND METHODS A total of 246 breast masses were included. US features and the final assessment categories of the breast masses were analysed by a radiologist and the CAC system according to the Breast Imaging Reporting and Data System (BI-RADS) lexicon. The CAC system evaluated the BI-RADS assessment from the fusion of multi-view and colour Doppler US images without (SmartBreast) or with combining clinical variables (m-CAC system). The diagnostic performance and agreement of US characteristics between the radiologist and the CAC system were compared. RESULTS The agreement between the radiologist and the CAC system was substantial for mass shape (κ = 0.673), orientation (κ = 0.682), margin (κ = 0.622), posterior features (κ = 0.629), calcifications in a mass (κ = 0.709) and vascularity (κ = 0.745), fair for echo pattern (κ = 0.379), and moderate for BI-RADS assessment (κ = 0.575). With BI-RADS 4a as the cut-off value, the specificity (52.5% versus 25%, p<0.0001) and accuracy (73.98% versus 62.6%, p=0.0002) of the m-CAC system were improved without significant loss of sensitivity (94.44% versus 98.41%, p=0.1250) compared with the SmartBreast. The m-CAC system showed similar specificity (52.5% versus 45.83%, p=0.2430) and accuracy (73.98% versus 73.58%, p=1.0000) as the radiologist, but a lower sensitivity (94.44% versus 100%, p=0.0156). CONCLUSION The CAC system showed an acceptable agreement with the radiologist for characterisation of breast lesions. It has the potential to mimic the decision-making behaviour of radiologists for the classification of breast lesions.
Collapse
Affiliation(s)
- Y Gu
- Department of Ultrasound, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, 100730, China
| | - W Xu
- Department of Ultrasound, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, 100730, China
| | - Y Liu
- Department of Medical Imaging Advanced Research, Beijing Research Institute, Shenzhen Mindray Bio-Medical Electronics Co., Ltd, Beijing, China
| | - X An
- Department of Medical Imaging Advanced Research, Beijing Research Institute, Shenzhen Mindray Bio-Medical Electronics Co., Ltd, Beijing, China
| | - J Li
- Department of Ultrasound, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, 100730, China
| | - L Cong
- Department of Medical Imaging Advanced Research, Beijing Research Institute, Shenzhen Mindray Bio-Medical Electronics Co., Ltd, Beijing, China
| | - L Zhu
- Shenzhen Mindray Bio-Medical Electronics Co., Ltd, Shenzhen, China
| | - X He
- Shenzhen Mindray Bio-Medical Electronics Co., Ltd, Shenzhen, China
| | - H Wang
- Department of Ultrasound, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, 100730, China.
| | - Y Jiang
- Department of Ultrasound, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No. 1 Shuai Fu Yuan, Dong Cheng District, Beijing, 100730, China.
| |
Collapse
|
8
|
Villa-Camacho JC, Baikpour M, Chou SHS. Artificial Intelligence for Breast US. JOURNAL OF BREAST IMAGING 2023; 5:11-20. [PMID: 38416959 DOI: 10.1093/jbi/wbac077] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Indexed: 03/01/2024]
Abstract
US is a widely available, commonly used, and indispensable imaging modality for breast evaluation. It is often the primary imaging modality for the detection and diagnosis of breast cancer in low-resource settings. In addition, it is frequently employed as a supplemental screening tool via either whole breast handheld US or automated breast US among women with dense breasts. In recent years, a variety of artificial intelligence systems have been developed to assist radiologists with the detection and diagnosis of breast lesions on US. This article reviews the background and evidence supporting the use of artificial intelligence tools for breast US, describes implementation strategies and impact on clinical workflow, and discusses potential emerging roles and future directions.
Collapse
Affiliation(s)
| | - Masoud Baikpour
- Massachusetts General Hospital, Department of Radiology, Boston, MA, USA
| | - Shinn-Huey S Chou
- Massachusetts General Hospital, Department of Radiology, Boston, MA, USA
| |
Collapse
|