New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs

Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Artificial Intelligence-Based Software with CE Mark for Chest X-ray Interpretation: Opportunities and Challenges

Chest X-ray (CXR) is the most important technique for performing chest imaging, despite its well-known limitations in terms of scope and sensitivity. These intrinsic limitations of CXR have prompted the development of several artificial intelligence (AI)-based software packages dedicated to CXR interpretation. The online database "AI for radiology" was queried to identify CE-marked AI-based software available for CXR interpretation. The returned studies were divided according to the targeted disease. AI-powered computer-aided detection software is already widely adopted in screening and triage for pulmonary tuberculosis, especially in countries with few resources and suffering from high a burden of this disease. AI-based software has also been demonstrated to be valuable for the detection of lung nodules detection, automated flagging of positive cases, and post-processing through the development of digital bone suppression software able to produce digital bone suppressed images. Finally, the majority of available CE-marked software packages for CXR are designed to recognize several findings, with potential differences in sensitivity and specificity for each of the recognized findings.

Role of chest radiographs in early lung cancer detection

Background

Lung cancer is the second most common and the most fatal form of cancer. Although annual low-dose computed tomography is used as the primary method of cancer screening, it presents challenges regarding resources as well as potential health risks from radiation exposure. Chest radiography (CXR), though less effective, is used frequently and commonly. Moreover, often in clinical settings, CXR is the first imaging modality used; computed tomography is subsequently performed if abnormalities are detected on CXRs. This study examined whether controlling for distractors and time constraints, as well as side-by-side comparison of multiple CXRs in clinical settings can aid earlier detection of radiological abnormalities indicative of lung cancer lesions.

Methods

Thirty-two attending physicians in the Republic of Korea examined 1,750 radiographs of 50 lung cancer cases. Using “hot spot” technology, participants indicated the possible locations of cancer lesions on each radiograph. Subsequently, the same radiographs, cropped to focus the anatomical regions where lung cancers were diagnosed, were shown side-by-side to the participants. The participants were asked to identify the radiograph which first enabled the diagnosis of lung cancer and which first showed a possible lesion.

Results

Removal of systemic constraints alone significantly improved lesion identification by 221.72±9.69 days. Presenting radiographs side-by-side, cropped to relevant areas, had an additional significant and positive impact on cancer detection in both hidden and open areas on CXRs. Also, lesions were detected at smaller sizes and earlier than when actually diagnosed.

Conclusions

CXR with improved methods and settings provides an easily accessible and low-risk imaging method for earlier detection of lung cancer compared to current clinical imaging settings. Further, this study demonstrates the potential effectiveness of programs that allow side-by-side comparisons of cropped areas of multiple radiographs to detect radiological abnormalities.

Validation of radiologists' findings by computer-aided detection (CAD) software in breast cancer detection with automated 3D breast ultrasound: a concept study in implementation of artificial intelligence software

Background

Computer-aided detection software for automated breast ultrasound has been shown to have potential in improving the accuracy of radiologists. Alternative ways of implementing computer-aided detection, such as independent validation or preselecting suspicious cases, might also improve radiologists’ accuracy.

Purpose

To investigate the effect of using computer-aided detection software to improve the performance of radiologists by validating findings reported by radiologists during screening with automated breast ultrasound.

Material and Methods

Unilateral automated breast ultrasound exams were performed in 120 women with dense breasts that included 60 randomly selected normal exams, 30 exams with benign lesions, and 30 malignant cases (20 mammography-negative). Eight radiologists were instructed to detect breast cancer and rate lesions using BI-RADS and level-of-suspiciousness scores. Computer-aided detection software was used to check the validity of radiologists' findings. Findings found negative by computer-aided detection were not included in the readers’ performance analysis; however, the nature of these findings were further analyzed. The area under the curve and the partial area under the curve for an interval in the range of 80%–100% specificity before and after validation of computer-aided detection were compared. Sensitivity was computed for all readers at a simulation of 90% specificity.

Results

Partial AUC improved significantly from 0.126 (95% confidence interval [CI] = 0.098–0.153) to 0.142 (95% CI = 0.115–0.169) (P  = 0.037) after computer-aided detection rejected mostly benign lesions and normal tissue scored BI-RADS 3 or 4. The full areas under the curve (0.823 vs. 0.833, respectively) were not significantly different (P  = 0.743). Four cancers detected by readers were completely missed by computer-aided detection and four other cancers were detected by both readers and computer-aided detection but falsely rejected due to technical limitations of our implementation of computer-aided detection validation. In this study, validation of computer-aided detection discarded 42.6% of findings that were scored BI-RADS ≥3 by the radiologists, of which 85.5% were non-malignant findings.

Conclusion

Validation of radiologists’ findings using computer-aided detection software for automated breast ultrasound has the potential to improve the performance of radiologists. Validation of computer-aided detection might be an efficient tool for double-reading strategies by limiting the amount of discordant cases needed to be double-read.

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists: A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis

RATIONALE AND OBJECTIVES

Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

METHODS

A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

RESULTS

The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808-0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766-0.811), P < .0001.

CONCLUSIONS

With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs

Purpose To develop and validate a deep learning-based automatic detection algorithm (DLAD) for malignant pulmonary nodules on chest radiographs and to compare its performance with physicians including thoracic radiologists. Materials and Methods For this retrospective study, DLAD was developed by using 43 292 chest radiographs (normal radiograph-to-nodule radiograph ratio, 34 067:9225) in 34 676 patients (healthy-to-nodule ratio, 30 784:3892; 19 230 men [mean age, 52.8 years; age range, 18-99 years]; 15 446 women [mean age, 52.3 years; age range, 18-98 years]) obtained between 2010 and 2015, which were labeled and partially annotated by 13 board-certified radiologists, in a convolutional neural network. Radiograph classification and nodule detection performances of DLAD were validated by using one internal and four external data sets from three South Korean hospitals and one U.S. hospital. For internal and external validation, radiograph classification and nodule detection performances of DLAD were evaluated by using the area under the receiver operating characteristic curve (AUROC) and jackknife alternative free-response receiver-operating characteristic (JAFROC) figure of merit (FOM), respectively. An observer performance test involving 18 physicians, including nine board-certified radiologists, was conducted by using one of the four external validation data sets. Performances of DLAD, physicians, and physicians assisted with DLAD were evaluated and compared. Results According to one internal and four external validation data sets, radiograph classification and nodule detection performances of DLAD were a range of 0.92-0.99 (AUROC) and 0.831-0.924 (JAFROC FOM), respectively. DLAD showed a higher AUROC and JAFROC FOM at the observer performance test than 17 of 18 and 15 of 18 physicians, respectively (P < .05), and all physicians showed improved nodule detection performances with DLAD (mean JAFROC FOM improvement, 0.043; range, 0.006-0.190; P < .05). Conclusion This deep learning-based automatic detection algorithm outperformed physicians in radiograph classification and nodule detection performance for malignant pulmonary nodules on chest radiographs, and it enhanced physicians' performances when used as a second reader. © RSNA, 2018 Online supplemental material is available for this article.

Improved cancer detection in automated breast ultrasound by radiologists using Computer Aided Detection

OBJECTIVE

To investigate the effect of dedicated Computer Aided Detection (CAD) software for automated breast ultrasound (ABUS) on the performance of radiologists screening for breast cancer.

METHODS

90 ABUS views of 90 patients were randomly selected from a multi-institutional archive of cases collected between 2010 and 2013. This dataset included normal cases (n=40) with >1year of follow up, benign (n=30) lesions that were either biopsied or remained stable, and malignant lesions (n=20). Six readers evaluated all cases with and without CAD in two sessions. CAD-software included conventional CAD-marks and an intelligent minimum intensity projection of the breast tissue. Readers reported using a likelihood-of-malignancy scale from 0 to 100. Alternative free-response ROC analysis was used to measure the performance.

RESULTS

Without CAD, the average area-under-the-curve (AUC) of the readers was 0.77 and significantly improved with CAD to 0.84 (p=0.001). Sensitivity of all readers improved (range 5.2-10.6%) by using CAD but specificity decreased in four out of six readers (range 1.4-5.7%). No significant difference was observed in the AUC between experienced radiologists and residents both with and without CAD.

CONCLUSIONS

Dedicated CAD-software for ABUS has the potential to improve the cancer detection rates of radiologists screening for breast cancer.