A comparison of follow-up recommendations by chest radiologists, general radiologists, and pulmonologists using computer-aided detection to assess radiographs for actionable pulmonary nodules

Computer-Aided Detection of Seven Chest Pathologies on Standard Posteroanterior Chest X-Rays Compared to Radiologists Reading Dual-Energy Subtracted Radiographs

RATIONALE AND OBJECTIVES

Retrospective performance evaluation of a computer-aided detection (CAD) system on standard posteroanterior (PA) chest radiographs (PA-CXR) in detection of pulmonary nodules, infectious consolidation, pneumothorax, pleural effusion, aortic calcification, cardiomegaly and rib fractures compared to radiologists analyzing PA-CXR including dual-energy subtraction radiography (further termed as DESR).

MATERIALS AND METHODS

PA-CXR/DESR images of 197 patients were included. All patients underwent chest CT (gold standard) within a short interval (mean 28 hours). All images were evaluated by three blinded readers for the presence of pulmonary nodules, infectious consolidation, pneumothorax, pleural effusion, aortic calcification, cardiomegaly, and rib fractures. Meanwhile PA-CXR were analyzed by a CAD software. CAD results were compared to the majority result of the three readers. Sensitivity and specificity were calculated. McNemar's test was applied to test for significant differences. Interobserver agreement was defined using Cohen's kappa (κ).

RESULTS

Sensitivity of the CAD software was significantly higher (p < 0.05) for detection of infectious consolidation and pulmonary nodules (67.9% vs 26.8% and 54% vs 35.6%, respectively; p < 0.001) compared to radiologists analyzing DESR images. For the residual evaluated pathologies no statistical significant differences could be found. Overall, mean inter observer agreement between the three radiologists was moderate (k = 0.534). The best interobserver agreement could be reached for pneumothorax (k = 0.708) and pleural effusion (k = 0.699), while the worst was obtained for rib fractures (k = 0.412).

CONCLUSION

The CAD system has the potential to improve the detection of infectious consolidation and pulmonary nodules on CXR images.

Is there an advantage to using computer aided detection for the early detection of pulmonary nodules within chest X-Ray imaging?

OBJECTIVE

Using published literature, this research examines whether Computer-aided Detection (CAD) identifies more Pulmonary Nodules (PN) within Chest X-ray (CXR) systems, compared to radiologist diagnosis without CAD.

KEY FINDINGS

Although the primary papers were pointing to CAD being a beneficial system in the diagnosis of PN detection, a regression analysis of the data available within these papers showed no correlation between the higher sensitivity of CAD against the detrimental high False Positives (FP) of CAD. Findings of the studies were deemed inconclusive.

CONCLUSION

Further research is recommended to review the potential of CAD on CXR PN detection.

IMPLICATIONS FOR PRACTICE

CAD acting as a second reader could potentially reduce interpreter error rate.

Analog Computer-Aided Detection (CAD) information can be more effective than binary marks

In socially important visual search tasks, such as baggage screening and diagnostic radiology, experts miss more targets than is desirable. Computer-aided detection (CAD) programs have been developed specifically to improve performance in these professional search tasks. For example, in breast cancer screening, many CAD systems are capable of detecting approximately 90% of breast cancer, with approximately 0.5 false-positive detections per image. Nevertheless, benefits of CAD in clinical settings tend to be small (Birdwell, 2009) or even absent (Meziane et al., 2011; Philpotts, 2009). The marks made by a CAD system can be "binary," giving the same signal to any location where the signal is above some threshold. Alternatively, a CAD system presents an analog signal that reflects strength of the signal at a location. In the experiments reported, we compare analog and binary CAD presentations using nonexpert observers and artificial stimuli defined by two noisy signals: a visible color signal and an "invisible" signal that informed our simulated CAD system. We found that analog CAD generally yielded better overall performance than binary CAD. The analog benefit is similar at high and low target prevalence. Our data suggest that the form of the CAD signal can directly influence performance. Analog CAD may allow the computer to be more helpful to the searcher.

Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images

PURPOSE

To evaluate the added value of computer-aided detection (CAD) for lung nodules on chest radiographs when radiologists have bone-suppressed images (BSIs) available.

MATERIALS AND METHODS

Written informed consent was waived by the institutional review board. Selection of study images and study setup was reviewed and approved by the institutional review boards. Three hundred posteroanterior (PA) and lateral chest radiographs (189 radiographs with negative findings and 111 radiographs with a solitary nodule) in 300 subjects were selected from image archives at four institutions. PA images were processed by using a commercially available CAD, and PA BSIs were generated. Five radiologists and three residents evaluated the radiographs with BSIs available, first, without CAD and, second, after inspection of the CAD marks. Readers marked locations suspicious for a nodule and provided a confidence score for that location to be a nodule. Location-based receiver operating characteristic analysis was performed by using jackknife alternative free-response receiver operating characteristic analysis. Area under the curve (AUC) functioned as figure of merit, and P values were computed with the Dorfman-Berbaum-Metz method.

RESULTS

Average nodule size was 16.2 mm. Stand-alone CAD reached a sensitivity of 74% at 1.0 false-positive mark per image. Without CAD, average AUC for observers was 0.812. With CAD, performance significantly improved to an AUC of 0.841 (P = .0001). CAD detected 127 of 239 nodules that were missed after evaluation of the radiographs together with BSIs pooled over all observers. Only 57 of these detections were eventually marked by the observers after review of CAD candidates.

CONCLUSION

CAD improved radiologists' performance for the detection of lung nodules on chest radiographs, even when baseline performance was optimized by providing lateral radiographs and BSIs. Still, most of the true-positive CAD candidates are dismissed by observers.

A comparison of computer-aided detection (CAD) effectiveness in pulmonary nodule identification using different methods of bone suppression in chest radiographs

This study aimed to compare the diagnostic effectiveness of computer-aided detection (CAD) software (OnGuard™ 5.2) in combination with hardware-based bone suppression (dual-energy subtraction radiography (DESR)), software-based bone suppression (SoftView™, version 2.4), and standard posteroanterior images with no bone suppression. A retrospective pilot study compared the diagnostic performance of two commercially available methods of bone suppression when used with commercially available CAD software. Chest images from 27 patients with computed tomography (CT) and pathology-proven malignant pulmonary nodules (8-34 mm) and 25 CT-negative patient controls were used for analysis. The Friedman, McNemar, and chi-square tests were used to compare diagnostic performance and the kappa statistic was used to evaluate method agreement. The average number of regions of interest and false-positives per image identified by CAD were not found to be significantly different regardless of the bone suppression methods evaluated. Similarly, the sensitivity, specificity, and test efficiency were not found to be significantly different. Agreement between the methods was between poor and excellent. The accuracy of CAD (OnGuard™, version 5.2) is not statistically different with either DESR or SoftView™ (version 2.4) bone suppression technology in digital chest images for pulmonary nodule identification. Low values for sensitivity (<80 %) and specificity (<50 %) may limit their utility for clinical radiology.

New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs

OBJECTIVE

To investigate two new methods of using computer-aided detection (CAD) system information for the detection of lung nodules on chest radiographs. We evaluated an interactive CAD application and an independent combination of radiologists and CAD scores.

METHODS

300 posteroanterior and lateral digital chest radiographs were selected, including 111 with a solitary pulmonary nodule (average diameter, 16 mm). Both nodule and control cases were verified by CT. Six radiologists and six residents reviewed the chest radiographs without CAD and with CAD (ClearRead +Detect™ 5.2; Riverain Technologies, Miamisburg, OH) in two reading sessions. The CAD system was used in an interactive manner; CAD marks, accompanied by a score of suspicion, remained hidden unless the location was queried by the radiologist. Jackknife alternative free response receiver operating characteristics multireader multicase analysis was used to measure detection performance. Area under the curve (AUC) and partial AUC (pAUC) between a specificity of 80% and 100% served as the measure for detection performance. We also evaluated the results of a weighted combination of CAD scores and reader scores, at the location of reader findings.

RESULTS

AUC for the observers without CAD was 0.824. No significant improvement was seen with interactive use of CAD (AUC = 0.834; p = 0.15). Independent combination significantly improved detection performance (AUC = 0.834; p = 0.006). pAUCs without and with interactive CAD were similar (0.128), but improved with independent combination (0.137).

CONCLUSION

Interactive CAD did not improve reader performance for the detection of lung nodules on chest radiographs. Independent combination of reader and CAD scores improved the detection performance of lung nodules.

ADVANCES IN KNOWLEDGE

(1) Interactive use of currently available CAD software did not improve the radiologists' detection performance of lung nodules on chest radiographs. (2) Independently combining the interpretations of the radiologist and the CAD system improved detection of lung nodules on chest radiographs.

Computerized characterization of lung nodule subtlety using thoracic CT images

The goal of this work is to design computerized image analysis techniques for automatically characterizing lung nodule subtlety in CT images. Automated subtlety estimation methods may help in computer-aided detection (CAD) assessment by quantifying dataset difficulty and facilitating comparisons among different CAD algorithms. A dataset containing 813 nodules from 499 patients was obtained from the Lung Image Database Consortium. Each nodule was evaluated by four radiologists regarding nodule subtlety using a 5-point rating scale (1: most subtle). We developed a 3D technique for segmenting lung nodules using a prespecified initial ROI. Texture and morphological features were automatically extracted from the segmented nodules and their margins. The dataset was partitioned into trainers and testers using a 1:1 ratio. An artificial neural network (ANN) was trained with average reader subtlety scores as the reference. Effective features for characterizing nodule subtlety were selected based on the training set using the ANN and a stepwise feature selection method. The performance of the classifier was evaluated using prediction probability (PK) as an agreement measure, which is considered a generalization of the area under the receiver operating characteristic curve when the reference standard is multi-level. Using an ANN classifier trained with a set of 2 features (selected from a total of 30 features), including compactness and average gray value, the test concordance between computer scores and the average reader scores was 0.789 ± 0.014. Our results show that the proposed method had strong agreement with the average of subtlety scores provided by radiologists.

Lung cancer screening with computer aided detection chest radiography: design and results of a randomized, controlled trial

Introduction

The sensitivity of CT based lung cancer screening for the detection of early lung cancer is balanced by the high number of benign lung nodules identified, the unknown consequences of radiation from the test, and the potential costs of a CT based screening program. CAD chest radiography may improve the sensitivity of standard chest radiography while minimizing the risks of CT based screening.

Methods

Study subjects were age 40–75 years with 10+ pack-years of smoking and/or an additional risk for developing lung cancer. Subjects were randomized to receive a PA view chest radiograph or placebo control (went through the process of being imaged but were not imaged). Images were reviewed first without then with the assistance of CAD. Actionable nodules were reported and additional evaluation was tracked. The primary outcome was the rate of developing symptomatic advanced stage lung cancer.

Results

1,424 subjects were enrolled. 710 received a CAD chest radiograph, 29 of whom were found to have an actionable lung nodule on prevalence screening. Of the 15 subjects who had a chest CT performed for additional evaluation, a lung nodule was confirmed in 4, 2 of which represented lung cancer. Both of the cancers were seen by the radiologist unaided and were identified by the CAD chest radiograph. The cumulative incidence of symptomatic advanced lung cancer was 0.42 cases per 100 person-years in the control arm; there were no events in the screening arm.

Conclusions

Further evaluation is necessary to determine if CAD chest radiography has a role as a lung cancer screening tool.

ClinicalTrials.gov identifier NCT01663155

Predicting readers' diagnostic accuracy with a new CAD algorithm

RATIONALE AND OBJECTIVES

Before computer-aided detection (CAD) algorithms can be used in clinical practice, they must be shown to improve readers' diagnostic accuracy over their unaided performance. This is usually accomplished through a large multireader, multicase (MRMC) clinical trial. It is burdensome, however, for an MRMC study to be performed with each new release of a CAD algorithm. The aim of this report is to present an approach for building models to predict readers' accuracy with a new CAD algorithm.

MATERIALS AND METHODS

A modeling approach for predicting readers' results with a new CAD algorithm is described. Multiple-variable logistic regression was used to build models for readers' sensitivity and false-positive rate, given the results of an MRMC study with an older CAD algorithm and the stand-alone performance results of a new CAD algorithm. Data from a large lung MRMC CAD trial are used to illustrate the modeling approach and test the ability of the models to predict readers' accuracy with the new CAD algorithm.

RESULTS

The model overestimated the readers' actual sensitivity with the new CAD algorithm, but this did not reach statistical significance (0.621 vs 0.603, P = .147). The observed and predicted false-positive rates also did not differ significantly (0.275 vs 0.285, P = .250).

CONCLUSIONS

Using one clinical study as a test case, it is shown that the modeling approach is feasible. More testing of the approach is needed to determine if and under what circumstances it can be used as an alternative to a full-scale MRMC study. Meanwhile, the approach can be used to determine if a new CAD algorithm is likely to improve readers' accuracy before embarking on a full-scale MRMC study.