Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study

Performance of artificial intelligence-based software for the automatic detection of lung lesions on chest radiographs of patients with suspected lung cancer

PURPOSE

This study aimed to evaluate the performance of the commercially available artificial intelligence-based software CXR-AID for the automatic detection of pulmonary nodules on the chest radiographs of patients suspected of having lung cancer.

MATERIALS AND METHODS

This retrospective study included 399 patients with clinically suspected lung cancer who underwent CT and chest radiography within 1 month between June 2020 and May 2022. The candidate areas on chest radiographs identified by CXR-AID were categorized into target (properly detected areas) and non-target (improperly detected areas) areas. The non-target areas were further divided into non-target normal areas (false positives for normal structures) and non-target abnormal areas. The visibility score, characteristics and location of the nodules, presence of overlapping structures, and background lung score and presence of pulmonary disease were manually evaluated and compared between the nodules detected or undetected by CXR-AID. The probability indices calculated by CXR-AID were compared between the target and non-target areas.

RESULTS

Among the 450 nodules detected in 399 patients, 331 nodules detected in 313 patients were visible on chest radiographs during manual evaluation. CXR-AID detected 264 of these 331 nodules with a sensitivity of 0.80. The detection sensitivity increased significantly with the visibility score. No significant correlation was observed between the background lung score and sensitivity. The non-target area per image was 0.85, and the probability index of the non-target area was lower than that of the target area. The non-target normal area per image was 0.24. Larger and more solid nodules exhibited higher sensitivities, while nodules with overlapping structures demonstrated lower detection sensitivities.

CONCLUSION

The nodule detection sensitivity of CXR-AID on chest radiographs was 0.80, and the non-target and non-target normal areas per image were 0.85 and 0.24, respectively. Larger, solid nodules without overlapping structures were detected more readily by CXR-AID.

Deep learning-based automatic detection for pulmonary nodules on chest radiographs: The relationship with background lung condition, nodule characteristics, and location

PURPOSE

Computer-aided diagnosis (CAD), which assists in the interpretation of chest radiographs, is becoming common. However, few studies have evaluated the benefits and pitfalls of CAD in the real world. This study aimed to evaluate the independent performance of commercially available deep learning-based automatic detection (DLAD) software, EIRL Chest X-ray Lung Nodule, in a cohort that included patients with background pulmonary abnormalities often encountered in clinical situations.

METHODS

Patients with clinically suspected lung cancer for whom chest radiography was performed within a month before or after CT scan between June 2020 and May 2022 in our institution were enrolled. The reference standard was created using a bounding box annotated by two radiologists with reference to the CT. The visibility score, characteristics, location of the pulmonary nodules, presence of overlapping structures or pulmonary disease, and background lung score were manually determined.

RESULTS

We included 388 patients. The DLAD software detected 222 of the 322 nodules visible on manual evaluation, with a sensitivity of 0.689 and a false-positive rate of 0.168. The detectability of the DLAD software was significantly lower for small and subsolid and nodules with overlapping structures. The visibility score and sensitivity of detection by the DLAD software were positively correlated. The relationship between the background lung score and detection by the DLAD software was unclear.

CONCLUSION

The standalone performance of DLAD in detecting pulmonary nodules exhibited a sensitivity of 0.689 and a false-positive rate of 0.168. Understanding the characteristics of DLAD is crucial when interpreting chest radiographs with the assistance of the DLAD.

Computer-Aided Detection of Seven Chest Pathologies on Standard Posteroanterior Chest X-Rays Compared to Radiologists Reading Dual-Energy Subtracted Radiographs

RATIONALE AND OBJECTIVES

Retrospective performance evaluation of a computer-aided detection (CAD) system on standard posteroanterior (PA) chest radiographs (PA-CXR) in detection of pulmonary nodules, infectious consolidation, pneumothorax, pleural effusion, aortic calcification, cardiomegaly and rib fractures compared to radiologists analyzing PA-CXR including dual-energy subtraction radiography (further termed as DESR).

MATERIALS AND METHODS

PA-CXR/DESR images of 197 patients were included. All patients underwent chest CT (gold standard) within a short interval (mean 28 hours). All images were evaluated by three blinded readers for the presence of pulmonary nodules, infectious consolidation, pneumothorax, pleural effusion, aortic calcification, cardiomegaly, and rib fractures. Meanwhile PA-CXR were analyzed by a CAD software. CAD results were compared to the majority result of the three readers. Sensitivity and specificity were calculated. McNemar's test was applied to test for significant differences. Interobserver agreement was defined using Cohen's kappa (κ).

RESULTS

Sensitivity of the CAD software was significantly higher (p < 0.05) for detection of infectious consolidation and pulmonary nodules (67.9% vs 26.8% and 54% vs 35.6%, respectively; p < 0.001) compared to radiologists analyzing DESR images. For the residual evaluated pathologies no statistical significant differences could be found. Overall, mean inter observer agreement between the three radiologists was moderate (k = 0.534). The best interobserver agreement could be reached for pneumothorax (k = 0.708) and pleural effusion (k = 0.699), while the worst was obtained for rib fractures (k = 0.412).

CONCLUSION

The CAD system has the potential to improve the detection of infectious consolidation and pulmonary nodules on CXR images.

Artificial intelligence-supported lung cancer detection by multi-institutional readers with multi-vendor chest radiographs: a retrospective clinical validation study

Background

We investigated the performance improvement of physicians with varying levels of chest radiology experience when using a commercially available artificial intelligence (AI)-based computer-assisted detection (CAD) software to detect lung cancer nodules on chest radiographs from multiple vendors.

Methods

Chest radiographs and their corresponding chest CT were retrospectively collected from one institution between July 2017 and June 2018. Two author radiologists annotated pathologically proven lung cancer nodules on the chest radiographs while referencing CT. Eighteen readers (nine general physicians and nine radiologists) from nine institutions interpreted the chest radiographs. The readers interpreted the radiographs alone and then reinterpreted them referencing the CAD output. Suspected nodules were enclosed with a bounding box. These bounding boxes were judged correct if there was significant overlap with the ground truth, specifically, if the intersection over union was 0.3 or higher. The sensitivity, specificity, accuracy, PPV, and NPV of the readers’ assessments were calculated.

Results

In total, 312 chest radiographs were collected as a test dataset, including 59 malignant images (59 nodules of lung cancer) and 253 normal images. The model provided a modest boost to the reader’s sensitivity, particularly helping general physicians. The performance of general physicians was improved from 0.47 to 0.60 for sensitivity, from 0.96 to 0.97 for specificity, from 0.87 to 0.90 for accuracy, from 0.75 to 0.82 for PPV, and from 0.89 to 0.91 for NPV while the performance of radiologists was improved from 0.51 to 0.60 for sensitivity, from 0.96 to 0.96 for specificity, from 0.87 to 0.90 for accuracy, from 0.76 to 0.80 for PPV, and from 0.89 to 0.91 for NPV. The overall increase in the ratios of sensitivity, specificity, accuracy, PPV, and NPV were 1.22 (1.14–1.30), 1.00 (1.00–1.01), 1.03 (1.02–1.04), 1.07 (1.03–1.11), and 1.02 (1.01–1.03) by using the CAD, respectively.

Conclusion

The AI-based CAD was able to improve the ability of physicians to detect nodules of lung cancer in chest radiographs. The use of a CAD model can indicate regions physicians may have overlooked during their initial assessment.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-021-08847-9.

Clinical Implementation of Deep Learning in Thoracic Radiology: Potential Applications and Challenges

Chest X-ray radiography and computed tomography, the two mainstay modalities in thoracic radiology, are under active investigation with deep learning technology, which has shown promising performance in various tasks, including detection, classification, segmentation, and image synthesis, outperforming conventional methods and suggesting its potential for clinical implementation. However, the implementation of deep learning in daily clinical practice is in its infancy and facing several challenges, such as its limited ability to explain the output results, uncertain benefits regarding patient outcomes, and incomplete integration in daily workflow. In this review article, we will introduce the potential clinical applications of deep learning technology in thoracic radiology and discuss several challenges for its implementation in daily clinical practice.

Is there an advantage to using computer aided detection for the early detection of pulmonary nodules within chest X-Ray imaging?

OBJECTIVE

Using published literature, this research examines whether Computer-aided Detection (CAD) identifies more Pulmonary Nodules (PN) within Chest X-ray (CXR) systems, compared to radiologist diagnosis without CAD.

KEY FINDINGS

Although the primary papers were pointing to CAD being a beneficial system in the diagnosis of PN detection, a regression analysis of the data available within these papers showed no correlation between the higher sensitivity of CAD against the detrimental high False Positives (FP) of CAD. Findings of the studies were deemed inconclusive.

CONCLUSION

Further research is recommended to review the potential of CAD on CXR PN detection.

IMPLICATIONS FOR PRACTICE

CAD acting as a second reader could potentially reduce interpreter error rate.

Deep Convolutional Neural Network-based Software Improves Radiologist Detection of Malignant Lung Nodules on Chest Radiographs

Background Multicenter studies are required to validate the added benefit of using deep convolutional neural network (DCNN) software for detecting malignant pulmonary nodules on chest radiographs. Purpose To compare the performance of radiologists in detecting malignant pulmonary nodules on chest radiographs when assisted by deep learning-based DCNN software with that of radiologists or DCNN software alone in a multicenter setting. Materials and Methods Investigators at four medical centers retrospectively identified 600 lung cancer-containing chest radiographs and 200 normal chest radiographs. Each radiograph with a lung cancer had at least one malignant nodule confirmed by CT and pathologic examination. Twelve radiologists from the four centers independently analyzed the chest radiographs and marked regions of interest. Commercially available deep learning-based computer-aided detection software separately trained, tested, and validated with 19 330 radiographs was used to find suspicious nodules. The radiologists then reviewed the images with the assistance of DCNN software. The sensitivity and number of false-positive findings per image of DCNN software, radiologists alone, and radiologists with the use of DCNN software were analyzed by using logistic regression and Poisson regression. Results The average sensitivity of radiologists improved (from 65.1% [1375 of 2112; 95% confidence interval {CI}: 62.0%, 68.1%] to 70.3% [1484 of 2112; 95% CI: 67.2%, 73.1%], P < .001) and the number of false-positive findings per radiograph declined (from 0.2 [488 of 2400; 95% CI: 0.18, 0.22] to 0.18 [422 of 2400; 95% CI: 0.16, 0.2], P < .001) when the radiologists re-reviewed radiographs with the DCNN software. For the 12 radiologists in this study, 104 of 2400 radiographs were positively changed (from false-negative to true-positive or from false-positive to true-negative) using the DCNN, while 56 of 2400 radiographs were changed negatively. Conclusion Radiologists had better performance with deep convolutional network software for the detection of malignant pulmonary nodules on chest radiographs than without. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Jacobson in this issue.

Computer-aided Detection Fidelity of Pulmonary Nodules in Chest Radiograph

Aim:

The most ubiquitous chest diagnostic method is the chest radiograph. A common radiographic finding, quite often incidental, is the nodular pulmonary lesion. The detection of small lesions out of complex parenchymal structure is a daily clinical challenge. In this study, we investigate the efficacy of the computer-aided detection (CAD) software package SoftView™ 2.4A for bone suppression and OnGuard™ 5.2 (Riverain Technologies, Miamisburg, OH, USA) for automated detection of pulmonary nodules in chest radiographs.

Subjects and Methods:

We retrospectively evaluated a dataset of 100 posteroanterior chest radiographs with pulmonary nodular lesions ranging from 5 to 85 mm. All nodules were confirmed with a consecutive computed tomography scan and histologically classified as 75% malignant. The number of detected lesions by observation in unprocessed images was compared to the number and dignity of CAD-detected lesions in bone-suppressed images (BSIs).

Results:

SoftView™ BSI does not affect the objective lesion-to-background contrast. OnGuard™ has a stand-alone sensitivity of 62% and specificity of 58% for nodular lesion detection in chest radiographs. The false positive rate is 0.88/image and the false negative (FN) rate is 0.35/image. From the true positive lesions, 20% were proven benign and 80% were malignant. FN lesions were 47% benign and 53% malignant.

Conclusion:

We conclude that CAD does not qualify for a stand-alone standard of diagnosis. The use of CAD accompanied with a critical radiological assessment of the software suggested pattern appears more realistic. Accordingly, it is essential to focus on studies assessing the quality-time-cost profile of real-time (as opposed to retrospective) CAD implementation in clinical diagnostics.

The imaging of small pulmonary nodules

Lung cancer is the leading cause of cancer death worldwide. The major goal in lung cancer research is the improvement of long-term survival. Pulmonary nodules have high clinical importance, they may not only prove to be an early manifestation of lung cancer, but decide to choose the right therapy. This review will introduce the development and current situation of several imaging examination methods: computed tomography (CT), positron emission tomography/computed tomography (PET/CT), endobronchial ultrasound (EBUS).

Computer-Aided Diagnosis for Chest Radiographs in Intensive Care

The chest radiograph is an essential tool for the diagnosis of several lung diseases in intensive care units (ICU). However, several factors make the interpretation of the chest radiograph difficult including the number of X-rays done daily in ICU, the quality of the chest radiograph, and the lack of a standardized interpretation. To overcome these limitations in the interpretation of chest radiographs, researchers have developed computer-aided diagnosis (CAD) systems. In this review, the authors report the methodology used to develop CAD systems including identification of the region of interest, analysis of these regions, and classification. Currently, only a few CAD systems for chest X-ray interpretation are commercially available. Some promising research is ongoing, but the involvement of the pediatric research community is needed for the development and validation of such CAD systems dedicated to pediatric intensive care.

[Detection of lung nodules. New opportunities in chest radiography]

BACKGROUND

Chest radiography still represents the most commonly performed X-ray examination because it is readily available, requires low radiation doses and is relatively inexpensive. However, as previously published, many initially undetected lung nodules are retrospectively visible in chest radiographs.

STANDARD RADIOLOGICAL METHODS

The great improvements in detector technology with the increasing dose efficiency and improved contrast resolution provide a better image quality and reduced dose needs.

METHODICAL INNOVATIONS

The dual energy acquisition technique and advanced image processing methods (e.g. digital bone subtraction and temporal subtraction) reduce the anatomical background noise by reduction of overlapping structures in chest radiography. Computer-aided detection (CAD) schemes increase the awareness of radiologists for suspicious areas.

RESULTS

The advanced image processing methods show clear improvements for the detection of pulmonary lung nodules in chest radiography and strengthen the role of this method in comparison to 3D acquisition techniques, such as computed tomography (CT).

ASSESSMENT

Many of these methods will probably be integrated into standard clinical treatment in the near future. Digital software solutions offer advantages as they can be easily incorporated into radiology departments and are often more affordable as compared to hardware solutions.

Improving the radiologist-CAD interaction: designing for appropriate trust

Computer-aided diagnosis (CAD) has great potential to improve radiologists' diagnostic performance. However, the reported performance of the radiologist-CAD team is lower than what might be expected based on the performance of the radiologist and the CAD system in isolation. This indicates that the interaction between radiologists and the CAD system is not optimal. An important factor in the interaction between humans and automated aids (such as CAD) is trust. Suboptimal performance of the human-automation team is often caused by an inappropriate level of trust in the automation. In this review, we examine the role of trust in the radiologist-CAD interaction and suggest ways to improve the output of the CAD system so that it allows radiologists to calibrate their trust in the CAD system more effectively. Observer studies of the CAD systems show that radiologists often have an inappropriate level of trust in the CAD system. They sometimes under-trust CAD, thereby reducing its potential benefits, and sometimes over-trust it, leading to diagnostic errors they would not have made without CAD. Based on the literature on trust in human-automation interaction and the results of CAD observer studies, we have identified four ways to improve the output of CAD so that it allows radiologists to form a more appropriate level of trust in CAD. Designing CAD systems for appropriate trust is important and can improve the performance of the radiologist-CAD team. Future CAD research and development should acknowledge the importance of the radiologist-CAD interaction, and specifically the role of trust therein, in order to create the perfect artificial partner for the radiologist. This review focuses on the role of trust in the radiologist-CAD interaction. The aim of the review is to encourage CAD developers to design for appropriate trust and thereby improve the performance of the radiologist-CAD team.

Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images

PURPOSE

To evaluate the added value of computer-aided detection (CAD) for lung nodules on chest radiographs when radiologists have bone-suppressed images (BSIs) available.

MATERIALS AND METHODS

Written informed consent was waived by the institutional review board. Selection of study images and study setup was reviewed and approved by the institutional review boards. Three hundred posteroanterior (PA) and lateral chest radiographs (189 radiographs with negative findings and 111 radiographs with a solitary nodule) in 300 subjects were selected from image archives at four institutions. PA images were processed by using a commercially available CAD, and PA BSIs were generated. Five radiologists and three residents evaluated the radiographs with BSIs available, first, without CAD and, second, after inspection of the CAD marks. Readers marked locations suspicious for a nodule and provided a confidence score for that location to be a nodule. Location-based receiver operating characteristic analysis was performed by using jackknife alternative free-response receiver operating characteristic analysis. Area under the curve (AUC) functioned as figure of merit, and P values were computed with the Dorfman-Berbaum-Metz method.

RESULTS

Average nodule size was 16.2 mm. Stand-alone CAD reached a sensitivity of 74% at 1.0 false-positive mark per image. Without CAD, average AUC for observers was 0.812. With CAD, performance significantly improved to an AUC of 0.841 (P = .0001). CAD detected 127 of 239 nodules that were missed after evaluation of the radiographs together with BSIs pooled over all observers. Only 57 of these detections were eventually marked by the observers after review of CAD candidates.

CONCLUSION

CAD improved radiologists' performance for the detection of lung nodules on chest radiographs, even when baseline performance was optimized by providing lateral radiographs and BSIs. Still, most of the true-positive CAD candidates are dismissed by observers.

A comparison of computer-aided detection (CAD) effectiveness in pulmonary nodule identification using different methods of bone suppression in chest radiographs

This study aimed to compare the diagnostic effectiveness of computer-aided detection (CAD) software (OnGuard™ 5.2) in combination with hardware-based bone suppression (dual-energy subtraction radiography (DESR)), software-based bone suppression (SoftView™, version 2.4), and standard posteroanterior images with no bone suppression. A retrospective pilot study compared the diagnostic performance of two commercially available methods of bone suppression when used with commercially available CAD software. Chest images from 27 patients with computed tomography (CT) and pathology-proven malignant pulmonary nodules (8-34 mm) and 25 CT-negative patient controls were used for analysis. The Friedman, McNemar, and chi-square tests were used to compare diagnostic performance and the kappa statistic was used to evaluate method agreement. The average number of regions of interest and false-positives per image identified by CAD were not found to be significantly different regardless of the bone suppression methods evaluated. Similarly, the sensitivity, specificity, and test efficiency were not found to be significantly different. Agreement between the methods was between poor and excellent. The accuracy of CAD (OnGuard™, version 5.2) is not statistically different with either DESR or SoftView™ (version 2.4) bone suppression technology in digital chest images for pulmonary nodule identification. Low values for sensitivity (<80 %) and specificity (<50 %) may limit their utility for clinical radiology.

New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs

OBJECTIVE

To investigate two new methods of using computer-aided detection (CAD) system information for the detection of lung nodules on chest radiographs. We evaluated an interactive CAD application and an independent combination of radiologists and CAD scores.

METHODS

300 posteroanterior and lateral digital chest radiographs were selected, including 111 with a solitary pulmonary nodule (average diameter, 16 mm). Both nodule and control cases were verified by CT. Six radiologists and six residents reviewed the chest radiographs without CAD and with CAD (ClearRead +Detect™ 5.2; Riverain Technologies, Miamisburg, OH) in two reading sessions. The CAD system was used in an interactive manner; CAD marks, accompanied by a score of suspicion, remained hidden unless the location was queried by the radiologist. Jackknife alternative free response receiver operating characteristics multireader multicase analysis was used to measure detection performance. Area under the curve (AUC) and partial AUC (pAUC) between a specificity of 80% and 100% served as the measure for detection performance. We also evaluated the results of a weighted combination of CAD scores and reader scores, at the location of reader findings.

RESULTS

AUC for the observers without CAD was 0.824. No significant improvement was seen with interactive use of CAD (AUC = 0.834; p = 0.15). Independent combination significantly improved detection performance (AUC = 0.834; p = 0.006). pAUCs without and with interactive CAD were similar (0.128), but improved with independent combination (0.137).

CONCLUSION

Interactive CAD did not improve reader performance for the detection of lung nodules on chest radiographs. Independent combination of reader and CAD scores improved the detection performance of lung nodules.

ADVANCES IN KNOWLEDGE

(1) Interactive use of currently available CAD software did not improve the radiologists' detection performance of lung nodules on chest radiographs. (2) Independently combining the interpretations of the radiologist and the CAD system improved detection of lung nodules on chest radiographs.

Comparison of dual-energy subtraction and electronic bone suppression combined with computer-aided detection on chest radiographs: effect on human observers' performance in nodule detection

OBJECTIVE

The objective of our study was to compare the effect of dual-energy subtraction and bone suppression software alone and in combination with computer-aided detection (CAD) on the performance of human observers in lung nodule detection.

MATERIALS AND METHODS

One hundred one patients with from one to five lung nodules measuring 5-29 mm and 42 subjects with no nodules were retrospectively selected and randomized. Three independent radiologists marked suspicious-appearing lesions on the original chest radiographs, dual-energy subtraction images, and bone-suppressed images before and after postprocessing with CAD. Marks of the observers and CAD marks were compared with CT as the reference standard. Data were analyzed using nonparametric tests and the jackknife alternative free-response receiver operating characteristic (JAFROC) method.

RESULTS

Using dual-energy subtraction alone (p = 0.0198) or CAD alone (p = 0.0095) improved the detection rate compared with using the original conventional chest radiograph. The combination of bone suppression and CAD provided the highest sensitivity (51.6%) and the original nonenhanced conventional chest radiograph alone provided the lowest (46.9%; p = 0.0049). Dual-energy subtraction and bone suppression provided the same false-positive (p = 0.2702) and true-positive (p = 0.8451) rates. Up to 22.9% of lesions were found only by the CAD program and were missed by the readers. JAFROC showed no difference in the performance between modalities (p = 0.2742-0.5442).

CONCLUSION

Dual-energy subtraction and the electronic bone suppression program used in this study provided similar detection rates for pulmonary nodules. Additionally, CAD alone or combined with bone suppression can significantly improve the sensitivity of human observers for pulmonary nodule detection.