Using Artificial Intelligence to Stratify Normal versus Abnormal Chest X-rays: External Validation of a Deep Learning Algorithm at East Kent Hospitals University NHS Foundation Trust

Post-deployment performance of a deep learning algorithm for normal and abnormal chest X-ray classification: A study at visa screening centers in the United Arab Emirates

Background

Chest radiographs (CXRs) are widely used to screen for infectious diseases like tuberculosis and COVID-19 among migrants. At such high-volume settings, manual CXR reporting is challenging and integrating artificial intelligence (AI) algorithms into the workflow help to rule out normal findings in minutes, allowing radiologists to focus on abnormal cases.

Methods

In this post-deployment study, all the CXRs acquired during the visa screening process across 33 centers in United Arab Emirates from January 2021 to June 2022 (18 months) were included. The qXR v2.1 chest X-ray interpretation software was used to classify the scans into normal and abnormal, and its agreement against radiologist was evaluated. Additionally, a digital survey was conducted among 20 healthcare professionals with prior AI experience to understand real-world implementation challenges and impact.

Results

The analysis of 1309,443 CXRs from 1309,431 patients (median age: 35 years; IQR [29-42]; 1030,071 males [78.7 %]) in this study revealed a Negative Predictive Value (NPV) of 99.92 % (95 % CI: 99.92, 99.93), Positive Predictive Value (PPV) of 5.06 % (95 % CI: 4.99, 5.13) and overall percent agreement of the AI with radiologists of 72.90 % (95 % CI: 72.82, 72.98). In the survey, majority (88.2 %) of the radiologists agreed to turnaround time reduction after AI integration, while 82 % suggested that the AI improved their diagnostic accuracy.

Discussion

In contrast with the existing studies, this research uses a substantially large data. A high NPV and satisfactory agreement with human readers indicate that AI can reliably identify normal CXRs, making it suitable for routine applications.

Effect of Artificial Intelligence as a Second Reader on the Lung Nodule Detection and Localization Accuracy of Radiologists and Non-radiology Physicians in Chest Radiographs: A Multicenter Reader Study

RATIONALE AND OBJECTIVES

Missed nodules in chest radiographs (CXRs) are common occurrences. We assessed the effect of artificial intelligence (AI) as a second reader on the accuracy of radiologists and non-radiology physicians in lung nodule detection and localization in CXRs.

MATERIALS AND METHODS

This retrospective study using the multi-reader multi-case design included 300 CXRs acquired from 40 hospitals across the US. All CXRs had a paired follow-up image (chest CT or CXR) to augment the ground truth establishment for the presence and location of nodules on CXRs by five independent thoracic radiologists. 15 readers (nine radiologists and six non-radiology physicians) read each CXR twice in a second-reader paradigm, once without AI and then immediately with AI assistance. The primary analysis assessed the difference in area-under-the-alternative-free-response-receiver-operating-characteristic-curve (AFROC) of readers with and without AI. Case-level area-under-the-receiver-operating-characteristic-curve (AUROC), sensitivity, and specificity were assessed in secondary analyses.

RESULTS

A total of 300 CXRs (147 with nodules, 153 without nodules) from 300 patients (mean age, 64 years ± 15 [standard deviation]; 174 women) were included. The mean AFROC of readers was 0.73 without AI and 0.81 with AI (95% CI of difference, 0.05-0.10). Case-level AUROC was 0.77 without AI and 0.84 with AI (95% CI of difference, 0.04-0.09). Case-level sensitivity was 72.8% and 83.5% (95% CI of difference, 6.8-14.6) and specificity was 71.1% and 72.0% (95% CI of difference, -0.8-2.6) without and with AI, respectively.

CONCLUSION

Using AI, readers detected and localized more nodules without any significant difference in false positive interpretations.

Deploying artificial intelligence software in an NHS trust: a how-to guide for clinicians

Over the past 10 years, artificial intelligence (AI) has become one of the fastest-growing sectors in healthcare. There are now numerous new technologies designed to cut costs and improve diagnoses and treatment pathways. However, there is significant scepticism amongst National Health Service (NHS) staff regarding the usefulness of AI and it's cost to the NHS. This has likely resulted in underuse and slow adoption of software that may revolutionize our healthcare system and ensure its continued survival and effectiveness. Several governing bodies have put forward guidance on the safe and effective adoption of AI tools, but this rarely covers the reality of selecting and deploying new software. This article set out clear guidance on the practicalities and pitfalls of deploying digital solutions in healthcare, using the example of a deep learning algorithm designed to improve the accuracy of chest X-ray (CXR) interpretation in the emergency department.

Accuracy of an artificial intelligence-enabled diagnostic assistance device in recognizing normal chest radiographs: a service evaluation

Objectives

Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR).

Methods

A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI.

Results

A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased.

Conclusions

The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity.

Advances in knowledge

There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.