Augmenting lung cancer diagnosis on chest radiographs: positioning artificial intelligence to improve radiologist performance

Implementation of Artificial Intelligence in Personalized Prognostic Assessment of Lung Cancer: A Narrative Review

Artificial Intelligence (AI) has revolutionized the management of non-small-cell lung cancer (NSCLC) by enhancing different aspects, including staging, prognosis assessment, treatment prediction, response evaluation, recurrence/prognosis prediction, and personalized prognostic assessment. AI algorithms may accurately classify NSCLC stages using machine learning techniques and deep imaging data analysis. This could potentially improve precision and efficiency in staging, facilitating personalized treatment decisions. Furthermore, there are data suggesting the potential application of AI-based models in predicting prognosis in terms of survival rates and disease progression by integrating clinical, imaging and molecular data. In the present narrative review, we will analyze the preliminary studies reporting on how AI algorithms could predict responses to various treatment modalities, such as surgery, radiotherapy, chemotherapy, targeted therapy, and immunotherapy. There is robust evidence suggesting that AI also plays a crucial role in predicting the likelihood of tumor recurrence after surgery and the pattern of failure, which has significant implications for tailoring adjuvant treatments. The successful implementation of AI in personalized prognostic assessment requires the integration of different data sources, including clinical, molecular, and imaging data. Machine learning (ML) and deep learning (DL) techniques enable AI models to analyze these data and generate personalized prognostic predictions, allowing for a precise and individualized approach to patient care. However, challenges relating to data quality, interpretability, and the ability of AI models to generalize need to be addressed. Collaboration among clinicians, data scientists, and regulators is critical for the responsible implementation of AI and for maximizing its benefits in providing a more personalized prognostic assessment. Continued research, validation, and collaboration are essential to fully exploit the potential of AI in NSCLC management and improve patient outcomes. Herein, we have summarized the state of the art of applications of AI in lung cancer for predicting staging, prognosis, and pattern of recurrence after treatment in order to provide to the readers a large comprehensive overview of this challenging issue.

AI-Driven Models for Diagnosing and Predicting Outcomes in Lung Cancer: A Systematic Review and Meta-Analysis

(1) Background: Lung cancer's high mortality due to late diagnosis highlights a need for early detection strategies. Artificial intelligence (AI) in healthcare, particularly for lung cancer, offers promise by analyzing medical data for early identification and personalized treatment. This systematic review evaluates AI's performance in early lung cancer detection, analyzing its techniques, strengths, limitations, and comparative edge over traditional methods. (2) Methods: This systematic review and meta-analysis followed the PRISMA guidelines rigorously, outlining a comprehensive protocol and employing tailored search strategies across diverse databases. Two reviewers independently screened studies based on predefined criteria, ensuring the selection of high-quality data relevant to AI's role in lung cancer detection. The extraction of key study details and performance metrics, followed by quality assessment, facilitated a robust analysis using R software (Version 4.3.0). The process, depicted via a PRISMA flow diagram, allowed for the meticulous evaluation and synthesis of the findings in this review. (3) Results: From 1024 records, 39 studies met the inclusion criteria, showcasing diverse AI model applications for lung cancer detection, emphasizing varying strengths among the studies. These findings underscore AI's potential for early lung cancer diagnosis but highlight the need for standardization amidst study variations. The results demonstrate promising pooled sensitivity and specificity of 0.87, signifying AI's accuracy in identifying true positives and negatives, despite the observed heterogeneity attributed to diverse study parameters. (4) Conclusions: AI demonstrates promise in early lung cancer detection, showing high accuracy levels in this systematic review. However, study variations underline the need for standardized protocols to fully leverage AI's potential in revolutionizing early diagnosis, ultimately benefiting patients and healthcare professionals. As the field progresses, validated AI models from large-scale perspective studies will greatly benefit clinical practice and patient care in the future.

Evaluating the performance of artificial intelligence software for lung nodule detection on chest radiographs in a retrospective real-world UK population

OBJECTIVES

Early identification of lung cancer on chest radiographs improves patient outcomes. Artificial intelligence (AI) tools may increase diagnostic accuracy and streamline this pathway. This study evaluated the performance of commercially available AI-based software trained to identify cancerous lung nodules on chest radiographs.

DESIGN

This retrospective study included primary care chest radiographs acquired in a UK centre. The software evaluated each radiograph independently and outputs were compared with two reference standards: (1) the radiologist report and (2) the diagnosis of cancer by multidisciplinary team decision. Failure analysis was performed by interrogating the software marker locations on radiographs.

PARTICIPANTS

5722 consecutive chest radiographs were included from 5592 patients (median age 59 years, 53.8% women, 1.6% prevalence of cancer).

RESULTS

Compared with radiologist reports for nodule detection, the software demonstrated sensitivity 54.5% (95% CI 44.2% to 64.4%), specificity 83.2% (82.2% to 84.1%), positive predictive value (PPV) 5.5% (4.6% to 6.6%) and negative predictive value (NPV) 99.0% (98.8% to 99.2%). Compared with cancer diagnosis, the software demonstrated sensitivity 60.9% (50.1% to 70.9%), specificity 83.3% (82.3% to 84.2%), PPV 5.6% (4.8% to 6.6%) and NPV 99.2% (99.0% to 99.4%). Normal or variant anatomy was misidentified as an abnormality in 69.9% of the 943 false positive cases.

CONCLUSIONS

The software demonstrated considerable underperformance in this real-world patient cohort. Failure analysis suggested a lack of generalisability in the training and testing datasets as a potential factor. The low PPV carries the risk of over-investigation and limits the translation of the software to clinical practice. Our findings highlight the importance of training and testing software in representative datasets, with broader implications for the implementation of AI tools in imaging.

A Narrative Review of the Use of Artificial Intelligence in Breast, Lung, and Prostate Cancer

Artificial intelligence (AI) has been an important topic within radiology. Currently, AI is used clinically to assist with the detection of lesions through detection systems. However, a number of recent studies have demonstrated the increased value of neural networks in radiology. With an increasing number of screening requirements for cancers, this review aims to study the accuracy of the numerous AI models used in the detection and diagnosis of breast, lung, and prostate cancers. This study summarizes pertinent findings from reviewed articles and provides analysis on the relevancy to clinical radiology. This study found that whereas AI is showing continual improvement in radiology, AI alone does not surpass the effectiveness of a radiologist. Additionally, it was found that there are multiple variations on how AI should be integrated with a radiologist's workflow.

Criticality and clinical department prediction of ED patients using machine learning based on heterogeneous medical data

PROBLEM

Emergency triage faces multiple challenges, including limited medical resources and inadequate manual triage nurses, which cause incorrect triage, overcrowding in the emergency department (ED), and long patient waiting time.

OBJECTIVE

This paper aims to propose and validate an accurate and efficient artificial intelligence-based method for effectively ED triage and alleviating the pressure on medical resources.

METHODS

We propose two novel machine learning models, TransNet and TextRNN, for predicting patient severity levels and clinical departments using heterogeneous medical data in ED triage. Our models employ a parallel structure for feature extraction and incorporate an attention mechanism to extract essential information from the fused features, enabling accurate predictions. The models analyze the triage data (2020-2022) from the ED of Beijing University People's Hospital, incorporating variables (demographics, triage vital signs, and chief complaints) to identify patient severity levels and clinical departments. We performed data cleaning, categorization, and encoding first. Then, we divided the available data into a training set (56%), a validation set (24%), and a test set (20%) by random sampling. Finally, our models underwent 5-fold cross-validation and were compared with other state-of-the-art models.

RESULTS

We comprehensively evaluated the proposed models against various Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN), Traditional Machine Learning (TML), and Transformer-based (TF) models, achieving excellent performance in predicting triage outcomes. Specifically, TextRNN achieved a prediction success rate of 86.23% [85.86-86.70] for severity levels and 94.30% [94.00-94.46] for clinical departments among 161,198 ED visits. Moreover, TransNet demonstrated higher sensitivities of 84.08% and 90.05% for severity levels and clinical departments, respectively, with specificities of 76.48% and 95.16%. The accuracy of our model is 0.87%, 0.18%, 4.29%, and 1.96%, higher than that of the above four family models on average. Furthermore, our method significantly reduced under-triage by 12.06% and over-triage by 17.92% compared to manual triage.

CONCLUSIONS

Experimental results demonstrated that the proposed models fuse heterogeneous medical data in the triage process, successfully predicting patients' triage outcomes. Our models can improve triage efficiency, reduce the under/over-triage rate, and provide physicians with valuable decision-making support.

Artificial Intelligence-Based Software with CE Mark for Chest X-ray Interpretation: Opportunities and Challenges

Chest X-ray (CXR) is the most important technique for performing chest imaging, despite its well-known limitations in terms of scope and sensitivity. These intrinsic limitations of CXR have prompted the development of several artificial intelligence (AI)-based software packages dedicated to CXR interpretation. The online database "AI for radiology" was queried to identify CE-marked AI-based software available for CXR interpretation. The returned studies were divided according to the targeted disease. AI-powered computer-aided detection software is already widely adopted in screening and triage for pulmonary tuberculosis, especially in countries with few resources and suffering from high a burden of this disease. AI-based software has also been demonstrated to be valuable for the detection of lung nodules detection, automated flagging of positive cases, and post-processing through the development of digital bone suppression software able to produce digital bone suppressed images. Finally, the majority of available CE-marked software packages for CXR are designed to recognize several findings, with potential differences in sensitivity and specificity for each of the recognized findings.

Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance

Background Automation bias (the propensity for humans to favor suggestions from automated decision-making systems) is a known source of error in human-machine interactions, but its implications regarding artificial intelligence (AI)-aided mammography reading are unknown. Purpose To determine how automation bias can affect inexperienced, moderately experienced, and very experienced radiologists when reading mammograms with the aid of an artificial intelligence (AI) system. Materials and Methods In this prospective experiment, 27 radiologists read 50 mammograms and provided their Breast Imaging Reporting and Data System (BI-RADS) assessment assisted by a purported AI system. Mammograms were obtained between January 2017 and December 2019 and were presented in two randomized sets. The first was a training set of 10 mammograms, with the correct BI-RADS category suggested by the AI system. The second was a set of 40 mammograms in which an incorrect BI-RADS category was suggested for 12 mammograms. Reader performance, degree of bias in BI-RADS scoring, perceived accuracy of the AI system, and reader confidence in their own BI-RADS ratings were assessed using analysis of variance (ANOVA) and repeated-measures ANOVA followed by post hoc tests and Kruskal-Wallis tests followed by the Dunn post hoc test. Results The percentage of correctly rated mammograms by inexperienced (mean, 79.7% ± 11.7 [SD] vs 19.8% ± 14.0; P < .001; r = 0.93), moderately experienced (mean, 81.3% ± 10.1 vs 24.8% ± 11.6; P < .001; r = 0.96), and very experienced (mean, 82.3% ± 4.2 vs 45.5% ± 9.1; P = .003; r = 0.97) radiologists was significantly impacted by the correctness of the AI prediction of BI-RADS category. Inexperienced radiologists were significantly more likely to follow the suggestions of the purported AI when it incorrectly suggested a higher BI-RADS category than the actual ground truth compared with both moderately (mean degree of bias, 4.0 ± 1.8 vs 2.4 ± 1.5; P = .044; r = 0.46) and very (mean degree of bias, 4.0 ± 1.8 vs 1.2 ± 0.8; P = .009; r = 0.65) experienced readers. Conclusion The results show that inexperienced, moderately experienced, and very experienced radiologists reading mammograms are prone to automation bias when being supported by an AI-based system. This and other effects of human and machine interaction must be considered to ensure safe deployment and accurate diagnostic performance when combining human readers and AI. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Baltzer in this issue.

Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence

OBJECTIVES

To externally validate the performance of a commercial AI software program for interpreting CXRs in a large, consecutive, real-world cohort from primary healthcare centres.

METHODS

A total of 3047 CXRs were collected from two primary healthcare centres, characterised by low disease prevalence, between January and December 2018. All CXRs were labelled as normal or abnormal according to CT findings. Four radiology residents read all CXRs twice with and without AI assistance. The performances of the AI and readers with and without AI assistance were measured in terms of area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity.

RESULTS

The prevalence of clinically significant lesions was 2.2% (68 of 3047). The AUROC, sensitivity, and specificity of the AI were 0.648 (95% confidence interval [CI] 0.630-0.665), 35.3% (CI, 24.7-47.8), and 94.2% (CI, 93.3-95.0), respectively. AI detected 12 of 41 pneumonia, 3 of 5 tuberculosis, and 9 of 22 tumours. AI-undetected lesions tended to be smaller than true-positive lesions. The readers' AUROCs ranged from 0.534-0.676 without AI and 0.571-0.688 with AI (all p values < 0.05). For all readers, the mean reading time was 2.96-10.27 s longer with AI assistance (all p values < 0.05).

CONCLUSIONS

The performance of commercial AI in these high-volume, low-prevalence settings was poorer than expected, although it modestly boosted the performance of less-experienced readers. The technical prowess of AI demonstrated in experimental settings and approved by regulatory bodies may not directly translate to real-world practice, especially where the demand for AI assistance is highest.

KEY POINTS

• This study shows the limited applicability of commercial AI software for detecting abnormalities in CXRs in a health screening population. • When using AI software in a specific clinical setting that differs from the training setting, it is necessary to adjust the threshold or perform additional training with such data that reflects this environment well. • Prospective test accuracy studies, randomised controlled trials, or cohort studies are needed to examine AI software to be implemented in real clinical practice.

Unassisted Clinicians Versus Deep Learning-Assisted Clinicians in Image-Based Cancer Diagnostics: Systematic Review With Meta-analysis

BACKGROUND

A number of publications have demonstrated that deep learning (DL) algorithms matched or outperformed clinicians in image-based cancer diagnostics, but these algorithms are frequently considered as opponents rather than partners. Despite the clinicians-in-the-loop DL approach having great potential, no study has systematically quantified the diagnostic accuracy of clinicians with and without the assistance of DL in image-based cancer identification.

OBJECTIVE

We systematically quantified the diagnostic accuracy of clinicians with and without the assistance of DL in image-based cancer identification.

METHODS

PubMed, Embase, IEEEXplore, and the Cochrane Library were searched for studies published between January 1, 2012, and December 7, 2021. Any type of study design was permitted that focused on comparing unassisted clinicians and DL-assisted clinicians in cancer identification using medical imaging. Studies using medical waveform-data graphics material and those investigating image segmentation rather than classification were excluded. Studies providing binary diagnostic accuracy data and contingency tables were included for further meta-analysis. Two subgroups were defined and analyzed, including cancer type and imaging modality.

RESULTS

In total, 9796 studies were identified, of which 48 were deemed eligible for systematic review. Twenty-five of these studies made comparisons between unassisted clinicians and DL-assisted clinicians and provided sufficient data for statistical synthesis. We found a pooled sensitivity of 83% (95% CI 80%-86%) for unassisted clinicians and 88% (95% CI 86%-90%) for DL-assisted clinicians. Pooled specificity was 86% (95% CI 83%-88%) for unassisted clinicians and 88% (95% CI 85%-90%) for DL-assisted clinicians. The pooled sensitivity and specificity values for DL-assisted clinicians were higher than for unassisted clinicians, at ratios of 1.07 (95% CI 1.05-1.09) and 1.03 (95% CI 1.02-1.05), respectively. Similar diagnostic performance by DL-assisted clinicians was also observed across the predefined subgroups.

CONCLUSIONS

The diagnostic performance of DL-assisted clinicians appears better than unassisted clinicians in image-based cancer identification. However, caution should be exercised, because the evidence provided in the reviewed studies does not cover all the minutiae involved in real-world clinical practice. Combining qualitative insights from clinical practice with data-science approaches may improve DL-assisted practice, although further research is required.

TRIAL REGISTRATION

PROSPERO CRD42021281372; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=281372.

Commonly Missed Findings on Chest Radiographs: Causes and Consequences

Chest radiography (CXR) continues to be the most frequently performed imaging examination worldwide, yet it remains prone to frequent errors in interpretation. These pose potential adverse consequences to patients and are a leading motivation for medical malpractice lawsuits. Commonly missed CXR findings and the principal causes of these errors are reviewed and illustrated. Perceptual errors are the predominant source of these missed findings. The medicolegal implications of such errors are explained. Awareness of commonly missed CXR findings, their causes, and their consequences are important in developing approaches to reduce and mitigate these errors.

Reducing Errors Resulting From Commonly Missed Chest Radiography Findings

Chest radiography (CXR), the most frequently performed imaging examination, is vulnerable to interpretation errors resulting from commonly missed findings. Methods to reduce these errors are presented. A practical approach using a systematic and comprehensive visual search strategy is described. The use of a checklist for quality control in the interpretation of CXR images is proposed to avoid overlooking commonly missed findings of clinical importance. Artificial intelligence is among the emerging and promising methods to enhance detection of CXR abnormalities. Despite their potential adverse consequences, errors offer opportunities for continued education and quality improvements in patient care, if managed within a just, supportive culture.

Comparison of detection performance of soft tissue calcifications using artificial intelligence in panoramic radiography

Artificial intelligence (AI) is limited to teeth and periodontal disease in the dental field, and is used for diagnosis assistance or data analysis, and there has been no research conducted in actual clinical situations. So, we created an environment similar to actual clinical practice and conducted research by selecting three of the soft tissue diseases (carotid artery calcification, lymph node calcification, and sialolith) that are difficult for general dentists to see. Therefore, in this study, the accuracy and reading time are evaluated using panoramic images and AI. A total of 20,000 panoramic images including three diseases were used to develop and train a fast R-CNN model. To compare the performance of the developed model, two oral and maxillofacial radiologists (OMRs) and two general dentists (GDs) read 352 images, excluding the panoramic images used in development for soft tissue calcification diagnosis. On the first visit, the observers read images without AI; on the second visit, the same observers used AI to read the same image. The diagnostic accuracy and specificity for soft tissue calcification of AI were high from 0.727 to 0.926 and from 0.171 to 1.000, whereas the sensitivity for lymph node calcification and sialolith were low at 0.250 and 0.188, respectively. The reading time of AI increased in the GD group (619 to 1049) and decreased in the OMR group (1347 to 1372). In addition, reading scores increased in both groups (GD from 11.4 to 39.8 and OMR from 3.4 to 10.8). Using AI, although the detection sensitivity of sialolith and lymph node calcification was lower than that of carotid artery calcification, the total reading time of the OMR specialists was reduced and the GDs reading accuracy was improved. The AI used in this study helped to improve the diagnostic accuracy of the GD group, who were not familiar with the soft tissue calcification diagnosis, but more data sets are needed to improve the detection performance of the two diseases with low sensitivity of AI.

Frequency of Missed Findings on Chest Radiographs (CXRs) in an International, Multicenter Study: Application of AI to Reduce Missed Findings

Background: Missed findings in chest X-ray interpretation are common and can have serious consequences. Methods: Our study included 2407 chest radiographs (CXRs) acquired at three Indian and five US sites. To identify CXRs reported as normal, we used a proprietary radiology report search engine based on natural language processing (mPower, Nuance). Two thoracic radiologists reviewed all CXRs and recorded the presence and clinical significance of abnormal findings on a 5-point scale (1—not important; 5—critical importance). All CXRs were processed with the AI model (Qure.ai) and outputs were recorded for the presence of findings. Data were analyzed to obtain area under the ROC curve (AUC). Results: Of 410 CXRs (410/2407, 18.9%) with unreported/missed findings, 312 (312/410, 76.1%) findings were clinically important: pulmonary nodules (n = 157), consolidation (60), linear opacities (37), mediastinal widening (21), hilar enlargement (17), pleural effusions (11), rib fractures (6) and pneumothoraces (3). AI detected 69 missed findings (69/131, 53%) with an AUC of up to 0.935. The AI model was generalizable across different sites, geographic locations, patient genders and age groups. Conclusion: A substantial number of important CXR findings are missed; the AI model can help to identify and reduce the frequency of important missed findings in a generalizable manner.

Application of Artificial Intelligence in Lung Cancer

Lung cancer is the leading cause of malignancy-related mortality worldwide due to its heterogeneous features and diagnosis at a late stage. Artificial intelligence (AI) is good at handling a large volume of computational and repeated labor work and is suitable for assisting doctors in analyzing image-dominant diseases like lung cancer. Scientists have shown long-standing efforts to apply AI in lung cancer screening via CXR and chest CT since the 1960s. Several grand challenges were held to find the best AI model. Currently, the FDA have approved several AI programs in CXR and chest CT reading, which enables AI systems to take part in lung cancer detection. Following the success of AI application in the radiology field, AI was applied to digitalized whole slide imaging (WSI) annotation. Integrating with more information, like demographics and clinical data, the AI systems could play a role in decision-making by classifying EGFR mutations and PD-L1 expression. AI systems also help clinicians to estimate the patient's prognosis by predicting drug response, the tumor recurrence rate after surgery, radiotherapy response, and side effects. Though there are still some obstacles, deploying AI systems in the clinical workflow is vital for the foreseeable future.