Performance of artificial intelligence-based software for the automatic detection of lung lesions on chest radiographs of patients with suspected lung cancer

PURPOSE

This study aimed to evaluate the performance of the commercially available artificial intelligence-based software CXR-AID for the automatic detection of pulmonary nodules on the chest radiographs of patients suspected of having lung cancer.

MATERIALS AND METHODS

This retrospective study included 399 patients with clinically suspected lung cancer who underwent CT and chest radiography within 1 month between June 2020 and May 2022. The candidate areas on chest radiographs identified by CXR-AID were categorized into target (properly detected areas) and non-target (improperly detected areas) areas. The non-target areas were further divided into non-target normal areas (false positives for normal structures) and non-target abnormal areas. The visibility score, characteristics and location of the nodules, presence of overlapping structures, and background lung score and presence of pulmonary disease were manually evaluated and compared between the nodules detected or undetected by CXR-AID. The probability indices calculated by CXR-AID were compared between the target and non-target areas.

RESULTS

Among the 450 nodules detected in 399 patients, 331 nodules detected in 313 patients were visible on chest radiographs during manual evaluation. CXR-AID detected 264 of these 331 nodules with a sensitivity of 0.80. The detection sensitivity increased significantly with the visibility score. No significant correlation was observed between the background lung score and sensitivity. The non-target area per image was 0.85, and the probability index of the non-target area was lower than that of the target area. The non-target normal area per image was 0.24. Larger and more solid nodules exhibited higher sensitivities, while nodules with overlapping structures demonstrated lower detection sensitivities.

CONCLUSION

The nodule detection sensitivity of CXR-AID on chest radiographs was 0.80, and the non-target and non-target normal areas per image were 0.85 and 0.24, respectively. Larger, solid nodules without overlapping structures were detected more readily by CXR-AID.

Dallal Bashi O, Nik Ali N, Mahmood Al Kubaisi Y. Lung disease recognition methods using audio-based analysis with machine learning

The use of computer-based automated approaches and improvements in lung sound recording techniques have made lung sound-based diagnostics even better and devoid of subjectivity errors. Using a computer to evaluate lung sound features more thoroughly with the use of analyzing changes in lung sound behavior, recording measurements, suppressing the presence of noise contaminations, and graphical representations are all made possible by computer-based lung sound analysis. This paper starts with a discussion of the need for this research area, providing an overview of the field and the motivations behind it. Following that, it details the survey methodology used in this work. It presents a discussion on the elements of sound-based lung disease classification using machine learning algorithms. This includes commonly prior considered datasets, feature extraction techniques, pre-processing methods, artifact removal methods, lung-heart sound separation, deep learning algorithms, and wavelet transform of lung audio signals. The study introduces studies that review lung screening including a summary table of these references and discusses the literature gaps in the existing studies. It is concluded that the use of sound-based machine learning in the classification of respiratory diseases has promising results. While we believe this material will prove valuable to physicians and researchers exploring sound-signal-based machine learning, large-scale investigations remain essential to solidify the findings and foster wider adoption within the medical community.

Use of artificial intelligence in triaging of chest radiographs to reduce radiologists' workload

OBJECTIVES

To evaluate whether deep learning-based detection algorithms (DLD)-based triaging can reduce outpatient chest radiograph interpretation workload while maintaining noninferior sensitivity.

METHODS

This retrospective study included patients who underwent initial chest radiography at the outpatient clinic between June 1 and June 30, 2017. Readers interpreted radiographs with/without a commercially available DLD that detects nine radiologic findings (atelectasis, calcification, cardiomegaly, consolidation, fibrosis, nodules, pneumothorax, pleural effusion, and pneumoperitoneum). The reading order was determined in a randomized, crossover manner. The radiographs were classified into negative and positive examinations. In a 50% worklist reduction scenario, radiographs were sorted in descending order of probability scores: the lower half was regarded as negative exams, while the remaining were read with DLD by radiologists. The primary analysis evaluated noninferiority in sensitivity between radiologists reading all radiographs and simulating a 50% worklist reduction, with the inferiority margin of 5%. The specificities were compared using McNemar's test.

RESULTS

The study included 1964 patients (median age [interquartile range], 55 years [40-67 years]). The sensitivity was 82.6% (195 of 236; 95% CI: 77.5%, 87.3%) when readers interpreted all chest radiographs without DLD and 83.5% (197 of 236; 95% CI: 78.8%, 88.1%) in the 50% worklist reduction scenario. The difference in sensitivity was 0.8% (95% CI: - 3.8%, 5.5%), establishing noninferiority of 50% worklist reduction (p = 0.01). The specificity increased from 86.7% (1498 of 1728) to 90.4% (1562 of 1728) (p < 0.001) with DLD-based triage.

CONCLUSION

Deep learning-based triaging may substantially reduce workload without lowering sensitivity while improving specificity.

CLINICAL RELEVANCE STATEMENT

Substantial workload reduction without lowering sensitivity was feasible using deep learning-based triaging of outpatient chest radiograph; however, the legal responsibility for incorrect diagnoses based on AI-standalone interpretation remains an issue that should be defined before clinical implementation.

KEY POINTS

• A 50% workload reduction simulation using deep learning-based detection algorithm maintained noninferior sensitivity while improving specificity. • The CT recommendation rate significantly decreased in the disease-negative patients, whereas it slightly increased in the disease-positive group without statistical significance. • In the exploratory analysis, the noninferiority of sensitivity was maintained until 70% of the workload was reduced; the difference in sensitivity was 0%.

Can Artificial Intelligence Replace Humans for Detecting Lung Tumors on Radiographs? An Examination of Resected Malignant Lung Tumors

OBJECTIVE

Although lung cancer screening trials have showed the efficacy of computed tomography to decrease mortality compared with chest radiography, the two are widely taken as different kinds of clinical practices. Artificial intelligence can improve outcomes by detecting lung tumors in chest radiographs. Currently, artificial intelligence is used as an aid for physicians to interpret radiograms, but with the future evolution of artificial intelligence, it may become a modality that replaces physicians. Therefore, in this study, we investigated the current situation of lung cancer diagnosis by artificial intelligence.

METHODS

In total, we recruited 174 consecutive patients with malignant pulmonary tumors who underwent surgery after chest radiography that was checked by artificial intelligence before surgery. Artificial intelligence diagnoses were performed using the medical image analysis software EIRL X-ray Lung Nodule version 1.12, (LPIXEL Inc., Tokyo, Japan).

RESULTS

The artificial intelligence determined pulmonary tumors in 90 cases (51.7% for all patients and 57.7% excluding 18 patients with adenocarcinoma in situ). There was no significant difference in the detection rate by the artificial intelligence among histological types. All eighteen cases of adenocarcinoma in situ were not detected by either the artificial intelligence or the physicians. In a univariate analysis, the artificial intelligence could detect cases with larger histopathological tumor size (p < 0.0001), larger histopathological invasion size (p < 0.0001), and higher maximum standardized uptake values of positron emission tomography-computed tomography (p < 0.0001). In a multivariate analysis, detection by AI was significantly higher in cases with a large histopathological invasive size (p = 0.006). In 156 cases excluding adenocarcinoma in situ, we examined the rate of artificial intelligence detection based on the tumor site. Tumors in the lower lung field area were less frequently detected (p = 0.019) and tumors in the middle lung field area were more frequently detected (p = 0.014) compared with tumors in the upper lung field area.

CONCLUSIONS

Our study showed that using artificial intelligence, the diagnosis of tumor-associated findings and the diagnosis of areas that overlap with anatomical structures is not satisfactory. While the current standing of artificial intelligence diagnostics is to assist physicians in making diagnoses, there is the possibility that artificial intelligence can substitute for humans in the future. However, artificial intelligence should be used in the future as an enhancement, to aid physicians in the role of a radiologist in the workflow.

Retrospectively assessing evaluation and management of artificial-intelligence detected nodules on uninterpreted chest radiographs in the era of radiologists shortage

PURPOSE

High volumes of chest radiographs (CXR) remain uninterpreted due to severe shortage of radiologists. These CXRs may be informally reported by non-radiologist physicians, or not reviewed at all. Artificial intelligence (AI) software can aid lung nodule detection. Our aim was to assess evaluation and management by non-radiologists of uninterpreted CXRs with AI detected nodules, compared to retrospective radiology reports.

MATERIALS AND METHODS

AI detected nodules on uninterpreted CXRs of adults, performed 30/6/2022-31/1/2023, were evaluated. Excluded were patients with known active malignancy and duplicate CXRs of the same patient. The electronic medical records (EMR) were reviewed, and the clinicians' notes on the CXR and AI detected nodule were documented. Dedicated thoracic radiologists retrospectively interpreted all CXRs, and similarly to the clinicians, they had access to the AI findings, prior imaging and EMR. The radiologists' interpretation served as the ground truth, and determined if the AI-detected nodule was a true lung nodule and if further workup was required.

RESULTS

A total of 683 patients met the inclusion criteria. The clinicians commented on 386 (56.5%) CXRs, identified true nodules on 113 CXRs (16.5%), incorrectly mentioned 31 (4.5%) false nodules as real nodules, and did not mention the AI detected nodule on 242 (35%) CXRs, of which 68 (10%) patients were retrospectively referred for further workup by the radiologist. For 297 patients (43.5%) there were no comments regarding the CXR in the EMR. Of these, 77 nodules (11.3%) were retrospectively referred for further workup by the radiologist.

CONCLUSION

AI software for lung nodule detection may be insufficient without a formal radiology report, and may lead to over diagnosis or misdiagnosis of nodules.

Deep learning-based reconstruction for acceleration of lumbar spine MRI: a prospective comparison with standard MRI

OBJECTIVE

To compare the image quality and diagnostic performance between standard turbo spin-echo MRI and accelerated MRI with deep learning (DL)-based image reconstruction for degenerative lumbar spine diseases.

MATERIALS AND METHODS

Fifty patients who underwent both the standard and accelerated lumbar MRIs at a 1.5-T scanner for degenerative lumbar spine diseases were prospectively enrolled. DL reconstruction algorithm generated coarse (DL_coarse) and fine (DL_fine) images from the accelerated protocol. Image quality was quantitatively assessed in terms of signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) and qualitatively assessed using five-point visual scoring systems. The sensitivity and specificity of four radiologists for the diagnosis of degenerative diseases in both protocols were compared.

RESULTS

The accelerated protocol reduced the average MRI acquisition time by 32.3% as compared to the standard protocol. As compared with standard images, DL_coarse and DL_fine showed significantly higher SNRs on T1-weighted images (T1WI; both p < .001) and T2-weighted images (T2WI; p = .002 and p < 0.001), higher CNRs on T1WI (both p < 0.001), and similar CNRs on T2WI (p = .49 and p = .27). The average radiologist assessment of overall image quality for DL_coarse and DL_fine was higher on sagittal T1WI (p = .04 and p < .001) and axial T2WI (p = .006 and p = .01) and similar on sagittal T2WI (p = .90 and p = .91). Both DL_coarse and DL_fine had better image quality of cauda equina and paraspinal muscles on axial T2WI (both p = .04 for cauda equina; p = .008 and p = .002 for paraspinal muscles). Differences in sensitivity and specificity for the detection of central canal stenosis and neural foraminal stenosis between standard and DL-reconstructed images were all statistically nonsignificant (p ≥ 0.05).

CONCLUSION

DL-based protocol reduced MRI acquisition time without degrading image quality and diagnostic performance of readers for degenerative lumbar spine diseases.

CLINICAL RELEVANCE STATEMENT

The deep learning (DL)-based reconstruction algorithm may be used to further accelerate spine MRI imaging to reduce patient discomfort and increase the cost efficiency of spine MRI imaging.

KEY POINTS

• By using deep learning (DL)-based reconstruction algorithm in combination with the accelerated MRI protocol, the average acquisition time was reduced by 32.3% as compared with the standard protocol. • DL-reconstructed images had similar or better quantitative/qualitative overall image quality and similar or better image quality for the delineation of most individual anatomical structures. • The average radiologist's sensitivity and specificity for the detection of major degenerative lumbar spine diseases, including central canal stenosis, neural foraminal stenosis, and disc herniation, on standard and DL-reconstructed images, were similar.

Clinical outcomes and actual consequence of lung nodules incidentally detected on chest radiographs by artificial intelligence

This study evaluated how often clinically significant lung nodules were detected unexpectedly on chest radiographs (CXR) by artificial intelligence (AI)-based detection software, and whether co-existing findings can aid in differential diagnosis of lung nodules. Patients (> 18 years old) with AI-detected lung nodules at their first visit from March 2021 to February 2022, except for those in the pulmonology or thoracic surgery departments, were retrospectively included. Three radiologists categorized nodules into malignancy, active inflammation, post-inflammatory sequelae, or "other" groups. Characteristics of the nodule and abnormality scores of co-existing lung lesions were compared. Approximately 1% of patients (152/14,563) had unexpected lung nodules. Among 73 patients with follow-up exams, 69.9% had true positive nodules. Increased abnormality scores for nodules were significantly associated with malignancy (odds ratio [OR] 1.076, P = 0.001). Increased abnormality scores for consolidation (OR 1.033, P = 0.040) and pleural effusion (OR 1.025, P = 0.041) were significantly correlated with active inflammation-type nodules. Abnormality scores for fibrosis (OR 1.036, P = 0.013) and nodules (OR 0.940, P = 0.001) were significantly associated with post-inflammatory sequelae categorization. AI-based lesion-detection software of CXRs in daily practice can help identify clinically significant incidental lung nodules, and referring accompanying lung lesions may help classify the nodule.

Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography

OBJECTIVE

To examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error.

METHODS

Multi-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes/no) on four occasions (09/20-01/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2-4, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions.

RESULTS

Relative to the No AI condition (FN = 2.7%, FP = 51.4%), FN and FPs were higher in the Keep AI (No Box) (FN = 33.0%, FP = 86.0%), Delete AI (No Box) (FN = 26.7%, FP = 80.5%), and Keep AI (Box) (FN = to 20.7%, FP = 80.5%) conditions (all ps < 0.05). FNs were higher in the Keep AI (No Box) condition (33.0%) than in the Keep AI (Box) condition (20.7%) (p = 0.04). FPs were higher in the Keep AI (No Box) (86.0%) condition than in the Delete AI (No Box) condition (80.5%) (p = 0.03).

CONCLUSION

Incorrect AI causes radiologists to make incorrect follow-up decisions when they were correct without AI. This effect is mitigated when radiologists believe AI will be deleted from the patient's file or a box is provided around the region of interest.

CLINICAL RELEVANCE STATEMENT

When AI is wrong, radiologists make more errors than they would have without AI. Based on human factors psychology, our manuscript provides evidence for two AI implementation strategies that reduce the deleterious effects of incorrect AI.

KEY POINTS

• When AI provided incorrect results, false negative and false positive rates among the radiologists increased. • False positives decreased when AI results were deleted, versus kept, in the patient's record. • False negatives and false positives decreased when AI visually outlined the region of suspicion.

The diagnostic performance and clinical value of deep learning-based nodule detection system concerning influence of location of pulmonary nodule

BACKGROUND

The deep learning-based nodule detection (DLD) system improves nodule detection performance of observers on chest radiographs (CXRs). However, its performance in different pulmonary nodule (PN) locations remains unknown.

METHODS

We divided the CXR intrathoracic region into non-danger zone (NDZ) and danger zone (DZ). The DZ included the lung apices, paramediastinal areas, and retrodiaphragmatic areas, where nodules could be missed. We used a dataset of 300 CXRs (100 normal and 200 abnormal images with 216 PNs [107 NDZ and 109 DZ nodules]). Eight observers (two thoracic radiologists [TRs], two non-thoracic radiologists [NTRs], and four radiology residents [RRs]) interpreted each radiograph with and without the DLD system. The metric of lesion localization fraction (LLF; the number of correctly localized lesions divided by the total number of true lesions) was used to evaluate the diagnostic performance according to the nodule location.

RESULTS

The DLD system demonstrated a lower LLF for the detection of DZ nodules (64.2) than that of NDZ nodules (83.2, p = 0.008). For DZ nodule detection, the LLF of the DLD system (64.2) was lower than that of TRs (81.7, p < 0.001), which was comparable to that of NTRs (56.4, p = 0.531) and RRs (56.7, p = 0.459). Nonetheless, the LLF of RRs significantly improved from 56.7 to 65.6 using the DLD system (p = 0.021) for DZ nodule detection.

CONCLUSION

The performance of the DLD system was lower in the detection of DZ nodules compared to that of NDZ nodules. Nonetheless, RR performance in detecting DZ nodules improved upon using the DLD system.

CRITICAL RELEVANCE STATEMENT

Despite the deep learning-based nodule detection system's limitations in detecting danger zone nodules, it proves beneficial for less-experienced observers by providing valuable assistance in identifying these nodules, thereby advancing nodule detection in clinical practice.

KEY POINTS

• The deep learning-based nodule detection (DLD) system can improve the diagnostic performance of observers in nodule detection. • The DLD system shows poor diagnostic performance in detecting danger zone nodules. • For less-experienced observers, the DLD system is helpful in detecting danger zone nodules.

Deep learning-based automatic detection for pulmonary nodules on chest radiographs: The relationship with background lung condition, nodule characteristics, and location

PURPOSE

Computer-aided diagnosis (CAD), which assists in the interpretation of chest radiographs, is becoming common. However, few studies have evaluated the benefits and pitfalls of CAD in the real world. This study aimed to evaluate the independent performance of commercially available deep learning-based automatic detection (DLAD) software, EIRL Chest X-ray Lung Nodule, in a cohort that included patients with background pulmonary abnormalities often encountered in clinical situations.

METHODS

Patients with clinically suspected lung cancer for whom chest radiography was performed within a month before or after CT scan between June 2020 and May 2022 in our institution were enrolled. The reference standard was created using a bounding box annotated by two radiologists with reference to the CT. The visibility score, characteristics, location of the pulmonary nodules, presence of overlapping structures or pulmonary disease, and background lung score were manually determined.

RESULTS

We included 388 patients. The DLAD software detected 222 of the 322 nodules visible on manual evaluation, with a sensitivity of 0.689 and a false-positive rate of 0.168. The detectability of the DLAD software was significantly lower for small and subsolid and nodules with overlapping structures. The visibility score and sensitivity of detection by the DLAD software were positively correlated. The relationship between the background lung score and detection by the DLAD software was unclear.

CONCLUSION

The standalone performance of DLAD in detecting pulmonary nodules exhibited a sensitivity of 0.689 and a false-positive rate of 0.168. Understanding the characteristics of DLAD is crucial when interpreting chest radiographs with the assistance of the DLAD.

Associations of chest X-ray trajectories, smoking, and the risk of lung cancer in two population-based cohort studies

Objectives

Despite the increasing use of computed tomography (CT), chest X-ray (CXR) remains the first-line investigation for suspected lung cancer (LC) in primary care. However, the associations of CXR trajectories, smoking and LC risk remain unknown.

Methods

A total of 52,486 participants from the PLCO and 22,194 participants from the NLST were included. The associations of CXR trajectories with LC risk were evaluated with multivariable COX regression models and pooled with meta-analyses. Further analyses were conducted to explore the stratified associations by smoking status and the factors associated with progression and regression in CXR.

Results

Compared to stable negative CXR (CXRSN), HRs (95%CIs) of LC incidence were 2.88(1.50-5.52), 3.86(2.03-7.35), and 1.08(0.80-1.46) for gain of positive CXR (CXRGP), stable positive CXR (CXRSP), and loss of positive CXR (CXRLP), while the risk of LC mortality were 1.58(1.33-1.87), 2.56(1.53-4.29), and 1.05(0.89-1.25). Similar trends were observed across different smoking status. However, LC risk with CXRGP overweighed that with CXRSP among ever smokers [2.95(2.25-3.88) vs. 2.59(1.33-5.02)] and current smokers [2.33(1.70-3.18) vs. 2.26(1.06-4.83)]. Moreover, compared to CXRSN among never smokers, even no progression in CXR, the HRs(95%CIs) of LC incidence were 7.39(5.60-9.75) and 31.45(23.58-41.95) for ever and current smokers, while risks of LC mortality were 6.30(5.07-7.81) and 27.17(21.65-34.11). If participants gained positive CXR, LC incidence risk significantly climbed to 22.04(15.37-31.60) and 71.97(48.82-106.09) for ever and current smokers, while LC mortality risk climbed to 11.90(8.58-16.50) and 38.92(27.04-56.02). CXRLP was associated with decreased LC risk. However, even smokers lost their positive CXR, and the increased risks of LC incidence and mortality did not decrease to non-significant level. Additionally, smoking was significantly associated with increased risk of CXRGP but not CXRLP.

Conclusion

LC risk differed across CXR trajectories and would be modified by smoking status. Comprehensive intervention incorporating CXR trajectories and smoking status should be recommended to reduce LC risk.

Estimating the Volume of Nodules and Masses on Serial Chest Radiography Using a Deep-Learning-Based Automatic Detection Algorithm: A Preliminary Study

BACKGROUND

The purpose of this study was to assess the volume of the pulmonary nodules and masses on serial chest X-rays (CXRs) from deep-learning-based automatic detection algorithm (DLAD)-based parameters.

METHODS

In a retrospective single-institutional study, 72 patients, who obtained serial CXRs (n = 147) for pulmonary nodules or masses with corresponding chest CT images as the reference standards, were included. A pre-trained DLAD based on a convolutional neural network was developed to detect and localize nodules using 13,710 radiographs and to calculate a localization map and the derived parameters (e.g., the area and mean probability value of pulmonary nodules) for each CXR, including serial follow-ups. For validation, reference 3D CT volumes were measured semi-automatically. Volume prediction models for pulmonary nodules were established through univariable or multivariable, and linear or non-linear regression analyses with the parameters. A polynomial regression analysis was performed as a method of a non-linear regression model.

RESULTS

Of the 147 CXRs and 208 nodules of 72 patients, the mean volume of nodules or masses was measured as 9.37 ± 11.69 cm³ (mean ± standard deviation). The area and CT volume demonstrated a linear correlation of moderate strength (i.e., R = 0.58, RMSE: 9449.9 mm³ m³ in a linear regression analysis). The area and mean probability values exhibited a strong linear correlation (R = 0.73). The volume prediction performance based on a multivariable regression model was best with a mean probability and unit-adjusted area (i.e.

, RMSE

7975.6 mm³, the smallest among the other variable parameters).

CONCLUSIONS

The prediction model with the area and the mean probability based on the DLAD showed a rather accurate quantitative estimation of pulmonary nodule or mass volume and the change in serial CXRs.

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system

Background

For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective

The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods

The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results

The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion

The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

A review on lung disease recognition by acoustic signal analysis with deep learning networks

Recently, assistive explanations for difficulties in the health check area have been made viable thanks in considerable portion to technologies like deep learning and machine learning. Using auditory analysis and medical imaging, they also increase the predictive accuracy for prompt and early disease detection. Medical professionals are thankful for such technological support since it helps them manage further patients because of the shortage of skilled human resources. In addition to serious illnesses like lung cancer and respiratory diseases, the plurality of breathing difficulties is gradually rising and endangering society. Because early prediction and immediate treatment are crucial for respiratory disorders, chest X-rays and respiratory sound audio are proving to be quite helpful together. Compared to related review studies on lung disease classification/detection using deep learning algorithms, only two review studies based on signal analysis for lung disease diagnosis have been conducted in 2011 and 2018. This work provides a review of lung disease recognition with acoustic signal analysis with deep learning networks. We anticipate that physicians and researchers working with sound-signal-based machine learning will find this material beneficial.

Acoustic-Based Deep Learning Architectures for Lung Disease Diagnosis: A Comprehensive Overview

Lung auscultation has long been used as a valuable medical tool to assess respiratory health and has gotten a lot of attention in recent years, notably following the coronavirus epidemic. Lung auscultation is used to assess a patient's respiratory role. Modern technological progress has guided the growth of computer-based respiratory speech investigation, a valuable tool for detecting lung abnormalities and diseases. Several recent studies have reviewed this important area, but none are specific to lung sound-based analysis with deep-learning architectures from one side and the provided information was not sufficient for a good understanding of these techniques. This paper gives a complete review of prior deep-learning-based architecture lung sound analysis. Deep-learning-based respiratory sound analysis articles are found in different databases including the Plos, ACM Digital Libraries, Elsevier, PubMed, MDPI, Springer, and IEEE. More than 160 publications were extracted and submitted for assessment. This paper discusses different trends in pathology/lung sound, the common features for classifying lung sounds, several considered datasets, classification methods, signal processing techniques, and some statistical information based on previous study findings. Finally, the assessment concludes with a discussion of potential future improvements and recommendations.

Prediction of model generalizability for unseen data: Methodology and case study in brain metastases detection in T1-Weighted contrast-enhanced 3D MRI

BACKGROUND AND PURPOSE

A medical AI system's generalizability describes the continuity of its performance acquired from varying geographic, historical, and methodologic settings. Previous literature on this topic has mostly focused on "how" to achieve high generalizability (e.g., via larger datasets, transfer learning, data augmentation, model regularization schemes), with limited success. Instead, we aim to understand "when" the generalizability is achieved: Our study presents a medical AI system that could estimate its generalizability status for unseen data on-the-fly.

MATERIALS AND METHODS

We introduce a latent space mapping (LSM) approach utilizing Fréchet distance loss to force the underlying training data distribution into a multivariate normal distribution. During the deployment, a given test data's LSM distribution is processed to detect its deviation from the forced distribution; hence, the AI system could predict its generalizability status for any previously unseen data set. If low model generalizability is detected, then the user is informed by a warning message integrated into a sample deployment workflow. While the approach is applicable for most classification deep neural networks (DNNs), we demonstrate its application to a brain metastases (BM) detector for T1-weighted contrast-enhanced (T1c) 3D MRI. The BM detection model was trained using 175 T1c studies acquired internally (from the authors' institution) and tested using (1) 42 internally acquired exams and (2) 72 externally acquired exams from the publicly distributed Brain Mets dataset provided by the Stanford University School of Medicine. Generalizability scores, false positive (FP) rates, and sensitivities of the BM detector were computed for the test datasets.

RESULTS AND CONCLUSION

The model predicted its generalizability to be low for 31% of the testing data (i.e., two of the internally and 33 of the externally acquired exams), where it produced (1) ∼13.5 false positives (FPs) at 76.1% BM detection sensitivity for the low and (2) ∼10.5 FPs at 89.2% BM detection sensitivity for the high generalizability groups respectively. These results suggest that the proposed formulation enables a model to predict its generalizability for unseen data.

Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact

Background Automated interpretation of normal chest radiographs could alleviate the workload of radiologists. However, the performance of such an artificial intelligence (AI) tool compared with clinical radiology reports has not been established. Purpose To perform an external evaluation of a commercially available AI tool for (a) the number of chest radiographs autonomously reported, (b) the sensitivity for AI detection of abnormal chest radiographs, and (c) the performance of AI compared with that of the clinical radiology reports. Materials and Methods In this retrospective study, consecutive posteroanterior chest radiographs from adult patients in four hospitals in the capital region of Denmark were obtained in January 2020, including images from emergency department patients, in-hospital patients, and outpatients. Three thoracic radiologists labeled chest radiographs in a reference standard based on chest radiograph findings into the following categories: critical, other remarkable, unremarkable, or normal (no abnormalities). AI classified chest radiographs as high confidence normal (normal) or not high confidence normal (abnormal). Results A total of 1529 patients were included for analysis (median age, 69 years [IQR, 55-69 years]; 776 women), with 1100 (72%) classified by the reference standard as having abnormal radiographs, 617 (40%) as having critical abnormal radiographs, and 429 (28%) as having normal radiographs. For comparison, clinical radiology reports were classified based on the text and insufficient reports excluded (n = 22). The sensitivity of AI was 99.1% (95% CI: 98.3, 99.6; 1090 of 1100 patients) for abnormal radiographs and 99.8% (95% CI: 99.1, 99.9; 616 of 617 patients) for critical radiographs. Corresponding sensitivities for radiologist reports were 72.3% (95% CI: 69.5, 74.9; 779 of 1078 patients) and 93.5% (95% CI: 91.2, 95.3; 558 of 597 patients), respectively. Specificity of AI, and hence the potential autonomous reporting rate, was 28.0% of all normal posteroanterior chest radiographs (95% CI: 23.8, 32.5; 120 of 429 patients), or 7.8% (120 of 1529 patients) of all posteroanterior chest radiographs. Conclusion Of all normal posteroanterior chest radiographs, 28% were autonomously reported by AI with a sensitivity for any abnormalities higher than 99%. This corresponded to 7.8% of the entire posteroanterior chest radiograph production. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Park in this issue.

Intracranial steno-occlusive lesion detection on time-of-flight MR angiography using multi-task learning

Steno-occlusive lesions in intracranial arteries refer to segments of narrowed or occluded blood vessels that increase the risk of ischemic strokes. Steno-occlusive lesion detection is crucial in clinical settings; however, automatic detection methods have hardly been studied. Therefore, we propose a novel automatic method to detect steno-occlusive lesions in sequential transverse slices on time-of-flight magnetic resonance angiography. Our method simultaneously detects lesions while segmenting blood vessels based on end-to-end multi-task learning, reflecting that the lesions are closely related to the connectivity of blood vessels. We design classification and localization modules that can be attached to arbitrary segmentation network. As blood vessels are segmented, both modules simultaneously predict the presence and location of lesions for each transverse slice. By combining outputs from the two modules, we devise a simple operation that boosts the performance of lesion localization. Experimental results show that lesion prediction and localization performance is improved by incorporating blood vessel extraction. Our ablation study demonstrates that the proposed operation enhances lesion localization accuracy. We also verify the effectiveness of multi-task learning by comparing our approach with those that individually detect lesions with extracted blood vessels.

Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review

Limitations of the chest X-ray (CXR) have resulted in attempts to create machine learning systems to assist clinicians and improve interpretation accuracy. An understanding of the capabilities and limitations of modern machine learning systems is necessary for clinicians as these tools begin to permeate practice. This systematic review aimed to provide an overview of machine learning applications designed to facilitate CXR interpretation. A systematic search strategy was executed to identify research into machine learning algorithms capable of detecting >2 radiographic findings on CXRs published between January 2020 and September 2022. Model details and study characteristics, including risk of bias and quality, were summarized. Initially, 2248 articles were retrieved, with 46 included in the final review. Published models demonstrated strong standalone performance and were typically as accurate, or more accurate, than radiologists or non-radiologist clinicians. Multiple studies demonstrated an improvement in the clinical finding classification performance of clinicians when models acted as a diagnostic assistance device. Device performance was compared with that of clinicians in 30% of studies, while effects on clinical perception and diagnosis were evaluated in 19%. Only one study was prospectively run. On average, 128,662 images were used to train and validate models. Most classified less than eight clinical findings, while the three most comprehensive models classified 54, 72, and 124 findings. This review suggests that machine learning devices designed to facilitate CXR interpretation perform strongly, improve the detection performance of clinicians, and improve the efficiency of radiology workflow. Several limitations were identified, and clinician involvement and expertise will be key to driving the safe implementation of quality CXR machine learning systems.

AI Improves Nodule Detection on Chest Radiographs in a Health Screening Population: A Randomized Controlled Trial

Background The impact of artificial intelligence (AI)-based computer-aided detection (CAD) software has not been prospectively explored in real-world populations. Purpose To investigate whether commercial AI-based CAD software could improve the detection rate of actionable lung nodules on chest radiographs in participants undergoing health checkups. Materials and Methods In this single-center, pragmatic, open-label randomized controlled trial, participants who underwent chest radiography between July 2020 and December 2021 in a health screening center were enrolled and randomized into intervention (AI group) and control (non-AI group) arms. One of three designated radiologists with 13-36 years of experience interpreted each radiograph, referring to the AI-based CAD results for the AI group. The primary outcome was the detection rate, that is, the number of true-positive radiographs divided by the total number of radiographs, of actionable lung nodules confirmed on CT scans obtained within 3 months. Actionable nodules were defined as solid nodules larger than 8 mm or subsolid nodules with a solid portion larger than 6 mm (Lung Imaging Reporting and Data System, or Lung-RADS, category 4). Secondary outcomes included the positive-report rate, sensitivity, false-referral rate, and malignant lung nodule detection rate. Clinical outcomes were compared between the two groups using univariable logistic regression analyses. Results A total of 10 476 participants (median age, 59 years [IQR, 50-66 years]; 5121 men) were randomized to an AI group (n = 5238) or non-AI group (n = 5238). The trial met the predefined primary outcome, demonstrating an improved detection rate of actionable nodules in the AI group compared with the non-AI group (0.59% [31 of 5238 participants] vs 0.25% [13 of 5238 participants], respectively; odds ratio, 2.4; 95% CI: 1.3, 4.7; P = .008). The detection rate for malignant lung nodules was higher in the AI group compared with the non-AI group (0.15% [eight of 5238 participants] vs 0.0% [0 of 5238 participants], respectively; P = .008). The AI and non-AI groups showed similar false-referral rates (45.9% [56 of 122 participants] vs 56.0% [56 of 100 participants], respectively; P = .14) and positive-report rates (2.3% [122 of 5238 participants] vs 1.9% [100 of 5238 participants]; P = .14). Conclusion In health checkup participants, artificial intelligence-based software improved the detection of actionable lung nodules on chest radiographs. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Auffermann in this isssue.

Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists

IMPORTANCE

Artificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value.

OBJECTIVE

To prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists.

DESIGN, SETTING, AND PARTICIPANTS

A multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021.

EXPOSURES

A bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved.

MAIN OUTCOMES AND MEASURES

Time to write reports based on different caption generation models.

RESULTS

The training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents' mean (SD) reporting time using the NLP-generated model was 283 (37) seconds-significantly shorter than the normal template (347 [58] seconds; P < .001) and the rule-based model (296 [46] seconds; P < .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)-significantly higher than the normal template (0.37 [0.09]; P < .001) and the rule-based model (0.57 [0.19]; P < .001).

CONCLUSIONS AND RELEVANCE

In this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.

Artificial intelligence: A critical review of applications for lung nodule and lung cancer

Artificial intelligence (AI) is a broad concept that usually refers to computer programs that can learn from data and perform certain specific tasks. In the recent years, the growth of deep learning, a successful technique for computer vision tasks that does not require explicit programming, coupled with the availability of large imaging databases fostered the development of multiple applications in the medical imaging field, especially for lung nodules and lung cancer, mostly through convolutional neural networks (CNN). Some of the first applications of AI is this field were dedicated to automated detection of lung nodules on X-ray and computed tomography (CT) examinations, with performances now reaching or exceeding those of radiologists. For lung nodule segmentation, CNN-based algorithms applied to CT images show excellent spatial overlap index with manual segmentation, even for irregular and ground glass nodules. A third application of AI is the classification of lung nodules between malignant and benign, which could limit the number of follow-up CT examinations for less suspicious lesions. Several algorithms have demonstrated excellent capabilities for the prediction of the malignancy risk when a nodule is discovered. These different applications of AI for lung nodules are particularly appealing in the context of lung cancer screening. In the field of lung cancer, AI tools applied to lung imaging have been investigated for distinct aims. First, they could play a role for the non-invasive characterization of tumors, especially for histological subtype and somatic mutation predictions, with a potential therapeutic impact. Additionally, they could help predict the patient prognosis, in combination to clinical data. Despite these encouraging perspectives, clinical implementation of AI tools is only beginning because of the lack of generalizability of published studies, of an inner obscure working and because of limited data about the impact of such tools on the radiologists' decision and on the patient outcome. Radiologists must be active participants in the process of evaluating AI tools, as such tools could support their daily work and offer them more time for high added value tasks.

IEViT: An enhanced vision transformer architecture for chest X-ray image classification

BACKGROUND AND OBJECTIVE

Chest X-ray imaging is a relatively cheap and accessible diagnostic tool that can assist in the diagnosis of various conditions, including pneumonia, tuberculosis, COVID-19, and others. However, the requirement for expert radiologists to view and interpret chest X-ray images can be a bottleneck, especially in remote and deprived areas. Recent advances in machine learning have made possible the automated diagnosis of chest X-ray scans. In this work, we examine the use of a novel Transformer-based deep learning model for the task of chest X-ray image classification.

METHODS

We first examine the performance of the Vision Transformer (ViT) state-of-the-art image classification machine learning model for the task of chest X-ray image classification, and then propose and evaluate the Input Enhanced Vision Transformer (IEViT), a novel enhanced Vision Transformer model that can achieve improved performance on chest X-ray images associated with various pathologies.

RESULTS

Experiments on four chest X-ray image data sets containing various pathologies (tuberculosis, pneumonia, COVID-19) demonstrated that the proposed IEViT model outperformed ViT for all the data sets and variants examined, achieving an F1-score between 96.39% and 100%, and an improvement over ViT of up to +5.82% in terms of F1-score across the four examined data sets. IEViT's maximum sensitivity (recall) ranged between 93.50% and 100% across the four data sets, with an improvement over ViT of up to +3%, whereas IEViT's maximum precision ranged between 97.96% and 100% across the four data sets, with an improvement over ViT of up to +6.41%.

CONCLUSIONS

Results showed that the proposed IEViT model outperformed all ViT's variants for all the examined chest X-ray image data sets, demonstrating its superiority and generalisation ability. Given the relatively low cost and the widespread accessibility of chest X-ray imaging, the use of the proposed IEViT model can potentially offer a powerful, but relatively cheap and accessible method for assisting diagnosis using chest X-ray images.

Artificial Intelligence-Based Identification of Normal Chest Radiographs: A Simulation Study in a Multicenter Health Screening Cohort

Objective

This study aimed to investigate the feasibility of using artificial intelligence (AI) to identify normal chest radiography (CXR) from the worklist of radiologists in a health-screening environment.

Materials and Methods

This retrospective simulation study was conducted using the CXRs of 5887 adults (mean age ± standard deviation, 55.4 ± 11.8 years; male, 4329) from three health screening centers in South Korea using a commercial AI (Lunit INSIGHT CXR3, version 3.5.8.8). Three board-certified thoracic radiologists reviewed CXR images for referable thoracic abnormalities and grouped the images into those with visible referable abnormalities (identified as abnormal by at least one reader) and those with clearly visible referable abnormalities (identified as abnormal by at least two readers). With AI-based simulated exclusion of normal CXR images, the percentages of normal images sorted and abnormal images erroneously removed were analyzed. Additionally, in a random subsample of 480 patients, the ability to identify visible referable abnormalities was compared among AI-unassisted reading (i.e., all images read by human readers without AI), AI-assisted reading (i.e., all images read by human readers with AI assistance as concurrent readers), and reading with AI triage (i.e., human reading of only those rendered abnormal by AI).

Results

Of 5887 CXR images, 405 (6.9%) and 227 (3.9%) contained visible and clearly visible abnormalities, respectively. With AI-based triage, 42.9% (2354/5482) of normal CXR images were removed at the cost of erroneous removal of 3.5% (14/405) and 1.8% (4/227) of CXR images with visible and clearly visible abnormalities, respectively. In the diagnostic performance study, AI triage removed 41.6% (188/452) of normal images from the worklist without missing visible abnormalities and increased the specificity for some readers without decreasing sensitivity.

Conclusion

This study suggests the feasibility of sorting and removing normal CXRs using AI with a tailored cut-off to increase efficiency and reduce the workload of radiologists.

Artificial intelligence system for identification of false-negative interpretations in chest radiographs

OBJECTIVES

To investigate the efficacy of an artificial intelligence (AI) system for the identification of false negatives in chest radiographs that were interpreted as normal by radiologists.

METHODS

We consecutively collected chest radiographs that were read as normal during 1 month (March 2020) in a single institution. A commercialized AI system was retrospectively applied to these radiographs. Radiographs with abnormal AI results were then re-interpreted by the radiologist who initially read the radiograph ("AI as the advisor" scenario). The reference standards for the true presence of relevant abnormalities in radiographs were defined by majority voting of three thoracic radiologists. The efficacy of the AI system was evaluated by detection yield (proportion of true-positive identification among the entire examination) and false-referral rate (FRR, proportion of false-positive identification among all examinations). Decision curve analyses were performed to evaluate the net benefits of applying the AI system.

RESULTS

A total of 4208 radiographs from 3778 patients (M:F = 1542:2236; median age, 56 years) were included. The AI system identified initially overlooked relevant abnormalities with a detection yield and an FRR of 2.4% and 14.0%, respectively. In the "AI as the advisor" scenario, radiologists detected initially overlooked relevant abnormalities with a detection yield and FRR of 1.2% and 0.97%, respectively. In a decision curve analysis, AI as an advisor scenario exhibited a positive net benefit when the cost-to-benefit ratio was below 1:0.8.

CONCLUSION

An AI system could identify relevant abnormalities overlooked by radiologists and could enable radiologists to correct their false-negative interpretations by providing feedback to radiologists.

KEY POINTS

• In consecutive chest radiographs with normal interpretations, an artificial intelligence system could identify relevant abnormalities that were initially overlooked by radiologists. • The artificial intelligence system could enable radiologists to correct their initial false-negative interpretations by providing feedback to radiologists when overlooked abnormalities were present.

Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study

OBJECTIVES

Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists.

DESIGN

This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting.

SETTING

The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020.

PARTICIPANTS

Eleven consultant diagnostic radiologists of varying levels of experience participated in this study.

PRIMARY AND SECONDARY OUTCOME MEASURES

Proportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed.

RESULTS

Of 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy.

CONCLUSIONS

Use of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.

Toward Generalizability in the Deployment of Artificial Intelligence in Radiology: Role of Computation Stress Testing to Overcome Underspecification

The clinical deployment of artificial intelligence (AI) applications in medical imaging is perhaps the greatest challenge facing radiology in the next decade. One of the main obstacles to the incorporation of automated AI-based decision-making tools in medicine is the failure of models to generalize when deployed across institutions with heterogeneous populations and imaging protocols. The most well-understood pitfall in developing these AI models is overfitting, which has, in part, been overcome by optimizing training protocols. However, overfitting is not the only obstacle to the success and generalizability of AI. Underspecification is also a serious impediment that requires conceptual understanding and correction. It is well known that a single AI pipeline, with prescribed training and testing sets, can produce several models with various levels of generalizability. Underspecification defines the inability of the pipeline to identify whether these models have embedded the structure of the underlying system by using a test set independent of, but distributed identically, to the training set. An underspecified pipeline is unable to assess the degree to which the models will be generalizable. Stress testing is a known tool in AI that can limit underspecification and, importantly, assure broad generalizability of AI models. However, the application of stress tests is new in radiologic applications. This report describes the concept of underspecification from a radiologist perspective, discusses stress testing as a specific strategy to overcome underspecification, and explains how stress tests could be designed in radiology-by modifying medical images or stratifying testing datasets. In the upcoming years, stress tests should become in radiology the standard that crash tests have become in the automotive industry. Keywords: Computer Applications-General, Informatics, Computer-aided Diagnosis © RSNA, 2021.

An Artificial Intelligence-Based Chest X-ray Model on Human Nodule Detection Accuracy From a Multicenter Study

IMPORTANCE

Most early lung cancers present as pulmonary nodules on imaging, but these can be easily missed on chest radiographs.

OBJECTIVE

To assess if a novel artificial intelligence (AI) algorithm can help detect pulmonary nodules on radiographs at different levels of detection difficulty.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study included 100 posteroanterior chest radiograph images taken between 2000 and 2010 of adult patients from an ambulatory health care center in Germany and a lung image database in the US. Included images were selected to represent nodules with different levels of detection difficulties (from easy to difficult), and comprised both normal and nonnormal control.

EXPOSURES

All images were processed with a novel AI algorithm, the AI Rad Companion Chest X-ray. Two thoracic radiologists established the ground truth and 9 test radiologists from Germany and the US independently reviewed all images in 2 sessions (unaided and AI-aided mode) with at least a 1-month washout period.

MAIN OUTCOMES AND MEASURES

Each test radiologist recorded the presence of 5 findings (pulmonary nodules, atelectasis, consolidation, pneumothorax, and pleural effusion) and their level of confidence for detecting the individual finding on a scale of 1 to 10 (1 representing lowest confidence; 10, highest confidence). The analyzed metrics for nodules included sensitivity, specificity, accuracy, and receiver operating characteristics curve area under the curve (AUC).

RESULTS

Images from 100 patients were included, with a mean (SD) age of 55 (20) years and including 64 men and 36 women. Mean detection accuracy across the 9 radiologists improved by 6.4% (95% CI, 2.3% to 10.6%) with AI-aided interpretation compared with unaided interpretation. Partial AUCs within the effective interval range of 0 to 0.2 false positive rate improved by 5.6% (95% CI, -1.4% to 12.0%) with AI-aided interpretation. Junior radiologists saw greater improvement in sensitivity for nodule detection with AI-aided interpretation as compared with their senior counterparts (12%; 95% CI, 4% to 19% vs 9%; 95% CI, 1% to 17%) while senior radiologists experienced similar improvement in specificity (4%; 95% CI, -2% to 9%) as compared with junior radiologists (4%; 95% CI, -3% to 5%).

CONCLUSIONS AND RELEVANCE

In this diagnostic study, an AI algorithm was associated with improved detection of pulmonary nodules on chest radiographs compared with unaided interpretation for different levels of detection difficulty and for readers with different experience.

A Highly Reliable Convolutional Neural Network Based Soft Tissue Sarcoma Metastasis Detection from Chest X-ray Images: A Retrospective Cohort Study

Simple Summary

Soft tissue sarcomas are relatively rare malignant diseases. Part of the diagnosis and follow-up includes medical imaging of the thorax for detection of lung metastases. A Python script was created and trained using a set of lung X-rays and concordant CT scans from a high-volume German-speaking sarcoma center. It is capable of detecting malignant metastasis in the lung with a precision of 71.2%, specificity of 90.5%, sensitivity of 94% and accuracy of 91.2%. Furthermore, the program was able to detect even small nodules with a size <1 cm in conventional X-rays of the thorax. This algorithm was implemented into our daily clinical practice alongside with the radiologists’ findings. With this tool we aim to improve the quality of our service and reduce the expenditure of time.

Abstract

Introduction: soft tissue sarcomas are a subset of malignant tumors that are relatively rare and make up 1% of all malignant tumors in adulthood. Due to the rarity of these tumors, there are significant differences in quality in the diagnosis and treatment of these tumors. One paramount aspect is the diagnosis of hematogenous metastases in the lungs. Guidelines recommend routine lung imaging by means of X-rays. With the ever advancing AI-based diagnostic support, there has so far been no implementation for sarcomas. The aim of the study was to utilize AI to obtain analyzes regarding metastasis on lung X-rays in the most possible sensitive and specific manner in sarcoma patients. Methods: a Python script was created and trained using a set of lung X-rays with sarcoma metastases from a high-volume German-speaking sarcoma center. 26 patients with lung metastasis were included. For all patients chest X-ray with corresponding lung CT scans, and histological biopsies were available. The number of trainable images were expanded to 600. In order to evaluate the biological sensitivity and specificity, the script was tested on lung X-rays with a lung CT as control. Results: in this study we present a new type of convolutional neural network-based system with a precision of 71.2%, specificity of 90.5%, sensitivity of 94%, recall of 94% and accuracy of 91.2%. A good detection of even small findings was determined. Discussion: the created script establishes the option to check lung X-rays for metastases at a safe level, especially given this rare tumor entity.

Chest radiographs and machine learning - Past, present and future

Despite its simple acquisition technique, the chest X-ray remains the most common first-line imaging tool for chest assessment globally. Recent evidence for image analysis using modern machine learning points to possible improvements in both the efficiency and the accuracy of chest X-ray interpretation. While promising, these machine learning algorithms have not provided comprehensive assessment of findings in an image and do not account for clinical history or other relevant clinical information. However, the rapid evolution in technology and evidence base for its use suggests that the next generation of comprehensive, well-tested machine learning algorithms will be a revolution akin to early advances in X-ray technology. Current use cases, strengths, limitations and applications of chest X-ray machine learning systems are discussed.

Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

BACKGROUND

Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model.

METHODS

In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior.

FINDINGS

Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645-0·785) across the 127 clinical findings, compared with 0·808 (0·763-0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645-0·785) across all findings, compared with 0·957 (0·954-0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings.

INTERPRETATION

This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice.

FUNDING

Annalise.ai.

Deep learning applied to automatic disease detection using chest X-rays

Deep learning (DL) has shown rapid advancement and considerable promise when applied to the automatic detection of diseases using CXRs. This is important given the widespread use of CXRs across the world in diagnosing significant pathologies, and the lack of trained radiologists to report them. This review article introduces the basic concepts of DL as applied to CXR image analysis including basic deep neural network (DNN) structure, the use of transfer learning and the application of data augmentation. It then reviews the current literature on how DNN models have been applied to the detection of common CXR abnormalities (e.g. lung nodules, pneumonia, tuberculosis and pneumothorax) over the last few years. This includes DL approaches employed for the classification of multiple different diseases (multi-class classification). Performance of different techniques and models and their comparison with human observers are presented. Some of the challenges facing DNN models, including their future implementation and relationships to radiologists, are also discussed.

AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset

OBJECTIVE

Assess if deep learning-based artificial intelligence (AI) algorithm improves reader performance for lung cancer detection on chest X-rays (CXRs).

METHODS

This reader study included 173 images from cancer-positive patients (n = 98) and 346 images from cancer-negative patients (n = 196) selected from National Lung Screening Trial (NLST). Eight readers, including three radiology residents, and five board-certified radiologists, participated in the observer performance test. AI algorithm provided image-level probability of pulmonary nodule or mass on CXRs and a heatmap of detected lesions. Reader performance was compared with AUC, sensitivity, specificity, false-positives per image (FPPI), and rates of chest CT recommendations.

RESULTS

With AI, the average sensitivity of readers for the detection of visible lung cancer increased for residents, but was similar for radiologists compared to that without AI (0.61 [95% CI, 0.55-0.67] vs. 0.72 [95% CI, 0.66-0.77], p = 0.016 for residents, and 0.76 [95% CI, 0.72-0.81] vs. 0.76 [95% CI, 0.72-0.81, p = 1.00 for radiologists), while false-positive findings per image (FPPI) was similar for residents, but decreased for radiologists (0.15 [95% CI, 0.11-0.18] vs. 0.12 [95% CI, 0.09-0.16], p = 0.13 for residents, and 0.24 [95% CI, 0.20-0.29] vs. 0.17 [95% CI, 0.13-0.20], p < 0.001 for radiologists). With AI, the average rate of chest CT recommendation in patients positive for visible cancer increased for residents, but was similar for radiologists (54.7% [95% CI, 48.2-61.2%] vs. 70.2% [95% CI, 64.2-76.2%], p < 0.001 for residents and 72.5% [95% CI, 68.0-77.1%] vs. 73.9% [95% CI, 69.4-78.3%], p = 0.68 for radiologists), while that in cancer-negative patients was similar for residents, but decreased for radiologists (11.2% [95% CI, 9.6-13.1%] vs. 9.8% [95% CI, 8.0-11.6%], p = 0.32 for residents and 16.4% [95% CI, 14.7-18.2%] vs. 11.7% [95% CI, 10.2-13.3%], p < 0.001 for radiologists).

CONCLUSIONS

AI algorithm can enhance the performance of readers for the detection of lung cancers on chest radiographs when used as second reader.

KEY POINTS

• Reader study in the NLST dataset shows that AI algorithm had sensitivity benefit for residents and specificity benefit for radiologists for the detection of visible lung cancer. • With AI, radiology residents were able to recommend more chest CT examinations (54.7% vs 70.2%, p < 0.001) for patients with visible lung cancer. • With AI, radiologists recommended significantly less proportion of unnecessary chest CT examinations (16.4% vs. 11.7%, p < 0.001) in cancer-negative patients.

Application of N Descriptors Proposed by the International Association for the Study of Lung Cancer in Clinical Staging

Background Patients with N1 or N2 non-small cell lung cancer exhibit prognostic heterogeneity. To refine the current N staging system, new N stages were proposed by the International Association for the Study of Lung Cancer. However, those proposed new N stages have not been validated. Purpose To evaluate the prognostic performance of the proposed N descriptors for clinical staging. Materials and Methods Participants with non-small cell lung cancer without distant metastasis from January 2010 to December 2014 were retrospectively included. Each patient's clinical N (cN) stage was assigned to one of seven categories (cN0, cN1a, cN1b, cN2a1, cN2a2, cN2b, cN3). The 5-year overall survival rates were estimated with the Kaplan-Meier method. The adjusted hazard ratios (HRs) and their 95% CIs were estimated by using a multivariable Cox proportional hazard model. Ad hoc analyses according to lymph node (LN) size were performed. Results A total of 1271 patients (median age, 66 years; interquartile range, 59-73 years; 812 men) were included. The 5-year overall survival rates were 77.3%, 53.7%, 36.0%, 29.2%, 34.4%, 18.0%, and 12.4% for stages cN0, cN1a, cN1b, cN2a1, cN2a2, cN2b, and cN3, respectively. Patients with cN2b disease had a worse prognosis than patients with cN2a disease (HR, 1.53; 95% CI: 1.06, 2.22; P = .02). There was no prognostic difference between cN1b and cN1a (HR, 1.13; 95% CI: 0.61, 2.09; P = .71); however, there was a difference between cN1 subgroups when stratified by LN size (≥2 cm; HR, 2.26; 95% CI: 1.16, 4.44; P = .02). Within cN2a disease, there were no differences between cN2a1 and cN2a2 (HR, 0.98; 95% CI: 0.61, 1.56; P = .93) or between subgroups according to LN size (HR, 0.74; 95% CI: 0.40, 1.37; P = .34). Conclusion A survival difference was observed between single- and multistation involvement among cN2 disease. The number of involved lymph node stations in patients with cN1 disease and the presence of skip metastasis in patients with cN2 disease were not associated with survival differences. © RSNA, 2021 Online supplemental material is available for this article.