Validation of a Deep Learning Tool in the Detection of Intracranial Hemorrhage and Large Vessel Occlusion

Purpose: Recently developed machine-learning algorithms have demonstrated strong performance in the detection of intracranial hemorrhage (ICH) and large vessel occlusion (LVO). However, their generalizability is often limited by geographic bias of studies. The aim of this study was to validate a commercially available deep learning-based tool in the detection of both ICH and LVO across multiple hospital sites and vendors throughout the U.S.

Materials and Methods: This was a retrospective and multicenter study using anonymized data from two institutions. Eight hundred fourteen non-contrast CT cases and 378 CT angiography cases were analyzed to evaluate ICH and LVO, respectively. The tool's ability to detect and quantify ICH, LVO, and their various subtypes was assessed among multiple CT vendors and hospitals across the United States. Ground truth was based off imaging interpretations from two board-certified neuroradiologists.

Results: There were 255 positive and 559 negative ICH cases. Accuracy was 95.6%, sensitivity was 91.4%, and specificity was 97.5% for the ICH tool. ICH was further stratified into the following subtypes: intraparenchymal, intraventricular, epidural/subdural, and subarachnoid with true positive rates of 92.9, 100, 94.3, and 89.9%, respectively. ICH true positive rates by volume [small (<5 mL), medium (5–25 mL), and large (>25 mL)] were 71.8, 100, and 100%, respectively. There were 156 positive and 222 negative LVO cases. The LVO tool demonstrated an accuracy of 98.1%, sensitivity of 98.1%, and specificity of 98.2%. A subset of 55 randomly selected cases were also assessed for LVO detection at various sites, including the distal internal carotid artery, middle cerebral artery M1 segment, proximal middle cerebral artery M2 segment, and distal middle cerebral artery M2 segment with an accuracy of 97.0%, sensitivity of 94.3%, and specificity of 97.4%.

Conclusion: Deep learning tools can be effective in the detection of both ICH and LVO across a wide variety of hospital systems. While some limitations were identified, specifically in the detection of small ICH and distal M2 occlusion, this study highlights a deep learning tool that can assist radiologists in the detection of emergent findings in a variety of practice settings.

Deep Learning-Based Algorithm for Automatic Detection of Pulmonary Embolism in Chest CT Angiograms

Purpose: Since the prompt recognition of acute pulmonary embolism (PE) and the immediate initiation of treatment can significantly reduce the risk of death, we developed a deep learning (DL)-based application aimed to automatically detect PEs on chest computed tomography angiograms (CTAs) and alert radiologists for an urgent interpretation. Convolutional neural networks (CNNs) were used to design the application. The associated algorithm used a hybrid 3D/2D UNet topology. The training phase was performed on datasets adequately distributed in terms of vendors, patient age, slice thickness, and kVp. The objective of this study was to validate the performance of the algorithm in detecting suspected PEs on CTAs. Methods: The validation dataset included 387 anonymized real-world chest CTAs from multiple clinical sites (228 U.S. cities). The data were acquired on 41 different scanner models from five different scanner makers. The ground truth (presence or absence of PE on CTA images) was established by three independent U.S. board-certified radiologists. Results: The algorithm correctly identified 170 of 186 exams positive for PE (sensitivity 91.4% [95% CI: 86.4–95.0%]) and 184 of 201 exams negative for PE (specificity 91.5% [95% CI: 86.8–95.0%]), leading to an accuracy of 91.5%. False negative cases were either chronic PEs or PEs at the limit of subsegmental arteries and close to partial volume effect artifacts. Most of the false positive findings were due to contrast agent-related fluid artifacts, pulmonary veins, and lymph nodes. Conclusions: The DL-based algorithm has a high degree of diagnostic accuracy with balanced sensitivity and specificity for the detection of PE on CTAs.

Enhancing Radiologist Efficiency with AI: A Multi-Reader Multi-Case Study on Aortic Dissection Detection and Prioritization

BACKGROUND AND OBJECTIVES

Acute aortic dissection (AD) is a life-threatening condition in which early detection can significantly improve patient outcomes and survival. This study evaluates the clinical benefits of integrating a deep learning (DL)-based application for the automated detection and prioritization of AD on chest CT angiographies (CTAs) with a focus on the reduction in the scan-to-assessment time (STAT) and interpretation time (IT).

MATERIALS AND METHODS

This retrospective Multi-Reader Multi-Case (MRMC) study compared AD detection with and without artificial intelligence (AI) assistance. The ground truth was established by two U.S. board-certified radiologists, while three additional expert radiologists served as readers. Each reader assessed the same CTAs in two phases: assessment unaided by AI assistance (pre-AI arm) and, after a 1-month washout period, assessment aided by device outputs (post-AI arm). STAT and IT metrics were compared between the two arms.

RESULTS

This study included 285 CTAs (95 per reader, per arm) with a mean patient age of 58.5 years ±14.7 (SD), of which 52% were male and 37% had a prevalence of AD. AI assistance significantly reduced the STAT for detecting 33 true positive AD cases from 15.84 min (95% CI: 13.37-18.31 min) without AI to 5.07 min (95% CI: 4.23-5.91 min) with AI, representing a 68% reduction (p < 0.01). The IT also reduced significantly from 21.22 s (95% CI: 19.87-22.58 s) without AI to 14.17 s (95% CI: 13.39-14.95 s) with AI (p < 0.05).

CONCLUSIONS

The integration of a DL-based algorithm for AD detection on chest CTAs significantly reduces both the STAT and IT. By prioritizing urgent cases, the AI-assisted approach outperforms the standard First-In, First-Out (FIFO) workflow.

Performance and clinical utility of an artificial intelligence-enabled tool for pulmonary embolism detection

PURPOSE

Diagnosing pulmonary embolism (PE) is still challenging due to other conditions that can mimic its appearance, leading to incomplete or delayed management and several inter-observer variabilities. This study evaluated the performance and clinical utility of an artificial intelligence (AI)-based application designed to assist clinicians in the detection of PE on CT pulmonary angiography (CTPA).

PATIENTS AND METHODS

CTPAs from 230 US cities acquired on 57 scanner models from 6 different vendors were retrospectively collected. Three US board certified expert radiologists defined the ground truth by majority agreement. The same cases were analyzed by CINA-PE, an AI-driven algorithm capable of detecting and highlighting suspected PE locations. The algorithm's performance at a per-case and per-finding level was evaluated. Furthermore, cases with PE not mentioned in the clinical report but correctly detected by the algorithm were analyzed.

RESULTS

A total of 1204 CTPAs (mean age 62.1 years ± 16.6[SD], 44.4 % female, 14.9 % positive) were included in the study. Per-case sensitivity and specificity were 93.9 % (95%CI: 89.3 %-96.9 %) and 94.8 % (95%CI: 93.3 %-96.1 %), respectively. Per-finding positive predictive value was 89.5 % (95%CI: 86.7 %-91.9 %). Among the 196 positive cases, 29 (15.6 %) were not mentioned in the clinical report. The algorithm detected 22/29 (76 %) of these cases, leading to a reduction in the miss rate from 15.6 % to 3.8 % (7/186).

CONCLUSIONS

The AI-based application may improve diagnostic accuracy in detecting PE and enhance patient outcomes through timely intervention. Integrating AI tools in clinical workflows can reduce missed or delayed diagnoses, and positively impact healthcare delivery and patient care.

Diagnostic Performance of a Deep Learning-Powered Application for Aortic Dissection Triage Prioritization and Classification

This multicenter retrospective study evaluated the diagnostic performance of a deep learning (DL)-based application for detecting, classifying, and highlighting suspected aortic dissections (ADs) on chest and thoraco-abdominal CT angiography (CTA) scans. CTA scans from over 200 U.S. and European cities acquired on 52 scanner models from six manufacturers were retrospectively collected and processed by CINA-CHEST (AD) (Avicenna.AI, La Ciotat, France) device. The diagnostic performance of the device was compared with the ground truth established by the majority agreement of three U.S. board-certified radiologists. Furthermore, the DL algorithm's time to notification was evaluated to demonstrate clinical effectiveness. The study included 1303 CTAs (mean age 58.8 ± 16.4 years old, 46.7% male, 10.5% positive). The device demonstrated a sensitivity of 94.2% [95% CI: 88.8-97.5%] and a specificity of 97.3% [95% CI: 96.2-98.1%]. The application classified positive cases by the AD type with an accuracy of 99.5% [95% CI: 98.9-99.8%] for type A and 97.5 [95% CI: 96.4-98.3%] for type B. The application did not miss any type A cases. The device flagged 32 cases incorrectly, primarily due to acquisition artefacts and aortic pathologies mimicking AD. The mean time to process and notify of potential AD cases was 27.9 ± 8.7 s. This deep learning-based application demonstrated a strong performance in detecting and classifying aortic dissection cases, potentially enabling faster triage of these urgent cases in clinical settings.

Validation of a Deep Learning Tool for Detection of Incidental Vertebral Compression Fractures

OBJECTIVE

This study evaluated the performance of a deep learning-based vertebral compression fracture (VCF) detection tool in patients with incidental VCF. The purpose of this study was to validate this tool across multiple sites and multiple vendors.

METHODS

This was a retrospective, multicenter, multinational blinded study using anonymized chest and abdominal CT scans performed for indications other than VCF in patients ≥50 years old. Images were obtained from 2 teleradiology companies in France and United States and were processed by CINA-VCF v1.0, a deep learning algorithm designed for VCF detection. Ground truth was established by majority consensus across 3 board-certified radiologists. Overall performance of CINA-VCF was evaluated, as well as subset analyses based on imaging acquisition parameters, baseline patient characteristics, and VCF severity. A subgroup was also analyzed and compared with available clinical radiology reports.

RESULTS

Four hundred seventy-four CT scans were included in this study, comprising 166 (35.0%) positive and 308 (65.0%) negative VCF cases. CINA-VCF demonstrated an area under the curve (AUC) of 0.97 (95% CI: 0.96-0.99), accuracy of 93.7% (95% CI: 91.1%-95.7%), sensitivity of 95.2% (95% CI: 90.7%-97.9%), and specificity of 92.9% (95% CI: 89.4%-96.5%). Subset analysis based on VCF severity resulted in a specificity of 94.2% (95% CI: 90.9%-96.6%) for grade 0 negative cases and a specificity of 64.3% (95% CI: 35.1%-87.2%) for grade 1 negative cases. For grades 2 and 3 positive cases, sensitivity was 89.7% (95% CI: 79.9%-95.8%) and 99.0% (95% CI: 94.4%-100.0%), respectively.

CONCLUSIONS

CINA-VCF successfully detected incidental VCF and even outperformed clinical reports. The performance was consistent among all subgroups analyzed. Limitations of the tool included various confounding pathologies such as Schmorl's nodes and borderline cases. Despite these limitations, this study validates the applicability and generalizability of the tool in the clinical setting.

Contribution of an Artificial Intelligence Tool in the Detection of Incidental Pulmonary Embolism on Oncology Assessment Scans

INTRODUCTION

The incidence of venous thromboembolism is estimated to be around 3% of cancer patients. However, a majority of incidental pulmonary embolism (iPE) can be overlooked by radiologists in asymptomatic patients, performing CT scans for disease surveillance, which may significantly impact the patient's health and management. Routine imaging in oncology is usually reviewed with delayed hours after the acquisition of images. Nevertheless, the advent of AI in radiology could reduce the risk of the diagnostic delay of iPE by an optimal triage immediately at the acquisition console. This study aimed to determine the accuracy rate of an AI algorithm (CINA-iPE) in detecting iPE and the duration until the management of cancer patients in our center, in addition to describing the characteristics of patients with a confirmed pulmonary embolism (PE).

MATERIALS AND METHODS

This is a retrospective analysis of the role of Avicenna's CE-certified and FDA-cleared CINA-iPE algorithm in oncology patients treated at Gustave Roussy Cancer Campus. The results obtained from the AI algorithm were compared with the attending radiologist's report and were analyzed by both a radiology resident and a senior radiologist. In case of any discordant results, the reason for this discrepancy was further investigated. The duration between the exact time of the CT scan and analysis was assessed, as well as the duration from the result's report and the start of active management.

RESULTS

Out of 3047 patients, 104 alerts were detected for iPE (prevalence of 1.3%), while 2942 had negative findings. In total, 36 of the 104 patients had confirmed PE, while 68 alerts were false positives. Only one patient reported as negative by the AI tool was deemed to have a PE by the radiologist. The sensitivity and specificity of the AI model were 97.3% and 97.74%, while the PPV and NPV were 34.62% and 99.97%, respectively. Most causes of FP were artifacts (22 cases, 32.3%) and lymph nodes (11 cases, 16.2%). Seven patients experienced delayed diagnosis, requiring them to return to the ER for treatment after being sent home following their scan. The remaining patients received prompt care immediately after their testing, with a mean delay time of 8.13 h.

CONCLUSIONS

The addition of an AI system for the detection of unsuspected PEs on chest CT scans in routine oncology care demonstrated a promising efficacy in comparison to human performance. Despite a low prevalence, the sensitivity and specificity of the AI tool reached 97.3% and 97.7%, respectively, with detection of all the reported clinical PEs, except one single case. This study describes the potential synergy between AI and radiologists for an optimal diagnosis of iPE in routine clinical cancer care.

CLINICAL RELEVANCE STATEMENT

In the oncology field, iPEs are common, with an increased risk of morbidity when missed with a delayed diagnosis. With the assistance of a reliable AI tool, the radiologist can focus on the challenging analysis of oncology results while dealing with urgent diagnosis such as PE by sending the patient straight to the ER (Emergency Room) for prompt treatment.

Deep Learning-Based ASPECTS Algorithm Enhances Reader Performance and Reduces Interpretation Time

BACKGROUND AND PURPOSE

ASPECTS is a long-standing and well-documented selection criterion for acute ischemic stroke treatment; however, the interpretation of ASPECTS is a challenging and time-consuming task for physicians with notable interobserver variabilities. We conducted a multireader, multicase study in which readers assessed ASPECTS without and with the support of a deep learning (DL)-based algorithm to analyze the impact of the software on clinicians' performance and interpretation time.

MATERIALS AND METHODS

A total of 200 NCCT scans from 5 clinical sites (27 scanner models, 4 different vendors) were retrospectively collected. The reference standard was established through the consensus of 3 expert neuroradiologists who had access to baseline CTA and CTP data. Subsequently, 8 additional clinicians (4 typical ASPECTS readers and 4 senior neuroradiologists) analyzed the NCCT scans without and with the assistance of CINA-ASPECTS (Avicenna.AI), a DL-based, FDA-cleared, and CE-marked algorithm designed to compute ASPECTS automatically. Differences were evaluated in both performance and interpretation time between the assisted and unassisted assessments.

RESULTS

With software aid, readers demonstrated increased region-based accuracy from 72.4% to 76.5% (P < .05) and increased receiver operating characteristic area under the curve (ROC AUC) from 0.749 to 0.788 (P < .05). Notably, all readers exhibited an improved ROC AUC when utilizing the software. Moreover, the use of the algorithm improved the score-based interobserver reliability and correlation coefficient of ASPECTS evaluation by 0.222 and 0.087 (P < .0001), respectively. Additionally, the readers' mean time spent analyzing a case was significantly reduced by 6% (P < .05) when aided by the algorithm.

CONCLUSIONS

With the assistance of the algorithm, readers' analyses were not only more accurate but also faster. Additionally, the overall ASPECTS evaluation exhibited greater consistency, fewer variabilities, and higher precision compared with the reference standard. This novel tool has the potential to enhance patient selection for appropriate treatment by enabling physicians to deliver accurate and timely diagnoses of acute ischemic stroke.

Automated vertebral compression fracture detection and quantification on opportunistic CT scans: a performance evaluation

AIM

Since the majority of vertebral compression fractures (VCFs) are asymptomatic, they often go undetected on opportunistic CT scans. To reduce rates of undiagnosed osteoporosis, we developed a deep learning (DL)-based algorithm using 2D/3D U-Nets convolutional neural networks to opportunistically screen for VCF on CT scans. This study aimed to evaluate the performance of the algorithm using external real-world data.

MATERIALS AND METHODS

CT scans acquired for various indications other than a suspicion of VCF from January 2019 to August 2020 were retrospectively and consecutively collected. The algorithm was designed to label each vertebra, detect VCF, measure vertebral height loss (VHL) and calculate mean Hounsfield Units (mean HU) for vertebral bone attenuation. For the ground truth, two board-certified radiologists defined if VCF was present and performed the measurements. The algorithm analyzed the scans and the results were compared to the experts' assessments.

RESULTS

A total of 100 patients (mean age: 76.6 years ± 10.1[SD], 72% women) were evaluated. The overall labeling agreement was 94.9% (95%CI: 93.7%-95.9%). Regarding VHL, the 95% limits of agreement (LoA) between the algorithm and the radiologists was [-9.3, 8.6]; 94.1% of the differences lay within the radiologists' LoA and the intraclass correlation coefficient was 0.854 (95%CI: 0.822-0.881). For the mean HU, Pearson's correlation was 0.89 (95%CI: 0.84-0.92; p-value <0.0001). Finally, the algorithm's VCF screening sensitivity and specificity were 92.3% (95%CI: 81.5%-97.9%) and 91.7% (95%CI: 80.0%-97.7%), respectively.

CONCLUSIONS

This automated tool for screening and quantification of opportunistic VCF demonstrated high reliability and performance that may facilitate radiologists' task and improve opportunistic osteoporosis assessments.