Diagnostic accuracy of automated occlusion detection in CT angiography using e-CTA

Deep learning-assisted diagnosis of large vessel occlusion in acute ischemic stroke based on four-dimensional computed tomography angiography

Purpose

To develop deep learning models based on four-dimensional computed tomography angiography (4D-CTA) images for automatic detection of large vessel occlusion (LVO) in the anterior circulation that cause acute ischemic stroke.

Methods

This retrospective study included 104 LVO patients and 105 non-LVO patients for deep learning models development. Another 30 LVO patients and 31 non-LVO patients formed the time-independent validation set. Four phases of 4D-CTA (arterial phase P1, arterial-venous phase P2, venous phase P3 and late venous phase P4) were arranged and combined and two input methods was used: combined input and superimposed input. Totally 26 models were constructed using a modified HRNet network. Assessment metrics included the areas under the curve (AUC), accuracy, sensitivity, specificity and F1 score. Kappa analysis was performed to assess inter-rater agreement between the best model and radiologists of different seniority.

Results

The P1 + P2 model (combined input) had the best diagnostic performance. In the internal validation set, the AUC was 0.975 (95%CI: 0.878-0.999), accuracy was 0.911, sensitivity was 0.889, specificity was 0.944, and the F1 score was 0.909. In the time-independent validation set, the model demonstrated consistently high performance with an AUC of 0.942 (95%CI: 0.851-0.986), accuracy of 0.902, sensitivity of 0.867, specificity of 0.935, and an F1 score of 0.901. The best model showed strong consistency with the diagnostic efficacy of three radiologists of different seniority (k = 0.84, 0.80, 0.70, respectively).

Conclusion

The deep learning model, using combined arterial and arterial-venous phase, was highly effective in detecting LVO, alerting radiologists to speed up the diagnosis.

Artificial intelligence-based decision support software to improve the efficacy of acute stroke pathway in the NHS: an observational study

Introduction

In a drip-and-ship model for endovascular thrombectomy (EVT), early identification of large vessel occlusion (LVO) and timely referral to a comprehensive center (CSC) are crucial when patients are admitted to an acute stroke center (ASC). Several artificial intelligence (AI) decision-aid tools are increasingly being used to facilitate the rapid identification of LVO. This retrospective cohort study aimed to evaluate the impact of deploying e-Stroke AI decision support software in the hyperacute stroke pathway on process metrics and patient outcomes at an ASC in the United Kingdom.

Methods

Except for the deployment of e-Stroke on 01 March 2020, there were no significant changes made to the stroke pathway at the ASC. The data were obtained from a prospective stroke registry between 01 January 2019 and 31 March 2021. The outcomes were compared between the 14 months before and 12 months after the deployment of AI (pre-e-Stroke cohort vs. post-e-Stroke cohort) on 01 March 2020. Time window analyses were performed using Welch's t-test. Cochran-Mantel-Haenszel test was used to compare changes in disability at 3 months assessed by modified Rankin Score (mRS) ordinal shift analysis, and Fisher's exact test was used for dichotomised mRS analysis.

Results

In the pre-e-Stroke cohort, 19 of 22 patients referred received EVT. In the post-e-Stroke cohort, 21 of the 25 patients referred were treated. The mean door-in-door-out (DIDO) and door-to-referral times in pre-e-Stroke vs. post-e-Stroke cohorts were 141 vs. 79 min (difference 62 min, 95% CI 96.9-26.8 min, p < 0.001) and 71 vs. 44 min (difference 27 min, 95% CI 47.4-5.4 min, p = 0.01), respectively. The adjusted odds ratio (age and NIHSS) for mRS ordinal shift analysis at 3 months was 3.14 (95% CI 0.99-10.51, p = 0.06) and the dichotomized mRS 0-2 at 3 months was 16% vs. 48% (p = 0.04) in the pre- vs. post-e-Stroke cohorts, respectively.

Conclusion

In this single-center study in the United Kingdom, the DIDO time significantly decreased since the introduction of e-Stroke decision support software into an ASC hyperacute stroke pathway. The reduction in door-in to referral time indicates faster image interpretation and referral for EVT. There was an indication of an increased proportion of patients regaining independent function after EVT. However, this should be interpreted with caution given the small sample size. Larger, prospective studies and further systematic real-world evaluation are needed to demonstrate the widespread generalisability of these findings.

Deep-learning based detection of vessel occlusions on CT-angiography in patients with suspected acute ischemic stroke

Swift diagnosis and treatment play a decisive role in the clinical outcome of patients with acute ischemic stroke (AIS), and computer-aided diagnosis (CAD) systems can accelerate the underlying diagnostic processes. Here, we developed an artificial neural network (ANN) which allows automated detection of abnormal vessel findings without any a-priori restrictions and in <2 minutes. Pseudo-prospective external validation was performed in consecutive patients with suspected AIS from 4 different hospitals during a 6-month timeframe and demonstrated high sensitivity (≥87%) and negative predictive value (≥93%). Benchmarking against two CE- and FDA-approved software solutions showed significantly higher performance for our ANN with improvements of 25-45% for sensitivity and 4-11% for NPV (p ≤ 0.003 each). We provide an imaging platform ( https://stroke.neuroAI-HD.org ) for online processing of medical imaging data with the developed ANN, including provisions for data crowdsourcing, which will allow continuous refinements and serve as a blueprint to build robust and generalizable AI algorithms.

Duration and accuracy of automated stroke CT workflow with AI-supported intracranial large vessel occlusion detection

The Automation Platform (AP) is a software platform to support the workflow of radiologists and includes a stroke CT package with integrated artificial intelligence (AI) based tools. The aim of this study was to evaluate the diagnostic performance of the AP for the detection of intracranial large vessel occlusions (LVO) on conventional CT angiography (CTA), and the duration of CT processing in a cohort of acute stroke patients. The diagnostic performance for intracranial LVO detection on CTA by the AP was evaluated in a retrospective cohort of 100 acute stroke patients and compared to the diagnostic performance of five radiologists with different levels of experience. The reference standard was set by an independent neuroradiologist, with access to the readings of the different radiologists, clinical data, and follow-up. The data processing time of the AP for ICH detection on non-contrast CT, LVO detection on CTA, and the processing of CTP maps was assessed in a subset 60 patients of the retrospective cohort. This was compared to 13 radiologists, who were prospectively timed for the processing and reading of 21 stroke CTs. The AP showed shorter processing time of CTA (mean 60 versus 395 s) and CTP (mean 196 versus 243-349 s) as compared to radiologists, but showed lower sensitivity for LVO detection (sensitivity 77% of the AP vs mean sensitivity 87% of radiologists). If the AP would have been used as a stand-alone system, 1 ICA occlusion, 2 M1 occlusions and 8 M2 occlusions would have been missed, which would be eligible for mechanical thrombectomy. In conclusion, the AP showed shorter processing time of CTA and CTP as compared with radiologists, which illustrates the potential of the AP to speed-up the diagnostic work-up. However, its performance for LVO detection was lower as compared with radiologists, especially for M2 vessel occlusions.

Accuracy of artificial intelligence software for CT angiography in stroke

OBJECTIVE

Software developed using artificial intelligence may automatically identify arterial occlusion and provide collateral vessel scoring on CT angiography (CTA) performed acutely for ischemic stroke. We aimed to assess the diagnostic accuracy of e-CTA by Brainomix™ Ltd by large-scale independent testing using expert reading as the reference standard.

METHODS

We identified a large clinically representative sample of baseline CTA from 6 studies that recruited patients with acute stroke symptoms involving any arterial territory. We compared e-CTA results with masked expert interpretation of the same scans for the presence and location of laterality-matched arterial occlusion and/or abnormal collateral score combined into a single measure of arterial abnormality. We tested the diagnostic accuracy of e-CTA for identifying any arterial abnormality (and in a sensitivity analysis compliant with the manufacturer's guidance that software only be used to assess the anterior circulation).

RESULTS

We include CTA from 668 patients (50% female; median: age 71 years, NIHSS 9, 2.3 h from stroke onset). Experts identified arterial occlusion in 365 patients (55%); most (343, 94%) involved the anterior circulation. Software successfully processed 545/668 (82%) CTAs. The sensitivity, specificity and diagnostic accuracy of e-CTA for detecting arterial abnormality were each 72% (95% CI = 66-77%). Diagnostic accuracy was non-significantly improved in a sensitivity analysis excluding occlusions from outside the anterior circulation (76%, 95% CI = 72-80%).

INTERPRETATION

Compared to experts, the diagnostic accuracy of e-CTA for identifying acute arterial abnormality was 72-76%. Users of e-CTA should be competent in CTA interpretation to ensure all potential thrombectomy candidates are identified.

Diagnostic accuracy of a decision-support software for the detection of intracranial large-vessel occlusion in CT angiography

AIM

To investigate the real-world clinical performance of the decision-support software "e-CTA" (e-Stroke Suite, Brainomix Limited, Oxford UK) for the detection of acute intracranial large-vessel occlusion (LVO) on computed tomography (CT) angiography at a UK district general hospital.

MATERIALS AND METHODS

The retrospective study included 300 consecutive CT angiograms of the head and neck performed between 8 March 2021 and 20 May 2021. e-CTA findings were recorded and compared with the radiologist report. Cases in which there was disagreement between e-CTA and the radiologist were reviewed by a sub-specialist vascular radiologist as the reference standard.

RESULTS

The incidence of intracranial LVO was 7%. e-CTA correctly identified 18 of 21 intracranial proximal LVOs (86%). There were 34 false positives. The sensitivity was 0.86 (95% confidence interval [CI], 0.64-0.97), with specificity of 0.88 (95% CI, 0.83-0.91). The positive predictive value was 0.35 (95% CI, 0.27-0.43). The negative predictive value was 0.99 (95% CI, 0.96-1.00).

CONCLUSION

Sensitivity, specificity, and negative predictive values were similar to those reported in the literature (Seker et al., Int J Stroke. 2021; 17:77-82); however, the positive predictive value for e-CTA was significantly lower. In practice, this meant that over half of all reported occlusions by the software were false positives. Radiologists should be aware of these metrics in order to assign appropriate weight to software findings when formulating a report. Differences in population demographics, scanners, CT protocols, and incidence are all factors potentially influencing software accuracy. Local validation testing may help provide accuracy metrics more relevant to individual institutions.

Spatial CT perfusion data helpful in automatically locating vessel occlusions for acute ischemic stroke patients

Introduction

Locating a vessel occlusion is important for clinical decision support in stroke healthcare. The advent of endovascular thrombectomy beyond proximal large vessel occlusions spurs alternative approaches to locate vessel occlusions. We explore whether CT perfusion (CTP) data can help to automatically locate vessel occlusions.

Methods

We composed an atlas with the downstream regions of particular vessel segments. Occlusion of these segments should result in the hypoperfusion of the corresponding downstream region. We differentiated between seven-vessel occlusion locations (ICA, proximal M1, distal M1, M2, M3, ACA, and posterior circulation). We included 596 patients from the DUtch acute STroke (DUST) multicenter study. Each patient CTP data set was processed with perfusion software to determine the hypoperfused region. The downstream region with the highest overlap with the hypoperfused region was considered to indicate the vessel occlusion location. We assessed the indications from CTP against expert annotations from CTA.

Results

Our atlas-based model had a mean accuracy of 86% and could achieve substantial agreement with the annotations from CTA according to Cohen's kappa coefficient (up to 0.68). In particular, anterior large vessel occlusions and occlusions in the posterior circulation could be located with an accuracy of 80 and 92%, respectively.

Conclusion

The spatial layout of the hypoperfused region can help to automatically indicate the vessel occlusion location for acute ischemic stroke patients. However, variations in vessel architecture between patients seemed to limit the capacity of CTP data to distinguish between vessel occlusion locations more accurately.

Machine-learning algorithm in acute stroke: real-world experience

AIM

To assess the clinical performance of a commercially available machine learning (ML) algorithm in acute stroke.

MATERIALS AND METHODS

CT and CT angiography (CTA) studies of 104 consecutive patients (43 females, age range 19-93, median age 62) performed for suspected acute stroke at a single tertiary institution with real-time ML software analysis (RAPID™ ASPECTS and CTA) were included. Studies were retrospectively reviewed independently by two neuroradiologists in a blinded manner.

RESULTS

The cohort included 24 acute infarcts and 16 large vessel occlusions (LVO). RAPID™ ASPECTS interpretation demonstrated high sensitivity (87.5%) and NPV (87.5%) but very poor specificity (30.9%) and PPV (30.9%) for detection of acute ischaemic parenchymal changes. There was a high percentage of false positives (51.1%). In cases of proven LVO, RAPID™ ASPECTS showed good correlation with neuroradiologists' blinded independent interpretation, Pearson correlation coefficient = 0.96 (both readers), 0.63 (RAPID™ vs reader 1), 0.69 (RAPID™ vs reader 2). RAPID™ CTA interpretation demonstrated high sensitivity (92.3%), specificity (85.3%), and negative predictive (NPV) (98.5%) with moderate positive predictive value (PPV) (52.2%) for detection of LVO (N=13). False positives accounted for 12.5% of cases, of which 27.3% were attributed to arterial stenosis.

CONCLUSION

RAPID™ CTA was robust and reliable in detection of LVO. Although demonstrating high sensitivity and NPV, RAPID™ ASPECTS interpretation was associated with a high number of false positives, which decreased clinicians' confidence in the algorithm. However, in cases of proven LVO, RAPID™ ASPECTS performed well and had good correlation with neuroradiologists' blinded interpretation.

Automated CT angiography collateral scoring in anterior large vessel occlusion stroke: A multireader study

BACKGROUND

Computed tomography (CT) angiography collateral score (CTA-CS) is an important clinical outcome predictor following mechanical thrombectomy for ischemic stroke with large vessel occlusion (LVO). The present multireader study aimed to evaluate the performance of e-CTA software for automated assistance in CTA-CS scoring.

MATERIALS AND METHODS

Brain CTA images of 56 patients with anterior LVO were retrospectively processed. Twelve readers of various clinical training, including junior neuroradiologists, senior neuroradiologists, and neurologists graded collateral flow using visual CTA-CS scale in two sessions separated by a washout period. Reference standard was the consensus of three expert readers. Duration of reading time, inter-rater reliability, and statistical comparison of readers' performance metrics were analyzed between the e-CTA assisted and unassisted sessions.

RESULTS

e-CTA assistance resulted in significant increase in mean accuracy (58.6% to 67.5%, p = 0.003), mean F1 score (0.574 to 0.676, p = 0.002), mean precision (58.8% to 68%, p = 0.007), and mean recall (58.7% to 69.9%, p = 0.002), especially with slight filling deficit (CTA-CS 2 and 3). Mean reading time was reduced across all readers (103.4 to 59.7 s, p = 0.001), and inter-rater agreement in CTA-CS assessment was increased (Krippendorff's alpha 0.366 to 0.676). Optimized occlusion laterality detection was also noted with mean accuracy (92.9% to 96.8%, p = 0.009).

CONCLUSION

Automated assistance for CTA-CS using e-CTA software provided helpful decision support for readers in terms of improving scoring accuracy and reading efficiency for physicians with a range of experience and training backgrounds and leading to significant improvements in inter-rater agreement.

Accuracy of Automated Computer-Aided Diagnosis for Stroke Imaging: A Critical Evaluation of Current Evidence

There is increasing interest in computer applications, using artificial intelligence methodologies, to perform health care tasks previously performed by humans, particularly in medical imaging for diagnosis. In stroke, there are now commercial artificial intelligence software for use with computed tomography or MR imaging to identify acute ischemic brain tissue pathology, arterial obstruction on computed tomography angiography or as hyperattenuated arteries on computed tomography, brain hemorrhage, or size of perfusion defects. A rapid, accurate diagnosis may aid treatment decisions for individual patients and could improve outcome if it leads to effective and safe treatment; or conversely, to disaster if a delayed or incorrect diagnosis results in inappropriate treatment. Despite this potential clinical impact, diagnostic tools including artificial intelligence methods are not subjected to the same clinical evaluation standards as are mandatory for drugs. Here, we provide an evidence-based review of the pros and cons of commercially available automated methods for medical imaging diagnosis, including those based on artificial intelligence, to diagnose acute brain pathology on computed tomography or magnetic resonance imaging in patients with stroke.

Validation of a machine learning software tool for automated large vessel occlusion detection in patients with suspected acute stroke

PURPOSE

CT angiography (CTA) is the imaging standard for large vessel occlusion (LVO) detection in patients with acute ischemic stroke. StrokeSENS LVO is an automated tool that utilizes a machine learning algorithm to identify anterior large vessel occlusions (LVO) on CTA. The aim of this study was to test the algorithm's performance in LVO detection in an independent dataset.

METHODS

A total of 400 studies (217 LVO, 183 other/no occlusion) read by expert consensus were used for retrospective analysis. The LVO was defined as intracranial internal carotid artery (ICA) occlusion and M1 middle cerebral artery (MCA) occlusion. Software performance in detecting anterior LVO was evaluated using receiver operator characteristics (ROC) analysis, reporting area under the curve (AUC), sensitivity, and specificity. Subgroup analyses were performed to evaluate if performance in detecting LVO differed by subgroups, namely M1 MCA and ICA occlusion sites, and in data stratified by patient age, sex, and CTA acquisition characteristics (slice thickness, kilovoltage tube peak, and scanner manufacturer).

RESULTS

AUC, sensitivity, and specificity overall were as follows: 0.939, 0.894, and 0.874, respectively, in the full cohort; 0.927, 0.857, and 0.874, respectively, in the ICA occlusion cohort; 0.945, 0.914, and 0.874, respectively, in the M1 MCA occlusion cohort. Performance did not differ significantly by patient age, sex, or CTA acquisition characteristics.

CONCLUSION

The StrokeSENS LVO machine learning algorithm detects anterior LVO with high accuracy from a range of scans in a large dataset.

Interpretable Machine Learning Modeling for Ischemic Stroke Outcome Prediction

Background and Purpose

Mechanical thrombectomy greatly improves stroke outcomes. Nonetheless, some patients fall short of full recovery despite good reperfusion. The purpose of this study was to develop machine learning (ML) models for the pre-interventional prediction of functional outcome at 3 months of thrombectomy in acute ischemic stroke (AIS), using clinical and auto-extractable radiological information consistently available upon first emergency evaluation.

Materials and Methods

A two-center retrospective cohort of 293 patients with AIS who underwent thrombectomy was analyzed. ML models were developed to predict dichotomized modified Rankin score at 90 days (mRS-90) using clinical and imaging features, both separately and combined. Conventional and experimental imaging biomarkers were quantified using automated image-processing software from non-contract computed tomography (CT) and computed tomography angiography (CTA). Shapley Additive Explanation (SHAP) was applied for model interpretability and predictor importance analysis of the optimal model.

Results

Merging clinical and imaging features returned the best results for mRS-90 prediction. The best performing classifier was Extreme Gradient Boosting (XGB) with an area under the receiver operating characteristic curve (AUC) = 84% using selected features. The most important classifying features were age, baseline National Institutes of Health Stroke Scale (NIHSS), occlusion side, degree of brain atrophy [primarily represented by cortical cerebrospinal fluid (CSF) volume and lateral ventricle volume], early ischemic core [primarily represented by e-Alberta Stroke Program Early CT Score (ASPECTS)], and collateral circulation deficit volume on CTA.

Conclusion

Machine learning that is applied to quantifiable image features from CT and CTA alongside basic clinical characteristics constitutes a promising automated method in the pre-interventional prediction of stroke prognosis. Interpretable models allow for exploring which initial features contribute the most to post-thrombectomy outcome prediction overall and for each individual patient outcome.

Hyperdense Artery Sign in Patients With Acute Ischemic Stroke-Automated Detection With Artificial Intelligence-Driven Software

Background

Hyperdense artery sign (HAS) on non-contrast CT (NCCT) can indicate a large vessel occlusion (LVO) in patients with acute ischemic stroke. HAS detection belongs to routine reporting in patients with acute stroke and can help to identify patients in whom LVO is not initially suspected. We sought to evaluate automated HAS detection by commercial software and compared its performance to that of trained physicians against a reference standard.

Methods

Non-contrast CT scans from 154 patients with and without LVO proven by CT angiography (CTA) were independently rated for HAS by two blinded neuroradiologists and an AI-driven algorithm (Brainomix®). Sensitivity and specificity were analyzed for the clinicians and the software. As a secondary analysis, the clot length was automatically calculated by the software and compared with the length manually outlined on CTA images as the reference standard.

Results

Among 154 patients, 84 (54.5%) had CTA-proven LVO. HAS on the correct side was detected with a sensitivity and specificity of 0.77 (CI:0.66–0.85) and 0.87 (0.77–0.94), 0.8 (0.69–0.88) and 0.97 (0.89–0.99), and 0.93 (0.84–0.97) and 0.71 (0.59–0.81) by the software and readers 1 and 2, respectively. The automated estimation of the thrombus length was in moderate agreement with the CTA-based reference standard [intraclass correlation coefficient (ICC) 0.73].

Conclusion

Automated detection of HAS and estimation of thrombus length on NCCT by the tested software is feasible with a sensitivity and specificity comparable to that of trained neuroradiologists.

Diagnostic performance of an algorithm for automated large vessel occlusion detection on CT angiography

Background

Machine learning algorithms hold the potential to contribute to fast and accurate detection of large vessel occlusion (LVO) in patients with suspected acute ischemic stroke. We assessed the diagnostic performance of an automated LVO detection algorithm on CT angiography (CTA).

Methods

Data from the MR CLEAN Registry and PRESTO were used including patients with and without LVO. CTA data were analyzed by the algorithm for detection and localization of LVO (intracranial internal carotid artery (ICA)/ICA terminus (ICA-T), M1, or M2). Assessments done by expert neuroradiologists were used as reference. Diagnostic performance was assessed for detection of LVO and per occlusion location by means of sensitivity, specificity, and area under the curve (AUC).

Results

We analyzed CTAs of 1110 patients from the MR CLEAN Registry (median age (IQR) 71 years (60–80); 584 men; 1110 with LVO) and of 646 patients from PRESTO (median age (IQR) 73 years (62–82); 358 men; 141 with and 505 without LVO). For detection of LVO, the algorithm yielded a sensitivity of 89% in the MR CLEAN Registry and a sensitivity of 72%, specificity of 78%, and AUC of 0.75 in PRESTO. Sensitivity per occlusion location was 88% for ICA/ICA-T, 94% for M1, and 72% for M2 occlusion in the MR CLEAN Registry, and 80% for ICA/ICA-T, 95% for M1, and 49% for M2 occlusion in PRESTO.

Conclusion

The algorithm provided a high detection rate for proximal LVO, but performance varied significantly by occlusion location. Detection of M2 occlusion needs further improvement.