Diagnosis of recanalization of the intracranial artery has poor inter-rater reliability

Deep Learning-Based Automated Thrombolysis in Cerebral Infarction Scoring: A Timely Proof-of-Principle Study

Background and Purpose

Mechanical thrombectomy is an established procedure for treatment of acute ischemic stroke. Mechanical thrombectomy success is commonly assessed by the Thrombolysis in Cerebral Infarction (TICI) score, assigned by visual inspection of X-ray digital subtraction angiography data. However, expert-based TICI scoring is highly observer-dependent. This represents a major obstacle for mechanical thrombectomy outcome comparison in, for instance, multicentric clinical studies. Focusing on occlusions of the M1 segment of the middle cerebral artery, the present study aimed to develop a deep learning (DL) solution to automated and, therefore, objective TICI scoring, to evaluate the agreement of DL- and expert-based scoring, and to compare corresponding numbers to published scoring variability of clinical experts.

Methods

The study comprises 2 independent datasets. For DL system training and initial evaluation, an in-house dataset of 491 digital subtraction angiography series and modified TICI scores of 236 patients with M1 occlusions was collected. To test the model generalization capability, an independent external dataset with 95 digital subtraction angiography series was analyzed. Characteristics of the DL system were modeling TICI scoring as ordinal regression, explicit consideration of the temporal image information, integration of physiological knowledge, and modeling of inherent TICI scoring uncertainties.

Results

For the in-house dataset, the DL system yields Cohen’s kappa, overall accuracy, and specific agreement values of 0.61, 71%, and 63% to 84%, respectively, compared with the gold standard: the expert rating. Values slightly drop to 0.52/64%/43% to 87% when the model is, without changes, applied to the external dataset. After model updating, they increase to 0.65/74%/60% to 90%. Literature Cohen’s kappa values for expert-based TICI scoring agreement are in the order of 0.6.

Conclusions

The agreement of DL- and expert-based modified TICI scores in the range of published interobserver variability of clinical experts highlights the potential of the proposed DL solution to automated TICI scoring.

Reliability of the Modified TICI Score among Endovascular Neurosurgeons

BACKGROUND AND PURPOSE

The modified TICI score is the benchmark for quantifying reperfusion after mechanical thrombectomy. There has been limited investigation into the reliability of this score. We aim to identify intra-rater and inter-rater reliability of the mTICI score among endovascular neurosurgeons.

MATERIALS AND METHODS

Four independent endovascular neurosurgeons (raters) reviewed angiograms of 67 patients at 2 time points. κ statistics assessed inter- and intrarater reliability and compared raters'-versus-proceduralists' scores. Reliability was also assessed for occlusion location and by dichotomizing modified TICI scores (0-2a versus 2b-3).

RESULTS

Interrater reliability was moderate-to-substantial, weighted κ = 0.417-0.703, overall κ = 0.374 (P < .001). The dichotomized modified TICI score had moderate-to-substantial interrater agreement, κ statistics = 0.468-0.715, overall κ = 0.582 (P < .001). Intrarater reliability was moderate-to-almost perfect, weighted κ = 0.594-0.81. The dichotomized modified TICI score had substantial-to-almost perfect reliability, κ = 0.632-0.82. Proceduralists had fair-to-moderate agreement with raters, weighted κ = 0.348-0.574, and the dichotomized modified TICI score had fair-to-moderate agreement, κ = 0.365-0.544. When proceduralists and raters disagreed, proceduralists' scores were higher in 79.6% of cases. M1 followed by ICA occlusions had the highest agreement.

CONCLUSIONS

The modified TICI score is a practical metric for assessing reperfusion after mechanical thrombectomy, though not without limitations. Agreement improved when scores were dichotomized around the clinically relevant threshold of successful revascularization. Interrater reliability improved with time, suggesting that formal training of interventionalists may improve reporting reliability. Agreement of the modified TICI scale is best with M1 and ICA occlusion and becomes less reliable with more distal or posterior circulation occlusions. These findings should be considered when developing research trials.

Early Recanalization Postintravenous Thrombolysis in Ischemic Stroke with Large Vessel Occlusion: A Digital Subtraction Angiography Study

AIMS

We aimed to evaluate early recanalization postintravenous (i.v.) tissue plasminogen activator (t-PA) by digital subtraction angiography (DSA) in acute ischemic stroke (AIS) with large vessel occlusion (LVO).

METHODS

We performed baseline CT angiography to identify LVO in AIS. Recanalization pre- and post-intra-arterial therapy (IAT) was categorized to none, partial, and global recanalization (GR). Modified Rankin Scale score ≤2 at 3 months was considered a favorable outcome.

RESULTS

Among 1610 patients with AIS, 286 received IV t-PA. Of these, 55 patients with LVO were included. The median time from IV t-PA to DSA was 120 min (interquartile range, 79-152). Recanalization post-IV t-PA was observed in seven patients (12.7%). By occlusion sites, the recanalization rates were as follows: extracranial internal carotid artery 2 of 14 (14.3%); intracranial internal carotid artery 3 of 24 (12.5%); M1 of middle cerebral artery 3 of 39 (7.7%); M2 of middle cerebral artery 1 of 40 (2.5%); vertebral artery 0 of 4; and basilar artery 0 of 7. GR post-IAT was associated with favorable outcomes (odds ratio: 8.6; 95% confidence interval, 1.5-48.0; P = 0.014).

CONCLUSION

Early recanalization assessed by DSA post-IV t-PA is rarely observed in acute ischemic stroke patients with LVO.

The Revascularization Scales Dilemma: Is It Right to Apply the Treatment in Cerebral Ischemia Scale in Posterior Circulation Stroke?

BACKGROUND AND PURPOSE

Although various revascularization scales are used in the angiographic evaluation of acute ischemic stroke, observer reliability tests of these scales have been rarely performed for posterior circulation stroke. We aimed to evaluate inter- and intraobserver variability of 2 scales, the modified Treatment in Cerebral Ischemia and the Arterial Occlusive Lesion, in posterior circulation stroke.

MATERIALS AND METHODS

Three independent readers interpreted pre- and postthrombolytic angiographies of 62 patients with posterior circulation stroke by using the modified Treatment in Cerebral Ischemia and Arterial Occlusive Lesion scales. The κ statistic was used to measure observer agreement for both scales, and κ > 0.6 was considered substantial agreement.

RESULTS

For the Arterial Occlusive Lesion scale, inter- and intraobserver agreement was >0.6. While intraobserver agreement of the modified Treatment in Cerebral Ischemia scale was >0.6 except for 1 reader, interobserver agreement was lower in dichotomized and original scales. In 49 cases with solely basilar artery occlusion, inter- and intraobserver agreement of both scales was similar to that in all 62 patients with posterior circulation stroke. In 2 consecutive readings, there was a significant decrease in the proportion of mTICI 2a reads (22.58% in the first versus 13.44% in the second session, P < .03) and a reciprocal increase in the sum of proportions for modified Treatment in Cerebral Ischemia 2b and modified Treatment in Cerebral Ischemia 3 reads (62.37% in the first versus 72.58% in the second session, P < .046).

CONCLUSIONS

In angiographic assessment of posterior circulation stroke, inter- and intraobserver agreement for the Arterial Occlusive Lesion scale was reliable, while the modified Treatment in Cerebral Ischemia failed to achieve substantial interobserver agreement. The clinical impact of this result needs to be validated in future studies.

Intermediate Catheters Reduce the Length of Mechanical Thrombectomy Procedures in Acute Basilar Artery Occlusions

BACKGROUND AND PURPOSE

In the past years, technical developments have raised recanalization rates of endovascular treatments of intracerebral artery occlusions in acute ischemic stroke. By using stent retrievers, several prospective trials have reported recanalization rates up to 79 % as well as good neurological outcome in up to 58 % of the cases. The degree of the recanalization and the length of the procedure are factors known to influence the clinical outcome of patients treated endovascularly. Yet, still little is known about factors influencing the angiographic results of thrombectomy procedures. The purpose of this study was to investigate whether the use of intermediate catheters affects the angiographic results of thrombectomy procedures in basilar artery occlusions.

MATERIALS AND METHODS

A total of 47 consecutive patients with acute basilar artery occlusions who underwent endovascular treatment with stent retrievers in our department were retrospectively identified. We analyzed the angiographic data regarding the use of intermediate catheters, the lengths of the procedures, the number of passes of the stent retrievers, the angiographic results, and the site of access to the basilar artery.

RESULTS

Recanalization with modified thrombolysis in cerebral infarction (mTICI) ≥ 2b was achieved in 74.5 %. Intermediate catheters were used in 13 cases. The mean length of the procedures was significantly shorter when intermediate catheters were used (44.8 ± 27.6 vs. 70.7 ± 41.4 min, P = .043). There were no significant differences in the number of passes or in the final mTICI scores.

CONCLUSIONS

The use of intermediate catheters significantly reduces the length of mechanical thrombectomy procedures in acute basilar artery occlusions.

Scoring flow restoration in cerebral angiograms after endovascular revascularization in acute ischemic stroke patients

Endovascular revascularization techniques are increasingly used to treat arterial occlusions in patients with acute ischemic stroke. To monitor and communicate treatment results, a valid, reproducible, and clinically relevant, yet easy to use grading scheme of arterial recanalization and tissue reperfusion for digital subtraction angiography is needed. An ideal scoring system would consider the target arterial lesion, the perfusion deficit, and the collateral status before treatment and measure recanalization, reperfusion, early venous shunting, vasospasm, as well as distal embolization after flow restoration. Currently, a variety of different flow restoration scales are in use, including the Thrombolysis in Myocardial Infarction scoring system, the Thrombolysis in Cerebral Infarction score, and the Arterial Occlusive Lesion score, which describe the local recanalization result. These scores are not used homogeneously throughout the literature, are often modified and not fully documented, which make them inept to compare treatment effects across studies. In addition, none of these scores cover all of the above-mentioned aspects, nor are they able to describe satisfactorily all relevant angiographic findings, and data on their reliability and predictive power regarding clinical outcome are sparse. We aimed to review and illustrate the different revascularization scales, discuss their advantages and limitations as well as the available data regarding standardization, reliability testing, and outcome prediction. In addition, we give examples for the use of the scales and show potential pitfalls.

Inter- and intraobserver agreement in scoring angiographic results of intra-arterial stroke therapy

BACKGROUND AND PURPOSE

Angiographic results are commonly used as surrogate markers of the success of intra-arterial therapies for acute stroke. Inter- and intraobserver agreement in judging angiographic results remain poorly characterized. Our goal was to assess 2 commonly used revascularization scales.

MATERIALS AND METHODS

A portfolio of 148 pre- and post treatment images of 37 cases of proximal anterior circulation occlusions was electronically sent to 12 expert observers who were asked to grade treatment outcomes according to recanalization (of arterial occlusive lesion) or reperfusion (TICI) scales. Three expert observers had to score treatment outcomes by using a similar portfolio of 32 patients or when they had full access to all angiographic data, twice for each method 3-12 months apart. Results were analyzed by using κ statistics.

RESULTS

Agreement among 9 responding observers was moderate for both the TICI (κ = 0.45 ± 0.01) and arterial occlusive lesion (κ = 0.39 ± 0.16) scales. Agreement was similar (moderate) when 3 observers had access to a portfolio (κ = 0.59 ± 0.06 and 0.49 ± 0.07, respectively) or to the full angiographic data (κ = 0.54 ± 0.06 and 0.59 ± 0.07, respectively). Intraobserver agreement was "fair to moderate" for both methods. Interobserver agreement became "substantial" (>0.6) when outcomes were dichotomized into "success" (TICI 2b, 3; arterial occlusive lesion II, III or "failure"; the results were judged more favorably when the arterial occlusive lesion rather than the TICI scale was used.

CONCLUSIONS

There is an important variability in the assessment of angiographic outcomes of endovascular treatments, invalidating comparisons among publications. A simple dichotomous judgment can be used as a surrogate outcome when treatments are assessed by the same observers in randomized trials.

Peri-interventional Subarachnoid Hemorrhage During Mechanical Thrombectomy with stent retrievers in Acute Stroke: A Retrospective Case-Control Study

BACKGROUND

Mechanical thrombectomy with stent retrievers in acute stroke has emerged as a promising new technique with the highest recanalization rate of the therapeutic procedures available so far. However, endovascular treatment is also associated with the risk of specific complications. One of those is the occurrence of peri-interventional subarachnoid hemorrhage (SAH), which has been reported in 5-16 % of the cases. Interestingly, this rate is higher than that of angiographically detectable perforations (0-3 %), leaving the majority of peri-interventional SAH to be due to angiographically occult perforations. Little is known about the influence of this finding on clinical outcome. The purpose of this study was to investigate the clinical relevance of SAH due to occult perforations during thrombectomy with stent retrievers.

METHODS

Postinterventional computed tomography (CT) scans of 217 consecutive patients with acute occlusions of intracerebral arteries who were treated with stent retrievers in our department between October 2009 and October 2012 were retrospectively analyzed.

RESULTS

SAH was found on postinterventional CT scans in 5.5 % of the cases. Seven cases were included for further analysis and matched to controls by the following characteristics: (1) site of occlusion, (2) result of the recanalization procedure according to the modified thrombolysis in cerebral infarction score, (3) administration of intravenous recombinant tissue plasminogen activator, (4) presence of proximal extracranial occlusion, (5) age, and (6) sex. Comparison of the angiographic data of the two cohorts showed no significant difference in the length of the procedures or the number of maneuvers needed for recanalization, nor were there significant differences in clinical outcomes as measured by NIHSS and mRS scores. Secondary symptomatic ICH occurred in one case in either cohort and led to death in both cases. The rate of asymptomatic ICH within the first 24 h after recanalization was significantly higher in the group with peri-interventional SAH (57 vs. 0 %, P = 0.018).

CONCLUSIONS

This small retrospective case-control study did not reveal a significant influence of peri-interventional SAH due to angiographically occult perforations on neurologic outcome of patients treated with stent retrievers.

Clarifying differences among thrombolysis in cerebral infarction scale variants: is the artery half open or half closed?

BACKGROUND AND PURPOSE

Although thrombolysis in cerebral infarction (TICI) 2b/3 has been regarded as a successful angiographic outcome, the definition or subclassification of TICI 2 has differed between the original (o-TICI) and modified TICI (m-TICI). We sought to compare interobserver variability for both scores and analyze the subgroups of the TICI 2.

METHODS

Five readers interpreted angiographies independently using a 6-point scale as follows: grade 0, no antegrade flow; grade 1, flow past the initial occlusion without tissue reperfusion; grade 2, partial reperfusion in <50% of the affected territory; grade 3, partial reperfusion in 50% to 66%; grade 4, partial reperfusion in ≥ 67%; grade 5, complete perfusion. Readings using this scale were then converted into o-TICI and m-TICI score. Statistical analysis was performed according to TICI 2 subgroups.

RESULTS

Interobserver agreement was good for the o-TICI and m-TICI scores (intraclass correlation coefficient, 0.73 and 0.67, respectively). Our grade 3 (partial perfusion with 50% to 66%) occupied 19% of total readings, which would have been classified as grade 2a in o-TICI, but as 2b in m-TICI. The m-TICI was more likely to predict good clinical outcome than o-TICI (odds ratio, 2.01 versus 1.63, in reads with TICI 2b/3 versus 0/2a).

CONCLUSIONS

Both TICI scales showed good agreement among readers. However, the variability in partial perfusion thresholds leads to different grading in ≈ 20% of cases and may result in significantly different rates of accurate outcome prediction.