Prediction of 60-Day Case Fatality After Aneurysmal Subarachnoid Hemorrhage: External Validation of a Prediction Model*

Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage

BACKGROUND

Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients.

METHODS

We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell's c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a "membership model" to obtain a gross estimate of transportability.

RESULTS

Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70-0.78, range 0.52-0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37-0.24, range -1.40-0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78-1.13, range 0.53-1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model.

CONCLUSIONS

We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.

Declining trend in the estimated annual rupture rate of unruptured cerebral aneurysms in Japan: a nationwide study

The incidence of subarachnoid hemorrhages (SAHs) in Japan has recently decreased. However, trends in the annual rates of unruptured cerebral aneurysms (UCAs) are unclear because calculations based on follow-up periods are limited to patients with ruptured cerebral aneurysms. We aimed to clarify current trends in the estimated annual rupture rates of UCAs in Japan and to identify the most relevant contributing factors. We analyzed data from the Japanese Ministry of Health, Labour and Welfare and records of the Japan Neurosurgical Society. The estimated annual rupture rates of UCAs between 2003 and 2018 were calculated according to age-adjusted mortality rates of SAH and number of treated ruptured cerebral aneurysms (RCAs). We estimated trends in annual rupture rates using sensitivity analysis and assessed associations between estimated annual rupture rates and the prevalences of hypertension and current smoking. The estimated annual rupture rate of UCAs significantly decreased from 1.44 to 0.87% and from 0.92 to 0.76%, respectively, in terms of age-adjusted mortality rates of SAH and number of treated RCAs (p < 0.001). The range of changes in estimated annual rupture rates of UCAs was - 1.13%‒0.83%, representing a declining UCA trend of 88%. The estimated annual rupture rates of UCAs declined by 0.02-0.10% and 0.01-0.05% with every percent decrease in hypertension and current smoking prevalence, respectively. The estimated annual rupture rate of UCAs has recently decreased in Japan, possibly due to a decrease in the prevalence of hypertension rather than smoking.

Clinical prediction score for prolonged length of hospital stay in aneurysmal subarachnoid hemorrhage

BACKGROUND

Length of stay (LOS) is an important indicator of the optimization of health services and hospital financing efficiency in aneurysmal subarachnoid hemorrhage (aSAH) patients. The purpose of this study was to develop a scoring model to predict the LOS of patients with aSAH.

METHOD

A clinical scoring was developed based on retrospectively collected data from the cerebral aneurysm registry of the National Brain Center Hospital, Jakarta, from January 2019 to June 2022. Multivariate logistic regression was used to determine the odds ratio for risk-adjusted prolonged LOS. LOS predictors were obtained based on the regression coefficients and converted into a point score model.

RESULTS

Of the 209 aSAH patients observed, 117 patients had prolonged LOS (> 14 days of hospital stay). A clinical score was developed with a range of 0-7 points. Four variables were chosen as predictors of prolonged LOS: the presence of high-grade aSAH (1 point), aneurysm treatment (endovascular coiling: 1 point; surgical clipping: 2 points), cardiovascular comorbidities (1 point), and hospital-acquired pneumonia (3 points). The score showed good discrimination with an area under the receiving operating characteristics curve (AUC) of 0.8183 (SE 0.0278) and a p-value for the Hosmer-Lemeshow (HL) goodness-of-fit of 0.9322.

CONCLUSION

This simple clinical score reliably predicted prolonged LOS in aneurysmal subarachnoid hemorrhage cases and may aid clinicians in improving patient outcomes and decreasing healthcare costs.

Mortality among patients treated for aneurysmal subarachnoid hemorrhage in Eastern Denmark 2017-2019

OBJECTIVE

The aim of the study was to investigate (1) the 30-day, 3-month, and 12-month cumulative mortalities for patients who underwent aneurysm occlusion, and (2) the causes of death, and (3) the potential risk factors for death.

METHODS

All patients who underwent surgical clipping or endovascular treatment of a ruptured aneurysm at Copenhagen University Hospital, during the period of January 1, 2017-December 31, 2019, were included and followed up for 12 months. Data regarding vital status, causes of death, comorbidities, treatment, and clinical presentations on admission was collected. The absolute mortality risk was estimated as a function of time with a 95% confidence interval. The associations between potential risk factors and death were estimated as odds ratios with 95% confidence intervals using logistic regression models.

RESULTS

A total of 317 patients were included. The overall cumulative mortalities after 30 days, 3 months, and 12 months were 10.7%, 12.9%, and 16.1%, respectively. The most common cause of death was severe primary hemorrhage (52.9%), followed by infections (15.7%) and rebleeding (11.8%). WFNS score > 3 and Fisher score > 3 on admission, preprocedural hydrocephalus, and preprocedural rebleeding were found significantly associated with higher risk of death.

CONCLUSIONS

Considerable mortality was seen. Possible preventable causes accounted for approximately 22% of the deaths. The occurrence of both pre- and postprocedural rebleeding's indicates an opportunity of further improvement of the mortality by (1) further reduction of time from aSAH to aneurysm occlusion and (2) continuous efforts in improving methods of aneurysm occlusion.

Methodological quality of multivariate prognostic models for intracranial haemorrhages in intensive care units: a systematic review

OBJECTIVES

Patients with severe spontaneous intracranial haemorrhages, managed in intensive care units, face ethical issues regarding the difficulty of anticipating their recovery. Prognostic tools help clinicians in counselling patients and relatives and guide therapeutic decisions. We aimed to methodologically assess prognostic tools for functional outcomes in severe spontaneous intracranial haemorrhages.

DATA SOURCES

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses recommendations, we conducted a systematic review querying Medline, Embase, Web of Science, and the Cochrane in January 2020.

STUDY SELECTION

We included development or validation of multivariate prognostic models for severe intracerebral or subarachnoid haemorrhage.

DATA EXTRACTION

We evaluated the articles following the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies and Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis statements to assess the tools' methodological reporting.

RESULTS

Of the 6149 references retrieved, we identified 85 articles eligible. We discarded 43 articles due to the absence of prognostic performance or predictor selection. Among the 42 articles included, 22 did not validate models, 6 developed and validated models and 14 only externally validated models. When adding 11 articles comparing developed models to existing ones, 25 articles externally validated models. We identified methodological pitfalls, notably the lack of adequate validations or insufficient performance levels. We finally retained three scores predicting mortality and unfavourable outcomes: the IntraCerebral Haemorrhages (ICH) score and the max-ICH score for intracerebral haemorrhages, the SubArachnoid Haemorrhage International Trialists score for subarachnoid haemorrhages.

CONCLUSIONS

Although prognostic studies on intracranial haemorrhages abound in the literature, they lack methodological robustness or show incomplete reporting. Rather than developing new scores, future authors should focus on externally validating and updating existing scores with large and recent cohorts.

Systemic Inflammation Response Index and Systemic Immune-inflammation Index for Predicting the Prognosis of Patients with Aneurysmal Subarachnoid Hemorrhage

OBJECTIVES

Inflammatory response plays a pivotal role in the progress of aneurysmal subarachnoid hemorrhage (aSAH). As novel inflammatory markers, systemic inflammation response index (SIRI) and systemic immune-inflammation (SII) index could reflect clinical outcomes of patients with various diseases. The aim of this study was to ascertain whether initial SIRI and SII index were associated with prognosis of aSAH patients.

METHODS

A total of 680 patients with aSAH were enrolled. Their prognosis was evaluated with modified Rankin Scale (mRS) at 3 months, and unfavorable clinical outcome was defined as mRS score of 3-6. Receiver operating characteristic (ROC) curve analysis was performed to identify cutoff values of SIRI and SII index for predicting clinical outcomes. Univariate and multivariate regression analyses were performed to explore relationships of SIRI and SII index with prognosis of patients.

RESULTS

Optimal cutoff values of SIRI and SII index to discriminate between favorable and unfavorable clinical outcomes were 3.2 × 10⁹/L and 960 × 10⁹/L, respectively (P < 0.001 and 0.004, respectively). In multivariate analysis, SIRI value ≥ 3.2 × 10⁹/L (odds ratio [OR]: 1.82, 95% CI: 1.46-3.24; P = 0.021) and SII index value ≥ 960 × 10⁹/L (OR: 1.68, 95% CI: 1.24-2.74; P = 0.040) were independent predicting factors for poor prognosis after aSAH.

CONCLUSIONS

SIRI and SII index values are associated with clinical outcomes of patients with aSAH. Elevated SIRI and SII index could be independent predicting factors for a poor prognosis after aSAH.

Between-center and between-country differences in outcome after aneurysmal subarachnoid hemorrhage in the Subarachnoid Hemorrhage International Trialists (SAHIT) repository

OBJECTIVE

Differences in clinical outcomes between centers and countries may reflect variation in patient characteristics, diagnostic and therapeutic policies, or quality of care. The purpose of this study was to investigate the presence and magnitude of between-center and between-country differences in outcome after aneurysmal subarachnoid hemorrhage (aSAH).

METHODS

The authors analyzed data from 5972 aSAH patients enrolled in randomized clinical trials of 3 different treatments from the Subarachnoid Hemorrhage International Trialists (SAHIT) repository, including data from 179 centers and 20 countries. They used random effects logistic regression adjusted for patient characteristics and timing of aneurysm treatment to estimate between-center and between-country differences in unfavorable outcome, defined as a Glasgow Outcome Scale score of 1-3 (severe disability, vegetative state, or death) or modified Rankin Scale score of 4-6 (moderately severe disability, severe disability, or death) at 3 months. Between-center and between-country differences were quantified with the median odds ratio (MOR), which can be interpreted as the ratio of odds of unfavorable outcome between a typical high-risk and a typical low-risk center or country.

RESULTS

The proportion of patients with unfavorable outcome was 27% (n = 1599). The authors found substantial between-center differences (MOR 1.26, 95% CI 1.16-1.52), which could not be explained by patient characteristics and timing of aneurysm treatment (adjusted MOR 1.21, 95% CI 1.11-1.44). They observed no between-country differences (adjusted MOR 1.13, 95% CI 1.00-1.40).

CONCLUSIONS

Clinical outcomes after aSAH differ between centers. These differences could not be explained by patient characteristics or timing of aneurysm treatment. Further research is needed to confirm the presence of differences in outcome after aSAH between hospitals in more recent data and to investigate potential causes.

External Validation of the Subarachnoid Hemorrhage International Trialists (SAHIT) Predictive Model Using the Barrow Ruptured Aneurysm Trial (BRAT) Cohort

BACKGROUND

The Subarachnoid Hemorrhage International Trialists (SAHIT) repository is a collection of randomized clinical trials, prospective observational studies, and hospital registries that was used to create a predictive model of unfavorable outcome/mortality following aneurysmal SAH.

OBJECTIVE

To externally validate the SAHIT model using Barrow Ruptured Aneurysm Trial (BRAT) data, which was not included in the SAHIT repository.

METHODS

This is a post hoc analysis of the prospective, randomized BRAT. Three models were created: (1) Core (age, hypertension, World Federation of Neurosurgical Societies grade), (2) neuroimaging (aneurysm size/location, Fisher score), and (3) full model (model 1 and 2 plus treatment type). The performance of the models was evaluated by measures of model discrimination (area under the curve [AUC]) and model calibration (goodness of fit test, calibration in-the-large, calibration slope).

RESULTS

A total of 338 patients (average age 54 years; 67.7% good clinical grade; average aneurysm size 6.7 mm; 84.1% anterior circulation) were included. Due to a large number of crossovers, more aneurysms were clipped than coiled (67.5% vs 32.5%, respectively). A total of 10.1% of the patients died and 29.6% experienced an unfavorable outcome. For unfavorable outcome, the AUCs for the three models were: 0.728, 0.732, and 0.734, respectively. For mortality, the AUCs for the three models were: 0.721, 0.739, and 0.744, respectively. Overall, all models showed good calibration, and the measures of calibration fell within 95% CI of those produced in the SAHIT study.

CONCLUSION

Using the BRAT data, we have externally validated the SAHIT model for predicting unfavorable outcome and mortality after SAH. The model may be used to counsel patients and families on prognosis following aneurysmal SAH.

Prediction of Outcome After Aneurysmal Subarachnoid Hemorrhage

Background and Purpose- Early prediction of clinical outcome after aneurysmal subarachnoid hemorrhage (aSAH) is still lacking accuracy. In this observational cohort study, we aimed to develop and validate an accurate bedside prediction model for clinical outcome after aSAH, to aid decision-making at an early stage. Methods- For the development of the prediction model, a prospectively kept single-center cohort of 1215 aSAH patients, admitted between 1998 and 2014, was used. For temporal validation, a prospective cohort of 224 consecutive aSAH patients from the same center, admitted between 2015 and 2017, was used. External validation was performed using the ISAT (International Subarachnoid Aneurysm Trial) database (2143 patients). Primary outcome measure was poor functional outcome 2 months after aSAH, defined as modified Rankin Scale score 4-6. The model was constructed using multivariate regression analyses. Performance of the model was examined in terms of discrimination and calibration. Results- The final model included 4 predictors independently associated with poor outcome after 2 months: age, World Federation of Neurosurgical Societies grade after resuscitation, aneurysm size, and Fisher grade. Temporal validation showed high discrimination (area under the receiver operating characteristic curve, 0.90; 95% CI, 0.85-0.94), external validation showed fair to good discrimination (area under the receiver operating characteristic curve, 0.73; 95% CI, 0.70-0.76). The model showed satisfactory calibration in both validation cohorts. The SAFIRE grading scale was derived from the final model: size of the aneurysm, age, Fisher grade, world federation of neurosurgical societies after resuscitation. Conclusions- The SAFIRE grading scale is an accurate, generalizable, and easily applicable model for early prediction of clinical outcome after aSAH.

Proving cortical death after vascular coma: Evoked potentials, EEG and neuroimaging

OBJECTIVES

Several studies have shown that bilateral abolition of somatosensory evoked potentials after a nontraumatic coma has 100% specificity for nonawakening with ethical consequences for active care withdrawal. We propose to evaluate the prognostic value of bilateral abolished cortical components of SEPs in severe vascular coma.

METHODS

A total of 144 comatose patients after subarachnoid haemorrhage were evaluated by multimodal evoked potentials (EPs); 7 patients presented a bilateral abolition of somatosensory and auditory EPs. Their prognosis value was interpreted with respect to brainstem auditory EPs, EEG, and structural imaging.

RESULTS

One patient emerged from vegetative state during follow-up; 6 patients did not return to consciousness. The main neurophysiological difference was a cortical reactivity to pain preserved in the patient who returned to consciousness. This patient had focal sub-cortical lesions, which could explain the abolition of primary cortical components by a bilateral deafferentation of somatosensory and auditory pathways.

CONCLUSIONS

This is the first report of a favourable outcome after a multimodal abolition of primary cortex EPs in vascular coma. For the 3 cases of vascular coma with preserved brainstem function, EEG reactivity and cortical EPs were abolished by a diffuse ischaemia close to cerebral anoxia.

SIGNIFICANCE

The complementarity of EPs, EEG, and imaging must be emphasised if therapeutic limitations are considered to avoid over-interpretation of the prognosis value of EPs.

Mortality after Spontaneous Subarachnoid Hemorrhage: Causality and Validation of a Prediction Model

OBJECTIVE

To evaluate primary causes of death after spontaneous subarachnoid hemorrhage (SAH) and externally validate the HAIR score, a prognostication tool, in a single academic institution.

METHODS

We reviewed all patients with SAH admitted to our neuro-intensive care unit between 2010 and 2016. Univariate and multivariate logistic regressions were performed to identify predictors of in-hospital mortality. The HAIR score predictors were Hunt and Hess grade at treatment decision, age, intraventricular hemorrhage, and rebleeding within 24 hours. Validation of the HAIR score was characterized with the receiver operating curve, the area under the curve, and a calibration plot.

RESULTS

Among 434 patients with SAH, in-hospital mortality was 14.1%. Of the 61 mortalities, 54 (88.5%) had a neurologic cause of death or withdrawal of care and 7 (11.5%) had cardiac death. Median time from SAH to death was 6 days. The main causes of death were effect of the initial hemorrhage (26.2%), rebleeding (23%) and refractory cerebral edema (19.7%). Factors significantly associated with in-hospital mortality in the multivariate analysis were age, Hunt and Hess grade, and intracerebral hemorrhage. Maximum lumen size was also a significant risk factor after aneurysmal SAH. The HAIR score had a satisfactory discriminative ability, with an area under the curve of 0.89.

CONCLUSIONS

The in-hospital mortality is lower than in previous reports, attesting to the continuing improvement of our institutional SAH care. The major causes are the same as in previous reports. Despite a different therapeutic protocol, the HAIR score showed good discrimination and could be a useful tool for predicting mortality.