A New Risk Index for Predicting Outcomes among Patients Undergoing Carotid Endarterectomy in Large Administrative Data Sets

Using machine learning to predict outcomes following carotid endarterectomy

OBJECTIVE

Prediction of outcomes following carotid endarterectomy (CEA) remains challenging, with a lack of standardized tools to guide perioperative management. We used machine learning (ML) to develop automated algorithms that predict outcomes following CEA.

METHODS

The Vascular Quality Initiative (VQI) database was used to identify patients who underwent CEA between 2003 and 2022. We identified 71 potential predictor variables (features) from the index hospitalization (43 preoperative [demographic/clinical], 21 intraoperative [procedural], and 7 postoperative [in-hospital complications]). The primary outcome was stroke or death at 1 year following CEA. Our data were split into training (70%) and test (30%) sets. Using 10-fold cross-validation, we trained six ML models using preoperative features (Extreme Gradient Boosting [XGBoost], random forest, Naïve Bayes classifier, support vector machine, artificial neural network, and logistic regression). The primary model evaluation metric was area under the receiver operating characteristic curve (AUROC). After selecting the best performing algorithm, additional models were built using intra- and postoperative data. Model robustness was evaluated using calibration plots and Brier scores. Performance was assessed on subgroups based on age, sex, race, ethnicity, insurance status, symptom status, and urgency of surgery.

RESULTS

Overall, 166,369 patients underwent CEA during the study period. In total, 7749 patients (4.7%) had the primary outcome of stroke or death at 1 year. Patients with an outcome were older with more comorbidities, had poorer functional status, and demonstrated higher risk anatomic features. They were also more likely to undergo intraoperative surgical re-exploration and have in-hospital complications. Our best performing prediction model at the preoperative stage was XGBoost, achieving an AUROC of 0.90 (95% confidence interval [CI], 0.89-0.91). In comparison, logistic regression had an AUROC of 0.65 (95% CI, 0.63-0.67), and existing tools in the literature demonstrate AUROCs ranging from 0.58 to 0.74. Our XGBoost models maintained excellent performance at the intra- and postoperative stages, with AUROCs of 0.90 (95% CI, 0.89-0.91) and 0.94 (95% CI, 0.93-0.95), respectively. Calibration plots showed good agreement between predicted and observed event probabilities with Brier scores of 0.15 (preoperative), 0.14 (intraoperative), and 0.11 (postoperative). Of the top 10 predictors, eight were preoperative features, including comorbidities, functional status, and previous procedures. Model performance remained robust on all subgroup analyses.

CONCLUSIONS

We developed ML models that accurately predict outcomes following CEA. Our algorithms perform better than logistic regression and existing tools, and therefore, have potential for important utility in guiding perioperative risk mitigation strategies to prevent adverse outcomes.

Accurate classification of carotid endarterectomy indication using physician claims and hospital discharge data

BACKGROUND AND PURPOSE

Studies of carotid endarterectomy (CEA) require stratification by symptomatic vs asymptomatic status because of marked differences in benefits and harms. In administrative datasets, this classification has been done using hospital discharge diagnosis codes of uncertain accuracy. This study aims to develop and evaluate algorithms for classifying symptomatic status using hospital discharge and physician claims data.

METHODS

A single center's administrative database was used to assemble a retrospective cohort of participants with CEA. Symptomatic status was ascertained by chart review prior to linkage with physician claims and hospital discharge data. Accuracy of rule-based classification by discharge diagnosis codes was measured by sensitivity and specificity. Elastic net logistic regression and random forest models combining physician claims and discharge data were generated from the training set and assessed in a test set of final year participants. Models were compared to rule-based classification using sensitivity at fixed specificity.

RESULTS

We identified 971 participants undergoing CEA at the Vancouver General Hospital (Vancouver, Canada) between January 1, 2008 and December 31, 2016. Of these, 729 met inclusion/exclusion criteria (n = 615 training, n = 114 test). Classification of symptomatic status using hospital discharge diagnosis codes was 32.8% (95% CI 29-37%) sensitive and 98.6% specific (96-100%). At matched 98.6% specificity, models that incorporated physician claims data were significantly more sensitive: elastic net 69.4% (59-82%) and random forest 78.8% (69-88%).

CONCLUSION

Discharge diagnoses were specific but insensitive for the classification of CEA symptomatic status. Elastic net and random forest machine learning algorithms that included physician claims data were sensitive and specific, and are likely an improvement over current state of classification by discharge diagnosis alone.

External Validation of Risk Prediction Models to Improve Selection of Patients for Carotid Endarterectomy

BACKGROUND AND PURPOSE

The net benefit of carotid endarterectomy (CEA) is determined partly by the risk of procedural stroke or death. Current guidelines recommend CEA if 30-day risks are <6% for symptomatic stenosis and <3% for asymptomatic stenosis. We aimed to identify prediction models for procedural stroke or death after CEA and to externally validate these models in a large registry of patients from the United States.

METHODS

We conducted a systematic search in MEDLINE and EMBASE for prediction models of procedural outcomes after CEA. We validated these models with data from patients who underwent CEA in the American College of Surgeons National Surgical Quality Improvement Program (2011-2017). We assessed discrimination using C statistics and calibration graphically. We determined the number of patients with predicted risks that exceeded recommended thresholds of procedural risks to perform CEA.

RESULTS

After screening 788 reports, 15 studies describing 17 prediction models were included. Nine were developed in populations including both asymptomatic and symptomatic patients, 2 in symptomatic and 5 in asymptomatic populations. In the external validation cohort of 26 293 patients who underwent CEA, 702 (2.7%) developed a stroke or died within 30-days. C statistics varied between 0.52 and 0.64 using all patients, between 0.51 and 0.59 using symptomatic patients, and between 0.49 to 0.58 using asymptomatic patients. The Ontario Carotid Endarterectomy Registry model that included symptomatic status, diabetes, heart failure, and contralateral occlusion as predictors, had C statistic of 0.64 and the best concordance between predicted and observed risks. This model identified 4.5% of symptomatic and 2.1% of asymptomatic patients with procedural risks that exceeded recommended thresholds.

CONCLUSIONS

Of the 17 externally validated prediction models, the Ontario Carotid Endarterectomy Registry risk model had most reliable predictions of procedural stroke or death after CEA and can inform patients about procedural hazards and help focus CEA toward patients who would benefit most from it.

Safety of carotid artery revascularization procedures in patients with atrial fibrillation

BACKGROUND

Atrial fibrillation (Afib) is a major contributor to cerebrovascular events. Coexisting carotid artery disease is not uncommon in Afib patients, yet they have been excluded from major randomized clinical trials. Therefore, the aim of this study was to evaluate the safety of carotid endarterectomy (CEA) and carotid artery stenting (CAS) in Afib patients.

METHODS

The Premier Healthcare Database was queried (2009-2015). Patients who underwent CEA or CAS were captured by International Classification of Diseases, Ninth Revision, Clinical Modification codes. Multivariable logistic modeling was implemented to examine the outcomes: in-hospital stroke, intracerebral hemorrhage (ICH), mortality, and stroke/death.

RESULTS

There were 86,778 patients included. The majority were asymptomatic (n = 82,128 [94.6%]). Afib was reported in 6743 patients (7.8%). In terms of absolute outcomes in both asymptomatic and symptomatic patients, Afib patients (vs non-Afib patients) had higher mortality and stroke/death (asymptomatic: mortality, 0.4% vs 0.2%; stroke/death, 1.7% vs 1.2%; symptomatic: mortality, 6.9% vs 2.1%; stroke/death, 10.6% vs 4.5%; all P < .05). Adjusted analysis yielded higher odds of ICH (adjusted odds ratio [aOR], 1.29; 95% confidence interval [CI], 1.00-1.67), mortality (aOR, 1.59; 95% CI, 1.11-2.26), and stroke/death (aOR, 1.30; 95% CI, 1.08-1.58) in Afib patients. Although univariable analysis found Afib to be a statistically significant predictor of ischemic stroke, similar results could not be elucidated in the multivariable analysis (aOR, 1.17; 95% CI, 0.93-1.47). In Afib patients, important predictors of stroke/death included CAS (aOR, 1.80; 95% CI, 1.21-2.68) and symptomatic presentation (aOR, 5.00; 95% CI, 3.20-7.83). Other important predictors were type of preoperative medication use, age, and hospital size.

CONCLUSIONS

Afib was associated with worse postoperative outcomes in patients with carotid artery disease. Symptomatic status in Afib patients is associated with a stroke/death risk that is higher than in recommended guidelines for CEA and particularly for CAS. Overall, CEA was associated with lower periprocedural ICH, mortality, and stroke/death in Afib patients compared with CAS.

Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

Importance

Randomized clinical trials have demonstrated that patients with asymptomatic carotid stenosis are eligible for carotid endarterectomy (CEA) if the 30-day surgical complication rate is less than 3% and the patient's life expectancy is at least 5 years.

Objective

To develop a risk prediction tool to improve patient selection for CEA among patients with asymptomatic carotid stenosis.

Design, Setting, and Participants

In this cohort study, veterans 65 years and older who received both carotid imaging and CEA in the Veterans Administration between January 1, 2005, and December 31, 2009 (n = 2325) were followed up for 5 years. Data were analyzed from January 2005 to December 2015. A risk prediction tool (the Carotid Mortality Index [CMI]) based on 23 candidate variables identified in the literature was developed using Veterans Administration and Medicare data. A simpler model based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality was also developed (any cancer in the past 5 years, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]). Model performance was assessed using measures of discrimination (eg, area under the curve [AUC]) and calibration. Internal validation was performed by correcting for optimism using 500 bootstrapped samples.

Main Outcome and Measure

Five-year mortality.

Results

Among 2325 veterans, the mean (SD) age was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. On the basis of a backward selection algorithm, 9 patient characteristics were selected (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model, which yielded an optimism-corrected AUC of 0.687 for the CMI. The 4C model had slightly worse discrimination (AUC, 0.657) compared with the CMI model; however, the calibration curve was similar to the full model in most of the range of predicted probabilities.

Conclusions and Relevance

According to results of this study, use of the CMI or the simpler 4C model may improve patient selection for CEA among patients with asymptomatic carotid stenosis.

How to better identify patients at high risk of postoperative complications?

PURPOSE OF REVIEW

Preoperative risk assessment and perioperative factors may help identify patients at increased risk of postoperative complications and allow postoperative management strategies that improve patient outcomes. This review summarizes historical and more recent scoring systems for predicting patients with increased morbidity and mortality in the postoperative period.

RECENT FINDINGS

Most prediction scores predict postoperative mortality with, at best, moderate accuracy. Scores that incorporate surgery-specific and intraoperative covariates may improve the accuracy of traditional scores. Traditional risk factors including increased ASA physical status score, emergent surgery, intraoperative blood loss and hemodynamic instability are consistently associated with increased mortality using most scoring systems.

SUMMARY

Preoperative clinical risk indices and risk calculators estimate surgical risk with moderate accuracy. Surgery-specific risk calculators are helpful in identifying patients at increased risk of 30-day mortality. Particular attention should be paid to intraoperative hemodynamic instability, blood loss, extent of surgical excision and volume of resection.