Predictive performance of the American College of Surgeons universal risk calculator in neurosurgical patients

American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator as a Predictor of Postoperative Outcomes After Adult Spinal Deformity Surgery: A Retrospective Cohort Analysis

BACKGROUND AND OBJECTIVES

In recent years, there has been an outpouring of scoring systems that were built to predict outcomes after various surgical procedures; however, research validating these studies in spinal surgery is quite limited. In this study, we evaluated the predictability of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator (ACS NSQIP SRC) for various postoperative outcomes after spinal deformity surgery.

METHODS

A retrospective chart review was conducted to identify patients who underwent spinal deformity surgery at our hospital between January 1, 2014, and December 31, 2022. Demographic and clinical data necessary to use the ACS NSQIP SRC and postoperative outcomes were collected for these patients. Predictability was analyzed using the area under the curve (AUC) of receiver operating characteristic curves and Brier scores.

RESULTS

Among the 159 study patients, the mean age was 64.5 ± 9.5 years, mean body mass index was 31.9 ± 6.6, and 95 (59.7%) patients were women. The outcome most accurately predicted by the ACS NSQIP SRC was postoperative pneumonia (observed = 5.0% vs predicted = 3.2%, AUC = 0.75, Brier score = 0.05), but its predictability still fell below the acceptable threshold. Other outcomes that were underpredicted by the ACS NSQIP SRC were readmission within 30 days (observed = 13.8% vs predicted = 9.0%, AUC = 0.63, Brier score = 0.12), rate of discharge to nursing home or rehabilitation facilities (observed = 56.0% vs predicted = 46.6%, AUC = 0.59, Brier = 0.26), reoperation (observed 11.9% vs predicted 5.4%, AUC = 0.60, Brier = 0.11), surgical site infection (observed 9.4% vs predicted 3.5%, AUC = 0.61, Brier = 0.05), and any complication (observed 33.3% vs 19%, AUC = 0.65, Brier = 0.23). Predicted and observed length of stay were not significantly associated (β = 0.132, P = .47).

CONCLUSION

The ACS NSQIP SRC is a poor predictor of outcomes after spinal deformity surgery.

Evaluation of venous thromboembolism risk assessment models for hospital inpatients: the VTEAM evidence synthesis

Background

Pharmacological prophylaxis during hospital admission can reduce the risk of acquired blood clots (venous thromboembolism) but may cause complications, such as bleeding. Using a risk assessment model to predict the risk of blood clots could facilitate selection of patients for prophylaxis and optimise the balance of benefits, risks and costs.

Objectives

We aimed to identify validated risk assessment models and estimate their prognostic accuracy, evaluate the cost-effectiveness of different strategies for selecting hospitalised patients for prophylaxis, assess the feasibility of using efficient research methods and estimate key parameters for future research.

Design

We undertook a systematic review, decision-analytic modelling and observational cohort study conducted in accordance with Enhancing the QUAlity and Transparency Of health Research (EQUATOR) guidelines.

Setting

NHS hospitals, with primary data collection at four sites.

Participants

Medical and surgical hospital inpatients, excluding paediatric, critical care and pregnancy-related admissions.

Interventions

Prophylaxis for all patients, none and according to selected risk assessment models.

Main outcome measures

Model accuracy for predicting blood clots, lifetime costs and quality-adjusted life-years associated with alternative strategies, accuracy of efficient methods for identifying key outcomes and proportion of inpatients recommended prophylaxis using different models.

Results

We identified 24 validated risk assessment models, but low-quality heterogeneous data suggested weak accuracy for prediction of blood clots and generally high risk of bias in all studies. Decision-analytic modelling showed that pharmacological prophylaxis for all eligible is generally more cost-effective than model-based strategies for both medical and surgical inpatients, when valuing a quality-adjusted life-year at £20,000. The findings were more sensitive to uncertainties in the surgical population; strategies using risk assessment models were more cost-effective if the model was assumed to have a very high sensitivity, or the long-term risks of post-thrombotic complications were lower. Efficient methods using routine data did not accurately identify blood clots or bleeding events and several pre-specified feasibility criteria were not met. Theoretical prophylaxis rates across an inpatient cohort based on existing risk assessment models ranged from 13% to 91%.

Limitations

Existing studies may underestimate the accuracy of risk assessment models, leading to underestimation of their cost-effectiveness. The cost-effectiveness findings do not apply to patients with an increased risk of bleeding. Mechanical thromboprophylaxis options were excluded from the modelling. Primary data collection was predominately retrospective, risking case ascertainment bias.

Conclusions

Thromboprophylaxis for all patients appears to be generally more cost-effective than using a risk assessment model, in hospitalised patients at low risk of bleeding. To be cost-effective, any risk assessment model would need to be highly sensitive. Current evidence on risk assessment models is at high risk of bias and our findings should be interpreted in this context. We were unable to demonstrate the feasibility of using efficient methods to accurately detect relevant outcomes for future research.

Future work

Further research should evaluate routine prophylaxis strategies for all eligible hospitalised patients. Models that could accurately identify individuals at very low risk of blood clots (who could discontinue prophylaxis) warrant further evaluation.

Study registration

This study is registered as PROSPERO CRD42020165778 and Researchregistry5216.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR127454) and will be published in full in Health Technology Assessment; Vol. 28, No. 20. See the NIHR Funding and Awards website for further award information.

Can the American College of Surgeons NSQIP Surgical Risk Calculator Accurately Predict Adverse Postoperative Outcomes in Emergency Abdominal Surgery? An Italian Multicenter Analysis

BACKGROUND

The American College of Surgeons NSQIP surgical risk calculator provides an estimation of 30-day postoperative adverse outcomes. It is useful in the identification of high-risk patients needing clinical optimization and supports the informed consent process. The purpose of this study is to validate its predictive value in the Italian emergency setting.

STUDY DESIGN

Six Italian institutions were included. Inclusion diagnoses were acute cholecystitis, appendicitis, gastrointestinal perforation or obstruction. Areas under the receiving operating characteristic curves, Brier score, Hosmer-Lemeshow index, and observed-to-expected event ratio were measured to assess both discrimination and calibration. Effect of the Surgeon Adjustment Score on calibration was then tested. A patient's personal risk ratio was obtained, and a cutoff was chosen to predict mortality with a high negative predicted value.

RESULTS

A total of 2,749 emergency procedures were considered for the analysis. The areas under the receiving operating characteristic curve were 0.932 for death (0.921 to 0.941, p < 0.0001; Brier 0.041) and 0.918 for discharge to nursing or rehabilitation facility (0.907 to 0.929, p < 0.0001; 0.070). Discrimination was also strong (area under the receiving operating characteristic curve >0.8) for renal failure, cardiac complication, pneumonia, venous thromboembolism, serious complication, and any complication. Brier score was informative (<0.25) for all the presented variables. The observed-to-expected event ratios were 1.0 for death and 0.8 for discharge to facility. For almost all other variables, there was a general risk underestimation, but the use of the Surgeon Adjustment Score permitted a better calibration of the model. A risk ratio >3.00 predicted the onset of death with sensitivity = 86%, specificity = 77%, and negative predicted value = 99%.

CONCLUSIONS

The American College of Surgeons NSQIP surgical risk calculator has proved to be a reliable predictor of adverse postoperative outcomes also in Italian emergency settings, with particular regard to mortality. We therefore recommend the use of the surgical risk calculator in the multidisciplinary care of patients undergoing emergency abdominal surgery.

Update on preoperative evaluation and optimisation

The patients presenting for surgery today often belong to the extremes of age, have multiple co-morbidities, and undergo complex surgeries. This makes them more prone to morbidity and mortality. A detailed preoperative evaluation of the patient can contribute to reducing this mortality and morbidity. There are various risk indices and validated scoring systems and many of them need to be calculated using preoperative parameters. Their key objective is to identify patients vulnerable to complications and to return them to desirable functional activity as soon as possible. Any individual undergoing surgery should be optimised preoperatively, but special considerations should be given to patients with comorbidity, on multiple drugs, and undergoing high-risk surgery. The objective of this review is to put forth the latest trends in the preoperative evaluation and optimisation of patients undergoing noncardiac surgery and emphasise the importance of risk stratification in these patients.

Validation of a risk prediction calculator in Australian patients undergoing head and neck microsurgery reconstruction

BACKGROUND

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) surgical risk calculator (SRC) is an open access calculator predicting patients' risk of postoperative complications. This study aims to assess the validity of the SRC in patients undergoing microsurgical free flap reconstruction at an Australian tertiary referral centre.

METHODS

This is a retrospective cohort study of 200 consecutive patients treated up to November 2020. SRC-predicted rates of postoperative complications and hospital length of stay (LOS) were compared to those observed for the ablative and reconstructive components of the procedure. The performance of the SRC was assessed using Brier scores, area under the receiver operating characteristic (ROC) curve (AUC), and the Hosmer-Lemeshow test.

RESULTS

For both ablative and reconstructive components, the SRC discriminates well for pneumonia and urinary tract infection, and it is calibrated well for readmission and sepsis, but it does not discriminate and calibrate well for any single outcome. SRC-predicted hospital LOS and actual LOS did not correlate well for the reconstructive component, but they correlated strongly for the ablative component.

CONCLUSIONS

The SRC is a poor predictor of postoperative complication rates and hospital LOS in patients undergoing head and neck microsurgical reconstruction.

Comparison of National Surgical Quality Improvement Program Surgical Risk Calculator, Trauma and Injury Severity Score, and American Society of Anesthesiologists Physical Status to predict operative trauma mortality in elderly patients

BACKGROUND

The Trauma and Injury Severity Score (TRISS) uses anatomical and physiologic variables to predict mortality. Elderly (65 years or older) trauma patients have increased mortality and morbidity for a given TRISS, in part because of functional status and comorbidities. These factors are incorporated into the American Society of Anesthesiologists Physical Status (ASA-PS) and National Surgical Quality Improvement Program Surgical Risk Calculator (NSQIP-SRC). We hypothesized scoring tools using comorbidities and functional status to be superior at predicting mortality, hospital length of stay (LOS), and complications in elderly trauma patients undergoing operation.

METHODS

Four level I trauma centers prospectively collected data on elderly trauma patients undergoing surgery within 24 hours of admission. Using logistic regression, five scoring models were compared: ASA-PS, NSQIP-SRC, TRISS, TRISS-ASA-PS, and TRISS-NSQIP-SRC.Brier scores and area under the receiver operator characteristics curve were calculated to compare mortality prediction. Adjusted R2 and root mean squared error were used to compare LOS and predictive ability for number of complications.

RESULTS

From 122 subjects, 9 (7.4%) died, and the average LOS was 12.9 days (range, 1-110 days). National Surgical Quality Improvement Program Surgical Risk Calculator was superior to ASA-PS and TRISS at predicting mortality (area under the receiver operator characteristics curve, 0.978 vs. 0.768 vs. 0.903; p = 0.007). Furthermore, NSQIP-SRC was more accurate predicting LOS (R2, 25.9% vs. 13.3% vs. 20.5%) and complications (R2, 34.0% vs. 22.6% vs. 29.4%) compared with TRISS and ASA-PS. Adding TRISS to NSQIP-SRC improved predictive ability compared with NSQIP-SRC alone for complications (R2, 35.5% vs. 34.0%; p = 0.046). However, adding ASA-PS or TRISS to NSQIP-SRC did not improve the predictive ability for mortality or LOS.

CONCLUSION

The NSQIP-SRC, which includes comorbidities and functional status, had superior ability to predict mortality, LOS, and complications compared with TRISS alone in elderly trauma patients undergoing surgery.

LEVEL OF EVIDENCE

Prognostic and Epidemiologic; Level III.

Inaccurate Risk Assessment by the ACS NSQIP Risk Calculator in Aortic Surgery

OBJECTIVES

The aim of this retrospective study was to assess the predictive performance of the American College of Surgeons (ACS) risk calculator for aortic aneurysm repair for the patient population of a Dutch tertiary referral hospital.

METHODS

This retrospective study included all patients who underwent elective endovascular or open aortic aneurysm repair at our institution between the years 2013 and 2019. Preoperative patient demographics and postoperative complication data were collected, and individual risk assessments were generated using five different current procedural terminology (CPT) codes. Receiver operating characteristic (ROC) curves, calibration plots, Brier scores, and Index of Prediction Accuracy (IPA) values were generated to evaluate the predictive performance of the ACS risk calculator in terms of discrimination and calibration.

RESULTS

Two hundred thirty-four patients who underwent elective endovascular or open aortic aneurysm repair were identified. Only five out of thirteen risk predictions were found to be sufficiently discriminative. Furthermore, the ACS risk calculator showed a structurally insufficient calibration. Most Brier scores were close to 0; however, comparison to a null model though IPA-scores showed the predictions generated by the ACS risk calculator to be inaccurate. Overall, the ACS risk calculator showed a consistent underestimation of the risk of complications.

CONCLUSIONS

The ACS risk calculator proved to be inaccurate within the framework of endovascular and open aortic aneurysm repair in our medical center. To minimize the effects of patient selection and cultural differences, multicenter collaboration is necessary to assess the performance of the ACS risk calculator in aortic surgery.

Risk assessment models for venous thromboembolism in hospitalised adult patients: a systematic review

INTRODUCTION

Hospital-acquired thrombosis accounts for a large proportion of all venous thromboembolism (VTE), with significant morbidity and mortality. This subset of VTE can be reduced through accurate risk assessment and tailored pharmacological thromboprophylaxis. This systematic review aimed to determine the comparative accuracy of risk assessment models (RAMs) for predicting VTE in patients admitted to hospital.

METHODS

A systematic search was performed across five electronic databases (including MEDLINE, EMBASE and the Cochrane Library) from inception to February 2021. All primary validation studies were eligible if they examined the accuracy of a multivariable RAM (or scoring system) for predicting the risk of developing VTE in hospitalised inpatients. Two or more reviewers independently undertook study selection, data extraction and risk of bias assessments using the PROBAST (Prediction model Risk Of Bias ASsessment Tool) tool. We used narrative synthesis to summarise the findings.

RESULTS

Among 6355 records, we included 51 studies, comprising 24 unique validated RAMs. The majority of studies included hospital inpatients who required medical care (21 studies), were undergoing surgery (15 studies) or receiving care for trauma (4 studies). The most widely evaluated RAMs were the Caprini RAM (22 studies), Padua prediction score (16 studies), IMPROVE models (8 studies), the Geneva risk score (4 studies) and the Kucher score (4 studies). C-statistics varied markedly between studies and between models, with no one RAM performing obviously better than other models. Across all models, C-statistics were often weak (<0.7), sometimes good (0.7-0.8) and a few were excellent (>0.8). Similarly, estimates for sensitivity and specificity were highly variable. Sensitivity estimates ranged from 12.0% to 100% and specificity estimates ranged from 7.2% to 100%.

CONCLUSION

Available data suggest that RAMs have generally weak predictive accuracy for VTE. There is insufficient evidence and too much heterogeneity to recommend the use of any particular RAM.

PROSPERO REGISTRATION NUMBER

Steve Goodacre, Abdullah Pandor, Katie Sworn, Daniel Horner, Mark Clowes. A systematic review of venous thromboembolism RAMs for hospital inpatients. PROSPERO 2020 CRD42020165778. Available from https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=165778https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=165778.

Meningioma surgical outcomes and complications in patients aged 75 years and older

OBJECTIVE

Meningioma incidence increases with age, yet limited data exist on how comorbidities impact complication rates in elderly patients undergoing meningioma resection. The objective of this study was to report surgical outcomes and identify risk factors for perioperative complications.

METHODS

We performed a retrospective study of patients 75 years and older undergoing meningioma resection. Outcomes included survival and complications. Major complications were those requiring surgical intervention or causing permanent neurological deficit. Recursive partitioning, Kaplan-Meier survival, univariate and multi-variate (MVA) analyses were performed.

RESULTS

From 1996 to 2014, 103 patients with a median age of 79 years (IQR 77-83 years) underwent cranial meningioma resection. Median follow-up was 5.8 years (IQR 1.7-8.7 years). Median actuarial survival was 10.5 years. Complications occurred in 32 patients (31.1%), and 13 patients (12.6%) had multiple complications. Major complications occurred in 16 patients (15.5%). Increasing age was not a significant predictor of any (p = 0.6408) or major complication (p = 0.8081). On univariate analysis, male sex, Charlson Comorbidity Index greater than 8, and cardiovascular comorbidities were significantly associated with major complications. On MVA only cardiovascular comorbidities (OR 3.94, 95% CI 1.05-14.76, p = 0.0238) were significantly associated with any complication. All patients with major complications had cardiovascular comorbidities, and on MVA male gender (OR 3.78, 95%CI 1.20-11.93, p = 0.0212) was associated with major complications.

CONCLUSIONS

Cardiovascular comorbidities and male gender are significant risk factors for complications after meningioma resection in patients aged 75 years and older. While there is morbidity associated with meningioma resection in this cohort, there is also excellent long-term survival.

Leveraging Decision Curve Analysis to Improve Clinical Application of Surgical Risk Calculators

BACKGROUND

Surgical risk calculators (SRCs) have been developed for estimation of postoperative complications but do not directly inform decision-making. Decision curve analysis (DCA) is a method for evaluating prediction models, measuring their utility in guiding decisions. We aimed to analyze the utility of SRCs to guide both preoperative and postoperative management of patients undergoing hepatopancreaticobiliary surgery by using DCA.

METHODS

A single-institution, retrospective review of patients undergoing hepatopancreaticobiliary operations between 2015 and 2017 was performed. Estimation of postoperative complications was conducted using the American College of Surgeons SRC [ACS-SRC] and the Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) calculator; risks were compared with observed outcomes. DCA was used to model optimal patient selection for risk prevention strategies and to compare the relative performance of the ACS-SRC and POTTER calculators.

RESULTS

A total of 994 patients were included in the analysis. C-statistics for the ACS-SRC prediction of 12 postoperative complications ranged from 0.546 to 0.782. DCA revealed that an ACS-SRC-guided readmission prevention intervention, when compared with an all-or-none approach, yielded a superior net benefit for patients with estimated risk between 5% and 20%. Comparison of SRCs for venous thromboembolism intervention demonstrated superiority of the ACS-SRC for thresholds for intervention between 2% and 4% with the POTTER calculator performing superiorly between 4% and 8% estimated risk.

CONCLUSIONS

SRCs can be used not only to predict complication risk but also to guide risk prevention strategies. This methodology should be incorporated into external validations of future risk calculators and can be applied for institution-specific quality improvement initiatives to improve patient outcomes.

Perioperative Morbidity and Mortality of Patients With COVID-19 Who Undergo Urgent and Emergent Surgical Procedures

Supplemental Digital Content is available in the text

To evaluate the perioperative morbidity and mortality of patients with COVID-19 who undergo urgent and emergent surgery.

Ensemble machine learning for the prediction of patient-level outcomes following thyroidectomy

BACKGROUND

Accurate prediction of thyroidectomy complications is necessary to inform treatment decisions. Ensemble machine learning provides one approach to improve prediction.

METHODS

We applied the Super Learner (SL) algorithm to the 2016-2018 thyroidectomy-specific NSQIP database to predict complications following thyroidectomy. Cross-validation was used to assess model discrimination and precision.

RESULTS

For the 17,987 patients undergoing thyroidectomy, rates of recurrent laryngeal nerve injury, post-operative hypocalcemia prior to discharge or within 30 days, and neck hematoma were 6.1%, 6.4%, 9.0%, and 1.8%, respectively. SL improved prediction of thyroidectomy-specific outcomes when compared with benchmark logistic regression approaches. For postoperative hypocalcemia prior to discharge, SL improved the cross-validated AUROC to 0.72 (95%CI 0.70-0.74) compared to 0.70 (95%CI 0.68-0.72; p < 0.001) when using a manually curated logistic regression algorithm.

CONCLUSION

Ensemble machine learning modestly improves prediction for thyroidectomy-specific outcomes. SL holds promise to provide more accurate patient-level risk prediction to inform treatment decisions.

Risk-adjustment models in patients undergoing head and neck surgery with reconstruction

BACKGROUND

With the current focus on value-based outcomes and reimbursement models, perioperative risk adjustment is essential. Specialty surgical outcomes are not well predicted by the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP); the Head and Neck-Reconstructive Surgery NSQIP was created as a specialty-specific platform for patients undergoing head and neck surgery with flap reconstruction. This study aims to investigate risk prediction models in these patients.

METHODS

The Head and Neck-Reconstructive Surgery NSQIP collected data on patients undergoing head and neck surgery with flap reconstruction from August 1, 2012 to October 20, 2016. Multivariable logistic regression models were created for 9 outcomes (postoperative ventilator dependence, pneumonia, superficial recipient surgical site infection, presence of tracheostomy/nasoenteric (NE)/gastrostomy/gastrojejunostomy(G/GJ) tube 30 days postoperatively, conversion from NE to G/GJ tube, unplanned return to the operating room, length of stay > 7 days). External validation was completed with a more contemporary cohort.

RESULTS

A total of 1095 patients were included in the modelling cohort and 407 in the validation cohort. Models performed well predicting tracheostomy, NE, G/GJ tube presence at 30 days postoperatively and conversion from NE to G/GJ tube (c-indices = 0.75-0.91). Models for postoperative pneumonia, superficial recipient surgical site infection, ventilator dependence > 48 h, and length of stay > 7 days were fair (concordance [c]-indices = 0.63-0.69). The predictive model for unplanned return to the operating room was poor (c-index = 0.58).

CONCLUSIONS AND RELEVANCE

Reliable and discriminant risk prediction models were able to be created for postoperative outcomes using the specialty-specific Head and Neck-Reconstructive Surgery Specific NSQIP.

Predicting morbidity and mortality after radical cystectomy using risk calculators: A comprehensive review of the literature

INTRODUCTION

Radical cystectomy (RC) with urinary diversion is associated with significant perioperative morbidity and mortality, varying between 30% and 70% and between 0.3% and 10.6%, respectively. Risk calculators have been extensively studied in the general surgery literature to predict 30- and 90-day postoperative morbidity and mortality but have not been widely accepted in the RC literature.

MATERIALS AND METHODS

We performed a search of MEDLINE and Embase databases during May 2020 to identify all relevant studies using the following keywords: radical cystectomy, surgical complication predictive model, surgical complication predictive equation, surgical complication predictive nomogram, surgical risk calculator, morbidity, and mortality. We determined the existing surgical predictive nomograms, calculators, and indices and their accuracy in predicting morbidity, mortality, and major complications after RC.

RESULTS

National Surgical Quality Improvement Program had poor accuracy at predicting 30-day morbidity at mortality (AUC 0.5-0.6). LACE index showed good discrimination at predicting 90-day mortality (AUC 0.7). The various frailty and sarcopenia indices have shown poor to fair accuracy at predicting (AUC 0.5-0.7). The Isbarn and Aziz nomograms have equivalent accuracy at predicting 90-day mortality (AUC 0.7) but are limited by inclusion of tumor histology and presence of metastatic disease as variables. POSSUM and P-POSSUM have poor ability at predicting morbidity and mortality (AUC 0.5) and are cumbersome calculators. The surgical Apgar score has been able to predict 30-day morbidity and mortality but can only be used in the postoperative setting.

DISCUSSION

The currently available surgical risk calculators have either poor accuracy at predicting post-RC morbidity and mortality or are limited by types of variables included. An ideal risk calculator would be comprised of preoperative factors only and have a high accuracy to serve as a tool for preoperative patient counseling prior to surgery.

CONCLUSION

There exists a strong need to develop a comprehensive and accurate preoperative risk calculator that predicts morbidity and mortality after RC.

Development and Validation of a Multivariate Prediction Model of Perioperative Mortality in Neurosurgery: The New Zealand Neurosurgical Risk Tool (NZRISK-NEURO)

BACKGROUND

Multivariate risk prediction models individualize prediction of adverse outcomes, assisting perioperative decision-making. There are currently no models specifically designed for the neurosurgical population.

OBJECTIVE

To develop and validate a neurosurgical risk prediction model, with 30-d, 1-yr, and 2-yr mortality endpoints.

METHODS

We accessed information on all adults in New Zealand who underwent neurosurgery or spinal surgery between July 1, 2011, and June 30, 2016, from an administrative database. Our dataset comprised of 18 375 participants, split randomly into derivation (75%) and validation (25%) datasets. Previously established covariates tested included American Society of Anesthesiologists physical status grade (ASA-PS), surgical acuity, operative severity, cancer status, and age. Exploratory covariates included anatomical site, gender, diabetes, trauma, ethnicity, and socioeconomic status. Least absolute shrinkage and selection operator (LASSO) regression analysis was used to construct 30-d, 1-yr, and 2-yr mortality models.

RESULTS

Our final models included 8 covariates: age, ASA-PS grade, surgical acuity, cancer status, anatomical site, diabetes, ethnicity, and trauma. The area under the receiver operating curve for the 30-d, 1-yr, and 2-yr mortality models was 0.90, 0.91, and 0.91 indicating excellent discrimination, respectively. Calibration also showed excellent performance with McFadden's pseudo R2 statistics of 0.28, 0.37, and 0.41 and calibration plot slopes of 0.93, 0.95, and 0.94, respectively. The strongest predictors of mortality were ASA-PS 4 and 5 (30 d) and cancer (1 and 2 yr).

CONCLUSION

NZRISK-NEURO is a robust multivariate calculator created specifically for neurosurgery, enabling physicians to generate data-driven individualized risk estimates, assisting shared decision-making and perioperative planning.

Is the ACS-NSQIP Risk Calculator Accurate in Predicting Adverse Postoperative Outcomes in the Emergency Setting? An Italian Single-center Preliminary Study

Background

The ACS-NSQIP surgical risk calculator (SRC) is an open-access online tool that estimates the chance for adverse postoperative outcomes. The risk is estimated based on 21 patient-related variables and customized for specific surgical procedures. The purpose of this monocentric retrospective study is to validate its predictive value in an Italian emergency setting.

Methods

From January to December 2018, 317 patients underwent surgical procedures for acute cholecystitis (n = 103), appendicitis (n = 83), gastrointestinal perforation (n = 45), and intestinal obstruction (n = 86). Patients’ personal risk was obtained and divided by the average risk to calculate a personal risk ratio (RR). Areas under the ROC curves (AUC) and Brier score were measured to assess both the discrimination and calibration of the predictive model.

Results

The AUC was 0.772 (95%CI 0.722–0.817, p < 0.0001; Brier 0.161) for serious complications, 0.887 (95%CI 0.847–0.919, p < 0.0001; Brier 0.072) for death, and 0.887 (95%CI 0.847–0.919, p < 0.0001; Brier 0.106) for discharge to nursing or rehab facility. Pneumonia, cardiac complications, and surgical site infection presented an AUC of 0.794 (95%CI 0.746–0.838, p < 0.001; Brier 0.103), 0.836 (95%CI 0.790–0.875, p < 0.0001; Brier 0.081), and 0.729 (95%CI 0.676–0.777, p < 0.0001; Brier 0.131), respectively. A RR > 1.24, RR > 1.52, and RR > 2.63 predicted the onset of serious complications (sensitivity = 60.47%, specificity = 64.07%; NPV = 81%), death (sensitivity = 82.76%, specificity = 62.85%; NPV = 97%), and discharge to nursing or rehab facility (sensitivity = 80.00%, specificity = 69.12%; NPV = 95%), respectively.

Conclusions

The calculator appears to be accurate in predicting adverse postoperative outcomes in our emergency setting. A RR cutoff provides a much more practical method to forecast the onset of a specific type of complication in a single patient.

Assessment of the American College of Surgeons National Surgical Quality Improvement Program Calculator in Predicting Outcomes and Length of Stay After Ivor Lewis Esophagectomy: A Single-Center Experience

BACKGROUND

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) calculator is a useful tool used by physicians to better inform patients on the surgical risk of postoperative complications. It makes use of the NSQIP database to derive the chance for several adverse outcomes to occur postoperatively given certain patient's factors. The aim of this study was to assess its applicability in a series of patients undergoing an Ivor Lewis esophagectomy.

METHODS

Data from 100 consecutive patients who underwent an Ivor Lewis esophagectomy between September 2013 and November 2017 at our institution were reviewed. Each patient was assessed using the ACS NSQIP surgical risk calculator. Actual events in this group were compared with their particular NSQIP-assessed risk. Logistic regression models were used to compare surgical risk calculator estimates binary outcomes such as incidence of postoperative complications such as cardiac events, renal events, surgical site infection, and death. Mixed linear model was used for length of stay (LOS) duration versus observed LOS. C-statistic was for predictive accuracy each binary outcome and intraclass correlation was used for LOS.

RESULTS

C-statistic values were higher than the cutoff (0.75) for surgical site infection and death. The ACS NSQIP risk calculator was poorly predictive of other reported outcomes by the calculator such as cardiac or renal complications. Corroboration between observed LOS and predicted LOS was weak (8 d versus 11 d, respectively, intraclass coefficient 0.04).

CONCLUSIONS

This study suggests that the risk calculator is useful for identifying risk of death or surgical site infection but poor at discriminating likelihood of other reported outcomes occurring, such as pneumonia, acute renal failure and cardiac complications for patients who underwent an Ivor Lewis esophagectomy. Estimations for procedure-specific complications for esophagectomy may need reevaluated.

Development of a Risk Prediction Model With Improved Clinical Utility in Elective Cervical and Lumbar Spine Surgery

STUDY DESIGN

Retrospective cohort.

OBJECTIVE

We present a universal model of risk prediction for patients undergoing elective cervical and lumbar spine surgery.

SUMMARY OF BACKGROUND DATA

Previous studies illustrate predictive risk models as possible tools to identify individuals at increased risk for postoperative complications and high resource utilization following spine surgery. Many are specific to one condition or procedure, cumbersome to calculate, or include subjective variables limiting applicability and utility.

METHODS

A retrospective cohort of 177,928 spine surgeries (lumbar (L) Ln = 129,800; cervical (C) Cn = 48,128) was constructed from the 2012 to 2016 American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) database. Cases were identified by Current Procedural Terminology (CPT) codes for cervical fusion, lumbar fusion, and lumbar decompression laminectomy. Significant preoperative risk factors for postoperative complications were identified and included in logistic regression. Sum of odds ratios from each factor was used to develop the Universal Spine Surgery (USS) score. Model performance was assessed using receiver-operating characteristic (ROC) curves and tested on 20% of the total sample.

RESULTS

Eighteen risk factors were identified, including sixteen found to be significant outcomes predictors. At least one complication was present among 11.1% of patients, the most common of which included bleeding requiring transfusion (4.86%), surgical site infection (1.54%), and urinary tract infection (1.08%). Complication rate increased as a function of the model score and ROC area under the curve analyses demonstrated fair predictive accuracy (lumbar = 0.741; cervical = 0.776). There were no significant deviations between score development and testing datasets.

CONCLUSION

We present the Universal Spine Surgery score as a robust, easily administered, and cross-validated instrument to quickly identify spine surgery candidates at increased risk for postoperative complications and high resource utilization without need for algorithmic software. This may serve as a useful adjunct in preoperative patient counseling and perioperative resource allocation.

LEVEL OF EVIDENCE

3.

The Japan Neurosurgical Database: Overview and Results of the First-year Survey

The Japan Neurosurgical Database (JND) is a prospective observational study registry established in 2017 by the Japan Neurosurgical Society (JNS) to visualize real-world clinical practice, promote science, and improve the quality of care and neurosurgery board certification in Japan. We summarize JND’s aims and methods, and describes the 2018 survey results. The JND registered in-hospital patients’ clinical data mainly from JNS training institutions in 2018. Caseload, patient demographics, and in-hospital outcomes of the overall cohort and a neurosurgical subgroup were examined according to major classifications of main diagnosis. Neurosurgical caseload per neurosurgeon in training in core hospitals in 2018 was calculated as an indicator of neurosurgical training. Of 523,283 cases (male 55.3%) registered from 1360 participating institutions, the neurosurgical subgroup comprised of 33.9%. Among the major classifications, cerebrovascular diseases comprised the largest proportion overall and in the neurosurgical subgroup (53.1%, 41.0%, respectively), followed by neurotrauma (19.1%, 25.5%), and brain tumor (10.4%, 12.8%). Functional neurosurgery (6.4%, 3.7%), spinal and peripheral nerve disorders (5.1%, 10.1%), hydrocephalus/developmental anomalies (2.9%, 5.3%), and encephalitis/infection/inflammatory and miscellaneous diseases (2.9%, 1.6%) comprised smaller proportions. Most patients were aged 70–79 years in the overall cohort and neurosurgical subgroup (27.8%, 29.4%). Neurotrauma and cerebrovascular diseases in the neurosurgical subgroup comprised a higher and lower proportion, respectively, than in the overall cohort in elderly patients (e.g. 80 years, 46.9% vs. 33.5%, 26.8% vs. 54.4%). The 2018 median neurosurgical caseload per neurosurgeon in training was 80.7 (25–75th percentile 51.5–117.5). These initial results from 2018 reveal unique aspects of neurosurgical practice in Japan.

Correlation of perioperative risk scores with hospital costs in neurosurgical patients

OBJECTIVE

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) online surgical risk calculator uses inherent patient characteristics to provide predictive risk scores for adverse postoperative events. The purpose of this study was to determine if predicted perioperative risk scores correlate with actual hospital costs.

METHODS

A single-center retrospective review of 1005 neurosurgical patients treated between September 1, 2011, and December 31, 2014, was performed. Individual patient characteristics were entered into the NSQIP calculator. Predicted risk scores were compared with actual in-hospital costs obtained from a billing database. Correlational statistics were used to determine if patients with higher risk scores were associated with increased in-hospital costs.

RESULTS

The Pearson correlation coefficient (R) was used to assess the correlation between 11 types of predicted complication risk scores and 5 types of encounter costs from 1005 health encounters involving neurosurgical procedures. Risk scores in categories such as any complication, serious complication, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, return to operating room, death, and discharge to nursing home or rehabilitation facility were obtained. Patients with higher predicted risk scores in all measures except surgical site infection were found to have a statistically significant association with increased actual in-hospital costs (p < 0.0005).

CONCLUSIONS

Previous work has demonstrated that the ACS NSQIP surgical risk calculator can accurately predict mortality after neurosurgery but is poorly predictive of other potential adverse events and clinical outcomes. However, this study demonstrates that predicted high-risk patients identified by the ACS NSQIP surgical risk calculator have a statistically significant moderate correlation to increased actual in-hospital costs. The NSQIP calculator may not accurately predict the occurrence of surgical complications (as demonstrated previously), but future iterations of the ACS universal risk calculator may be effective in predicting actual in-hospital costs, which could be advantageous in the current value-based healthcare environment.

Predicting nonroutine discharge after elective spine surgery: external validation of machine learning algorithms

OBJECTIVE

Nonroutine discharge after elective spine surgery increases healthcare costs, negatively impacts patient satisfaction, and exposes patients to additional hospital-acquired complications. Therefore, prediction of nonroutine discharge in this population may improve clinical management. The authors previously developed a machine learning algorithm from national data that predicts risk of nonhome discharge for patients undergoing surgery for lumbar disc disorders. In this paper the authors externally validate their algorithm in an independent institutional population of neurosurgical spine patients.

METHODS

Medical records from elective inpatient surgery for lumbar disc herniation or degeneration in the Transitional Care Program at Brigham and Women's Hospital (2013-2015) were retrospectively reviewed. Variables included age, sex, BMI, American Society of Anesthesiologists (ASA) class, preoperative functional status, number of fusion levels, comorbidities, preoperative laboratory values, and discharge disposition. Nonroutine discharge was defined as postoperative discharge to any setting other than home. The discrimination (c-statistic), calibration, and positive and negative predictive values (PPVs and NPVs) of the algorithm were assessed in the institutional sample.

RESULTS

Overall, 144 patients underwent elective inpatient surgery for lumbar disc disorders with a nonroutine discharge rate of 6.9% (n = 10). The median patient age was 50 years and 45.1% of patients were female. Most patients were ASA class II (66.0%), had 1 or 2 levels fused (80.6%), and had no diabetes (91.7%). The median hematocrit level was 41.2%. The neural network algorithm generalized well to the institutional data, with a c-statistic (area under the receiver operating characteristic curve) of 0.89, calibration slope of 1.09, and calibration intercept of -0.08. At a threshold of 0.25, the PPV was 0.50 and the NPV was 0.97.

CONCLUSIONS

This institutional external validation of a previously developed machine learning algorithm suggests a reliable method for identifying patients with lumbar disc disorder at risk for nonroutine discharge. Performance in the institutional cohort was comparable to performance in the derivation cohort and represents an improved predictive value over clinician intuition. This finding substantiates initial use of this algorithm in clinical practice. This tool may be used by multidisciplinary teams of case managers and spine surgeons to strategically invest additional time and resources into postoperative plans for this population.

A Systems Approach to Design and Implementation of Patient Assessment Tools in the Inpatient Setting

Failure to rescue events, or events involving preventable deaths from complications, are a significant contributor to inpatient mortality. While many interventions have been designed and implemented over several decades, this patient safety issue remains at the forefront of concern for most hospitals. In the first part of this study, the development and implementation of one type of highly studied and widely adopted rescue intervention, algorithm-based patient assessment tools, is examined. The analysis summarizes how a lack of systems-oriented approaches in the design and implementation of these tools has resulted in suboptimal understanding of patient risk of mortality and complications and the early recognition of patient deterioration. The gaps identified impact several critical aspects of excellent patient care, including information-sharing across care settings, support for the development of shared mental models within care teams, and access to timely and accurate patient information. This chapter describes the use of several system-oriented design and implementation activities to establish design objectives, model clinical processes and workflows, and create an extensible information system model to maximize the benefits of patient state and risk assessment tools in the inpatient setting. A prototype based on the product of the design activities is discussed along with system-level considerations for implementation. This study also demonstrates the effectiveness and impact of applying systems design principles and practices to real-world clinical applications.

Quality of British and American Nationwide Quality of Care and Patient Safety Benchmarking Programs: Case Neurosurgery

BACKGROUND

Multiple nationwide outcome registries are utilized for quality benchmarking between institutions and individual surgeons.

OBJECTIVE

To evaluate whether nationwide quality of care programs in the United Kingdom and United States can measure differences in neurosurgical quality.

METHODS

This prospective observational study comprised 418 consecutive adult patients undergoing elective craniotomy at Helsinki University Hospital between December 7, 2011 and December 31, 2012.We recorded outcome event rates and categorized them according to British Neurosurgical National Audit Programme (NNAP), American National Surgical Quality Improvement Program (NSQIP), and American National Neurosurgery Quality and Outcomes Database (N2QOD) to assess the applicability of these programs for quality benchmarking and estimated sample sizes required for reliable quality comparisons.

RESULTS

The rate of in-hospital major and minor morbidity was 18.7% and 38.0%, respectively, and 30-d mortality rate was 2.4%. The NSQIP criteria identified 96.2% of major but only 38.4% of minor complications. N2QOD performed better, but almost one-fourth (23.2%) of all patients with adverse outcomes, mostly minor, went unnoticed. For NNAP, a sample size of over 4200 patients per surgeon is required to detect a 50.0% increase in mortality rates between surgeons. The sample size required for reliable comparisons between the rates of complications exceeds 600 patients per center per year.

CONCLUSION

The implemented benchmarking programs in the United Kingdom and United States fail to identify a considerable number of complications in a high-volume center. Health care policy makers should be cautious as outcome comparisons between most centers and individual surgeons are questionable if based on the programs.

Reporting Methodology of Neurosurgical Studies Utilizing the American College of Surgeons-National Surgical Quality Improvement Program Database: A Systematic Review and Critical Appraisal

BACKGROUND

Use of large databases such as the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) has become increasingly common in neurosurgical research.

OBJECTIVE

To perform a critical appraisal and evaluation of the methodological reporting for studies in neurosurgical literature that utilize the ACS-NSQIP database.

METHODS

We queried Ovid MEDLINE, EMBASE, and PubMed databases for all neurosurgical studies utilizing the ACS-NSQIP. We assessed each study according to number of criteria fulfilled with respect to Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement, REporting of studies Conducted using Observational Routinely-collected Health Data (RECORD) Statement, and Journal of American Medical Association–Surgical Section (JAMA-Surgery) Checklist. A separate analysis was conducted among papers published in core and noncore journals in neurosurgery according to Bradford's law.

RESULTS

A total of 117 studies were included. Median (interquartile range [IQR]) scores for number of fulfilled criteria for STROBE Statement, RECORD Statement, and JAMA-Surgery Checklist were 20 (IQR:19-21), 9 (IQR:8-9), and 6 (IQR:5-6), respectively. For STROBE Statement, RECORD Statement, and JAMA-Surgery Checklist, item 9 (potential sources of bias), item 13 (supplemental information), and item 9 (missing data/sensitivity analysis) had the highest number of studies with no fulfillment among all studies (56, 68, 50%), respectively. When comparing core journals vs noncore journals, no significant difference was found (STROBE, P = .94; RECORD, P = .24; JAMA-Surgery checklist, P = .60).

CONCLUSION

While we observed an overall satisfactory reporting of methodology, most studies lacked mention of potential sources of bias, data cleaning methods, supplemental information, and external validity. Given the pervasive role of national databases and registries for research and health care policy, the surgical community needs to ensure the credibility and quality of such studies that ultimately aim to improve the value of surgical care delivery to patients.

Accuracy of the NSQIP risk calculator for predicting complications following adrenalectomy

PURPOSE

Adrenalectomy is performed to treat functional pathology and remove tumors of malignant concern. The National Surgical Quality Improvement Program (NSQIP) risk calculator predicts 30-day complications and length of stay following index surgical procedures. We assess whether this tool accurately predicts complications following adrenalectomy procedures at a tertiary care academic medical center.

METHODS

A retrospective review was performed for all adrenalectomies at a single institution from 2004 to 2016. 197 patients underwent adrenalectomy without concurrent resections. Predicted risk for NSQIP complications was calculated for each patient. The mean predicted and observed risks (%) at 30 days across all patients within each category were determined, and these were compared with two-sided one-sample t tests.

RESULTS

Of 197 adrenalectomies, 180 were laparoscopic and 17 were open. For laparoscopic adrenalectomy, ten (5.5%) complications were observed including nine (5%) graded Clavien III or greater. All observed complication rates were significantly different than predicted (p values for all < 0.005). Mean observed length of stay was also significantly less than predicted (1.6 versus 2.1 days, p < 0.001). In the open adrenalectomy subgroup, there were no observed complications with observed mean length of stay equivalent to predicted (5.8 versus 5.3, p = 0.08) without a higher readmission rate (5.9 versus 6.0%).

CONCLUSIONS

Statistical differences were noted between the actual complication rates of adrenalectomy versus those predicted by the NSQIP calculator. Certain observed differences may not necessarily have clinical significance. Urology procedure-specific calculators may better refine predictions for sub-specialty procedures with future work requisite to determine performance across all practice settings.

Validation of the American College of Surgeons National Surgical Quality Improvement Program Risk Model for Patients Undergoing Panniculectomy

Panniculectomy procedures have been reported to significantly improve quality of life, increase mobility, and improve hygiene in patients with a significant pannus formation. The primary aims of this study were to determine which preoperative risk factors may be used to differentiate postoperative complication rate among patient cohorts and to validate utilization of the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) risk calculator in patients undergoing panniculectomies.

METHODS

This retrospective study included all patients who underwent a panniculectomy procedure at our institution from 2005 to 2016. Baseline characteristics, preoperative risk factors, medical comorbidities, and postoperative complications were collected via retrospective chart review.

RESULTS

Two hundred sixty-four patients who underwent a panniculectomy were identified. The odds ratios of any postoperative complication were 8.26, 7.76, and 16.6 for patients with classes 1, 2, and 3 obesity, respectively (P < 0.05). Statistical modeling was utilized to evaluate the predictive performance of the ACS-NSQIP Surgical Risk Calculator. We calculated the C-statistic for the ACS-NSQIP model to be only 0.61, indicating that although the model is associated with the risk of complication, it does not have a strong predictive value for this particular procedure.

DISCUSSION

This study is one of the first to characterize postoperative complication rate based on extremum of body mass index for panniculectomy patients. Our results show that the utilization of the ACS-NSQIP Risk Calculator in this particular patient population underestimates the complication risk as a whole, which may necessitate the future development of a separate risk assessment model for this procedure.

Predicting complications of major head and neck oncological surgery: an evaluation of the ACS NSQIP surgical risk calculator

Background

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) universal surgical risk calculator is an online tool intended to improve the informed consent process and surgical decision-making. The risk calculator uses a database of information from 585 hospitals to predict a patient’s risk of developing specific postoperative outcomes.

Methods

Patient records at a major Canadian tertiary care referral center between July 2015 and March 2017 were reviewed for surgical cases including one of six major head and neck oncologic surgeries: total thyroidectomy, total laryngectomy, hemiglossectomy, partial glossectomy, laryngopharyngectomy, and composite resection. Preoperative information for 107 patients was entered into the risk calculator and compared to observed postoperative outcomes. Statistical analysis of the risk calculator was completed for the entire study population, for stratification by procedure, and by utilization of microvascular reconstruction. Accuracy was assessed using the ratio of predicted to observed outcomes, Receiver Operating Characteristics (ROC), Brier score, and the Wilcoxon signed–ranked test.

Results

The risk calculator accurately predicted the incidences for 11 of 12 outcomes for patients that did not undergo free flap reconstruction (NFF group), but was less accurate for patients that underwent free flap reconstruction (FF group). Length of stay (LOS) analysis showed similar results, with predicted and observed LOS statistically different in the overall population and FF group analyses (p = 0.001 for both), but not for the NFF group analysis (p = 0.764). All outcomes in the NFF group, when analyzed for calibration, met the threshold value (Brier scores < 0.09). Risk predictions for 8 of 12, and 10 of 12 outcomes were adequately calibrated in the FF group and the overall study population, respectively. Analyses by procedure were excellent, with the risk calculator showing adequate calibration for 7 of 8 procedural categories and adequate discrimination for all calculable categories (6 of 6).

Conclusion

The NSQIP-RC demonstrated efficacy for predicting postoperative complications in head and neck oncology surgeries that do not require microvascular reconstruction. The predictive value of the metric can be improved by inclusion of several factors important for risk stratification in head and neck oncology.

The American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator Has a Role in Predicting Discharge to Post-Acute Care in Total Joint Arthroplasty

BACKGROUND

Patient demand and increasing cost awareness have led to the creation of surgical risk calculators that attempt to predict the likelihood of adverse events and to facilitate risk mitigation. The American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator is an online tool available for a wide variety of surgical procedures, and has not yet been fully evaluated in total joint arthroplasty.

METHODS

A single-center, retrospective review was performed on 909 patients receiving a unilateral primary total knee (496) or hip (413) arthroplasty between January 2012 and December 2014. Patient characteristics were entered into the risk calculator, and predicted outcomes were compared with observed results. Discrimination was evaluated using the receiver-operator area under the curve (AUC) for 90-day readmission, return to operating room (OR), discharge to skilled nursing facility (SNF)/rehab, deep venous thrombosis (DVT), and periprosthetic joint infection (PJI).

RESULTS

The risk calculator demonstrated adequate performance in predicting discharge to SNF/rehab (AUC 0.72). Discrimination was relatively limited for DVT (AUC 0.70, P = .2), 90-day readmission (AUC 0.63), PJI (AUC 0.67), and return to OR (AUC 0.59). Risk score differences between those who did and did not experience discharge to SNF/rehab, 90-day readmission, and PJI reached significance (P < .01). Predicted length of stay performed adequately, only overestimating by 0.2 days on average (rho = 0.25, P < .001).

CONCLUSION

The American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator has fair utility in predicting discharge to SNF/rehab, but limited usefulness for 90-day readmission, return to OR, DVT, and PJI. Although length of stay predictions are similar to actual outcomes, statistical correlation remains relatively weak.