Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

PURPOSE

Preoperative prediction of postoperative complications (PCs) in inpatients with cancer is challenging. We developed an explainable machine learning (ML) model to predict PCs in a heterogenous population of inpatients with cancer undergoing same-hospitalization major operations.

METHODS

Consecutive inpatients who underwent same-hospitalization operations from December 2017 to June 2021 at a single institution were retrospectively reviewed. The ML model was developed and tested using electronic health record (EHR) data to predict 30-day PCs for patients with Clavien-Dindo grade 3 or higher (CD 3+) per the CD classification system. Model performance was assessed using area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), and calibration plots. Model explanation was performed using the Shapley additive explanations (SHAP) method at cohort and individual operation levels.

RESULTS

A total of 988 operations in 827 inpatients were included. The ML model was trained using 788 operations and tested using a holdout set of 200 operations. The CD 3+ complication rates were 28.6% and 27.5% in the training and holdout test sets, respectively. Training and holdout test sets' model performance in predicting CD 3+ complications yielded an AUROC of 0.77 and 0.73 and an AUPRC of 0.56 and 0.52, respectively. Calibration plots demonstrated good reliability. The SHAP method identified features and the contributions of the features to the risk of PCs.

CONCLUSION

We trained and tested an explainable ML model to predict the risk of developing PCs in patients with cancer. Using patient-specific EHR data, the ML model accurately discriminated the risk of developing CD 3+ complications and displayed top features at the individual operation and cohort level.

Preoperative Risk Prediction for Pancreatectomy: A Comparative Analysis of Three Scoring Systems

INTRODUCTION

Pancreatectomy is associated with high morbidity and mortality. Therefore, patient selection and risk prediction is paramount. In this study, three validated perioperative risk scoring systems were compared among patients undergoing pancreatectomy to identify the most clinically useful model.

MATERIALS AND METHODS

The 2014-2017 American College of Surgeons National Surgical Quality Improvement Program database was queried for pancreatectomy patients. Three models were evaluated: National Surgical Quality Improvement Program Universal Risk Calculator (URC), Model for End-Stage Liver Disease (MELD), and Modified Frailty Index-5 Factor (mFI-5). Outcomes were 30-d mortality and complications. Predictive performance of the models was compared using area under the receiver operating characteristic curve (AUC) and Brier scores.

RESULTS

Twenty two thousand one hundred twenty three pancreatectomy patients were identified. The 30-d mortality rate was 1.4% (n = 319). Complications occurred in 6020 cases (27.2%). AUC (95% CI) for 30-d mortality were 0.70 (0.67-0.73), 0.63 (0.60-0.67), and 0.60 (0.57-0.63) for URC, MELD, and mFI-5, respectively, with Brier score of 0.014 for all three models. AUC (95% confidence interval) for any complication was 0.59 (0.58-0.59) for URC, 0.53 (0.52-0.54) for MELD, and 0.53 (0.52-0.54) for mFI-5, with Brier scores 0.193 (URC), 0.200 (MELD), and 0.197 (mFI-5). For individual complications, URC was more predictive than MELD or mFI-5.

CONCLUSIONS

Of the validated preoperative risk scoring systems, URC was most predictive of both complications and 30-d mortality. None of the models performed better than fair to good. The lack of predictive accuracy of currently existing models highlights the need for development of improved perioperative risk models.

Evaluation of complications after laparoscopic and open appendectomy by the American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator

BACKGROUND

This study aims to evaluate the predictive level of the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) risk calculator for post-appendectomy complications.

METHODS

A total of 292 patients who were hospitalized for general appendectomy were included in the study. The age range of the patients was 18-76 years (mean: 35.3±13.6 years). The mean body mass index was 25.8±4.6. Twenty data points were entered into the ACS-NSQIP surgical risk calculator (SRC), which yielded the 17 most common complications and the average LOHS. Compli-cations encountered in 30-day follow-up were categorized according to the complications predicted by SRC. The actual and observed complication rates and LOHS were compared RESULTS: Post-operative complications developed in 13.4% of the patients, surgical site infection in 11.3%, serious complications in 3.1%, and readmission in 2.1%. Serious complications included pneumonia, sepsis, cardiac complications, and renal failure. The mean LOHS was 1.91±1.64 days (range: 1-14 days). No thromboembolism or mortality was observed. When the comparison of compli-cations using SRC was made with the ROC curve, the predictive value of SRC was 84.2% for any complication, 86.7% for serious complication, 47.6% for surgical site infection, 95.9% for renal failure, 99.0% for resurgery, and 88.3% for sepsis.

CONCLUSION

Although it is rare to see complications after simple appendectomy, it is known that complication rates increase sig-nificantly in the elderly, the obese, and those with comorbidities. Tools such as SRC will be beneficial for patients with these risk factors.

Resection of Non-Functional Pancreatic Neuroendocrine Neoplasms-A Single-Center Retrospective Outcome Analysis

Surgery remains the only curative treatment of pancreatic neuroendocrine neoplasms (pNEN). Here, we report the outcome after surgery for non-functional pNEN at a European Neuroendocrine Tumor Society (ENETS) center in Germany between 2000 and 2019; cases were analyzed for surgical (Clavien–Dindo classification; CDc) and oncological outcomes. Forty-nine patients (tumor grading G1 n = 25, G2 n = 22, G3 n = 2), with a median age of 56 years, were included. Severe complications (CDc ≥ grade 3b) occurred in 11 patients (22.4%) and type B/C pancreatic fistulas (POPFs) occurred in 5 patients (10.2%); in-hospital mortality was 2% (n = 1). Six of seven patients with tumor recurrence (14.3%) had G2 tumors in the pancreatic body/tail. The median survival was 5.7 years (68 months; [1–228 months]). Neither the occurrence (p = 0.683) nor the severity of complications had an influence on the relapse behavior (p = 0.086). This also applied for a POPF (≥B, p = 0.609). G2 pNEN patients (n = 22) with and without tumor recurrence had similar median tumor sizes (4 cm and 3.9 cm, respectively). Five of the six relapsed G2 patients (83.3%) had tumor-positive lymph nodes (N+); all G2 pNEN patients with recurrence had initially been treated with distal pancreatic resection. Pancreatic resections for pNEN are safe but associated with relevant postoperative morbidity. Future studies are needed to evaluate suitable resection strategies for G2 pNEN.

Machine learning predicts unpredicted deaths with high accuracy following hepatopancreatic surgery

Background

Machine learning to predict morbidity and mortality-especially in a population traditionally considered low risk-has not been previously examined. We sought to characterize the incidence of death among patients with a low estimated morbidity and mortality risk based on the National Surgical Quality Improvement Program (NSQIP) estimated probability (EP), as well as develop a machine learning model to identify individuals at risk for "unpredicted death" (UD) among patients undergoing hepatopancreatic (HP) procedures.

Methods

The NSQIP database was used to identify patients who underwent elective HP surgery between 2012-2017. The risk of morbidity and mortality was stratified into three tiers (low, intermediate, or high estimated) using a k-means clustering method with bin sorting. A machine learning classification tree and multivariable regression analyses were used to predict 30-day mortality with a 10-fold cross validation. C statistics were used to compare model performance.

Results

Among 63,507 patients who underwent an HP procedure, median patient age was 63 (IQR: 54-71) years. Patients underwent either pancreatectomy (n=38,209, 60.2%) or hepatic resection (n=25,298, 39.8%). Patients were stratified into three tiers of predicted morbidity and mortality risk based on the NSQIP EP: low (n=36,923, 58.1%), intermediate (n=23,609, 37.2%) and high risk (n=2,975, 4.7%). Among 36,923 patients with low estimated risk of morbidity and mortality, 237 patients (0.6%) experienced a UD. According to the classification tree analysis, age was the most important factor to predict UD (importance 16.9) followed by preoperative albumin level (importance: 10.8), disseminated cancer (importance: 6.5), preoperative platelet count (importance: 6.5), and sex (importance 5.9). Among patients deemed to be low risk, the c-statistic for the machine learning derived prediction model was 0.807 compared with an AUC of only 0.662 for the NSQIP EP.

Conclusions

A prognostic model derived using machine learning methodology performed better than the NSQIP EP in predicting 30-day UD among low risk patients undergoing HP surgery.

The American College of Surgeons Surgical Risk Calculator performs well for pulmonary resection: A validation study

OBJECTIVE

The American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator (NSQIP SRC) was developed to estimate the risk of postoperative morbidity and mortality within 30 days of an operation. We sought to externally evaluate the performance of the NSQIP SRC for patients undergoing pulmonary resection.

METHODS

Patients undergoing pulmonary resection at our center between January 2016 and December 2018 were included. Using data from our institution's prospectively maintained Society of Thoracic Surgeons General Thoracic Database, we identified 2514 patients. We entered requisite patient demographic information, preoperative risk factors, and procedural details into the online calculator. Predicted performance of the calculator versus observed outcomes was assessed by discrimination (concordance index [C-index]) and calibration.

RESULTS

The observed and predicted probabilities of any complication were 8.3% and 9.9%, respectively, and of serious complications were 7.4% and 9.2%, respectively. Observed and predicted 30-day mortality were 0.5% and 0.9%, respectively. The C-index for readmission was 0.644; the C-indices corresponding to all other outcomes in the NSQIP SRC ranged from 0.703 to 0.821. Calibration curves indicated excellent calibration for all binary end points, with the exception of renal failure (predicted underestimated observed probabilities), discharge to a nursing or rehabilitation facility (overestimated), and sepsis (overestimated). Correlation between predicted and observed length of stay was moderate (Spearman coefficient, 0.562), and calibration was good.

CONCLUSIONS

Except for readmission, renal failure, discharge to a location other than home, and sepsis, the NSQIP SRC can be used to reasonably predict postoperative complications in patients undergoing pulmonary resection.

Can the Emergency Surgery Score (ESS) predict outcomes in emergency general surgery patients with missing data elements? A nationwide analysis

BACKGROUND

The Emergency Surgery Score (ESS) is an accurate mortality risk calculator for emergency general surgery (EGS). We sought to assess whether ESS can accurately predict 30-day morbidity, mortality, and requirement for postoperative Intensive Care Unit (ICU) care in patients with missing data variables.

METHODS

All EGS patients with one or more missing ESS variables in the 2007-2015 ACS-NSQIP database were included. ESS was calculated assuming that a missing variable is normal (i.e. no additional ESS points). The correlation between ESS and morbidity, mortality, and postoperative ICU level of care was assessed using the c-statistics methodology.

RESULTS

Out of a total of 4,456,809 patients, 359,849 were EGS, and of those 256,278 (71.2%) patients had at least one ESS variable missing. ESS correlated extremely well with mortality (c-statistic = 0.94) and postoperative requirement of ICU care (c-statistic = 0.91) and well with morbidity (c-statistic = 0.77).

CONCLUSION

ESS performs well in predicting outcomes in EGS patients even when one or more data elements are missing and remains a useful bedside tool for counseling EGS patients and for benchmarking the quality of EGS care.

Risk-adjustment models in patients undergoing head and neck surgery with reconstruction

BACKGROUND

With the current focus on value-based outcomes and reimbursement models, perioperative risk adjustment is essential. Specialty surgical outcomes are not well predicted by the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP); the Head and Neck-Reconstructive Surgery NSQIP was created as a specialty-specific platform for patients undergoing head and neck surgery with flap reconstruction. This study aims to investigate risk prediction models in these patients.

METHODS

The Head and Neck-Reconstructive Surgery NSQIP collected data on patients undergoing head and neck surgery with flap reconstruction from August 1, 2012 to October 20, 2016. Multivariable logistic regression models were created for 9 outcomes (postoperative ventilator dependence, pneumonia, superficial recipient surgical site infection, presence of tracheostomy/nasoenteric (NE)/gastrostomy/gastrojejunostomy(G/GJ) tube 30 days postoperatively, conversion from NE to G/GJ tube, unplanned return to the operating room, length of stay > 7 days). External validation was completed with a more contemporary cohort.

RESULTS

A total of 1095 patients were included in the modelling cohort and 407 in the validation cohort. Models performed well predicting tracheostomy, NE, G/GJ tube presence at 30 days postoperatively and conversion from NE to G/GJ tube (c-indices = 0.75-0.91). Models for postoperative pneumonia, superficial recipient surgical site infection, ventilator dependence > 48 h, and length of stay > 7 days were fair (concordance [c]-indices = 0.63-0.69). The predictive model for unplanned return to the operating room was poor (c-index = 0.58).

CONCLUSIONS AND RELEVANCE

Reliable and discriminant risk prediction models were able to be created for postoperative outcomes using the specialty-specific Head and Neck-Reconstructive Surgery Specific NSQIP.

Retroperitoneal sarcoma perioperative risk stratification: A United States Sarcoma Collaborative evaluation of the ACS-NSQIP risk calculator

BACKGROUND

The ACS-NSQIP risk calculator predicts perioperative risk. This study tested the calculator's ability to predict risk for outcomes following retroperitoneal sarcoma (RPS) resection.

METHODS

The United States Sarcoma Collaborative database was queried for adults who underwent RPS resection. Estimated risk for outcomes was calculated twice in the risk calculator, once using sarcoma-specific CPT codes and once using codes indicative of most comorbid organ resection (eg nephrectomy). ROC curves were generated, with area under the curve (AUC) and Brier scores reported to assess discrimination and calibration. An AUC < 0.6 was considered ineffective discrimination. A negative ▲ Brier indicated improved performance relative to baseline outcome rates.

RESULTS

In total, 482 patients were identified with a 42.3% 90-day complication rate. Discrimination was poor for all outcomes except "all complications" and "renal failure." Baseline outcome rates were better predictors than calculator estimates except for "discharge to nursing or rehab facility" and "renal failure." Replacing sarcoma-specific CPT codes with resection-specific codes did not improve performance.

CONCLUSION

The ACS-NSQIP risk calculator poorly predicted outcomes following RPS resection. Changing sarcoma-specific CPT to resection-specific codes did not improve performance. Comorbidities in the calculator may not effectively capture perioperative risk. Future work should evaluate a sarcoma-specific calculator.