Prospective Validation of the Iowa Rectal Surgery Risk Calculator

The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators

Objective

To compare the performance of the ACS NSQIP "universal" risk calculator (N-RC) to operation-specific RCs.

Background

Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. However, operation-specific calculators may not provide sufficient improvements in accuracy to justify the costs in development, maintenance, and access.

Methods

For the N-RC, a cohort of 5,020,713 NSQIP patient records were randomly divided into 80% for machine learning algorithm training and 20% for validation. Operation-specific risk calculators (OS-RC) and OS-RCs with operation-specific predictors (OSP-RC) were independently developed for each of 6 operative groups (colectomy, whipple pancreatectomy, thyroidectomy, abdominal aortic aneurysm (open), hysterectomy/myomectomy, and total knee arthroplasty) and 14 outcomes using the same 80%/20% rule applied to the appropriate subsets of the 5M records. Predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and Hosmer-Lemeshow (H-L) P values, for 13 binary outcomes, and mean squared error for the length of stay outcome.

Results

The N-RC was found to have greater AUROC (P = 0.002) and greater AUPRC (P < 0.001) compared to the OS-RC. No other statistically significant differences in accuracy, across the 3 risk calculator types, were found. There was an inverse relationship between the operation group sample size and magnitude of the difference in AUROC (r = -0.278; P = 0.014) and in AUPRC (r = -0.425; P < 0.001) between N-RC and OS-RC. The smaller the sample size, the greater the superiority of the N-RC.

Conclusions

While operation-specific RCs might be assumed to have advantages over a universal RC, their reliance on smaller datasets may reduce their ability to accurately estimate predictor effects. In the present study, this tradeoff between operation specificity and accuracy, in estimating the effects of predictor variables, favors the N-R, though the clinical impact is likely to be negligible.

Training prediction models for individual risk assessment of postoperative complications after surgery for colorectal cancer

BACKGROUND

The occurrence of postoperative complications and anastomotic leakage are major drivers of mortality in the immediate phase after colorectal cancer surgery. We trained prediction models for calculating patients' individual risk of complications based only on preoperatively available data in a multidisciplinary team setting. Knowing prior to surgery the probability of developing a complication could aid in improving informed decision-making by surgeon and patient and individualize surgical treatment trajectories.

METHODS

All patients over 18 years of age undergoing any resection for colorectal cancer between January 1, 2014 and December 31, 2019 from the nationwide Danish Colorectal Cancer Group database were included. Data from the database were converted into Observational Medical Outcomes Partnership Common Data Model maintained by the Observation Health Data Science and Informatics initiative. Multiple machine learning models were trained to predict postoperative complications of Clavien-Dindo grade ≥ 3B and anastomotic leakage within 30 days after surgery.

RESULTS

Between 2014 and 2019, 23,907 patients underwent resection for colorectal cancer in Denmark. A Clavien-Dindo complication grade ≥ 3B occurred in 2,958 patients (12.4%). Of 17,190 patients that received an anastomosis, 929 experienced anastomotic leakage (5.4%). Among the compared machine learning models, Lasso Logistic Regression performed best. The predictive model for complications had an area under the receiver operating characteristic curve (AUROC) of 0.704 (95%CI 0.683-0.724) and an AUROC of 0.690 (95%CI 0.655-0.724) for anastomotic leakage.

CONCLUSIONS

The prediction of postoperative complications based only on preoperative variables using a national quality assurance colorectal cancer database shows promise for calculating patient's individual risk. Future work will focus on assessing the value of adding laboratory parameters and drug exposure as candidate predictors. Furthermore, we plan to assess the external validity of our proposed model.