Performance and usability of pre-operative prediction models for 30-day peri-operative mortality risk: a systematic review

Predictive modelling for postoperative acute kidney injury: big data enhancing quality or the Emperor's new clothes?

The increased availability of large clinical datasets together with increasingly sophisticated computing power has facilitated development of numerous risk prediction models for various adverse perioperative outcomes, including acute kidney injury (AKI). The rationale for developing such models is straightforward. However, despite numerous purported benefits, the uptake of preoperative prediction models into clinical practice has been limited. Barriers to implementation of predictive models, including limitations in their discrimination and accuracy, as well as their ability to meaningfully impact clinical practice and patient outcomes, are increasingly recognised. Some of the purported benefits of predictive modelling, particularly when applied to postoperative AKI, might not fare well under detailed scrutiny. Future research should address existing limitations and seek to demonstrate both benefit to patients and value to healthcare systems from implementation of these models in clinical practice.

Preoperative risk prediction models for acute kidney injury after noncardiac surgery: an independent external validation cohort study

BACKGROUND

Numerous models have been developed to predict acute kidney injury (AKI) after noncardiac surgery, yet there is a lack of independent validation and comparison among them.

METHODS

We conducted a systematic literature search to review published risk prediction models for AKI after noncardiac surgery. An independent external validation was performed using a retrospective surgical cohort at a large Chinese hospital from January 2019 to October 2022. The cohort included patients undergoing a wide range of noncardiac surgeries with perioperative creatinine measurements. Postoperative AKI was defined according to the Kidney Disease Improving Global Outcomes creatinine criteria. Model performance was assessed in terms of discrimination (area under the receiver operating characteristic curve, AUROC), calibration (calibration plot), and clinical utility (net benefit), before and after model recalibration through intercept and slope updates. A sensitivity analysis was conducted by including patients without postoperative creatinine measurements in the validation cohort and categorising them as non-AKI cases.

RESULTS

Nine prediction models were evaluated, each with varying clinical and methodological characteristics, including the types of surgical cohorts used for model development, AKI definitions, and predictors. In the validation cohort involving 13,186 patients, 650 (4.9%) developed AKI. Three models demonstrated fair discrimination (AUROC between 0.71 and 0.75); other models had poor or failed discrimination. All models exhibited some miscalibration; five of the nine models were well-calibrated after intercept and slope updates. Decision curve analysis indicated that the three models with fair discrimination consistently provided a positive net benefit after recalibration. The results were confirmed in the sensitivity analysis.

CONCLUSIONS

We identified three models with fair discrimination and potential clinical utility after recalibration for assessing the risk of acute kidney injury after noncardiac surgery.

Decision Curve Analysis of In-Hospital Mortality Prediction Models: The Relative Value of Pre- and Intraoperative Data For Decision-Making

BACKGROUND

Clinical prediction modeling plays a pivotal part in modern clinical care, particularly in predicting the risk of in-hospital mortality. Recent modeling efforts have focused on leveraging intraoperative data sources to improve model performance. However, the individual and collective benefit of pre- and intraoperative data for clinical decision-making remains unknown. We hypothesized that pre- and intraoperative predictors contribute equally to the net benefit in a decision curve analysis (DCA) of in-hospital mortality prediction models that include pre- and intraoperative predictors.

METHODS

Data from the VitalDB database featuring a subcohort of 6043 patients were used. A total of 141 predictors for in-hospital mortality were grouped into preoperative (demographics, intervention characteristics, and laboratory measurements) and intraoperative (laboratory and monitor data, drugs, and fluids) data. Prediction models using either preoperative, intraoperative, or all data were developed with multiple methods (logistic regression, neural network, random forest, gradient boosting machine, and a stacked learner). Predictive performance was evaluated by the area under the receiver-operating characteristic curve (AUROC) and under the precision-recall curve (AUPRC). Clinical utility was examined with a DCA in the predefined risk preference range (denoted by so-called treatment threshold probabilities) between 0% and 20%.

RESULTS

AUROC performance of the prediction models ranged from 0.53 to 0.78. AUPRC values ranged from 0.02 to 0.25 (compared to the incidence of 0.09 in our dataset) and high AUPRC values resulted from prediction models based on preoperative laboratory values. A DCA of pre- and intraoperative prediction models highlighted that preoperative data provide the largest overall benefit for decision-making, whereas intraoperative values provide only limited benefit for decision-making compared to preoperative data. While preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for low treatment thresholds up to 5% to 10%, preoperative laboratory measurements become the dominant source for decision support for higher thresholds.

CONCLUSIONS

When it comes to predicting in-hospital mortality and subsequent decision-making, preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for clinicians with risk-averse preferences, whereas preoperative laboratory values provide the largest benefit for decision-makers with more moderate risk preferences. Our decision-analytic investigation of different predictor categories moves beyond the question of whether certain predictors provide a benefit in traditional performance metrics (eg, AUROC). It offers a nuanced perspective on for whom these predictors might be beneficial in clinical decision-making. Follow-up studies requiring larger datasets and dedicated deep-learning models to handle continuous intraoperative data are essential to examine the robustness of our results.

Development and validation of the Ex-Care BR model: a multicentre initiative for identifying Brazilian surgical patients at risk of 30-day in-hospital mortality

BACKGROUND

Surgical risk stratification is crucial for enhancing perioperative assistance and allocating resources efficiently. However, existing models may not capture the complexity of surgical care in Brazil. Using data from various healthcare settings nationwide, we developed a new risk model for 30-day in-hospital mortality (the Ex-Care BR model).

METHODS

A retrospective cohort study was conducted in 10 hospitals from different geographic regions in Brazil. Data were analysed using multilevel logistic regression models. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC), Brier score, and calibration plots. Derivation and validation cohorts were randomly assigned.

RESULTS

A total of 107,372 patients were included, and 30-day in-hospital mortality was 2.1% (n=2261). The final risk model comprised four predictors related to the patient and surgery (age, ASA physical status classification, surgical urgency, and surgical size), and the random effect related to hospitals. The model showed excellent discrimination (AUROC=0.93, 95% confidence interval [CI], 0.93-0.94), calibration, and overall performance (Brier score=0.017) in the derivation cohort (n=75,094). Similar results were observed in the validation cohort (n=32,278) (AUROC=0.93, 95% CI, 0.92-0.93).

CONCLUSIONS

The Ex-Care BR is the first model to consider regional and organisational peculiarities of the Brazilian surgical scene, in addition to patient and surgical factors. It is particularly useful for identifying high-risk surgical patients in situations demanding efficient allocation of limited resources. However, a thorough exploration of mortality variations among hospitals is essential for a comprehensive understanding of risk.

CLINICAL TRIAL REGISTRATION

NCT05796024.

Rapid Response Systems

The hospital rapid response system (RRS) is a patient safety and quality intervention that responds quickly to clinical deteriorations on general wards with the goal of preventing cardiopulmonary arrests, reducing hospital mortality, and facilitating triage and level of care escalations. The RRS is one of the first organized, and systematic, elements of the "ICU without walls" model. RRSs have been shown to be effective in preventing deterioration to cardiopulmonary arrest on general hospital wards and reducing total and unexpected hospital mortality. Recent studies have demonstrated that this benefit can be enhanced through targeted improvements and modifications of existing RRSs.

Artificial intelligence in surgery

Artificial intelligence (AI) is rapidly emerging in healthcare, yet applications in surgery remain relatively nascent. Here we review the integration of AI in the field of surgery, centering our discussion on multifaceted improvements in surgical care in the preoperative, intraoperative and postoperative space. The emergence of foundation model architectures, wearable technologies and improving surgical data infrastructures is enabling rapid advances in AI interventions and utility. We discuss how maturing AI methods hold the potential to improve patient outcomes, facilitate surgical education and optimize surgical care. We review the current applications of deep learning approaches and outline a vision for future advances through multimodal foundation models.

Risk prediction models in emergency surgery: Protocol for a scoping review

BACKGROUND

Risk prediction models are used for many purposes in emergency surgery, including critical care triage and benchmarking. Several risk prediction models have been developed, and some are used for purposes other than those for which they were developed. We aim to provide an overview of the existing literature on risk prediction models used in emergency surgery and highlight knowledge gaps.

METHODS

We will conduct a scoping review on risk prediction models used for patients undergoing emergency surgery in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). We will search Medline, EMBASE, and the Cochrane Library and include all study designs. We aim to answer the following questions: (1) What risk prediction models are used in emergency surgery? (2) Which variables are used in these models? (3) Which surgical specialties are the models used for? (4) Have the models been externally validated? (5) Where have the models been externally validated? (6) What purposes were the models developed for? (7) What are the strengths and limitations of the included models? We will summarize the results descriptively. The certainty of evidence will be evaluated using a modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.

CONCLUSION

The outlined scoping review will summarize the existing literature on risk prediction models used in emergency surgery and highlight knowledge gaps.

Wearable Sensors as a Preoperative Assessment Tool: A Review

Surgery is a common first-line treatment for many types of disease, including cancer. Mortality rates after general elective surgery have seen significant decreases whilst postoperative complications remain a frequent occurrence. Preoperative assessment tools are used to support patient risk stratification but do not always provide a precise and accessible assessment. Wearable sensors (WS) provide an accessible alternative that offers continuous monitoring in a non-clinical setting. They have shown consistent uptake across the perioperative period but there has been no review of WS as a preoperative assessment tool. This paper reviews the developments in WS research that have application to the preoperative period. Accelerometers were consistently employed as sensors in research and were frequently combined with photoplethysmography or electrocardiography sensors. Pre-processing methods were discussed and missing data was a common theme; this was dealt with in several ways, commonly by employing an extraction threshold or using imputation techniques. Research rarely processed raw data; commercial devices that employ internal proprietary algorithms with pre-calculated heart rate and step count were most commonly employed limiting further feature extraction. A range of machine learning models were used to predict outcomes including support vector machines, random forests and regression models. No individual model clearly outperformed others. Deep learning proved successful for predicting exercise testing outcomes but only within large sample-size studies. This review outlines the challenges of WS and provides recommendations for future research to develop WS as a viable preoperative assessment tool.

Establishment of a pathomic-based machine learning model to predict CD276 (B7-H3) expression in colon cancer

CD276 is a promising prognostic indicator and an attractive therapeutic target in various malignancies. However, current methods for CD276 detection are time-consuming and expensive, limiting extensive studies and applications of CD276. We aimed to develop a pathomic model for CD276 prediction from H&E-stained pathological images, and explore the underlying mechanism of the pathomic features by associating the pathomic model with transcription profiles. A dataset of colon adenocarcinoma (COAD) patients was retrieved from the Cancer Genome Atlas (TCGA) database. The dataset was divided into the training and validation sets according to the ratio of 8:2 by a stratified sampling method. Using the gradient boosting machine (GBM) algorithm, we established a pathomic model to predict CD276 expression in COAD. Univariate and multivariate Cox regression analyses were conducted to assess the predictive performance of the pathomic model for overall survival in COAD. Gene Set Enrichment Analysis (GESA) was performed to explore the underlying biological mechanisms of the pathomic model. The pathomic model formed by three pathomic features for CD276 prediction showed an area under the curve (AUC) of 0.833 (95%CI: 0.784-0.882) in the training set and 0.758 (95%CI: 0.637-0.878) in the validation set, respectively. The calibration curves and Hosmer-Lemeshow goodness of fit test showed that the prediction probability of high/low expression of CD276 was in favorable agreement with the real situation in both the training and validation sets (P=0.176 and 0.255, respectively). The DCA curves suggested that the pathomic model acquired high clinical benefit. All the subjects were categorized into high pathomic score (PS) (PS-H) and low PS (PS-L) groups according to the cutoff value of PS. Univariate and multivariate Cox regression analysis indicated that PS was a risk factor for overall survival in COAD. Furthermore, through GESA analysis, we found several immune and inflammatory-related pathways and genes were associated with the pathomic model. We constructed a pathomics-based machine learning model for CD276 prediction directly from H&E-stained images in COAD. Through integrated analysis of the pathomic model and transcriptomics, the interpretability of the pathomic model provide a theoretical basis for further hypothesis and experimental research.

Machine-learning vs. logistic regression for preoperative prediction of medical morbidity after fast-track hip and knee arthroplasty-a comparative study

BACKGROUND

Machine-learning models may improve prediction of length of stay (LOS) and morbidity after surgery. However, few studies include fast-track programs, and most rely on administrative coding with limited follow-up and information on perioperative care. This study investigates potential benefits of a machine-learning model for prediction of postoperative morbidity in fast-track total hip (THA) and knee arthroplasty (TKA).

METHODS

Cohort study in consecutive unselected primary THA/TKA between 2014-2017 from seven Danish centers with established fast-track protocols. Preoperative comorbidity and prescribed medication were recorded prospectively and information on length of stay and readmissions was obtained through the Danish National Patient Registry and medical records. We used a machine-learning model (Boosted Decision Trees) based on boosted decision trees with 33 preoperative variables for predicting "medical" morbidity leading to LOS > 4 days or 90-days readmissions and compared to a logistical regression model based on the same variables. We also evaluated two parsimonious models, using the ten most important variables in the full machine-learning and logistic regression models. Data collected between 2014-2016 (n:18,013) was used for model training and data from 2017 (n:3913) was used for testing. Model performances were analyzed using precision, area under receiver operating (AUROC) and precision recall curves (AUPRC), as well as the Mathews Correlation Coefficient. Variable importance was analyzed using Shapley Additive Explanations values.

RESULTS

Using a threshold of 20% "risk-patients" (n:782), precision, AUROC and AUPRC were 13.6%, 76.3% and 15.5% vs. 12.4%, 74.7% and 15.6% for the machine-learning and logistic regression model, respectively. The parsimonious machine-learning model performed better than the full logistic regression model. Of the top ten variables, eight were shared between the machine-learning and logistic regression models, but with a considerable age-related variation in importance of specific types of medication.

CONCLUSION

A machine-learning model using preoperative characteristics and prescriptions slightly improved identification of patients in high-risk of "medical" complications after fast-track THA and TKA compared to a logistic regression model. Such algorithms could help find a manageable population of patients who may benefit most from intensified perioperative care.