Automated machine learning-based model for the prediction of delirium in patients after surgery for degenerative spinal disease

Construction and validation of a risk prediction model for postoperative delirium in patients with off‑pump coronary artery bypass grafting

Background

Compared with cardiopulmonary bypass surgery, off-pump coronary artery bypass grafting (OPCABG) reduces trauma to the body. However, there is still a risk of neurological complications, including postoperative delirium (POD). To date, few studies have been conducted on the risk of POD in OPCABG patients, and no standardized prediction model has been established. Thus, this study sought to analyze the factors influencing POD in OPCABG patients and to construct a risk prediction model.

Methods

A total of 1,258 patients with OPCABG were enrolled and divided into the training set for model construction (944 cases) and the test set for model validation (314 cases). A risk prediction model for POD in OPCABG patients was established by least absolute shrinkage and selection operator (LASSO) regression and multivariate logistic regression, and a nomogram was drawn. The discrimination and calibration degree of the model was evaluated by the receiver operator characteristic (ROC) curve and calibration curve.

Results

Eight variables [i.e., age, tissue oxygen saturation, mean arterial pressure (MAP), carotid stenosis, the anterior-posterior diameter of the aortic sinus, ventricular septum thickness, left ventricular ejection fraction (LVEF), and Mini-Mental State Examination (MMSE) scores] were screen out by the LASSO regression and multivariate logistic regression, and the model was constructed. The area under the ROC curve of the training set was 0.702 [95% confidence interval (CI): 0.662-0.743], and that of the test set was 0.658 (95% CI: 0.585-0.730). The results of the Hosmer-Lemeshow goodness-of-fit test showed that the predicted POD risk of OPCABG patients in the training and test sets was consistent with the actual POD risk (χ2=5.154, P=0.74).

Conclusions

The occurrence of POD in OPCABG patients is related to age, tissue oxygen saturation, MAP, carotid artery stenosis, the anterior-posterior diameter of aortic sinus, ventricular septal thickness, LVEF, and MMSE scores. The prediction model constructed with the above variables had high predictive performance, and thus may be helpful in the early identification of such patients.

Can Machine Learning Models Based on Computed Tomography Radiomics and Clinical Characteristics Provide Diagnostic Value for Epstein-Barr Virus-Associated Gastric Cancer?

OBJECTIVE

The aim of this study was to explore whether machine learning model based on computed tomography (CT) radiomics and clinical characteristics can differentiate Epstein-Barr virus-associated gastric cancer (EBVaGC) from non-EBVaGC.

METHODS

Contrast-enhanced CT images were collected from 158 patients with GC (46 EBV-positive, 112 EBV-negative) between April 2018 and February 2023. Radiomics features were extracted from the volumes of interest. A radiomics signature was built based on radiomics features by the least absolute shrinkage and selection operator logistic regression algorithm. Multivariate analyses were used to identify significant clinicoradiological variables. We developed 6 ML models for EBVaGC, including logistic regression, Extreme Gradient Boosting, random forest (RF), support vector machine, Gaussian Naive Bayes, and K-nearest neighbor algorithm. The area under the receiver operating characteristic curve (AUC), the area under the precision-recall curves (AP), calibration plots, and decision curve analysis were applied to assess the effectiveness of each model.

RESULTS

Six ML models achieved AUC of 0.706-0.854 and AP of 0.480-0.793 for predicting EBV status in GC. With an AUC of 0.854 and an AP of 0.793, the RF model performed the best. The forest plot of the AUC score revealed that the RF model had the most stable performance, with a standard deviation of 0.003 for AUC score. RF also performed well in the testing dataset, with an AUC of 0.832 (95% confidence interval: 0.679-0.951), accuracy of 0.833, sensitivity of 0.857, and specificity of 0.824, respectively.

CONCLUSIONS

The RF model based on clinical variables and Rad_score can serve as a noninvasive tool to evaluate the EBV status of gastric cancer.

Machine learning-based model to predict delirium in patients with advanced cancer treated with palliative care: a multicenter, patient-based registry cohort

This study aimed to present a new approach to predict to delirium admitted to the acute palliative care unit. To achieve this, this study employed machine learning model to predict delirium in patients in palliative care and identified the significant features that influenced the model. A multicenter, patient-based registry cohort study in South Korea between January 1, 2019, and December 31, 2020. Delirium was identified by reviewing the medical records based on the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. The study dataset included 165 patients with delirium among 2314 patients with advanced cancer admitted to the acute palliative care unit. Seven machine learning models, including extreme gradient boosting, adaptive boosting, gradient boosting, light gradient boosting, logistic regression, support vector machine, and random forest, were evaluated to predict delirium in patients with advanced cancer admitted to the acute palliative care unit. An ensemble approach was adopted to determine the optimal model. For k-fold cross-validation, the combination of extreme gradient boosting and random forest provided the best performance, achieving the following accuracy metrics: 68.83% sensitivity, 70.85% specificity, 69.84% balanced accuracy, and 74.55% area under the receiver operating characteristic curve. The performance of the isolated testing dataset was also validated, and the machine learning model was successfully deployed on a public website ( http://ai-wm.khu.ac.kr/Delirium/ ) to provide public access to delirium prediction results in patients with advanced cancer. Furthermore, using feature importance analysis, sex was determined to be the top contributor in predicting delirium, followed by a history of delirium, chemotherapy, smoking status, alcohol consumption, and living with family. Based on a large-scale, multicenter, patient-based registry cohort, a machine learning prediction model for delirium in patients with advanced cancer was developed in South Korea. We believe that this model will assist healthcare providers in treating patients with delirium and advanced cancer.

Establishment and validation of a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning

Objective

To develop and validate a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning methods.

Methods

A retrospective cohort study was conducted to select 303 elderly patients with severe coronary calcification as the study subjects. According to the occurrence of postoperative heart failure, the study subjects were divided into the heart failure group (n = 53) and the non-heart failure group (n = 250). Retrospective collection of clinical data from the study subjects during hospitalization. After processing the missing values in the original data and addressing sample imbalance using Adaptive Synthetic Sampling (ADASYN) method, the final dataset consists of 502 samples: 250 negative samples (i.e., patients not suffering from heart failure) and 252 positive samples (i.e., patients with heart failure). According to a 7:3 ratio, the datasets of 502 patients were randomly divided into a training set (n = 351) and a validation set (n = 151). On the training set, logistic regression (LR), extreme gradient boosting (XGBoost), support vector machine (SVM), and lightweight gradient boosting machine (LightGBM) algorithms were used to construct heart failure risk prediction models; Evaluate model performance on the validation set by calculating the area under the receiver operating characteristic curve (ROC) curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, F1-score, and prediction accuracy.

Result

A total of 17.49% of 303 patients occured postoperative heart failure. The AUC of LR, XGBoost, SVM, and LightGBM models in the training set were 0.872, 1.000, 0.699, and 1.000, respectively. After 10 fold cross validation, the AUC was 0.863, 0.972, 0.696, and 0.963 in the training set, respectively. Among them, XGBoost had the highest AUC and better predictive performance, while SVM models had the worst performance. The XGBoost model also showed good predictive performance in the validation set (AUC = 0.972, 95% CI [0.951-0.994]). The Shapley additive explanation (SHAP) method suggested that the six characteristic variables of blood cholesterol, serum creatinine, fasting blood glucose, age, triglyceride and NT-proBNP were important positive factors for the occurrence of heart failure, and LVEF was important negative factors for the occurrence of heart failure.

Conclusion

The seven characteristic variables of blood cholesterol, blood creatinine, fasting blood glucose, NT-proBNP, age, triglyceride and LVEF are all important factors affecting the occurrence of heart failure. The prediction model of heart failure risk for elderly patients after CRA based on the XGBoost algorithm is superior to SVM, LightGBM and the traditional LR model. This model could be used to assist clinical decision-making and improve the adverse outcomes of patients after CRA.

Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review

Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models' effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model's area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.

Interpretable Machine Learning Model to Predict Bone Cement Leakage in Percutaneous Vertebral Augmentation for Osteoporotic Vertebral Compression Fracture Based on SHapley Additive exPlanations

STUDY DESIGN

Retrospective study.

OBJECTIVES

Our objective is to create comprehensible machine learning (ML) models that can forecast bone cement leakage in percutaneous vertebral augmentation (PVA) for individuals with osteoporotic vertebral compression fracture (OVCF) while also identifying the associated risk factors.

METHODS

We incorporated data from patients (n = 425) which underwent PVA. To predict cement leakage, we devised six models based on a variety of parameters. Evaluate and juxtapose the predictive performances relied on measures of discrimination, calibration, and clinical utility. SHapley Additive exPlanations (SHAP) methodology was used to interpret model and evaluate the risk factors associated with cement leakage.

RESULTS

The occurrence rate of cement leakage was established at 50.4%. A binary logistic regression analysis identified cortical disruption (OR 6.880, 95% CI 4.209-11.246), the basivertebral foramen sign (OR 2.142, 95% CI 1.303-3.521), the fracture type (OR 1.683, 95% CI 1.083-2.617), and the volume of bone cement (OR 1.198, 95% CI 1.070-1.341) as independent predictors of cement leakage. The XGBoost model outperformed all others in predicting cement leakage in the testing set, with AUC of .8819, accuracy of .8025, recall score of .7872, F1 score of .8315, and a precision score of .881. Several important factors related to cement leakage were drawn based on the analysis of SHAP values and their clinical significance.

CONCLUSION

The ML based predictive model demonstrated significant accuracy in forecasting bone cement leakage for patients with OVCF undergoing PVA. When combined with SHAP, ML facilitated a personalized prediction and offered a visual interpretation of feature importance.

Temporal Generalizability of Machine Learning Models for Predicting Postoperative Delirium Using Electronic Health Record Data: Model Development and Validation Study

BACKGROUND

Although machine learning models demonstrate significant potential in predicting postoperative delirium, the advantages of their implementation in real-world settings remain unclear and require a comparison with conventional models in practical applications.

OBJECTIVE

The objective of this study was to validate the temporal generalizability of decision tree ensemble and sparse linear regression models for predicting delirium after surgery compared with that of the traditional logistic regression model.

METHODS

The health record data of patients hospitalized at an advanced emergency and critical care medical center in Kumamoto, Japan, were collected electronically. We developed a decision tree ensemble model using extreme gradient boosting (XGBoost) and a sparse linear regression model using least absolute shrinkage and selection operator (LASSO) regression. To evaluate the predictive performance of the model, we used the area under the receiver operating characteristic curve (AUROC) and the Matthews correlation coefficient (MCC) to measure discrimination and the slope and intercept of the regression between predicted and observed probabilities to measure calibration. The Brier score was evaluated as an overall performance metric. We included 11,863 consecutive patients who underwent surgery with general anesthesia between December 2017 and February 2022. The patients were divided into a derivation cohort before the COVID-19 pandemic and a validation cohort during the COVID-19 pandemic. Postoperative delirium was diagnosed according to the confusion assessment method.

RESULTS

A total of 6497 patients (68.5, SD 14.4 years, women n=2627, 40.4%) were included in the derivation cohort, and 5366 patients (67.8, SD 14.6 years, women n=2105, 39.2%) were included in the validation cohort. Regarding discrimination, the XGBoost model (AUROC 0.87-0.90 and MCC 0.34-0.44) did not significantly outperform the LASSO model (AUROC 0.86-0.89 and MCC 0.34-0.41). The logistic regression model (AUROC 0.84-0.88, MCC 0.33-0.40, slope 1.01-1.19, intercept -0.16 to 0.06, and Brier score 0.06-0.07), with 8 predictors (age, intensive care unit, neurosurgery, emergency admission, anesthesia time, BMI, blood loss during surgery, and use of an ambulance) achieved good predictive performance.

CONCLUSIONS

The XGBoost model did not significantly outperform the LASSO model in predicting postoperative delirium. Furthermore, a parsimonious logistic model with a few important predictors achieved comparable performance to machine learning models in predicting postoperative delirium.

Machine learning for prediction of delirium in patients with extensive burns after surgery

AIMS

Machine learning-based identification of key variables and prediction of postoperative delirium in patients with extensive burns.

METHODS

Five hundred and eighteen patients with extensive burns who underwent surgery were included and randomly divided into a training set, a validation set, and a testing set. Multifactorial logistic regression analysis was used to screen for significant variables. Nine prediction models were constructed in the training and validation sets (80% of dataset). The testing set (20% of dataset) was used to further evaluate the model. The area under the receiver operating curve (AUROC) was used to compare model performance. SHapley Additive exPlanations (SHAP) was used to interpret the best one and to externally validate it in another large tertiary hospital.

RESULTS

Seven variables were used in the development of nine prediction models: physical restraint, diabetes, sex, preoperative hemoglobin, acute physiological and chronic health assessment, time in the Burn Intensive Care Unit and total body surface area. Random Forest (RF) outperformed the other eight models in terms of predictive performance (ROC:84.00%) When external validation was performed, RF performed well (accuracy: 77.12%, sensitivity: 67.74% and specificity: 80.46%).

CONCLUSION

The first machine learning-based delirium prediction model for patients with extensive burns was successfully developed and validated. High-risk patients for delirium can be effectively identified and targeted interventions can be made to reduce the incidence of delirium.

Evaluating the performance of machine learning methods for risk estimation of delirium in patients hospitalized from the emergency department

INTRODUCTION

Delirium is a cerebral dysfunction seen commonly in the acute care setting. It is associated with increased mortality and morbidity and is frequently missed in the emergency department (ED) and inpatient care by clinical gestalt alone. Identifying those at risk of delirium may help prioritize screening and interventions in the hospital setting.

OBJECTIVE

Our objective was to leverage electronic health records to identify a clinically valuable risk estimation model for prevalent delirium in patients being transferred from the ED to inpatient units.

METHODS

This was a retrospective cohort study to develop and validate a risk model to detect delirium using patient data available from prior visits and ED encounter. Electronic health records were extracted for patients hospitalized from the ED between January 1, 2014, and December 31, 2020. Eligible patients were aged 65 or older, admitted to an inpatient unit from the emergency department, and had at least one DOSS assessment or CAM-ICU recorded within 72 h of hospitalization. Six machine learning models were developed to estimate the risk of delirium using clinical variables including demographic features, physiological measurements, medications administered, lab results, and diagnoses.

RESULTS

A total of 28,531 patients met the inclusion criteria with 8057 (28.4%) having a positive delirium screening within the outcome observation period. Machine learning models were compared using the area under the receiver operating curve (AUC). The gradient boosted machine achieved the best performance with an AUC of 0.839 (95% CI, 0.837-0.841). At a 90% sensitivity threshold, this model achieved a specificity of 53.5% (95% CI 53.0%-54.0%) a positive predictive value of 43.5% (95% CI 43.2%-43.9%), and a negative predictive value of 93.1% (95% CI 93.1%-93.2%). A random forest model and L1-penalized logistic regression also demonstrated notable performance with AUCs of 0.837 (95% CI, 0.835-0.838) and 0.831 (95% CI, 0.830-0.833) respectively.

CONCLUSION

This study demonstrated the use of machine learning algorithms to identify a combination of variables that enables an estimation of risk of positive delirium screens early in hospitalization to develop prevention or management protocols.