Automated machine learning-based model for the prediction of delirium in patients after surgery for degenerative spinal disease

A Supervised Explainable Machine Learning Model for Perioperative Neurocognitive Disorder in Liver-Transplantation Patients and External Validation on the Medical Information Mart for Intensive Care IV Database: Retrospective Study

BACKGROUND

Patients undergoing liver transplantation (LT) are at risk of perioperative neurocognitive dysfunction (PND), which significantly affects the patients' prognosis.

OBJECTIVE

This study used machine learning (ML) algorithms with an aim to extract critical predictors and develop an ML model to predict PND among LT recipients.

METHODS

In this retrospective study, data from 958 patients who underwent LT between January 2015 and January 2020 were extracted from the Third Affiliated Hospital of Sun Yat-sen University. Six ML algorithms were used to predict post-LT PND, and model performance was evaluated using area under the receiver operating curve (AUC), accuracy, sensitivity, specificity, and F1-scores. The best-performing model was additionally validated using a temporal external dataset including 309 LT cases from February 2020 to August 2022, and an independent external dataset extracted from the Medical Information Mart for Intensive Care Ⅳ (MIMIC-Ⅳ) database including 325 patients.

RESULTS

In the development cohort, 201 out of 751 (33.5%) patients were diagnosed with PND. The logistic regression model achieved the highest AUC (0.799) in the internal validation set, with comparable AUC in the temporal external (0.826) and MIMIC-Ⅳ validation sets (0.72). The top 3 features contributing to post-LT PND diagnosis were the preoperative overt hepatic encephalopathy, platelet level, and postoperative sequential organ failure assessment score, as revealed by the Shapley additive explanations method.

CONCLUSIONS

A real-time logistic regression model-based online predictor of post-LT PND was developed, providing a highly interoperable tool for use across medical institutions to support early risk stratification and decision making for the LT recipients.

Machine Learning for Prediction of Postoperative Delirium in Adult Patients: A Systematic Review and Meta-analysis

PURPOSE

This meta-analysis aimed to evaluate the performance of machine learning (ML) models in predicting postoperative delirium (POD) and to provide guidance for clinical application.

METHODS

PubMed, Embase, Cochrane Library, and Web of Science databases were searched from inception to April 29, 2024. Studies reported ML models for predicting POD in adult patients were included. Data extraction and risk of bias assessment were performed using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis - AI (TRIPOD-AI) and Prediction model Risk Of Bias ASsessment Tool (PROBAST) tools. Meta-analysis with the area under the curve (AUC) was performed using MedCalc software.

FINDINGS

A total of 23 studies were included after screening. Age (n = 20, 86.95%) and Random Forest (RF) (n = 24, 17.27%) were the most frequently used feature and ML algorithm, respectively. The meta-analysis showed an overall AUC of 0.792. The ensemble models (AUC = 0.805) showed better predictive performance than single models (AUC = 0.782). Additionally, considerable variations in AUC were found among different ML algorithms, with AdaBoost (AB) demonstrating good performance with AUC of 0.870. Notably, the generalizability of these models was uncertain due to limitations in external validation and bias assessment.

IMPLICATIONS

The performance of ensemble models were higher than single models, and the AB algorithms demonstrated better performance, compared with other algorithms. However, further research was needed to enhance the generalizability and transparency of ML models.

Revolutionizing spinal interventions: a systematic review of artificial intelligence technology applications in contemporary surgery

Leveraging its ability to handle large and complex datasets, artificial intelligence can uncover subtle patterns and correlations that human observation may overlook. This is particularly valuable for understanding the intricate dynamics of spinal surgery and its multifaceted impacts on patient prognosis. This review aims to delineate the role of artificial intelligence in spinal surgery. A search of the PubMed database from 1992 to 2023 was conducted using relevant English publications related to the application of artificial intelligence in spinal surgery. The search strategy involved a combination of the following keywords: "Artificial neural network," "deep learning," "artificial intelligence," "spinal," "musculoskeletal," "lumbar," "vertebra," "disc," "cervical," "cord," "stenosis," "procedure," "operation," "surgery," "preoperative," "postoperative," and "operative." A total of 1,182 articles were retrieved. After a careful evaluation of abstracts, 90 articles were found to meet the inclusion criteria for this review. Our review highlights various applications of artificial neural networks in spinal disease management, including (1) assessing surgical indications, (2) assisting in surgical procedures, (3) preoperatively predicting surgical outcomes, and (4) estimating the occurrence of various surgical complications and adverse events. By utilizing these technologies, surgical outcomes can be improved, ultimately enhancing the quality of life for patients.

Can Machine Learning Models Based on Computed Tomography Radiomics and Clinical Characteristics Provide Diagnostic Value for Epstein-Barr Virus-Associated Gastric Cancer?

OBJECTIVE

The aim of this study was to explore whether machine learning model based on computed tomography (CT) radiomics and clinical characteristics can differentiate Epstein-Barr virus-associated gastric cancer (EBVaGC) from non-EBVaGC.

METHODS

Contrast-enhanced CT images were collected from 158 patients with GC (46 EBV-positive, 112 EBV-negative) between April 2018 and February 2023. Radiomics features were extracted from the volumes of interest. A radiomics signature was built based on radiomics features by the least absolute shrinkage and selection operator logistic regression algorithm. Multivariate analyses were used to identify significant clinicoradiological variables. We developed 6 ML models for EBVaGC, including logistic regression, Extreme Gradient Boosting, random forest (RF), support vector machine, Gaussian Naive Bayes, and K-nearest neighbor algorithm. The area under the receiver operating characteristic curve (AUC), the area under the precision-recall curves (AP), calibration plots, and decision curve analysis were applied to assess the effectiveness of each model.

RESULTS

Six ML models achieved AUC of 0.706-0.854 and AP of 0.480-0.793 for predicting EBV status in GC. With an AUC of 0.854 and an AP of 0.793, the RF model performed the best. The forest plot of the AUC score revealed that the RF model had the most stable performance, with a standard deviation of 0.003 for AUC score. RF also performed well in the testing dataset, with an AUC of 0.832 (95% confidence interval: 0.679-0.951), accuracy of 0.833, sensitivity of 0.857, and specificity of 0.824, respectively.

CONCLUSIONS

The RF model based on clinical variables and Rad_score can serve as a noninvasive tool to evaluate the EBV status of gastric cancer.

Construction and validation of a risk prediction model for postoperative delirium in patients with off‑pump coronary artery bypass grafting

Background

Compared with cardiopulmonary bypass surgery, off-pump coronary artery bypass grafting (OPCABG) reduces trauma to the body. However, there is still a risk of neurological complications, including postoperative delirium (POD). To date, few studies have been conducted on the risk of POD in OPCABG patients, and no standardized prediction model has been established. Thus, this study sought to analyze the factors influencing POD in OPCABG patients and to construct a risk prediction model.

Methods

A total of 1,258 patients with OPCABG were enrolled and divided into the training set for model construction (944 cases) and the test set for model validation (314 cases). A risk prediction model for POD in OPCABG patients was established by least absolute shrinkage and selection operator (LASSO) regression and multivariate logistic regression, and a nomogram was drawn. The discrimination and calibration degree of the model was evaluated by the receiver operator characteristic (ROC) curve and calibration curve.

Results

Eight variables [i.e., age, tissue oxygen saturation, mean arterial pressure (MAP), carotid stenosis, the anterior-posterior diameter of the aortic sinus, ventricular septum thickness, left ventricular ejection fraction (LVEF), and Mini-Mental State Examination (MMSE) scores] were screen out by the LASSO regression and multivariate logistic regression, and the model was constructed. The area under the ROC curve of the training set was 0.702 [95% confidence interval (CI): 0.662-0.743], and that of the test set was 0.658 (95% CI: 0.585-0.730). The results of the Hosmer-Lemeshow goodness-of-fit test showed that the predicted POD risk of OPCABG patients in the training and test sets was consistent with the actual POD risk (χ2=5.154, P=0.74).

Conclusions

The occurrence of POD in OPCABG patients is related to age, tissue oxygen saturation, MAP, carotid artery stenosis, the anterior-posterior diameter of aortic sinus, ventricular septal thickness, LVEF, and MMSE scores. The prediction model constructed with the above variables had high predictive performance, and thus may be helpful in the early identification of such patients.

Machine learning-based model to predict delirium in patients with advanced cancer treated with palliative care: a multicenter, patient-based registry cohort

This study aimed to present a new approach to predict to delirium admitted to the acute palliative care unit. To achieve this, this study employed machine learning model to predict delirium in patients in palliative care and identified the significant features that influenced the model. A multicenter, patient-based registry cohort study in South Korea between January 1, 2019, and December 31, 2020. Delirium was identified by reviewing the medical records based on the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. The study dataset included 165 patients with delirium among 2314 patients with advanced cancer admitted to the acute palliative care unit. Seven machine learning models, including extreme gradient boosting, adaptive boosting, gradient boosting, light gradient boosting, logistic regression, support vector machine, and random forest, were evaluated to predict delirium in patients with advanced cancer admitted to the acute palliative care unit. An ensemble approach was adopted to determine the optimal model. For k-fold cross-validation, the combination of extreme gradient boosting and random forest provided the best performance, achieving the following accuracy metrics: 68.83% sensitivity, 70.85% specificity, 69.84% balanced accuracy, and 74.55% area under the receiver operating characteristic curve. The performance of the isolated testing dataset was also validated, and the machine learning model was successfully deployed on a public website ( http://ai-wm.khu.ac.kr/Delirium/ ) to provide public access to delirium prediction results in patients with advanced cancer. Furthermore, using feature importance analysis, sex was determined to be the top contributor in predicting delirium, followed by a history of delirium, chemotherapy, smoking status, alcohol consumption, and living with family. Based on a large-scale, multicenter, patient-based registry cohort, a machine learning prediction model for delirium in patients with advanced cancer was developed in South Korea. We believe that this model will assist healthcare providers in treating patients with delirium and advanced cancer.

Establishment and validation of a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning

Objective

To develop and validate a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning methods.

Methods

A retrospective cohort study was conducted to select 303 elderly patients with severe coronary calcification as the study subjects. According to the occurrence of postoperative heart failure, the study subjects were divided into the heart failure group (n = 53) and the non-heart failure group (n = 250). Retrospective collection of clinical data from the study subjects during hospitalization. After processing the missing values in the original data and addressing sample imbalance using Adaptive Synthetic Sampling (ADASYN) method, the final dataset consists of 502 samples: 250 negative samples (i.e., patients not suffering from heart failure) and 252 positive samples (i.e., patients with heart failure). According to a 7:3 ratio, the datasets of 502 patients were randomly divided into a training set (n = 351) and a validation set (n = 151). On the training set, logistic regression (LR), extreme gradient boosting (XGBoost), support vector machine (SVM), and lightweight gradient boosting machine (LightGBM) algorithms were used to construct heart failure risk prediction models; Evaluate model performance on the validation set by calculating the area under the receiver operating characteristic curve (ROC) curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, F1-score, and prediction accuracy.

Result

A total of 17.49% of 303 patients occured postoperative heart failure. The AUC of LR, XGBoost, SVM, and LightGBM models in the training set were 0.872, 1.000, 0.699, and 1.000, respectively. After 10 fold cross validation, the AUC was 0.863, 0.972, 0.696, and 0.963 in the training set, respectively. Among them, XGBoost had the highest AUC and better predictive performance, while SVM models had the worst performance. The XGBoost model also showed good predictive performance in the validation set (AUC = 0.972, 95% CI [0.951-0.994]). The Shapley additive explanation (SHAP) method suggested that the six characteristic variables of blood cholesterol, serum creatinine, fasting blood glucose, age, triglyceride and NT-proBNP were important positive factors for the occurrence of heart failure, and LVEF was important negative factors for the occurrence of heart failure.

Conclusion

The seven characteristic variables of blood cholesterol, blood creatinine, fasting blood glucose, NT-proBNP, age, triglyceride and LVEF are all important factors affecting the occurrence of heart failure. The prediction model of heart failure risk for elderly patients after CRA based on the XGBoost algorithm is superior to SVM, LightGBM and the traditional LR model. This model could be used to assist clinical decision-making and improve the adverse outcomes of patients after CRA.

Machine learning in female urinary incontinence: A scoping review

Introduction and Hypothesis

The aim was to conduct a scoping review of the literature on the use of machine learning (ML) in female urinary incontinence (UI) over the last decade.

Methods

A systematic search was performed among the Medline, Google Scholar, PubMed, and Web of Science databases using the following keywords: [Urinary incontinence] and [(Machine learning) or (Predict) or (Prediction model)]. Eligible studies were considered to have applied ML model to explore different management processes of female UI. Data analyzed included the field of application, type of ML, input variables, and results of model validation.

Results

A total of 798 papers were identified while 23 finally met the inclusion criteria. The vast majority of studies applied logistic regression to establish models (91.3%, 21/23). Most frequently ML was applied to predict postpartum UI (39.1%, 9/23), followed by de novo incontinence after pelvic floor surgery (34.8%, 8/23).There are also three papers using ML models to predict treatment outcomes and three papers using ML models to assist in diagnosis. Variables for modeling included demographic characteristics, clinical data, pelvic floor ultrasound, and urodynamic parameters. The area under receiver operating characteristic curve of these models fluctuated from 0.56 to 0.95, and only 11 studies reported sensitivity and specificity, with sensitivity ranging from 20% to 96.2% and specificity from 59.8% to 94.5%.

Conclusion

Machine learning modeling demonstrated good predictive and diagnostic abilities in some aspects of female UI, showing its promising prospects in near future. However, the lack of standardization and transparency in the validation and evaluation of the models, and the insufficient external validation greatly diminished the applicability and reproducibility, thus a focus on filling this gap is strongly recommended for future research.

Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review

Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models' effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model's area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.

Interpretable Machine Learning Model to Predict Bone Cement Leakage in Percutaneous Vertebral Augmentation for Osteoporotic Vertebral Compression Fracture Based on SHapley Additive exPlanations

STUDY DESIGN

Retrospective study.

OBJECTIVES

Our objective is to create comprehensible machine learning (ML) models that can forecast bone cement leakage in percutaneous vertebral augmentation (PVA) for individuals with osteoporotic vertebral compression fracture (OVCF) while also identifying the associated risk factors.

METHODS

We incorporated data from patients (n = 425) which underwent PVA. To predict cement leakage, we devised six models based on a variety of parameters. Evaluate and juxtapose the predictive performances relied on measures of discrimination, calibration, and clinical utility. SHapley Additive exPlanations (SHAP) methodology was used to interpret model and evaluate the risk factors associated with cement leakage.

RESULTS

The occurrence rate of cement leakage was established at 50.4%. A binary logistic regression analysis identified cortical disruption (OR 6.880, 95% CI 4.209-11.246), the basivertebral foramen sign (OR 2.142, 95% CI 1.303-3.521), the fracture type (OR 1.683, 95% CI 1.083-2.617), and the volume of bone cement (OR 1.198, 95% CI 1.070-1.341) as independent predictors of cement leakage. The XGBoost model outperformed all others in predicting cement leakage in the testing set, with AUC of .8819, accuracy of .8025, recall score of .7872, F1 score of .8315, and a precision score of .881. Several important factors related to cement leakage were drawn based on the analysis of SHAP values and their clinical significance.

CONCLUSION

The ML based predictive model demonstrated significant accuracy in forecasting bone cement leakage for patients with OVCF undergoing PVA. When combined with SHAP, ML facilitated a personalized prediction and offered a visual interpretation of feature importance.

Temporal Generalizability of Machine Learning Models for Predicting Postoperative Delirium Using Electronic Health Record Data: Model Development and Validation Study

BACKGROUND

Although machine learning models demonstrate significant potential in predicting postoperative delirium, the advantages of their implementation in real-world settings remain unclear and require a comparison with conventional models in practical applications.

OBJECTIVE

The objective of this study was to validate the temporal generalizability of decision tree ensemble and sparse linear regression models for predicting delirium after surgery compared with that of the traditional logistic regression model.

METHODS

The health record data of patients hospitalized at an advanced emergency and critical care medical center in Kumamoto, Japan, were collected electronically. We developed a decision tree ensemble model using extreme gradient boosting (XGBoost) and a sparse linear regression model using least absolute shrinkage and selection operator (LASSO) regression. To evaluate the predictive performance of the model, we used the area under the receiver operating characteristic curve (AUROC) and the Matthews correlation coefficient (MCC) to measure discrimination and the slope and intercept of the regression between predicted and observed probabilities to measure calibration. The Brier score was evaluated as an overall performance metric. We included 11,863 consecutive patients who underwent surgery with general anesthesia between December 2017 and February 2022. The patients were divided into a derivation cohort before the COVID-19 pandemic and a validation cohort during the COVID-19 pandemic. Postoperative delirium was diagnosed according to the confusion assessment method.

RESULTS

A total of 6497 patients (68.5, SD 14.4 years, women n=2627, 40.4%) were included in the derivation cohort, and 5366 patients (67.8, SD 14.6 years, women n=2105, 39.2%) were included in the validation cohort. Regarding discrimination, the XGBoost model (AUROC 0.87-0.90 and MCC 0.34-0.44) did not significantly outperform the LASSO model (AUROC 0.86-0.89 and MCC 0.34-0.41). The logistic regression model (AUROC 0.84-0.88, MCC 0.33-0.40, slope 1.01-1.19, intercept -0.16 to 0.06, and Brier score 0.06-0.07), with 8 predictors (age, intensive care unit, neurosurgery, emergency admission, anesthesia time, BMI, blood loss during surgery, and use of an ambulance) achieved good predictive performance.

CONCLUSIONS

The XGBoost model did not significantly outperform the LASSO model in predicting postoperative delirium. Furthermore, a parsimonious logistic model with a few important predictors achieved comparable performance to machine learning models in predicting postoperative delirium.

Machine learning for prediction of delirium in patients with extensive burns after surgery

AIMS

Machine learning-based identification of key variables and prediction of postoperative delirium in patients with extensive burns.

METHODS

Five hundred and eighteen patients with extensive burns who underwent surgery were included and randomly divided into a training set, a validation set, and a testing set. Multifactorial logistic regression analysis was used to screen for significant variables. Nine prediction models were constructed in the training and validation sets (80% of dataset). The testing set (20% of dataset) was used to further evaluate the model. The area under the receiver operating curve (AUROC) was used to compare model performance. SHapley Additive exPlanations (SHAP) was used to interpret the best one and to externally validate it in another large tertiary hospital.

RESULTS

Seven variables were used in the development of nine prediction models: physical restraint, diabetes, sex, preoperative hemoglobin, acute physiological and chronic health assessment, time in the Burn Intensive Care Unit and total body surface area. Random Forest (RF) outperformed the other eight models in terms of predictive performance (ROC:84.00%) When external validation was performed, RF performed well (accuracy: 77.12%, sensitivity: 67.74% and specificity: 80.46%).

CONCLUSION

The first machine learning-based delirium prediction model for patients with extensive burns was successfully developed and validated. High-risk patients for delirium can be effectively identified and targeted interventions can be made to reduce the incidence of delirium.

Evaluating the performance of machine learning methods for risk estimation of delirium in patients hospitalized from the emergency department

INTRODUCTION

Delirium is a cerebral dysfunction seen commonly in the acute care setting. It is associated with increased mortality and morbidity and is frequently missed in the emergency department (ED) and inpatient care by clinical gestalt alone. Identifying those at risk of delirium may help prioritize screening and interventions in the hospital setting.

OBJECTIVE

Our objective was to leverage electronic health records to identify a clinically valuable risk estimation model for prevalent delirium in patients being transferred from the ED to inpatient units.

METHODS

This was a retrospective cohort study to develop and validate a risk model to detect delirium using patient data available from prior visits and ED encounter. Electronic health records were extracted for patients hospitalized from the ED between January 1, 2014, and December 31, 2020. Eligible patients were aged 65 or older, admitted to an inpatient unit from the emergency department, and had at least one DOSS assessment or CAM-ICU recorded within 72 h of hospitalization. Six machine learning models were developed to estimate the risk of delirium using clinical variables including demographic features, physiological measurements, medications administered, lab results, and diagnoses.

RESULTS

A total of 28,531 patients met the inclusion criteria with 8057 (28.4%) having a positive delirium screening within the outcome observation period. Machine learning models were compared using the area under the receiver operating curve (AUC). The gradient boosted machine achieved the best performance with an AUC of 0.839 (95% CI, 0.837-0.841). At a 90% sensitivity threshold, this model achieved a specificity of 53.5% (95% CI 53.0%-54.0%) a positive predictive value of 43.5% (95% CI 43.2%-43.9%), and a negative predictive value of 93.1% (95% CI 93.1%-93.2%). A random forest model and L1-penalized logistic regression also demonstrated notable performance with AUCs of 0.837 (95% CI, 0.835-0.838) and 0.831 (95% CI, 0.830-0.833) respectively.

CONCLUSION

This study demonstrated the use of machine learning algorithms to identify a combination of variables that enables an estimation of risk of positive delirium screens early in hospitalization to develop prevention or management protocols.

Delirium Prediction Using Machine Learning Interpretation Method and Its Incorporation into a Clinical Workflow

Delirium in hospitalized patients is a worldwide problem, causing a burden on healthcare professionals and impacting patient prognosis. A machine learning interpretation method (ML interpretation method) presents the results of machine learning predictions and promotes guided decisions. This study focuses on visualizing the predictors of delirium using a ML interpretation method and implementing the analysis results in clinical practice. Retrospective data of 55,389 patients hospitalized in a single acute care center in Japan between December 2017 and February 2022 were collected. Patients were categorized into three analysis populations, according to inclusion and exclusion criteria, to develop delirium prediction models. The predictors were then visualized using Shapley additive explanation (SHAP) and fed back to clinical practice. The machine learning-based prediction of delirium in each population exhibited excellent predictive performance. SHAP was used to visualize the body mass index and albumin levels as critical contributors to delirium prediction. In addition, the cutoff value for age, which was previously unknown, was visualized, and the risk threshold for age was raised. By using the SHAP method, we demonstrated that data-driven decision support is possible using electronic medical record data.