Enabling personalized perioperative risk prediction by using a machine-learning model based on preoperative data

Machine learning-based model development for predicting risk factors of prolonged intra-aortic balloon pump therapy in patients with coronary artery bypass grafting

Machine learning algorithms are frequently used to clinical risk prediction. Our study was designed to predict risk factors of prolonged intra-aortic balloon pump (IABP) use in patients with coronary artery bypass grafting (CABG) through developing machine learning-based models. Patients who received perioperative IABP therapy were divided into two groups based on their length of IABP implantation longer than the 75th percentile for the whole cohort: normal (≤ 10 days) and prolonged (> 10 days) groups. Seven machine learning-based models were created and evaluated, and then the Shapley Additive exPlanations (SHAP) method was employed to further illustrate the influence of the features on model. In our study, a total of 143 patients were included, comprising 56 cases (38.16%) in the prolonged group. The logistic regression model was considered the final prediction model according to its most excellent performance. Furthermore, feature important analysis identified left ventricular end-systolic or diastolic diameter, preoperative IABP use, diabetes, and cardiac troponin T as the top five risk variables for prolonged IABP implantation in patients. The SHAP analysis further explained the features attributed to the model. Machine learning models were successfully developed and used to predict risk variables of prolonged IABP implantation in patients with CABG. This may help early identification for prolonged IABP use and initiate clinical interventions.

Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication

Importance

General-domain large language models may be able to perform risk stratification and predict postoperative outcome measures using a description of the procedure and a patient's electronic health record notes.

Objective

To examine predictive performance on 8 different tasks: prediction of American Society of Anesthesiologists Physical Status (ASA-PS), hospital admission, intensive care unit (ICU) admission, unplanned admission, hospital mortality, postanesthesia care unit (PACU) phase 1 duration, hospital duration, and ICU duration.

Design, Setting, and Participants

This prognostic study included task-specific datasets constructed from 2 years of retrospective electronic health records data collected during routine clinical care. Case and note data were formatted into prompts and given to the large language model GPT-4 Turbo (OpenAI) to generate a prediction and explanation. The setting included a quaternary care center comprising 3 academic hospitals and affiliated clinics in a single metropolitan area. Patients who had a surgery or procedure with anesthesia and at least 1 clinician-written note filed in the electronic health record before surgery were included in the study. Data were analyzed from November to December 2023.

Exposures

Compared original notes, note summaries, few-shot prompting, and chain-of-thought prompting strategies.

Main Outcomes and Measures

F1 score for binary and categorical outcomes. Mean absolute error for numerical duration outcomes.

Results

Study results were measured on task-specific datasets, each with 1000 cases with the exception of unplanned admission, which had 949 cases, and hospital mortality, which had 576 cases. The best results for each task included an F1 score of 0.50 (95% CI, 0.47-0.53) for ASA-PS, 0.64 (95% CI, 0.61-0.67) for hospital admission, 0.81 (95% CI, 0.78-0.83) for ICU admission, 0.61 (95% CI, 0.58-0.64) for unplanned admission, and 0.86 (95% CI, 0.83-0.89) for hospital mortality prediction. Performance on duration prediction tasks was universally poor across all prompt strategies for which the large language model achieved a mean absolute error of 49 minutes (95% CI, 46-51 minutes) for PACU phase 1 duration, 4.5 days (95% CI, 4.2-5.0 days) for hospital duration, and 1.1 days (95% CI, 0.9-1.3 days) for ICU duration prediction.

Conclusions and Relevance

Current general-domain large language models may assist clinicians in perioperative risk stratification on classification tasks but are inadequate for numerical duration predictions. Their ability to produce high-quality natural language explanations for the predictions may make them useful tools in clinical workflows and may be complementary to traditional risk prediction models.

[Artificial intelligence in intensive care medicine]

The integration of artificial intelligence (AI) into intensive care medicine has made considerable progress in recent studies, particularly in the areas of predictive analytics, early detection of complications, and the development of decision support systems. The main challenges remain availability and quality of data, reduction of bias and the need for explainable results from algorithms and models. Methods to explain these systems are essential to increase trust, understanding, and ethical considerations among healthcare professionals and patients. Proper training of healthcare professionals in AI principles, terminology, ethical considerations, and practical application is crucial for the successful use of AI. Careful assessment of the impact of AI on patient autonomy and data protection is essential for its responsible use in intensive care medicine. A balance between ethical and practical considerations must be maintained to ensure patient-centered care while complying with data protection regulations. Synergistic collaboration between clinicians, AI engineers, and regulators is critical to realizing the full potential of AI in intensive care medicine and maximizing its positive impact on patient care. Future research and development efforts should focus on improving AI models for real-time predictions, increasing the accuracy and utility of AI-based closed-loop systems, and overcoming ethical, technical, and regulatory challenges, especially in generative AI systems.

Susceptibility of AutoML mortality prediction algorithms to model drift caused by the COVID pandemic

BACKGROUND

Concept drift and covariate shift lead to a degradation of machine learning (ML) models. The objective of our study was to characterize sudden data drift as caused by the COVID pandemic. Furthermore, we investigated the suitability of certain methods in model training to prevent model degradation caused by data drift.

METHODS

We trained different ML models with the H2O AutoML method on a dataset comprising 102,666 cases of surgical patients collected in the years 2014-2019 to predict postoperative mortality using preoperatively available data. Models applied were Generalized Linear Model with regularization, Default Random Forest, Gradient Boosting Machine, eXtreme Gradient Boosting, Deep Learning and Stacked Ensembles comprising all base models. Further, we modified the original models by applying three different methods when training on the original pre-pandemic dataset: (Rahmani K, et al, Int J Med Inform 173:104930, 2023) we weighted older data weaker, (Morger A, et al, Sci Rep 12:7244, 2022) used only the most recent data for model training and (Dilmegani C, 2023) performed a z-transformation of the numerical input parameters. Afterwards, we tested model performance on a pre-pandemic and an in-pandemic data set not used in the training process, and analysed common features.

RESULTS

The models produced showed excellent areas under receiver-operating characteristic and acceptable precision-recall curves when tested on a dataset from January-March 2020, but significant degradation when tested on a dataset collected in the first wave of the COVID pandemic from April-May 2020. When comparing the probability distributions of the input parameters, significant differences between pre-pandemic and in-pandemic data were found. The endpoint of our models, in-hospital mortality after surgery, did not differ significantly between pre- and in-pandemic data and was about 1% in each case. However, the models varied considerably in the composition of their input parameters. None of our applied modifications prevented a loss of performance, although very different models emerged from it, using a large variety of parameters.

CONCLUSIONS

Our results show that none of our tested easy-to-implement measures in model training can prevent deterioration in the case of sudden external events. Therefore, we conclude that, in the presence of concept drift and covariate shift, close monitoring and critical review of model predictions are necessary.

[Big data and artificial intelligence in anesthesia : Reality or fiction?]

Big data and artificial intelligence are buzzwords that everyone is talking about and yet always provide a touch of science fiction to the scenery. What is the status of these topics in anesthesia? Are the first robots already rolling through the corridors while doctors are getting bored as all the work has been done? Spoiler alert! We are still far away from achieving this. Initially, paper charts and analogue notes stand in the way of comprehensive digitization. Source systems need to be merged and data standardized, harmonized and validated. Therefore, the friendly android that is rolling towards us, waving and holding a freshly brewed cup of coffee in our thoughts will have to wait; however, a glimpse of the future is already evident in some clinics and the first promising developments are already showing what could be the standard tomorrow. Learning algorithms calculate the length of stay individually for each patient in the intensive care unit (ICU), reducing negative consequences such as readmission and mortality. The field of ultrasound technology for regional anesthesia and closed-loop anesthesia systems is also demonstrating the benefits of artificial intelligence (AI)-assisted technologies in practice. The efforts are diverse and ambitious but they repeatedly collide with privacy challenges and significant capital expenditure, which weigh heavily on an already financially strained healthcare system; however, anyone who listens carefully to the medical staff knows that robots are not what they would expect and the buzzwords big data and artificial intelligence might be less science fiction than initially assumed.