Impact of Intraoperative Data on Risk Prediction for Mortality After Intra-Abdominal Surgery

Decision Curve Analysis of In-Hospital Mortality Prediction Models: The Relative Value of Pre- and Intraoperative Data For Decision-Making

BACKGROUND

Clinical prediction modeling plays a pivotal part in modern clinical care, particularly in predicting the risk of in-hospital mortality. Recent modeling efforts have focused on leveraging intraoperative data sources to improve model performance. However, the individual and collective benefit of pre- and intraoperative data for clinical decision-making remains unknown. We hypothesized that pre- and intraoperative predictors contribute equally to the net benefit in a decision curve analysis (DCA) of in-hospital mortality prediction models that include pre- and intraoperative predictors.

METHODS

Data from the VitalDB database featuring a subcohort of 6043 patients were used. A total of 141 predictors for in-hospital mortality were grouped into preoperative (demographics, intervention characteristics, and laboratory measurements) and intraoperative (laboratory and monitor data, drugs, and fluids) data. Prediction models using either preoperative, intraoperative, or all data were developed with multiple methods (logistic regression, neural network, random forest, gradient boosting machine, and a stacked learner). Predictive performance was evaluated by the area under the receiver-operating characteristic curve (AUROC) and under the precision-recall curve (AUPRC). Clinical utility was examined with a DCA in the predefined risk preference range (denoted by so-called treatment threshold probabilities) between 0% and 20%.

RESULTS

AUROC performance of the prediction models ranged from 0.53 to 0.78. AUPRC values ranged from 0.02 to 0.25 (compared to the incidence of 0.09 in our dataset) and high AUPRC values resulted from prediction models based on preoperative laboratory values. A DCA of pre- and intraoperative prediction models highlighted that preoperative data provide the largest overall benefit for decision-making, whereas intraoperative values provide only limited benefit for decision-making compared to preoperative data. While preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for low treatment thresholds up to 5% to 10%, preoperative laboratory measurements become the dominant source for decision support for higher thresholds.

CONCLUSIONS

When it comes to predicting in-hospital mortality and subsequent decision-making, preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for clinicians with risk-averse preferences, whereas preoperative laboratory values provide the largest benefit for decision-makers with more moderate risk preferences. Our decision-analytic investigation of different predictor categories moves beyond the question of whether certain predictors provide a benefit in traditional performance metrics (eg, AUROC). It offers a nuanced perspective on for whom these predictors might be beneficial in clinical decision-making. Follow-up studies requiring larger datasets and dedicated deep-learning models to handle continuous intraoperative data are essential to examine the robustness of our results.

Machine-learning-based models assist the prediction of pulmonary embolism in autoimmune diseases: A retrospective, multicenter study

BACKGROUND

Pulmonary embolism (PE) is a severe and acute cardiovascular syndrome with high mortality among patients with autoimmune inflammatory rheumatic diseases (AIIRDs). Accurate prediction and timely intervention play a pivotal role in enhancing survival rates. However, there is a notable scarcity of practical early prediction and risk assessment systems of PE in patients with AIIRD.

METHODS

In the training cohort, 60 AIIRD with PE cases and 180 age-, gender-, and disease-matched AIIRD non-PE cases were identified from 7254 AIIRD cases in Tongji Hospital from 2014 to 2022. Univariable logistic regression (LR) and least absolute shrinkage and selection operator (LASSO) were used to select the clinical features for further training with machine learning (ML) methods, including random forest (RF), support vector machines (SVM), neural network (NN), logistic regression (LR), gradient boosted decision tree (GBDT), classification and regression trees (CART), and C5.0 models. The performances of these models were subsequently validated using a multicenter validation cohort.

RESULTS

In the training cohort, 24 and 13 clinical features were selected by univariable LR and LASSO strategies, respectively. The five ML models (RF, SVM, NN, LR, and GBDT) showed promising performances, with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.962-1.000 in the training cohort and 0.969-0.999 in the validation cohort. CART and C5.0 models achieved AUCs of 0.850 and 0.932, respectively, in the training cohort. Using D-dimer as a pre-screening index, the refined C5.0 model achieved an AUC exceeding 0.948 in the training cohort and an AUC above 0.925 in the validation cohort. These results markedly outperformed the use of D-dimer levels alone.

CONCLUSION

ML-based models are proven to be precise for predicting the onset of PE in patients with AIIRD exhibiting clinical suspicion of PE.

TRIAL REGISTRATION

Chictr.org.cn : ChiCTR2200059599.

Predictive modeling of perioperative patient deterioration: combining unanticipated ICU admissions and mortality for improved risk prediction

OBJECTIVE

This paper presents a comprehensive analysis of perioperative patient deterioration by developing predictive models that evaluate unanticipated ICU admissions and in-hospital mortality both as distinct and combined outcomes.

MATERIALS AND METHODS

With less than 1% of cases resulting in at least one of these outcomes, we investigated 98 features to identify their role in predicting patient deterioration, using univariate analyses. Additionally, multivariate analyses were performed by employing logistic regression (LR) with LASSO regularization. We also assessed classification models, including non-linear classifiers like Support Vector Machines, Random Forest, and XGBoost.

RESULTS

During evaluation, careful attention was paid to the data imbalance therefore multiple evaluation metrics were used, which are less sensitive to imbalance. These metrics included the area under the receiver operating characteristics, precision-recall and kappa curves, and the precision, sensitivity, kappa, and F1-score. Combining unanticipated ICU admissions and mortality into a single outcome improved predictive performance overall. However, this led to reduced accuracy in predicting individual forms of deterioration, with LR showing the best performance for the combined prediction.

DISCUSSION

The study underscores the significance of specific perioperative features in predicting patient deterioration, especially revealed by univariate analysis. Importantly, interpretable models like logistic regression outperformed complex classifiers, suggesting their practicality. Especially, when combined in an ensemble model for predicting multiple forms of deterioration. These findings were mostly limited by the large imbalance in data as post-operative deterioration is a rare occurrence. Future research should therefore focus on capturing more deterioration events and possibly extending validation to multi-center studies.

CONCLUSIONS

This work demonstrates the potential for accurate prediction of perioperative patient deterioration, highlighting the importance of several perioperative features and the practicality of interpretable models like logistic regression, and ensemble models for the prediction of several outcome types. In future clinical practice these data-driven prediction models might form the basis for post-operative risk stratification by providing an evidence-based assessment of risk.

Machine learning improves prediction of postoperative outcomes after gastrointestinal surgery: a systematic review and meta-analysis

BACKGROUND

Machine learning (ML) approaches have become increasingly popular in predicting surgical outcomes. However, it is unknown whether they are superior to traditional statistical methods such as logistic regression (LR). This study aimed to perform a systematic review and meta-analysis to compare the performance of ML vs LR models in predicting postoperative outcomes for patients undergoing gastrointestinal (GI) surgery.

METHODS

A systematic search of Embase, MEDLINE, Cochrane, Web of Science, and Google Scholar was performed through December 2022. The primary outcome was the discriminatory performance of ML vs LR models as measured by the area under the receiver operating characteristic curve (AUC). A meta-analysis was then performed using a random effects model.

RESULTS

A total of 62 LR models and 143 ML models were included across 38 studies. On average, the best-performing ML models had a significantly higher AUC than the LR models (ΔAUC, 0.07; 95% CI, 0.04-0.09; P < .001). Similarly, on average, the best-performing ML models had a significantly higher logit (AUC) than the LR models (Δlogit [AUC], 0.41; 95% CI, 0.23-0.58; P < .001). Approximately half of studies (44%) were found to have a low risk of bias. Upon a subset analysis of only low-risk studies, the difference in logit (AUC) remained significant (ML vs LR, Δlogit [AUC], 0.40; 95% CI, 0.14-0.66; P = .009).

CONCLUSION

We found a significant improvement in discriminatory ability when using ML over LR algorithms in predicting postoperative outcomes for patients undergoing GI surgery. Subsequent efforts should establish standardized protocols for both developing and reporting studies using ML models and explore the practical implementation of these models.

Application of Machine Learning in Predicting Perioperative Outcomes in Patients with Cancer: A Narrative Review for Clinicians

This narrative review explores the utilization of machine learning (ML) and artificial intelligence (AI) models to enhance perioperative cancer care. ML and AI models offer significant potential to improve perioperative cancer care by predicting outcomes and supporting clinical decision-making. Tailored for perioperative professionals including anesthesiologists, surgeons, critical care physicians, nurse anesthetists, and perioperative nurses, this review provides a comprehensive framework for the integration of ML and AI models to enhance patient care delivery throughout the perioperative continuum.

Prediction of lactate concentrations after cardiac surgery using machine learning and deep learning approaches

Background

Although conventional prediction models for surgical patients often ignore intraoperative time-series data, deep learning approaches are well-suited to incorporate time-varying and non-linear data with complex interactions. Blood lactate concentration is one important clinical marker that can reflect the adequacy of systemic perfusion during cardiac surgery. During cardiac surgery and cardiopulmonary bypass, minute-level data is available on key parameters that affect perfusion. The goal of this study was to use machine learning and deep learning approaches to predict maximum blood lactate concentrations after cardiac surgery. We hypothesized that models using minute-level intraoperative data as inputs would have the best predictive performance.

Methods

Adults who underwent cardiac surgery with cardiopulmonary bypass were eligible. The primary outcome was maximum lactate concentration within 24 h postoperatively. We considered three classes of predictive models, using the performance metric of mean absolute error across testing folds: (1) static models using baseline preoperative variables, (2) augmentation of the static models with intraoperative statistics, and (3) a dynamic approach that integrates preoperative variables with intraoperative time series data.

Results

2,187 patients were included. For three models that only used baseline characteristics (linear regression, random forest, artificial neural network) to predict maximum postoperative lactate concentration, the prediction error ranged from a median of 2.52 mmol/L (IQR 2.46, 2.56) to 2.58 mmol/L (IQR 2.54, 2.60). The inclusion of intraoperative summary statistics (including intraoperative lactate concentration) improved model performance, with the prediction error ranging from a median of 2.09 mmol/L (IQR 2.04, 2.14) to 2.12 mmol/L (IQR 2.06, 2.16). For two modelling approaches (recurrent neural network, transformer) that can utilize intraoperative time-series data, the lowest prediction error was obtained with a range of median 1.96 mmol/L (IQR 1.87, 2.05) to 1.97 mmol/L (IQR 1.92, 2.05). Intraoperative lactate concentration was the most important predictive feature based on Shapley additive values. Anemia and weight were also important predictors, but there was heterogeneity in the importance of other features.

Conclusion

Postoperative lactate concentrations can be predicted using baseline and intraoperative data with moderate accuracy. These results reflect the value of intraoperative data in the prediction of clinically relevant outcomes to guide perioperative management.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions

OBJECTIVE

To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting.

BACKGROUND

To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown.

METHODS

Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

RESULTS

Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy.

CONCLUSIONS

Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

Predicting Intraoperative Hypothermia Burden during Non-Cardiac Surgery: A Retrospective Study Comparing Regression to Six Machine Learning Algorithms

BACKGROUND

Inadvertent intraoperative hypothermia is a common complication that affects patient comfort and morbidity. As the development of hypothermia is a complex phenomenon, predicting it using machine learning (ML) algorithms may be superior to logistic regression.

METHODS

We performed a single-center retrospective study and assembled a feature set comprised of 71 variables. The primary outcome was hypothermia burden, defined as the area under the intraoperative temperature curve below 37 °C over time. We built seven prediction models (logistic regression, extreme gradient boosting (XGBoost), random forest (RF), multi-layer perceptron neural network (MLP), linear discriminant analysis (LDA), k-nearest neighbor (KNN), and Gaussian naïve Bayes (GNB)) to predict whether patients would not develop hypothermia or would develop mild, moderate, or severe hypothermia. For each model, we assessed discrimination (F1 score, area under the receiver operating curve, precision, recall) and calibration (calibration-in-the-large, calibration intercept, calibration slope).

RESULTS

We included data from 87,116 anesthesia cases. Predicting the hypothermia burden group using logistic regression yielded a weighted F1 score of 0.397. Ranked from highest to lowest weighted F1 score, the ML algorithms performed as follows: XGBoost (0.44), RF (0.418), LDA (0.406), LDA (0.4), KNN (0.362), and GNB (0.32).

CONCLUSIONS

ML is suitable for predicting intraoperative hypothermia and could be applied in clinical practice.

On the Horizon: Specific Applications of Automation and Artificial Intelligence in Anesthesiology

Purpose of Review

The purpose of this review is to summarize the current research and critically examine artificial intelligence (AI) technologies and their applicability to the daily practice of anesthesiologists.

Recent Findings

Novel AI tools are developed using data from electronic health records, imaging, waveforms, clinical notes, and wearables. These tools can accurately predict the perioperative risk for adverse outcomes, the need for blood transfusion, and the risk of difficult intubation. Intraoperatively, AI models can assist with technical skill augmentation, patient monitoring, and management. Postoperatively, AI technology can aid in preventing complications and discharge planning. While further prospective validation is needed, these early applications demonstrate promise in every area of perioperative care.

Summary

The practice of anesthesiology is at a precipice fueled by technological innovation. The clinical AI implementation would enable personalized and safer patient care by offering actionable insights from the wealth of perioperative data.

Development and implementation of databases to track patient and safety outcomes

PURPOSE OF REVIEW

Recent advancements in big data analytical tools and large patient databases have expanded tremendously the opportunities to track patient and safety outcomes.We discuss the strengths and limitations of large databases and implementation in practice with a focus on the current opportunities to use technological advancements to improve patient safety.

RECENT FINDINGS

The most used sources of data for large patient safety observational studies are administrative databases, clinical registries, and electronic health records. These data sources have enabled research on patient safety topics ranging from rare adverse outcomes to large cohort studies of the modalities for pain control and safety of medications. Implementing the insights from big perioperative data research is augmented by automating data collection and tracking the safety outcomes on a provider, institutional, national, and global level. In the near future, big data from wearable devices, physiological waveforms, and genomics may lead to the development of personalized outcome measures.

SUMMARY

Patient safety research using large databases can provide actionable insights to improve outcomes in the perioperative setting. As datasets and methods to gain insights from those continue to grow, adopting novel technologies to implement personalized quality assurance initiatives can significantly improve patient care.

Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

Background

Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text.

Objective

This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed.

Methods

We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05.

Results

The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203).

Conclusions

Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.

A PBPK Model of Ternary Cyclodextrin Complex of ST-246 Was Built to Achieve a Reasonable IV Infusion Regimen for the Treatment of Human Severe Smallpox

ST-246 is an oral drug against pathogenic orthopoxvirus infections. An intravenous formulation is required for some critical patients. A ternary complex of ST-246/meglumine/hydroxypropyl-β-cyclodextrin with well-improved solubility was successfully developed in our institute. The aim of this study was to achieve a reasonable intravenous infusion regimen of this novel formulation by a robust PBPK model based on preclinical pharmacokinetic studies. The pharmacokinetics of ST-246 after intravenous injection at different doses in rats, dogs, and monkeys were conducted to obtain clearances. The clearance of humans was generated by using the allometric scaling approach. Tissue distribution of ST-246 was conducted in rats to obtain tissue partition coefficients (K_p). The PBPK model of the rat was first built using in vivo clearance and K_p combined with in vitro physicochemical properties, unbound fraction, and cyclodextrin effect parameters of ST-246. Then the PBPK model was transferred to a dog and monkey and validated simultaneously. Finally, pharmacokinetic profiles after IV infusion at different dosages utilizing the human PBPK model were compared to the observed oral PK profile of ST-246 at therapeutic dosage (600 mg). The mechanistic PBPK model described the animal PK behaviors of ST-246 via intravenous injection and infusion with fold errors within 1.2. It appeared that 6h-IV infusion at 5 mg/kg BID produced similar C_max and AUC as oral administration at 600 mg. A PBPK model of ST-246 was built to achieve a reasonable regimen of IV infusion for the treatment of severe smallpox, which will facilitate the clinical translation of this novel formulation.