Predicting mortality among septic patients presenting to the emergency department-a cross sectional analysis using machine learning

A comparative study of explainable ensemble learning and logistic regression for predicting in-hospital mortality in the emergency department

This study addresses the challenges associated with emergency department (ED) overcrowding and emphasizes the need for efficient risk stratification tools to identify high-risk patients for early intervention. While several scoring systems, often based on logistic regression (LR) models, have been proposed to indicate patient illness severity, this study aims to compare the predictive performance of ensemble learning (EL) models with LR for in-hospital mortality in the ED. A cross-sectional single-center study was conducted at the ED of Imam Reza Hospital in northeast Iran from March 2016 to March 2017. The study included adult patients with one to three levels of emergency severity index. EL models using Bagging, AdaBoost, random forests (RF), Stacking and extreme gradient boosting (XGB) algorithms, along with an LR model, were constructed. The training and validation visits from the ED were randomly divided into 80% and 20%, respectively. After training the proposed models using tenfold cross-validation, their predictive performance was evaluated. Model performance was compared using the Brier score (BS), The area under the receiver operating characteristics curve (AUROC), The area and precision-recall curve (AUCPR), Hosmer-Lemeshow (H-L) goodness-of-fit test, precision, sensitivity, accuracy, F1-score, and Matthews correlation coefficient (MCC). The study included 2025 unique patients admitted to the hospital's ED, with a total percentage of hospital deaths at approximately 19%. In the training group and the validation group, 274 of 1476 (18.6%) and 152 of 728 (20.8%) patients died during hospitalization, respectively. According to the evaluation of the presented framework, EL models, particularly Bagging, predicted in-hospital mortality with the highest AUROC (0.839, CI (0.802-0.875)) and AUCPR = 0.64 comparable in terms of discrimination power with LR (AUROC (0.826, CI (0.787-0.864)) and AUCPR = 0.61). XGB achieved the highest precision (0.83), sensitivity (0.831), accuracy (0.842), F1-score (0.833), and the highest MCC (0.48). Additionally, the most accurate models in the unbalanced dataset belonged to RF with the lowest BS (0.128). Although all studied models overestimate mortality risk and have insufficient calibration (P > 0.05), stacking demonstrated relatively good agreement between predicted and actual mortality. EL models are not superior to LR in predicting in-hospital mortality in the ED. Both EL and LR models can be considered as screening tools to identify patients at risk of mortality.

Machine learning for the prediction of sepsis-related death: a systematic review and meta-analysis

BACKGROUND AND OBJECTIVES

Sepsis is accompanied by a considerably high risk of mortality in the short term, despite the availability of recommended mortality risk assessment tools. However, these risk assessment tools seem to have limited predictive value. With the gradual integration of machine learning into clinical practice, some researchers have attempted to employ machine learning for early mortality risk prediction in sepsis patients. Nevertheless, there is a lack of comprehensive understanding regarding the construction of predictive variables using machine learning and the value of various machine learning methods. Thus, we carried out this systematic review and meta-analysis to explore the predictive value of machine learning for sepsis-related death at different time points.

METHODS

PubMed, Embase, Cochrane, and Web of Science databases were searched until August 9th, 2022. The risk of bias in predictive models was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). We also performed subgroup analysis according to time of death and type of model and summarized current predictive variables used to construct models for sepsis death prediction.

RESULTS

Fifty original studies were included, covering 104 models. The combined Concordance index (C-index), sensitivity, and specificity of machine learning models were 0.799, 0.81, and 0.80 in the training set, and 0.774, 0.71, and 0.68 in the validation set, respectively. Machine learning outperformed conventional clinical scoring tools and showed excellent C-index, sensitivity, and specificity in different subgroups. Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) are the preferred machine learning models because they showed more favorable accuracy with similar modeling variables. This study found that lactate was the most frequent predictor but was seriously ignored by current clinical scoring tools.

CONCLUSION

Machine learning methods demonstrate relatively favorable accuracy in predicting the mortality risk in sepsis patients. Given the limitations in accuracy and applicability of existing prediction scoring systems, there is an opportunity to explore updates based on existing machine learning approaches. Specifically, it is essential to develop or update more suitable mortality risk assessment tools based on the specific contexts of use, such as emergency departments, general wards, and intensive care units.

Predictors of outcomes in emergency department patients with suspected infections and without fulfillment of the sepsis criteria

BACKGROUND

Data on patient characteristics and determinants of serious outcomes for acutely admitted patients with infections who do not fulfill the sepsis criteria are sparse. The study aimed to characterize acutely admitted emergency department (ED) patients with infections and a composite outcome of in-hospital mortality or transfer to the intensive care unit without fulfilling the criteria for sepsis and to examine predictors of the composite outcome.

METHODS

This was a secondary analysis of data from a prospective observational study of patients with suspected bacterial infection admitted to the ED between October 1, 2017 and March 31, 2018. A National Early Warning Score 2 (NEWS2) ≥ 5 within the first 4 h in the ED was assumed to represent a sepsis-like condition with a high risk for the composite endpoint. Patients who achieved the composite outcome were grouped according to fulfillment of the NEWS2 ≥ 5 criteria. We used logistic regression analysis to estimate the unadjusted and adjusted odds ratio (OR) for the composite endpoint among patients with either NEWS2  < 5 (NEWS2-) or NEWS2  ≥ 5 (NEWS2+).

RESULTS

A total of 2055 patients with a median age of 73 years were included. Of these, 198 (9.6%) achieved the composite endpoint, including 59 (29.8%) NEWS2- and 139 (70.2%) NEWS2+ patients, respectively. Diabetes (OR 2.23;1.23-4.0), a Sequential Organ Failure Assessment (SOFA) score ≥ 2 (OR 2.57;1.37-4.79), and a Do-not-attempt-cardiopulmonary-resuscitation order (DNACPR) on admission (OR 3.70;1.75-7.79) were independent predictive variables for the composite endpoint in NEWS2- patients (goodness-of-fit test P = 0.291; area under the receiver operating characteristic curve for the model (AUROC) = 0.72). The regression model for NEWS2+ patients revealed that a SOFA score ≥ 2 (OR 2.79; 1.59-4.91), hypothermia (OR 2.48;1.30-4.75), and DNACPR order on admission were predictive variables for the composite endpoint (goodness-of-fit test P = 0.62; AUROC for the model = 0.70).

CONCLUSION

Approximately one-third of the patients with infections and serious outcomes during hospitalization did not meet the NEWS2 threshold for likely sepsis. Our study identified factors with independent predictive values for the development of serious outcomes that should be tested in future prediction models.

Machine learning techniques for mortality prediction in emergency departments: a systematic review

OBJECTIVES

This systematic review aimed to assess the performance and clinical feasibility of machine learning (ML) algorithms in prediction of in-hospital mortality for medical patients using vital signs at emergency departments (EDs).

DESIGN

A systematic review was performed.

SETTING

The databases including Medline (PubMed), Scopus and Embase (Ovid) were searched between 2010 and 2021, to extract published articles in English, describing ML-based models utilising vital sign variables to predict in-hospital mortality for patients admitted at EDs. Critical appraisal and data extraction for systematic reviews of prediction modelling studies checklist was used for study planning and data extraction. The risk of bias for included papers was assessed using the prediction risk of bias assessment tool.

PARTICIPANTS

Admitted patients to the ED.

MAIN OUTCOME MEASURE

In-hospital mortality.

RESULTS

Fifteen articles were included in the final review. We found that eight models including logistic regression, decision tree, K-nearest neighbours, support vector machine, gradient boosting, random forest, artificial neural networks and deep neural networks have been applied in this domain. Most studies failed to report essential main analysis steps such as data preprocessing and handling missing values. Fourteen included studies had a high risk of bias in the statistical analysis part, which could lead to poor performance in practice. Although the main aim of all studies was developing a predictive model for mortality, nine articles did not provide a time horizon for the prediction.

CONCLUSION

This review provided an updated overview of the state-of-the-art and revealed research gaps; based on these, we provide eight recommendations for future studies to make the use of ML more feasible in practice. By following these recommendations, we expect to see more robust ML models applied in the future to help clinicians identify patient deterioration earlier.