XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information

Development and validation of interpretable machine learning models for triage patients admitted to the intensive care unit

OBJECTIVES

Developing and validating interpretable machine learning (ML) models for predicting whether triaged patients need to be admitted to the intensive care unit (ICU).

MEASURES

The study analyzed 189,167 emergency patients from the Medical Information Mart for Intensive Care IV database, with the outcome being ICU admission. Three models were compared: Model 1 based on Emergency Severity Index (ESI), Model 2 on vital signs, and Model 3 on vital signs, demographic characteristics, medical history, and chief complaints. Nine ML algorithms were employed. The area under the receiver operating characteristic curve (AUC), F1 Score, Positive Predictive Value, Negative Predictive Value, Brier score, calibration curves, and decision curves analysis were used to evaluate the performance of the models. SHapley Additive exPlanations was used for explaining ML models.

RESULTS

The AUC of Model 3 was superior to that of Model 1 and Model 2. In Model 3, the top four algorithms with the highest AUC were Gradient Boosting (0.81), Logistic Regression (0.81), naive Bayes (0.80), and Random Forest (0.80). Upon further comparison of the four algorithms, Gradient Boosting was slightly superior to Random Forest and Logistic Regression, while naive Bayes performed the worst.

CONCLUSIONS

This study developed an interpretable ML triage model using vital signs, demographics, medical history, and chief complaints, proving more effective than traditional models in predicting ICU admission. Interpretable ML aids clinical decisions during triage.

Interpretable machine learning for predicting sepsis risk in emergency triage patients

The study aimed to develop and validate a sepsis prediction model using structured electronic medical records (sEMR) and machine learning (ML) methods in emergency triage. The goal was to enhance early sepsis screening by integrating comprehensive triage information beyond vital signs. This retrospective cohort study utilized data from the MIMIC-IV database. Two models were developed: Model 1 based on vital signs alone, and Model 2 incorporating vital signs, demographic characteristics, medical history, and chief complaints. Eight ML algorithms were employed, and model performance was evaluated using metrics such as AUC, F1 Score, and calibration curves. SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) methods were used to enhance model interpretability. The study included 189,617 patients, with 5.95% diagnosed with sepsis. Model 2 consistently outperformed Model 1 across most algorithms. In Model 2, Gradient Boosting achieved the highest AUC of 0.83, followed by Extra Tree, Random Forest, and Support Vector Machine (all 0.82). The SHAP method provided more comprehensible explanations for the Gradient Boosting algorithm. Modeling with comprehensive triage information using sEMR and ML methods was more effective in predicting sepsis at triage compared to using vital signs alone. Interpretable ML enhanced model transparency and provided sepsis prediction probabilities, offering a feasible approach for early sepsis screening and aiding healthcare professionals in making informed decisions during the triage process.

Assessing Risk in Implementing New Artificial Intelligence Triage Tools-How Much Risk is Reasonable in an Already Risky World?

Risk prediction in emergency medicine (EM) holds unique challenges due to issues surrounding urgency, blurry research-practise distinctions, and the high-pressure environment in emergency departments (ED). Artificial intelligence (AI) risk prediction tools have been developed with the aim of streamlining triaging processes and mitigating perennial issues affecting EDs globally, such as overcrowding and delays. The implementation of these tools is complicated by the potential risks associated with over-triage and under-triage, untraceable false positives, as well as the potential for the biases of healthcare professionals toward technology leading to the incorrect usage of such tools. This paper explores risk surrounding these issues in an analysis of a case study involving a machine learning triage tool called the Score for Emergency Risk Prediction (SERP) in Singapore. This tool is used for estimating mortality risk in presentation at the ED. After two successful retrospective studies demonstrating SERP's strong predictive accuracy, researchers decided that the pre-implementation randomised controlled trial (RCT) would not be feasible due to how the tool interacts with clinical judgement, complicating the blinded arm of the trial. This led them to consider other methods of testing SERP's real-world capabilities, such as ongoing-evaluation type studies. We discuss the outcomes of a risk-benefit analysis to argue that the proposed implementation strategy is ethically appropriate and aligns with improvement-focused and systemic approaches to implementation, especially the learning health systems framework (LHS) to ensure safety, efficacy, and ongoing learning.

Using machine learning and natural language processing in triage for prediction of clinical disposition in the emergency department

BACKGROUND

Accurate triage is required for efficient allocation of resources and to decrease patients' length of stay. Triage decisions are often subjective and vary by provider, leading to patients being over-triaged or under-triaged. This study developed machine learning models that incorporated natural language processing (NLP) to predict patient disposition. The models were assessed by comparing their performance with the judgements of emergency physicians (EPs).

METHOD

This retrospective study obtained data from patients visiting EDs between January 2018 and December 2019. Internal validation data came from China Medical University Hospital (CMUH), while external validation data were obtained from Asia University Hospital (AUH). Nontrauma patients aged ≥ 20 years were included. The models were trained using structured data and unstructured data (free-text notes) processed by NLP. The primary outcome was death in the ED or admission to the intensive care unit, and the secondary outcome was either admission to a general ward or transferal to another hospital. Six machine learning models (CatBoost, Light Gradient Boosting Machine, Logistic Regression, Random Forest, Extremely Randomized Trees, and Gradient Boosting) and one Logistic Regression derived from triage level were developed and evaluated using EPs' predictions as reference.

RESULT

A total of 17,2101 and 41,883 patients were enrolled from CMUH and AUH, respectively. EPs achieved F1 core of 0.361 and 0.498 for the primary and secondary outcomes, respectively. All machine learning models achieved higher F1 scores compared to EPs and Logistic Regression derived from triage level. Random Forest was selected for further evaluation and fine-tuning, because of its robust calibration and predictive performance. In internal validation, it achieved Brier scores of 0.072 and 0.089 for the primary and secondary outcomes, respectively, and 0.076 and 0.095 in external validation. Further analysis revealed that incorporating unstructured data significantly enhanced the model's performance. Threshold adjustments were applied to improve clinical applicability, aiming to balance the trade-off between sensitivity and positive predictive value.

CONCLUSION

This study developed and validated machine learning models that integrate structured and unstructured triage data to predict patient dispositions, distinguishing between general ward and critical conditions like ICU admissions and ED deaths. Integrating both structured and unstructured data significantly improved model performance.

Improving triage performance in emergency departments using machine learning and natural language processing: a systematic review

BACKGROUND

In Emergency Departments (EDs), triage is crucial for determining patient severity and prioritizing care, typically using the Manchester Triage Scale (MTS). Traditional triage systems, reliant on human judgment, are prone to under-triage and over-triage, resulting in variability, bias, and incorrect patient classification. Studies suggest that Machine Learning (ML) and Natural Language Processing (NLP) could enhance triage accuracy and consistency. This review analyzes studies on ML and/or NLP algorithms for ED patient triage.

METHODS

Following Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines, we conducted a systematic review across five databases: Web of Science, PubMed, Scopus, IEEE Xplore, and ACM Digital Library, from their inception of each database to October 2023. The risk of bias was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Only articles employing at least one ML and/or NLP method for patient triage classification were included.

RESULTS

Sixty studies covering 57 ML algorithms were included. Logistic Regression (LR) was the most used model, while eXtreme Gradient Boosting (XGBoost), decision tree-based algorithms with Gradient Boosting (GB), and Deep Neural Networks (DNNs) showed superior performance. Frequent predictive variables included demographics and vital signs, with oxygen saturation, chief complaints, systolic blood pressure, age, and mode of arrival being the most retained. The ML algorithms showed significant bias risk due to critical bias assessment in classification models.

CONCLUSION

NLP methods improved ML algorithms' classification capability using triage nursing and medical notes and structured clinical data compared to algorithms using only structured data. Feature engineering (FE) and class imbalance correction methods enhanced ML workflows' performance, but FE and eXplainable Artificial Intelligence (XAI) were underexplored in this field. Registration and funding. This systematic review has been registered (registration number: CRD42024604529) in the International Prospective Register of Systematic Reviews (PROSPERO) and can be accessed online at the following URL: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=604529 . Funding for this work was provided by the National Council for Scientific and Technological Development (CNPq), Brazil.

A Predictive Model of Pressure Injury in Children Undergoing Living Donor Liver Transplantation Based on Machine Learning Algorithm

AIMS

The aim of our study was to formulate and validate a prediction model using machine learning algorithms to forecast the risk of pressure injuries (PIs) in children undergoing living donor liver transplantation (LDLT).

DESIGN

A retrospective cohort study.

METHODS

The research was carried out at China's largest paediatric liver transplantation centre. A total of 438 children who underwent LDLT between June 2021 and December 2022 constituted the study cohort. The dataset was partitioned randomly into 70% for training datasets (306 cases) and 30% for testing datasets (132 cases). Utilising four machine learning algorithms-Decision Tree, Random Forest, Gradient Boosting Decision Tree and eXtreme Gradient Boosting-we identified risk factors and constructed predictive models.

RESULTS

Out of 438 children, 42 developed PIs, yielding an incidence rate of 9.6%. Notably, 94% of these cases were categorised as Stage 1, and 54% were localised on the occiput. Upon evaluating the four prediction models, the Decision Tree model emerged as the most effective. The primary contributors to pressure injury in the Decision Tree model were identified as operation time, intraoperative corticosteroid administration, preoperative skin protection measures and preoperative skin conditions. A visualisation elucidating the logical inference process for the 10 variables within the Decision Tree model was presented. Ultimately, based on the Decision Tree model, a predictive system was developed.

CONCLUSION

Machine learning algorithms facilitate the identification of crucial factors, enabling the creation of an effective Decision Tree model to forecast pressure injury development in children undergoing LDLT.

IMPACT

With this predictive model at their disposal, nurses can assess the pressure injury risk level in children more intuitively. Subsequently, they can implement tailored preventive strategies to mitigate the occurrence of PIs.

PATIENT OR PUBLIC CONTRIBUTION

Paediatric patients contributed electronic health records datasets.

Forecasting hospital discharges for respiratory conditions in Costa Rica using climate and pollution data

Respiratory diseases represent one of the most significant economic burdens on healthcare systems worldwide. The variation in the increasing number of cases depends greatly on climatic seasonal effects, socioeconomic factors, and pollution. Therefore, understanding these variations and obtaining precise forecasts allows health authorities to make correct decisions regarding the allocation of limited economic and human resources. We aimed to model and forecast weekly hospitalizations due to respiratory conditions in seven regional hospitals in Costa Rica using four statistical learning techniques (Random Forest, XGboost, Facebook's Prophet forecasting model, and an ensemble method combining the above methods), along with 22 climate change indices and aerosol optical depth as an indicator of pollution. Models were trained using data from 2000 to 2018 and were evaluated using data from 2019 as testing data. During the training period, we set up 2-year sliding windows and a 1-year assessment period, along with the grid search method to optimize hyperparameters for each model. The best model for each region was selected using testing data, based on predictive precision and to prevent overfitting. Prediction intervals were then computed using conformal inference. The relative importance of all climatic variables was computed for the best model, and similar patterns in some of the seven regions were observed based on the selected model. Finally, reliable predictions were obtained for each of the seven regional hospitals.

Inter hospital external validation of interpretable machine learning based triage score for the emergency department using common data model

Emergency departments (ED) are complex, triage is a main task in the ED to prioritize patient with limited medical resources who need them most. Machine learning (ML) based ED triage tool, Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable ML framework with single center. We aimed to develop SERP with 3 Korean multicenter cohorts based on common data model (CDM) without data sharing and compare performance with inter-hospital validation design. This retrospective cohort study included all adult emergency visit patients of 3 hospitals in Korea from 2016 to 2017. We adopted CDM for the standardized multicenter research. The outcome of interest was 2-day mortality after the patients' ED visit. We developed each hospital SERP using interpretable ML framework and validated inter-hospital wisely. We accessed the performance of each hospital's score based on some metrics considering data imbalance strategy. The study population for each hospital included 87,670, 83,363 and 54,423 ED visits from 2016 to 2017. The 2-day mortality rate were 0.51%, 0.56% and 0.65%. Validation results showed accurate for inter hospital validation which has at least AUROC of 0.899 (0.858-0.940). We developed multicenter based Interpretable ML model using CDM for 2-day mortality prediction and executed Inter-hospital external validation which showed enough high accuracy.

A conformal regressor for predicting negative conversion time of Omicron patients

In light of the situation and the characteristics of Omicron, the country has continuously optimized the rules for the prevention and control of COVID-19. The global epidemic is still spreading, and new cases of infection continue to emerge in China. To facilitate the infected person to estimate the course of virus infection, a prediction model for predicting negative conversion time is proposed in this article. The clinical features of Omicron-infected patients in Shandong Province in the first half of 2022 are retrospectively studied. These features are grouped by disease diagnosis result, clinical sign, traditional Chinese medicine symptoms, and drug use. These features are input to the eXtreme Gradient Boosting (XGBoost) model, and the output is the predicted number of negative conversion days. At the same time, XGBoost is used as the underlying algorithm of the conformal prediction (CP) framework, which can realize the probability interval estimation with a controllable error rate. The results show that the proposed model has a mean absolute error of 3.54 days and has the shortest interval prediction result. This shows that the method in this paper can carry more decision-making information and help people better understand the disease and self-estimate the course of the disease to a certain extent.

Machine learning models to predict end-stage kidney disease in chronic kidney disease stage 4

INTRODUCTION

End-stage kidney disease (ESKD) is associated with increased morbidity and mortality. Identifying patients with stage 4 CKD (CKD4) at risk of rapid progression to ESKD remains challenging. Accurate prediction of CKD4 progression can improve patient outcomes by improving advanced care planning and optimizing healthcare resource allocation.

METHODS

We obtained electronic health record data from patients with CKD4 in a large health system between January 1, 2006, and December 31, 2016. We developed and validated four models, including Least Absolute Shrinkage and Selection Operator (LASSO) regression, random forest, eXtreme Gradient Boosting (XGBoost), and artificial neural network (ANN), to predict ESKD at 3 years. We utilized area under the receiver operating characteristic curve (AUROC) to evaluate model performances and utilized Shapley additive explanation (SHAP) values and plots to define feature dependence of the best performance model.

RESULTS

We included 3,160 patients with CKD4. ESKD was observed in 538 patients (21%). All approaches had similar AUROCs; ANN yielded the highest AUROC (0.77; 95%CI 0.75 to 0.79) and LASSO regression (0.77; 95%CI 0.75 to 0.79), followed by random forest (0.76; 95% CI 0.74 to 0.79), and XGBoost (0.76; 95% CI 0.74 to 0.78).

CONCLUSIONS

We developed and validated several models for near-term prediction of kidney failure in CKD4. ANN, random forest, and XGBoost demonstrated similar predictive performances. Using this suite of models, interventions can be customized based on risk, and population health and resources appropriately allocated.

Using emergency department triage for machine learning-based admission and mortality prediction

AIMS

Patient admission is a decision relying on sparsely available data. This study aims to provide prediction models for discharge versus admission for ward observation or intensive care, and 30 day-mortality for patients triaged with the Manchester Triage System.

METHODS

This is a single-centre, observational, retrospective cohort study from data within ten minutes of patient presentation at the interdisciplinary emergency department of the Kepler University Hospital, Linz, Austria. We trained machine learning models including Random Forests and Neural Networks individually to predict discharge versus ward observation or intensive care admission, and 30 day-mortality. For analysis of the features' relevance, we used permutation feature importance.

RESULTS

A total of 58323 adult patients between 1 December 2015 and 31 August 2020 were included. Neural Networks and Random Forests predicted admission to ward observation with an AUC-ROC of 0.842 ± 0.00 with the most important features being age and chief complaint. For admission to intensive care, the models had an AUC-ROC of 0.819 ± 0.002 with the most important features being the Manchester Triage category and heart rate, and for the outcome 30 day-mortality an AUC-ROC of 0.925 ± 0.001. The most important features for the prediction of 30 day-mortality were age and general ward admission.

CONCLUSION

Machine learning can provide prediction on discharge versus admission to general wards and intensive care and inform about risk on 30 day-mortality for patients in the emergency department.

Development of a population-level prediction model for intensive care unit (ICU) survivorship and mortality in older adults: A population-based cohort study

Background and Aims

Given the growing utilization of critical care services by an aging population, development of population-level risk models which predict intensive care unit (ICU) survivorship and mortality may offer advantages for researchers and health systems. Our objective was to develop a risk model for ICU survivorship and mortality among community dwelling older adults.

Methods

This was a population-based cohort study of 48,127 patients who were 50 years and older with at least one primary care visit between January 1, 2017, and December 31, 2017. We used electronic health record (EHR) data to identify variables predictive of ICU survivorship.

Results

ICU admission and mortality within 2 years after index primary care visit date were used to divide patients into three groups of "alive without ICU admission", "ICU survivors," and "death." Multinomial logistic regression was used to identify EHR predictive variables for the three patient outcomes. Cross-validation by randomly splitting the data into derivation and validation data sets (60:40 split) was used to identify predictor variables and validate model performance using area under the receiver operating characteristics (AUC) curve. In our overall sample, 92.2% of patients were alive without ICU admission, 6.2% were admitted to the ICU at least once and survived, and 1.6% died. Greater deciles of age over 50 years, diagnoses of chronic obstructive pulmonary disorder or chronic heart failure, and laboratory abnormalities in alkaline phosphatase, hematocrit, and albumin contributed highest risk score weights for mortality. Risk scores derived from the model discriminated between patients that died versus remained alive without ICU admission (AUC = 0.858), and between ICU survivors versus alive without ICU admission (AUC = 0.765).

Conclusion

Our risk scores provide a feasible and scalable tool for researchers and health systems to identify patient cohorts at increased risk for ICU admission and survivorship. Further studies are needed to prospectively validate the risk scores in other patient populations.

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients' ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.

Machine learning-based suggestion for critical interventions in the management of potentially severe conditioned patients in emergency department triage

Providing timely intervention to critically ill patients is a challenging task in emergency departments (ED). Our study aimed to predict early critical interventions (CrIs), which can be used as clinical recommendations. This retrospective observational study was conducted in the ED of a tertiary hospital located in a Korean metropolitan city. Patient who visited ED from January 1, 2016, to December 31, 2018, were included. Need of six CrIs were selected as prediction outcomes, namely, arterial line (A-line) insertion, oxygen therapy, high-flow nasal cannula (HFNC), intubation, Massive Transfusion Protocol (MTP), and inotropes and vasopressor. Extreme gradient boosting (XGBoost) prediction model was built by using only data available at the initial stage of ED. Overall, 137,883 patients were included in the study. The areas under the receiver operating characteristic curve for the prediction of A-line insertion was 0·913, oxygen therapy was 0.909, HFNC was 0.962, intubation was 0.945, MTP was 0.920, and inotropes or vasopressor administration was 0.899 in the XGBoost method. In addition, an increase in the need for CrIs was associated with worse ED outcomes. The CrIs model was integrated into the study site's electronic medical record and could be used to suggest early interventions for emergency physicians.

Development and Validation of an Artificial Intelligence Electrocardiogram Recommendation System in the Emergency Department

The machine learning-assisted electrocardiogram (ECG) is increasingly recognized for its unprecedented capabilities in diagnosing and predicting cardiovascular diseases. Identifying the need for ECG examination early in emergency department (ED) triage is key to timely artificial intelligence-assisted analysis. We used machine learning to develop and validate a clinical decision support tool to predict ED triage patients’ need for ECG. Data from 301,658 ED visits from August 2017 to November 2020 in a tertiary hospital were divided into a development cohort, validation cohort, and two test cohorts that included admissions before and during the COVID-19 pandemic. Models were developed using logistic regression, decision tree, random forest, and XGBoost methods. Their areas under the receiver operating characteristic curves (AUCs), positive predictive values (PPVs), and negative predictive values (NPVs) were compared and validated. In the validation cohort, the AUCs were 0.887 for the XGBoost model, 0.885 for the logistic regression model, 0.878 for the random forest model, and 0.845 for the decision tree model. The XGBoost model was selected for subsequent application. In test cohort 1, the AUC was 0.891, with sensitivity of 0.812, specificity of 0.814, PPV of 0.708 and NPV of 0.886. In test cohort 2, the AUC was 0.885, with sensitivity of 0.816, specificity of 0.812, PPV of 0.659, and NPV of 0.908. In the cumulative incidence analysis, patients not receiving an ECG yet positively predicted by the model had significantly higher probability of receiving the examination within 48 h compared with those negatively predicted by the model. A machine learning model based on triage datasets was developed to predict ECG acquisition with high accuracy. The ECG recommendation can effectively predict whether patients presenting at ED triage will require an ECG, prompting subsequent analysis and decision-making in the ED.

Artificial intelligence, machine learning, and deep learning for clinical outcome prediction

AI is a broad concept, grouping initiatives that use a computer to perform tasks that would usually require a human to complete. AI methods are well suited to predict clinical outcomes. In practice, AI methods can be thought of as functions that learn the outcomes accompanying standardized input data to produce accurate outcome predictions when trialed with new data. Current methods for cleaning, creating, accessing, extracting, augmenting, and representing data for training AI clinical prediction models are well defined. The use of AI to predict clinical outcomes is a dynamic and rapidly evolving arena, with new methods and applications emerging. Extraction or accession of electronic health care records and combining these with patient genetic data is an area of present attention, with tremendous potential for future growth. Machine learning approaches, including decision tree methods of Random Forest and XGBoost, and deep learning techniques including deep multi-layer and recurrent neural networks, afford unique capabilities to accurately create predictions from high dimensional, multimodal data. Furthermore, AI methods are increasing our ability to accurately predict clinical outcomes that previously were difficult to model, including time-dependent and multi-class outcomes. Barriers to robust AI-based clinical outcome model deployment include changing AI product development interfaces, the specificity of regulation requirements, and limitations in ensuring model interpretability, generalizability, and adaptability over time.