Predicting 72-hour and 9-day return to the emergency department using machine learning

Aspiring to clinical significance: Insights from developing and evaluating a machine learning model to predict emergency department return visit admissions

Return visit admissions (RVA), which are instances where patients discharged from the emergency department (ED) rapidly return and require hospital admission, have been associated with quality issues and adverse outcomes. We developed and validated a machine learning model to predict 72-hour RVA using electronic health records (EHR) data. Study data were extracted from EHR data in 2019 from three urban EDs. The development and independent validation datasets included 62,154 patients from two EDs and 73,453 patients from one ED, respectively. Multiple machine learning algorithms were evaluated, including deep significance clustering (DICE), regularized logistic regression (LR), Gradient Boosting Decision Tree, and XGBoost. These machine learning models were also compared against an existing clinical risk score. To support clinical actionability, clinician investigators conducted manual chart reviews of the cases identified by the model. Chart reviews categorized predicted cases across index ED discharge diagnosis and RVA root cause classifications. The best-performing model achieved an AUC of 0.87 in the development site (test set) and 0.75 in the independent validation set. The model, which combined DICE and LR, boosted predictive performance while providing well-defined features. The model was relatively robust to sensitivity analyses regarding performance across age, race, and by varying predictor availability but less robust across diagnostic groups. Clinician examination demonstrated discrete model performance characteristics within clinical subtypes of RVA. This machine learning model demonstrated a strong predictive performance for 72- RVA. Despite the limited clinical actionability potentially due to model complexity, the rarity of the outcome, and variable relevance, the clinical examination offered guidance on further variable inclusion for enhanced predictive accuracy and actionability.

Using a clinical narrative-aware pre-trained language model for predicting emergency department patient disposition and unscheduled return visits

The increasing prevalence of overcrowding in Emergency Departments (EDs) threatens the effective delivery of urgent healthcare. Mitigation strategies include the deployment of monitoring systems capable of tracking and managing patient disposition to facilitate appropriate and timely care, which subsequently reduces patient revisits, optimizes resource allocation, and enhances patient outcomes. This study used ∼ 250,000 emergency department visit records from Taipei Medical University-Shuang Ho Hospital to develop a natural language processing model using BlueBERT, a biomedical domain-specific pre-trained language model, to predict patient disposition status and unplanned readmissions. Data preprocessing and the integration of both structured and unstructured data were central to our approach. Compared to other models, BlueBERT outperformed due to its pre-training on a diverse range of medical literature, enabling it to better comprehend the specialized terminology, relationships, and context present in ED data. We found that translating Chinese-English clinical narratives into English and textualizing numerical data into categorical representations significantly improved the prediction of patient disposition (AUROC = 0.9014) and 72-hour unscheduled return visits (AUROC = 0.6475). The study concludes that the BlueBERT-based model demonstrated superior prediction capabilities, surpassing the performance of prior patient disposition predictive models, thus offering promising applications in the realm of ED clinical practice.

How artificial intelligence could transform emergency care

Artificial intelligence (AI) in healthcare is the ability of a computer to perform tasks typically associated with clinical care (e.g. medical decision-making and documentation). AI will soon be integrated into an increasing number of healthcare applications, including elements of emergency department (ED) care. Here, we describe the basics of AI, various categories of its functions (including machine learning and natural language processing) and review emerging and potential future use-cases for emergency care. For example, AI-assisted symptom checkers could help direct patients to the appropriate setting, models could assist in assigning triage levels, and ambient AI systems could document clinical encounters. AI could also help provide focused summaries of charts, summarize encounters for hand-offs, and create discharge instructions with an appropriate language and reading level. Additional use cases include medical decision making for decision rules, real-time models that predict clinical deterioration or sepsis, and efficient extraction of unstructured data for coding, billing, research, and quality initiatives. We discuss the potential transformative benefits of AI, as well as the concerns regarding its use (e.g. privacy, data accuracy, and the potential for changing the doctor-patient relationship).

Prediction of high-risk emergency department revisits from a machine-learning algorithm: a proof-of-concept study

BACKGROUND

High-risk emergency department (ED) revisit is considered an important quality indicator that may reflect an increase in complications and medical burden. However, because of its multidimensional and highly complex nature, this factor has not been comprehensively investigated. This study aimed to predict high-risk ED revisit with a machine-learning (ML) approach.

METHODS

This 3-year retrospective cohort study assessed adult patients between January 2019 and December 2021 from National Taiwan University Hospital Hsin-Chu Branch with high-risk ED revisit, defined as hospital or intensive care unit admission after ED return within 72 hours. A total of 150 features were preliminarily screened, and 79 were used in the prediction model. Deep learning, random forest, extreme gradient boosting (XGBoost) and stacked ensemble algorithm were used. The stacked ensemble model combined multiple ML models and performed model stacking as a meta-level algorithm. Confusion matrix, accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUROC) were used to evaluate performance.

RESULTS

Analysis was performed for 6282 eligible adult patients: 5025 (80.0%) in the training set and 1257 (20.0%) in the testing set. High-risk ED revisit occurred for 971 (19.3%) of training set patients vs 252 (20.1%) in the testing set. Leading predictors of high-risk ED revisit were age, systolic blood pressure and heart rate. The stacked ensemble model showed more favourable prediction performance (AUROC 0.82) than the other models: deep learning (0.69), random forest (0.78) and XGBoost (0.79). Also, the stacked ensemble model achieved favourable accuracy and specificity.

CONCLUSION

The stacked ensemble algorithm exhibited better prediction performance in which the predictions were generated from different ML algorithms to optimally maximise the final set of results. Patients with older age and abnormal systolic blood pressure and heart rate at the index ED visit were vulnerable to high-risk ED revisit. Further studies should be conducted to externally validate the model.

Machine learning models for predicting unscheduled return visits of patients with abdominal pain at emergency department and validation during COVID-19 pandemic: A retrospective cohort study

Machine learning (ML) models for predicting 72-hour unscheduled return visits (URVs) for patients with abdominal pain in the emergency department (ED) were developed in a previous study. This study refined the data to adjust previous prediction models and evaluated the model performance in future data validation during the COVID-19 era. We aimed to evaluate the practicality of the ML models and compare the URVs before and during the COVID-19 pandemic. We used electronic health records from Chang Gung Memorial Hospital from 2018 to 2019 as a training dataset, and various machine learning models, including logistic regression (LR), random forest (RF), extreme gradient boosting (XGB), and voting classifier (VC) were developed and subsequently used to validate against the 2020 to 2021 data. The models highlighted several determinants for 72-hour URVs, including patient age, prior ER visits, specific vital signs, and medical interventions. The LR, XGB, and VC models exhibited the same AUC of 0.71 in the testing set, whereas the VC model displayed a higher F1 score (0.21). The XGB model demonstrated the highest specificity (0.99) and precision (0.64) but the lowest sensitivity (0.01). Among these models, the VC model showed the most favorable, balanced, and comprehensive performance. Despite the promising results, the study illuminated challenges in predictive modeling, such as the unforeseen influences of global events, such as the COVID-19 pandemic. These findings not only highlight the significant potential of machine learning in augmenting emergency care but also underline the importance of iterative refinement in response to changing real-world conditions.

Prediction of emergency department revisits among child and youth mental health outpatients using deep learning techniques

BACKGROUND

The proportion of Canadian youth seeking mental health support from an emergency department (ED) has risen in recent years. As EDs typically address urgent mental health crises, revisiting an ED may represent unmet mental health needs. Accurate ED revisit prediction could aid early intervention and ensure efficient healthcare resource allocation. We examine the potential increased accuracy and performance of graph neural network (GNN) machine learning models compared to recurrent neural network (RNN), and baseline conventional machine learning and regression models for predicting ED revisit in electronic health record (EHR) data.

METHODS

This study used EHR data for children and youth aged 4-17 seeking services at McMaster Children's Hospital's Child and Youth Mental Health Program outpatient service to develop and evaluate GNN and RNN models to predict whether a child/youth with an ED visit had an ED revisit within 30 days. GNN and RNN models were developed and compared against conventional baseline models. Model performance for GNN, RNN, XGBoost, decision tree and logistic regression models was evaluated using F1 scores.

RESULTS

The GNN model outperformed the RNN model by an F1-score increase of 0.0511 and the best performing conventional machine learning model by an F1-score increase of 0.0470. Precision, recall, receiver operating characteristic (ROC) curves, and positive and negative predictive values showed that the GNN model performed the best, and the RNN model performed similarly to the XGBoost model. Performance increases were most noticeable for recall and negative predictive value than for precision and positive predictive value.

CONCLUSIONS

This study demonstrates the improved accuracy and potential utility of GNN models in predicting ED revisits among children and youth, although model performance may not be sufficient for clinical implementation. Given the improvements in recall and negative predictive value, GNN models should be further explored to develop algorithms that can inform clinical decision-making in ways that facilitate targeted interventions, optimize resource allocation, and improve outcomes for children and youth.

Machine learning models for predicting unscheduled return visits to an emergency department: a scoping review

BACKGROUND

Unscheduled return visits (URVs) to emergency departments (EDs) are used to assess the quality of care in EDs. Machine learning (ML) models can incorporate a wide range of complex predictors to identify high-risk patients and reduce errors to save time and cost. However, the accuracy and practicality of such models are questionable. This review compares the predictive power of multiple ML models and examines the effects of multiple research factors on these models' performance in predicting URVs to EDs.

METHODS

We conducted the present scoping review by searching eight databases for data from 2010 to 2023. The criteria focused on eligible articles that used ML to predict ED return visits. The primary outcome was the predictive performances of the ML models, and results were analyzed on the basis of intervals of return visits, patient population, and research scale.

RESULTS

A total of 582 articles were identified through the database search, with 14 articles selected for detailed analysis. Logistic regression was the most widely used method; however, eXtreme Gradient Boosting generally exhibited superior performance. Variations in visit interval, target group, and research scale did not significantly affect the predictive power of the models.

CONCLUSION

This is the first study to summarize the use of ML for predicting URVs in ED patients. The development of practical ML prediction models for ED URVs is feasible, but improving the accuracy of predicting ED URVs to beyond 0.75 remains a challenge. Including multiple data sources and dimensions is key for enabling ML models to achieve high accuracy; however, such inclusion could be challenging within a limited timeframe. The application of ML models for predicting ED URVs may improve patient safety and reduce medical costs by decreasing the frequency of URVs. Further research is necessary to explore the real-world efficacy of ML models.

Predictive model for early functional outcomes following acute care after traumatic brain injuries: A machine learning-based development and validation study

INTRODUCTION

Few studies on early functional outcomes following acute care after traumatic brain injury (TBI) are available. The aim of this study was to develop and validate a predictive model for functional outcomes at discharge for TBI patients using machine learning methods.

PATIENTS AND METHODS

In this retrospective study, data from 5281 TBI patients admitted for acute care who were identified in the Beijing hospital discharge abstract database were analysed. Data from 4181 patients in 52 tertiary hospitals were used for model derivation and internal validation. Data from 1100 patients in 21 secondary hospitals were used for external validation. A poor outcome was defined as a Barthel Index (BI) score ≤ 60 at discharge. Logistic regression, XGBoost, random forest, decision tree, and back propagation neural network models were used to fit classification models. Performance was evaluated by the area under the receiver operating characteristic curve (AUC), the area under the precision-recall curve (AP), calibration plots, sensitivity/recall, specificity, positive predictive value (PPV)/precision, negative predictive value (NPV) and F1-score.

RESULTS

Compared to the other models, the random forest model demonstrated superior performance in internal validation (AUC of 0.856, AP of 0.786, and F1-score of 0.724) and external validation (AUC of 0.779, AP of 0.630, and F1-score of 0.604). The sensitivity/recall, specificity, PPV/precision, and NPV of the model were 71.8%, 69.2%, 52.2%, and 84.0%, respectively, in external validation. The BI score at admission, age, use of nonsurgical treatment, neurosurgery status, and modified Charlson Comorbidity Index were identified as the top 5 predictors for functional outcome at discharge.

CONCLUSIONS

We established a random forest model that performed well in predicting early functional outcomes following acute care after TBI. The model has utility for informing decision-making regarding patient management and discharge planning and for facilitating health care quality assessment and resource allocation for TBI treatment.

Use of Real-Time Information to Predict Future Arrivals in the Emergency Department

STUDY OBJECTIVE

We aimed to build prediction models for shift-level emergency department (ED) patient volume that could be used to facilitate prediction-driven staffing. We sought to evaluate the predictive power of rich real-time information and understand 1) which real-time information had predictive power and 2) what prediction techniques were appropriate for forecasting ED demand.

METHODS

We conducted a retrospective study in an ED site in a large academic hospital in New York City. We examined various prediction techniques, including linear regression, regression trees, extreme gradient boosting, and time series models. By comparing models with and without real-time predictors, we assessed the potential gain in prediction accuracy from real-time information.

RESULTS

Real-time predictors improved prediction accuracy on models without contemporary information from 5% to 11%. Among extensive real-time predictors examined, recent patient arrival counts, weather, Google trends, and concurrent patient comorbidity information had significant predictive power. Out of all the forecasting techniques explored, SARIMAX (Seasonal Autoregressive Integrated Moving Average with eXogenous factors) achieved the smallest out-of-sample the root mean square error (RMSE) of 14.656 and mean absolute prediction error (MAPE) of 8.703%. Linear regression was the second best, with out-of-sample RMSE and MAPE equal to 15.366 and 9.109%, respectively.

CONCLUSION

Real-time information was effective in improving the prediction accuracy of ED demand. Practice and policy implications for designing staffing paradigms with real-time demand forecasts to reduce ED congestion were discussed.

Veteran knowledge, perceptions, and receipt of care following visits to VA emergency departments for ambulatory care sensitive conditions

OBJECTIVE

Receipt of follow-up care after emergency department (ED) visits for chronic ambulatory care sensitive conditions (ACSCs)-asthma, chronic obstructive pulmonary disease, heart failure, diabetes, and/or hypertension-is crucial. We assessed Veterans' follow-up care knowledge, perceptions, and receipt of care after visits to Veterans Health Administration (VA) EDs for chronic ACSCs.

METHODS

Using explanatory sequential mixed methods, we interviewed Veterans with follow-up care needs after ACSC-related ED visits, and manually reviewed ED notes, abstracting interviewees' documented follow-up needs and care received.

RESULTS

We interviewed and reviewed ED notes of 35 Veterans, 12-27 (mean 19) days after ED visits. Follow-up care was completely received/scheduled in 20, partially received/scheduled in eight, and not received in seven Veterans. Among those who received care, it was received within specified time frames half the time. However, interviewees often did not recall these time frames or reported them to be longer than specified in the ED notes. Veterans who had not yet received or scheduled follow-up care commonly did not recall follow-up care instructions, believed that they did not need this care since they were not currently having symptoms, or thought that such care would be difficult to obtain due to appointment unavailability and/or difficulties communicating with follow-up care providers. Among the 28 Veterans in whom all or some follow-up care had been received/scheduled, for 25 cases VA staff reached out to the Veteran or the appointment was scheduled prior to or during the ED visit.

CONCLUSIONS

VA should prioritize implementing processes for EDs to efficiently communicate Veterans' needs to follow-up care providers and systems for reaching out to Veterans and/or arranging for care prior to Veterans leaving the ED. VA should also enhance practices using multimodal approaches for educating Veterans about recommended ED follow-up care and improve mechanisms for Veterans to communicate with follow-up care providers.

Benchmarking emergency department prediction models with machine learning and public electronic health records

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.

An explanatory machine learning framework for studying pandemics: The case of COVID-19 emergency department readmissions

One of the major challenges that confront medical experts during a pandemic is the time required to identify and validate the risk factors of the novel disease and to develop an effective treatment protocol. Traditionally, this process involves numerous clinical trials that may take up to several years, during which strict preventive measures must be in place to control the outbreak and reduce the deaths. Advanced data analytics techniques, however, can be leveraged to guide and speed up this process. In this study, we combine evolutionary search algorithms, deep learning, and advanced model interpretation methods to develop a holistic exploratory-predictive-explanatory machine learning framework that can assist clinical decision-makers in reacting to the challenges of a pandemic in a timely manner. The proposed framework is showcased in studying emergency department (ED) readmissions of COVID-19 patients using ED visits from a real-world electronic health records database. After an exploratory feature selection phase using genetic algorithm, we develop and train a deep artificial neural network to predict early (i.e., 7-day) readmissions (AUC = 0.883). Lastly, a SHAP model is formulated to estimate additive Shapley values (i.e., importance scores) of the features and to interpret the magnitude and direction of their effects. The findings are mostly in line with those reported by lengthy and expensive clinical trial studies.

Patterns in emergency department unscheduled return visits during the COVID-19 pandemic

Introduction

Fear surrounding nosocomial infections, expanded telehealth, and decreases in ED (emergency department) utilization altered the way patients sought emergency care during the COVID pandemic. This study aims to evaluate COVID-19's impact on the frequency and characteristics of unscheduled return visits (URVs) to the adult and pediatric ED.

Methods

In this retrospective cohort study, the electronic medical record was used to identify ≤9-day URVs at a tertiary adult and pediatric ED from 4/16/19–2/29/20 (control) and 4/16/20–2/28/21 (COVID). The primary outcome, proportion of total ED visits made up by URVs, and secondary outcomes, patient characteristics (age), illness acuity (emergency severity index (ESI)), disposition, and mortality were compared between the cohorts. Pediatric and adult data were analyzed separately. A sub-analysis was performed to exclude patients with suspected respiratory infections.

Results

For adults, n = 4265, there was no significant difference between the proportion of ED census made up by URVs (4.56% (control) vs 4.76% (COVID), p = 0.17), mean patient age (46.33 (control) vs 46.18 (COVID), p = 0.80), ESI acuity (2.95 (control) vs 2.95 (COVID), p = 0.83), disposition (admission 0.32% (control) vs 0.39% (COVID), p = 0.69), and mortality (0.23% (control) and 0.49% (COVID), p = 0.15). When excluding possible respiratory infections comparisons remained insignificant.

For pediatrics, n = 1214, there was a significant difference in the proportion of ED census made up by URVs (4.83% (control) to 3.55% (COVID), p < 0.01), age (5.52 (control) vs 6.43 (COVID), p = 0.01), and ESI acuity (3.31 (control) vs 3.17 (COVID), p < 0.01). There was no difference in disposition (admission 0.12% (control) vs 0% (COVID), p = 1). When excluding possible respiratory infections acuity (p = 0.03) remained significant.

Conclusion

In the adult population, COVID did not significantly alter any of our outcomes. For pediatric patients, a decrease in the proportion of URVs and increase in acuity during COVID suggests that patients may have had other means of accessing care, avoided the ED, received more adequate care at initial presentation, or represented when more acutely ill.

A Machine Learning Model for Predicting Unscheduled 72 h Return Visits to the Emergency Department by Patients with Abdominal Pain

Seventy-two-hour unscheduled return visits (URVs) by emergency department patients are a key clinical index for evaluating the quality of care in emergency departments (EDs). This study aimed to develop a machine learning model to predict 72 h URVs for ED patients with abdominal pain. Electronic health records data were collected from the Chang Gung Research Database (CGRD) for 25,151 ED visits by patients with abdominal pain and a total of 617 features were used for analysis. We used supervised machine learning models, namely logistic regression (LR), support vector machine (SVM), random forest (RF), extreme gradient boosting (XGB), and voting classifier (VC), to predict URVs. The VC model achieved more favorable overall performance than other models (AUROC: 0.74; 95% confidence interval (CI), 0.69–0.76; sensitivity, 0.39; specificity, 0.89; F1 score, 0.25). The reduced VC model achieved comparable performance (AUROC: 0.72; 95% CI, 0.69–0.74) to the full models using all clinical features. The VC model exhibited the most favorable performance in predicting 72 h URVs for patients with abdominal pain, both for all-features and reduced-features models. Application of the VC model in the clinical setting after validation may help physicians to make accurate decisions and decrease URVs.

Predicting 30-day return hospital admissions in patients with COVID-19 discharged from the emergency department: A national retrospective cohort study

OBJECTIVES

Identification of patients with coronavirus disease 2019 (COVID-19) at risk for deterioration after discharge from the emergency department (ED) remains a clinical challenge. Our objective was to develop a prediction model that identifies patients with COVID-19 at risk for return and hospital admission within 30 days of ED discharge.

METHODS

We performed a retrospective cohort study of discharged adult ED patients (n = 7529) with SARS-CoV-2 infection from 116 unique hospitals contributing to the National Registry of Suspected COVID-19 in Emergency Care. The primary outcome was return hospital admission within 30 days. Models were developed using classification and regression tree (CART), gradient boosted machine (GBM), random forest (RF), and least absolute shrinkage and selection (LASSO) approaches.

RESULTS

Among patients with COVID-19 discharged from the ED on their index encounter, 571 (7.6%) returned for hospital admission within 30 days. The machine-learning (ML) models (GBM, RF, and LASSO) performed similarly. The RF model yielded a test area under the receiver operating characteristic curve of 0.74 (95% confidence interval [CI], 0.71-0.78), with a sensitivity of 0.46 (95% CI, 0.39-0.54) and a specificity of 0.84 (95% CI, 0.82-0.85). Predictive variables, including lowest oxygen saturation, temperature, or history of hypertension, diabetes, hyperlipidemia, or obesity, were common to all ML models.

CONCLUSIONS

A predictive model identifying adult ED patients with COVID-19 at risk for return for return hospital admission within 30 days is feasible. Ensemble/boot-strapped classification methods (eg, GBM, RF, and LASSO) outperform the single-tree CART method. Future efforts may focus on the application of ML models in the hospital setting to optimize the allocation of follow-up resources.

Impact of diagnostic errors on adverse outcomes: learning from emergency department revisits with repeat CT or MRI

Background

To investigate diagnostic errors and their association with adverse outcomes (AOs) during patient revisits with repeat imaging (RVRIs) in the emergency department (ED).

Results

Diagnostic errors stemming from index imaging studies and AOs within 30 days in 1054 RVRIs (≤ 7 days) from 2005 to 2015 were retrospectively analyzed according to revisit timing (early [≤ 72 h] or late [> 72 h to 7 days] RVRIs). Risk factors for AOs were assessed using multivariable logistic analysis. The AO rate in the diagnostic error group was significantly higher than that in the non-error group (33.3% [77 of 231] vs. 14.8% [122 of 823], p < .001). The AO rate was the highest in early revisits within 72 h if diagnostic errors occurred (36.2%, 54 of 149). The most common diseases associated with diagnostic errors were digestive diseases in the radiologic misdiagnosis category (47.5%, 28 of 59) and neurologic diseases in the delayed radiology reporting time (46.8%, 29 of 62) and clinician error (27.3%, 30 of 110) categories. In the matched set of the AO and non-AO groups, multivariable logistic regression analysis revealed that the following diagnostic errors contributed to AO occurrence: radiologic error (odds ratio [OR] 3.56; p < .001) in total RVRIs, radiologic error (OR 3.70; p = .001) and clinician error (OR 4.82; p = .03) in early RVRIs, and radiologic error (OR 3.36; p = .02) in late RVRIs.

Conclusion

Diagnostic errors in index imaging studies are strongly associated with high AO rates in RVRIs in the ED.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13244-021-01108-0.

Using explainable machine learning to identify patients at risk of reattendance at discharge from emergency departments

Short-term reattendances to emergency departments are a key quality of care indicator. Identifying patients at increased risk of early reattendance could help reduce the number of missed critical illnesses and could reduce avoidable utilization of emergency departments by enabling targeted post-discharge intervention. In this manuscript, we present a retrospective, single-centre study where we created and evaluated an extreme gradient boosting decision tree model trained to identify patients at risk of reattendance within 72 h of discharge from an emergency department (University Hospitals Southampton Foundation Trust, UK). Our model was trained using 35,447 attendances by 28,945 patients and evaluated on a hold-out test set featuring 8847 attendances by 7237 patients. The set of attendances from a given patient appeared exclusively in either the training or the test set. Our model was trained using both visit level variables (e.g., vital signs, arrival mode, and chief complaint) and a set of variables available in a patients electronic patient record, such as age and any recorded medical conditions. On the hold-out test set, our highest performing model obtained an AUROC of 0.747 (95% CI 0.722–0.773) and an average precision of 0.233 (95% CI 0.194–0.277). These results demonstrate that machine-learning models can be used to classify patients, with moderate performance, into low and high-risk groups for reattendance. We explained our models predictions using SHAP values, a concept developed from coalitional game theory, capable of explaining predictions at an attendance level. We demonstrated how clustering techniques (the UMAP algorithm) can be used to investigate the different sub-groups of explanations present in our patient cohort.

Developing machine learning models to personalize care levels among emergency room patients for hospital admission

Objective

To develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data.

Materials and Methods

Using records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms—feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees—to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders.

Results

The best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87–0.89) and AUPRC of 0.65 (95%CI: 0.63–0.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65–0.70) and AUPRC of 0.37 (95%CI: 0.35–0.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors.

Discussion and Conclusions

Undertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.

Acuity patterns of heart failure among emergency departments in a large health system

BACKGROUND

The prognostic importance of Emergency Heart Failure Mortality Risk Grade (EHMRG) score in assessing short term mortality in Congestive Heart Failure (CHF) patients has been validated in the past, however, few studies have examined acuity patterns in the CHF population across healthcare settings. We aim to understand acuity patterns of CHF patients across a large health system for better resource utilization.

METHODS

Retrospective chart review of adult patients with acute CHF in a large Metropolitan health system was performed in 3 community and 3 academic hospitals between January 2014 and January 2016. We collected demographic data, setting type, and calculated EHMRG score. Descriptive analysis of each hospital and mixed-effects negative binomial models were created to see patterns of acuity versus hospital volume.

RESULTS

A total of 3312 Emergency Department (ED) visits among 2490 unique patients were included. Academic and community hospitals had 2168 patients and 1144 patients, respectively. Hospitals with higher volume treated a large amount of lower acuity patients. Academic hospitals had 30% of CHF ED visits in the lowest EHMRG quantile versus 20% at community hospital (p < 0.0001). Compared to EHMRG quantile 5b, hospital volume was 17%, 8% and 5% higher in quantile 1, 2, and 3 with a p-value less than 0.05 (IRR = 1.17; 1.08;1.05), respectively, but were not significant compared to quantile 4 and 5a. Revisit rates were lower in academic hospitals; admission rates were lower in community hospitals.

CONCLUSION

Academic hospitals had a higher number of Acute Heart Failure (AHF) patients, larger number of low acuity patients, higher admission rates, but less revisit rates to the ED as compared to community hospitals. We suggest acuity specific interventions will help decrease admission and revisit rates.

Generating contextual embeddings for emergency department chief complaints

Objective

We learn contextual embeddings for emergency department (ED) chief complaints using Bidirectional Encoder Representations from Transformers (BERT), a state-of-the-art language model, to derive a compact and computationally useful representation for free-text chief complaints.

Materials and methods

Retrospective data on 2.1 million adult and pediatric ED visits was obtained from a large healthcare system covering the period of March 2013 to July 2019. A total of 355 497 (16.4%) visits from 65 737 (8.9%) patients were removed for absence of either a structured or unstructured chief complaint. To ensure adequate training set size, chief complaint labels that comprised less than 0.01%, or 1 in 10 000, of all visits were excluded. The cutoff threshold was incremented on a log scale to create seven datasets of decreasing sparsity. The classification task was to predict the provider-assigned label from the free-text chief complaint using BERT, with Long Short-Term Memory (LSTM) and Embeddings from Language Models (ELMo) as baselines. Performance was measured as the Top-k accuracy from k = 1:5 on a hold-out test set comprising 5% of the samples. The embedding for each free-text chief complaint was extracted as the final 768-dimensional layer of the BERT model and visualized using t-distributed stochastic neighbor embedding (t-SNE).

Results

The models achieved increasing performance with datasets of decreasing sparsity, with BERT outperforming both LSTM and ELMo. The BERT model yielded Top-1 accuracies of 0.65 and 0.69, Top-3 accuracies of 0.87 and 0.90, and Top-5 accuracies of 0.92 and 0.94 on datasets comprised of 434 and 188 labels, respectively. Visualization using t-SNE mapped the learned embeddings in a clinically meaningful way, with related concepts embedded close to each other and broader types of chief complaints clustered together.

Discussion

Despite the inherent noise in the chief complaint label space, the model was able to learn a rich representation of chief complaints and generate reasonable predictions of their labels. The learned embeddings accurately predict provider-assigned chief complaint labels and map semantically similar chief complaints to nearby points in vector space.

Conclusion

Such a model may be used to automatically map free-text chief complaints to structured fields and to assist the development of a standardized, data-driven ontology of chief complaints for healthcare institutions.