Multiple performance measures are needed to evaluate triage systems in the emergency department

Machine learning-based prediction of early neurological deterioration after intravenous thrombolysis for stroke: insights from a large multicenter study

Background

This investigation seeks to ascertain the efficacy of various machine learning models in forecasting early neurological deterioration (END) following thrombolysis in patients with acute ischemic stroke (AIS).

Methods

Employing data from the Shenyang Stroke Emergency Map database, this multicenter study compiled information on 7,570 AIS patients from 29 comprehensive hospitals who received thrombolytic therapy between January 2019 and December 2021. An independent testing cohort was constituted from 2,046 patients at the First People's Hospital of Shenyang. The dataset incorporated 15 pertinent clinical and therapeutic variables. The principal outcome assessed was the occurrence of END post-thrombolysis. Model development was executed using an 80/20 split for training and internal validation, employing classifiers like logistic regression with lasso regularization (lasso regression), support vector machine (SVM), random forest (RF), gradient-boosted decision tree (GBDT), and multi-layer perceptron (MLP). The model with the highest area under the curve (AUC) was utilized to delineate feature significance.

Results

Baseline characteristics showed variability in END incidence between the training (n = 7,570; END incidence 22%) and external validation cohorts (n = 2,046; END incidence 10%; p < 0.001). Notably, all machine learning models demonstrated superior AUC values compared to the reference model, indicating their enhanced predictive capacity. The lasso regression model achieved the highest AUC at 0.829 (95% CI: 0.799-0.86; p < 0.001), closely followed by the MLP model with an AUC of 0.828 (95% CI: 0.799-0.858; p < 0.001). The SVM, RF, and GBDT models also showed commendable AUCs of 0.753, 0.797, and 0.774, respectively. Decision curve analysis revealed that the SVM and MLP models demonstrated a high net benefit. Feature importance analysis emphasized "Onset To Needle Time" and "Admission NIHSS Score" as significant predictors.

Conclusion

Our research establishes the MLP and lasso regression as robust tools for predicting early neurological deterioration in acute ischemic stroke patients following thrombolysis. Their superior predictive accuracy, compared to traditional models, highlights the significant potential of machine learning approaches in refining prognosis and enhancing clinical decisions in stroke care management. This advancement paves the way for more tailored therapeutic strategies, ultimately aiming to improve patient outcomes in clinical practice.

Comparison between the Smart Triage model and the Emergency Triage Assessment and Treatment guidelines in triaging children presenting to the emergency departments of two public hospitals in Kenya

Several triage systems have been developed, but little is known about their performance in low-resource settings. Evaluating and comparing novel triage systems to existing triage scales provides essential information about their added value, reliability, safety, and effectiveness before adoption. This study included children aged < 15 years who presented to the emergency departments of two public hospitals in Kenya between February and December 2021. We compared the performance of Emergency Triage Assessment and Treatment (ETAT) guidelines and Smart Triage (ST) models (ST model with independent triggers, and recalibrated ST model with independent triggers) in categorizing children into emergency, priority, and non-urgent triage categories. Sankey diagrams were used to visualize the distribution of children into similar or different triage categories by ETAT and ST models. Sensitivity, specificity, negative and positive predictive values for mortality and admission were calculated. 5618 children were enrolled, and the majority (3113, 55.4%) were aged between one and five years of age. Overall admission and mortality rates were 7% and 0.9%, respectively. ETAT classified 513 (9.2%) children into the emergency category compared to 1163 (20.8%) and 1161 (20.7%) by the ST model with independent triggers and recalibrated model with independent triggers, respectively. ETAT categorized 3089 (55.1%) children as non-urgent compared to 2097 (37.4%) and 2617 (46.7%) for the respective ST models. ETAT classified 191/395 (48.4%) admitted patients as emergencies compared to more than half by all the ST models. ETAT and ST models classified 25/49 (51%) and 39/49 (79.6%) deceased children as emergencies. Sensitivity for admission and mortality was 48.4% and 51% for ETAT and 74.9% and 79.6% for the ST models, respectively. Smart Triage shows potential for identifying critically ill children in low-resource settings, particularly when combined with independent triggers and performs comparably to ETAT. Evaluation of Smart Triage in other contexts and comparison to other triage systems is required.

Machine learning-based prediction of symptomatic intracerebral hemorrhage after intravenous thrombolysis for stroke: a large multicenter study

Background

This study aimed to compare the performance of different machine learning models in predicting symptomatic intracranial hemorrhage (sICH) after thrombolysis treatment for ischemic stroke.

Methods

This multicenter study utilized the Shenyang Stroke Emergency Map database, comprising 8,924 acute ischemic stroke patients from 29 comprehensive hospitals who underwent thrombolysis between January 2019 and December 2021. An independent testing cohort was further established, including 1,921 patients from the First People's Hospital of Shenyang. The structured dataset encompassed 15 variables, including clinical and therapeutic metrics. The primary outcome was the sICH occurrence post-thrombolysis. Models were developed using an 80/20 split for training and internal validation. Performance was assessed using machine learning classifiers, including logistic regression with lasso regularization, support vector machine (SVM), random forest, gradient-boosted decision tree (GBDT), and multilayer perceptron (MLP). The model boasting the highest area under the curve (AUC) was specifically employed to highlight feature importance.

Results

Baseline characteristics were compared between the training cohort (n = 6,369) and the external validation cohort (n = 1,921), with the sICH incidence being slightly higher in the training cohort (1.6%) compared to the validation cohort (1.1%). Among the evaluated models, the logistic regression with lasso regularization achieved the highest AUC of 0.87 (95% confidence interval [CI]: 0.79-0.95; p < 0.001), followed by the MLP model with an AUC of 0.766 (95% CI: 0.637-0.894; p = 0.04). The reference model and SVM showed AUCs of 0.575 and 0.582, respectively, while the random forest and GBDT models performed less optimally with AUCs of 0.536 and 0.436, respectively. Decision curve analysis revealed net benefits primarily for the SVM and MLP models. Feature importance from the logistic regression model emphasized anticoagulation therapy as the most significant negative predictor (coefficient: -2.0833) and recombinant tissue plasminogen activator as the principal positive predictor (coefficient: 0.5082).

Conclusion

After a comprehensive evaluation, the MLP model is recommended due to its superior ability to predict the risk of symptomatic hemorrhage post-thrombolysis in ischemic stroke patients. Based on decision curve analysis, the MLP-based model was chosen and demonstrated enhanced discriminative ability compared to the reference. This model serves as a valuable tool for clinicians, aiding in treatment planning and ensuring more precise forecasting of patient outcomes.

Strategies to improve the quality of nurse triage in emergency departments: A realist review protocol

AIM

The purpose of this realist review was to assess what works, for whom and in what context, regarding strategies that influence nurses' behaviour to improve triage quality in emergency departments (ED).

DESIGN

Realist review protocol.

METHODS

This protocol follows the PRISMA-P statement and will include any type of study on strategies to improve the triage process in the ED (using recognized and validated triage scales). The included studies were examined for scientific quality using the Mixed Methods Appraisal Tool. The framework for this realist review is based on the Behaviour Change Wheel (BCW) and the context-mechanism-outcome (CMO) models.

DISCUSSION

Nurses and ED decision makers will be informed on the evidence regarding strategies to improve the quality of triage and the factors required to maximize their effectiveness. Research gaps may also be identified to guide future research projects on the adoption of best practices in ED nursing triage.

FRENCH versus ESI: comparison between two nurse triage emergency scales with referent scenarios

OBJECTIVES

Acute triage is needed to prioritize care and achieve optimal resource allocation in busy emergency departments. The main objective is to compare the FRench Emergency Nurse Classification in Hospital scale (FRENCH) to the American scale Emergency Severity Index (ESI). Secondary objectives are to compare for each scale the over and under-triage, the triage matching to the gold standard and the inter-individual sorting reproducibility between the nurses.

METHODS

This is a prospective observational study conducting among the nursing staffs and nursing students, selected from Caen University College Hospital and Lisieux Hospital Center emergency departments between two months. Each group individually rank 60 referent clinical cases composed by scales designers. An assessment of scale practicality is collected after for each tool. The collected parameters are analyzed by a Cohen kappa concordance test (κ).

RESULTS

With 8151 triage results of gold standard scenarios sorting in two scales by the same nurses, the FRENCH scale seems to give better triage results than the US ESI scale (nurse: FRENCH 60% and ESI 53%, p = 0.003 ; nursing students: FRENCH 49% and ESI 42%, p < 0.001). In the two groups ESI has also a big tendency to under-sort (p = 0.01), particularly for the most severe patients (p < 0.01). The interobserver sorting concordance for any experience gives good results for the FRENCH and the ESI without any difference (nurses : FRENCH K_PQ=0.72 ESI K_PQ=0.78; p = 0.32 ; students K_PQ=0.44 K_PQ=0.55; p = 0.22).

CONCLUSION

The ESI and FRENCH scales comparison on 8151 sorting results shows direct validity in favor of FRENCH one and similar interobserver agreement for both scales.

Latent trajectories of frailty and risk prediction models among geriatric community dwellers: an interpretable machine learning perspective

BACKGROUND

This study aimed to identify long-term frailty trajectories among older adults (≥65) and construct interpretable prediction models to assess the risk of developing abnormal frailty trajectory among older adults and examine significant factors related to the progression of frailty.

METHODS

This study retrospectively collected data from the Chinese Longitudinal Healthy Longevity and Happy Family Study between 2002 and 2018 (N = 4083). Frailty was defined by the frailty index. The whole study consisted of two phases of tasks. First, group-based trajectory modeling was used to identify frailty trajectories. Second, easy-to-access epidemiological data was utilized to construct machine learning algorithms including naïve bayes, logistic regression, decision tree, support vector machine, random forest, artificial neural network, and extreme gradient boosting to predict the risk of long-term frailty trajectories. Further, Shapley additive explanations was employed to identify feature importance and open-up the black box model of machine learning to further strengthen decision makers' trust in the model.

RESULTS

Two distinct frailty trajectories (stable-growth: 82.54%, rapid-growth: 17.46%) were identified. Compared with other algorithms, random forest performed relatively better in distinguishing the stable-growth and rapid-growth groups. Physical function including activities of daily living and instrumental activities of daily living, marital status, weight, and cognitive function were top five predictors.

CONCLUSIONS

Interpretable machine learning can achieve the primary goal of risk stratification and make it more transparent in individual prediction beneficial to primary screening and tailored prevention.

Improving triage for children with comorbidity using the ED-PEWS: an observational study

OBJECTIVE

To assess the value of the Emergency Department-Pediatric Early Warning Score (ED-PEWS) for triage of children with comorbidity.

DESIGN

Secondary analysis of a prospective cohort.

SETTING AND PATIENTS

53 829 consecutive ED visits of children <16 years in three European hospitals (Netherlands, UK and Austria) participating in the TrIAGE (Triage Improvements Across General Emergency departments) project in different periods (2012-2015).

INTERVENTION

ED-PEWS, a score consisting of age and six physiological parameters.

MAIN OUTCOME MEASURE

A three-category reference standard as proxy for true patient urgency. We assessed discrimination and calibration of the ED-PEWS for children with comorbidity (complex and non-complex) and without comorbidity. In addition, we evaluated the value of adding the ED-PEWS to the routinely used Manchester Triage System (MTS).

RESULTS

5053 (9%) children had underlying non-complex morbidity and 5537 (10%) had complex comorbidity. The c-statistic for identification of high-urgency patients was 0.86 (95% prediction interval 0.84-0.88) for children without comorbidity, 0.87 (0.82-0.92) for non-complex and 0.86 (0.84-0.88) for complex comorbidity. For high and intermediate urgency, the c-statistic was 0.63 (0.62-0.63), 0.63 (0.61-0.65) and 0.63 (0.55-0.73) respectively. Sensitivity was slightly higher for children with comorbidity (0.73-0.75 vs 0.70) at the cost of a lower specificity (0.86-0.87 vs 0.92). Calibration was largely similar. Adding the ED-PEWS to the MTS for children with comorbidity improved performance, except in the setting with few high-urgency patients.

CONCLUSIONS

The ED-PEWS has a similar performance in children with and without comorbidity. Adding the ED-PEWS to the MTS for children with comorbidity improves triage, except in the setting with few high-urgency patients.

Agreement and validity of electronic patient self-triage (eTriage) with nurse triage in two UK emergency departments: a retrospective study

BACKGROUND

Triage and redirection of patients to alternative care providers is one tool used to overcome the growing issue of crowding in emergency departments (EDs). Electronic patient self-triage (eTriage) may reduce waiting times and required face-to-face contact. There are limited studies into its efficacy, accuracy and validity in an ED setting.

OBJECTIVES

The aim of this study was to assess the agreement and validity of eTriage with a reference standard of nurse face-to-face triage. A secondary aim was to assess the ability of both systems to predict high and low acuity outcomes.

DESIGN

This was a retrospective study conducted over 8 months in two UK hospitals. Inclusion criteria were all ambulatory patients aged ≥18. All patients completed an eTriage and nurse-led triage using the Manchester Triage System (MTS).

MAIN RESULTS

During the study period, 43 788 adult patients attended one of the two ED sites and 26 757 used eTriage. A total of 1424 patient episodes had no recorded MTS and were excluded from the study leaving 25 333 paired triages for the final cohort. Agreement between eTriage and nurse triage was low with a weighted Kappa coefficient of 0.14 (95% CI, 0.14-0.15) with an associated weak positive correlation (rs 0.321). Level of undertriage by eTriage compared with nurse triage was 10.1%, and overtriage was 59.2%. The sensitivity for prediction of high acuity outcomes was 88.5% (95% CI, 77.9-95.3%) for eTriage and 53.8% (95% CI 41.1-66.0%) for nurse MTS. The specificity for predicting low risk patients was 88.5% (95% CI, 87.4-89.5%) for eTriage and 80.6% (95% CI, 79.3-81.8%) for nurse MTS.

CONCLUSION

Agreement and correlation of eTriage with the reference standard of nurse MTS was low; patients using eTriage tended to over triage when compared to the triage nurse. eTriage had a higher sensitivity for high acuity presentations and demonstrated similar specificity for low acuity presentations when compared to triage nurse MTS. Further work is necessary to validate eTriage as a potential tool for safe redirection of ED attenders to alternative care providers.

Machine learning methods applied to triage in emergency services: A systematic review

BACKGROUND

In emergency services is important to accurately assess and classify symptoms, which may be improved with the help of technology. One mechanism that could help and improve predictions from health records or patient flow is machine learning (ML).

AIM

To analyse the effectiveness of ML systems in triage for making predictions at the emergency department in comparison with other triage scales/scores.

METHODS

Following the PRISMA recommendations, a systematic review was conducted using CINAHL, Cochrane, Cuiden, Medline and Scopus databases with the search equation "Machine learning AND triage AND emergency".

RESULTS

Eleven studies were identified. The studies show that the use of ML methods consistently predict important outcomes like mortality, critical care outcomes and admission, and the need for hospitalization in comparison with scales like Emergency Severity Index or others. Among the ML models considered, XGBoost and Deep Neural Networks obtained the highest levels of prediction accuracy, while Logistic Regression performed obtained the worst values.

CONCLUSIONS

Machine learning methods can be a good instrument for helping triage process with the prediction of important emergency variables like mortality or the need for critical care or hospitalization.

Use of machine learning approach to predict depression in the elderly in China: A longitudinal study

BACKGROUND

Early detection of potential depression among elderly people is conducive for timely preventive intervention and clinical care to improve quality of life. Therefore, depression prediction considering sequential progression patterns in elderly needs to be further explored.

METHODS

We selected 1,538 elderly people from Chinese Longitudinal Healthy Longevity Study (CLHLS) wave 3-7 survey. Long short-term memory (LSTM) and six machine learning (ML) models were used to predict different depression risk factors and the depression risks in the elderly population in the next two years. Receiver operating curve (ROC) and decision curve analysis (DCA) were used to evaluate the prediction accuracy of the reference model and ML models.

RESULTS

The area under the ROC curve (AUC) values of logistic regression with lasso regularisation (AUC=0.629, p-value=0.020) was the highest among ML models. DCA results showed that the net benefit of six ML models was similar (threshold: 0.00-0.10), the net benefit of lasso regression was the largest (threshold: 0.10-0.17 and 0.22-0.25), and the net benefit of DNN was the largest (threshold: 0.17-0.22 and 0.25-0.40). In two ML models, activities of daily living (ADL)/ instrumental ADL (IADL), self-rated health, marital status, arthritis, and number of cohabiting were the most important predictors for elderly with depression.

LIMITATIONS

The retrospective waves used in the LSTM model need to be further increased.

CONCLUSION

The decision support system based on the proposed LSTM+ML model may be very valuable for doctors, nurses and community medical providers for early diagnosis and intervention.

Improving the prioritization of children at the emergency department: Updating the Manchester Triage System using vital signs

Background

Vital signs are used in emergency care settings in the first assessment of children to identify those that need immediate attention. We aimed to develop and validate vital sign based Manchester Triage System (MTS) discriminators to improve triage of children at the emergency department.

Methods and findings

The TrIAGE project is a prospective observational study based on electronic health record data from five European EDs (Netherlands (n = 2), United Kingdom, Austria, and Portugal). In the current study, we included 117,438 consecutive children <16 years presenting to the ED during the study period (2012–2015). We derived new discriminators based on heart rate, respiratory rate, and/or capillary refill time for specific subgroups of MTS flowcharts. Moreover, we determined the optimal cut-off value for each vital sign. The main outcome measure was a previously developed 3-category reference standard (high, intermediate, low urgency) for the required urgency of care, based on mortality at the ED, immediate lifesaving interventions, disposition and resource use. We determined six new discriminators for children <1 year and ≥1 year: “Very abnormal respiratory rate”, “Abnormal heart rate”, and “Abnormal respiratory rate”, with optimal cut-offs, and specific subgroups of flowcharts. Application of the modified MTS reclassified 744 patients (2.5%). Sensitivity increased from 0.66 (95%CI 0.60–0.72) to 0.71 (0.66–0.75) for high urgency patients and from 0.67 (0.54–0.76) to 0.70 (0.58–0.80) for high and intermediate urgency patients. Specificity decreased from 0.90 (0.86–0.93) to 0.89 (0.85–0.92) for high and 0.66 (0.52–0.78) to 0.63 (0.50–0.75) for high and intermediate urgency patients. These differences were statistically significant. Overall performance improved (R² 0.199 versus 0.204).

Conclusions

Six new discriminators based on vital signs lead to a small but relevant increase in performance and should be implemented in the MTS.

Contribution of social factors to readmissions within 30 days after hospitalization for COPD exacerbation

BACKGROUND

To investigate whether, in patients hospitalized for COPD, the addition of social factors improves the predictive ability for the risk of overall 30-day readmissions, early readmissions (within 7 days after discharge), and late readmissions (8-30 days after discharge).

METHODS

Patients (aged ≥40 years) hospitalized for COPD were identified in the Medicare Current Beneficiary Survey from 2006 through 2012. With the use of 1000 bootstrap resampling from the original cohort (training-set), two prediction models were derived: 1) the reference model including age, comorbidities, and mechanical ventilation use, and 2) the optimized model including social factors (e.g., educational level, marital status) in addition to the covariates in the reference model. Prediction performance was examined separately for 30-day, early, and late readmissions.

RESULTS

Following 905 index hospitalizations for COPD, 18.5% were readmitted within 30 days. In the test-set, for overall 30-day readmissions, the discrimination ability between reference and optimized models did not change materially (C-statistic, 0.57 vs. 0.58). By contrast, for early readmissions, the optimized model had significantly improved discrimination (C-statistic, 0.57 vs. 0.63; integrated discrimination improvement [IDI], 0.018 [95%CI, 0.003-0.032]) and reclassification (continuous net reclassification index [NRI], 0.298 [95%CI 0.060-0.537]). Likewise, for late readmissions, the optimized model also had significantly improved discrimination (C-statistic, 0.65 vs. 0.68; IDI, 0.026 [95%CI 0.009-0.042]) and reclassification (continuous NRI, 0.243 [95%CI 0.028-0.459]).

CONCLUSIONS

In a nationally-representative sample of Medicare beneficiaries hospitalized for COPD, we found that the addition of social factors improved the predictive ability for readmissions when early and late readmissions were examined separately.

["Triage"-primary assessment of patients in the emergency department : An overview with a systematic review]

BACKGROUND

"Triage" means the primary assessment of a previously unknown patient with an acute health disorder, initially considered a medical emergency. The initial triage is part of the primary assessment, which also includes the registration of administrative data and patient's mode of arrival.

OBJECTIVES

The aim of the work is to provide an overview of frequently used structured primary assessment tools and the underlying evidence for their use in the emergency room.

METHODS

Based on a systematic literature search in PubMed, 41 articles were selected according to predefined criteria.

RESULTS

The most frequently used primary assessment systems in Germany are the Emergency Severity Index (ESI) and the Manchester Triage System (MTS). Scientific evidence exists for the accuracy and reliability of the primary assessment with these instruments. However, there are no gold standards for measuring urgency, so that separate criteria must be defined. Sufficient data to determine a treatment sector or the necessary staffing levels are lacking.

CONCLUSIONS

Structured primary assessment using formalized systems alone is inadequate to categorize the urgency of emergency and acute patients. In fact, a combination of different measures in an interprofessional team is required. Primary assessment systems and processes generally do not allow patients to be referred to downstream structures without a thorough medical examination.

Machine Learning-Based Prediction of Clinical Outcomes for Children During Emergency Department Triage

IMPORTANCE

While machine learning approaches may enhance prediction ability, little is known about their utility in emergency department (ED) triage.

OBJECTIVES

To examine the performance of machine learning approaches to predict clinical outcomes and disposition in children in the ED and to compare their performance with conventional triage approaches.

DESIGN, SETTING, AND PARTICIPANTS

Prognostic study of ED data from the National Hospital Ambulatory Medical Care Survey from January 1, 2007, through December 31, 2015. A nationally representative sample of 52 037 children aged 18 years or younger who presented to the ED were included. Data analysis was performed in August 2018.

MAIN OUTCOMES AND MEASURES

The outcomes were critical care (admission to an intensive care unit and/or in-hospital death) and hospitalization (direct hospital admission or transfer). In the training set (70% random sample), using routinely available triage data as predictors (eg, demographic characteristics and vital signs), we derived 4 machine learning-based models: lasso regression, random forest, gradient-boosted decision tree, and deep neural network. In the test set (the remaining 30% of the sample), we measured the models' prediction performance by computing C statistics, prospective prediction results, and decision curves. These machine learning models were built for each outcome and compared with the reference model using the conventional triage classification information.

RESULTS

Of 52 037 eligible ED visits by children (median [interquartile range] age, 6 [2-14] years; 24 929 [48.0%] female), 163 (0.3%) had the critical care outcome and 2352 (4.5%) had the hospitalization outcome. For the critical care prediction, all machine learning approaches had higher discriminative ability compared with the reference model, although the difference was not statistically significant (eg, C statistics of 0.85 [95% CI, 0.78-0.92] for the deep neural network vs 0.78 [95% CI, 0.71-0.85] for the reference; P = .16), and lower number of undertriaged critically ill children in the conventional triage levels 3 to 5 (urgent to nonurgent). For the hospitalization prediction, all machine learning approaches had significantly higher discrimination ability (eg, C statistic, 0.80 [95% CI, 0.78-0.81] for the deep neural network vs 0.73 [95% CI, 0.71-0.75] for the reference; P < .001) and fewer overtriaged children who did not require inpatient management in the conventional triage levels 1 to 3 (immediate to urgent). The decision curve analysis demonstrated a greater net benefit of machine learning models over ranges of clinical thresholds.

CONCLUSIONS AND RELEVANCE

Machine learning-based triage had better discrimination ability to predict clinical outcomes and disposition, with reduction in undertriaging critically ill children and overtriaging children who are less ill.