Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario-based cross-sectional study

OBJECTIVES

Artificial intelligence companies have been increasing their initiatives recently to improve the results of chatbots, which are software programs that can converse with a human in natural language. The role of chatbots in health care is deemed worthy of research. OpenAI's ChatGPT is a supervised and empowered machine learning-based chatbot. The aim of this study was to determine the performance of ChatGPT in emergency medicine (EM) triage prediction.

METHODS

This was a preliminary, cross-sectional study conducted with case scenarios generated by the researchers based on the emergency severity index (ESI) handbook v4 cases. Two independent EM specialists who were experts in the ESI triage scale determined the triage categories for each case. A third independent EM specialist was consulted as arbiter, if necessary. Consensus results for each case scenario were assumed as the reference triage category. Subsequently, each case scenario was queried with ChatGPT and the answer was recorded as the index triage category. Inconsistent classifications between the ChatGPT and reference category were defined as over-triage (false positive) or under-triage (false negative).

RESULTS

Fifty case scenarios were assessed in the study. Reliability analysis showed a fair agreement between EM specialists and ChatGPT (Cohen's Kappa: 0.341). Eleven cases (22%) were over triaged and 9 (18%) cases were under triaged by ChatGPT. In 9 cases (18%), ChatGPT reported two consecutive triage categories, one of which matched the expert consensus. It had an overall sensitivity of 57.1% (95% confidence interval [CI]: 34-78.2), specificity of 34.5% (95% CI: 17.9-54.3), positive predictive value (PPV) of 38.7% (95% CI: 21.8-57.8), negative predictive value (NPV) of 52.6 (95% CI: 28.9-75.6), and an F1 score of 0.461. In high acuity cases (ESI-1 and ESI-2), ChatGPT showed a sensitivity of 76.2% (95% CI: 52.8-91.8), specificity of 93.1% (95% CI: 77.2-99.2), PPV of 88.9% (95% CI: 65.3-98.6), NPV of 84.4 (95% CI: 67.2-94.7), and an F1 score of 0.821. The receiver operating characteristic curve showed an area under the curve of 0.846 (95% CI: 0.724-0.969, P < 0.001) for high acuity cases.

CONCLUSION

The performance of ChatGPT was best when predicting high acuity cases (ESI-1 and ESI-2). It may be useful when determining the cases requiring critical care. When trained with more medical knowledge, ChatGPT may be more accurate for other triage category predictions.

Clinical Acuity in the Emergency Department and Injury Severity Determine Hospital Admission of Older Patients with Low Energy Falls: Outcomes from a Prospective Feasibility Study

BACKGROUND

Low energy falls (LEF) in older adults constitute a relevant cause for emergency department (ED) visits, hospital admission and in-hospital mortality. Patient-reported outcome measures containing information about patients' medical, mental and social health problems might support disposition and therapy decisions. We investigated the value of a tablet-based (self-)assessment in predicting hospital admission and in-hospital mortality.

METHODS

Patients 65 years or older, consecutively presenting with LEF to our level I trauma center ED (from November 2020 to March 2021), were eligible for inclusion in this prospective observational study. The primary endpoint was hospital admission; secondary endpoints were in-hospital mortality and the use of the tablet for self-reported assessment. Multivariate logistic regression models were calculated to measure the association between clinical findings and endpoints.

RESULTS

Of 618 eligible patients, 201 patients were included. The median age was 82 years (62.7% women). The hospital admission rate was 45.3% (110/201), with an in-hospital mortality rate of 3.6% (4/110). Polypharmacy (odds ratio (OR): 8.48; 95% confidence interval (95%CI) 1.21-59.37, p = 0.03), lower emergency severity index (ESI) scores (OR: 0.33; 95%CI 0.17-0.64, p = 0.001) and increasing injury severity score (ISS) (OR: 1.54; 95%CI 1.32-1.79, p < 0.001) were associated with hospital admission. The Charlson comorbidity index (CCI) was significantly associated with in-hospital mortality (OR: 2.60; 95%CI: 1.17-5.81, p = 0.03). Increasing age (OR: 0.94; 95%CI: 0.89-0.99, p = 0.03) and frailty (OR: 0.71; 95%CI: 0.51-0.99, p = 0.04) were associated with the incapability of tablet use.

CONCLUSIONS

The severity of fall-related injuries and the clinical acuity are easily accessible, relevant predictors for hospital admission. Tablet-based (self-)assessment may be feasible and acceptable during ED visits and might help facilitate comprehensive geriatric assessments during ED stay.

Accuracy of Emergency Severity Index in older adults

BACKGROUND AND IMPORTANCE

Emergency Severity Index is a five-level triage tool in the emergency department that predicts the need for emergency department resources and the degree of emergency. However, it is unknown whether this is valid in patients aged greater than or equal to 65 years.

OBJECTIVE

The aim of the study was to compare the accuracy of the Emergency Severity Index triage system between emergency department patients aged 18-64 and greater than or equal to 65 years.

DESIGN, SETTINGS, AND PARTICIPANTS

This was a retrospective observational cohort study of adults who presented to a Finnish emergency department between 1 February 2018 and 28 February 2018. All data were collected from electronic health records.

OUTCOME MEASURES AND ANALYSIS

The primary outcome was 3-day mortality. The secondary outcomes were 30-day mortality, hospital admission, high dependency unit or ICU admission, and emergency department length of stay. The area under the receiver operating characteristic curve and cutoff performances were used to investigate significant associations between triage categories and outcomes. The results of the two age groups were compared.

MAIN RESULTS

There were 3141 emergency department patients aged 18-64 years and 2370 patients aged greater than or equal to 65 years. The 3-day mortality area under the curve in patients aged greater than or equal to 65 years was greater than that in patients aged 18-64 years. The Emergency Severity Index was associated with high dependency unit/ICU admissions in both groups, with moderate sensitivity [18-64 years: 61.8% (50.9-71.9%); greater than or equal to 65 years: 73.3% (63.5-81.6%)] and high specificity [18-64 years: 93.0% (92.0-93.8%); greater than or equal to 65 years: 90.9% (90.0-92.1%)]. The sensitivity was high and specificity was low for 30-day mortality and hospital admission in both age groups. The emergency department length of stay was the longest in Emergency Severity Index category 3 for both age groups. There was no significant difference in accuracy between age groups for any outcome.

CONCLUSION

Emergency Severity Index performed well in predicting high dependency unit/ICU admission rates for both 18-64 years and greater than or equal to 65-year-old patients. It predicted the 3-day mortality for patients aged greater than or equal to 65 years with high accuracy. It was inaccurate in predicting 30-day mortality and hospital admission for both age groups.

Prevalence and pattern of geriatric emergencies in a teaching hospital of North India

Introduction

Changing demographic patterns worldwide and improvement in healthcarehas contributed to increasing visits to the emergency department byelderly patients. Geriatric patients usually have multiple co-morbidities and declining physiological functional status. This complex interplay of various factors requires a specific and curated approach from the emergency physicians. Our aim was to study the pattern and prevalence of geriatric emergencies and the profile of infectious and non-infectious causes of fever in geriatric population in our tertiary care center.

Materials and Methods

This retrospective descriptive study was carried out at a tertiary care hospital of north India and included all patients aged more than 18 years who visited the emergency department over a period of six months (July 2018 to December 2018). Detailed data regarding demographic, clinical and diagnosis was obtained retrospectively from the hospital records system. The patients were divided into two groups, age less than 60 years and elderly patients more than 60 years of age for comparison.

Results

A total of 24768 patients above the age of 18 years visited the emergency department over a period of six months. Out of which 5399 (27.5%) patients belonged to the geriatric age group more than 60 years of age. 2474 (45.8%) geriatric age group patientswere triaged to critical areas level one and level two as compared to 4668 (24.1%) patients aged less than 60 years. Ninety (1.8%) geriatric patients succumbed to death as compared to 77 (0.4%) patients aged less than 60 years. 651 (21.9%) geriatric patients were shifted to intensive care unit as opposed to 1038 (14.8%) patients of the younger age group.226 (4.2%) geriatric patients presented with fever in the emergency department.116 (73.4%) patients having underlying co-morbidities , had fever due to infectious causes whereas 42 ( 26.5%) patients had fever due to non-infectious causes.

Conclusion

The clinical presentation mortality and morbidity pattern of geriatric patients differs significantly from that of younger population and requires a customized approach and dedicated emergency setups.

Disposition Decision Support by Laboratory Based Outcome Prediction

Disposition is one of the main tasks in the emergency department. However, there is a lack of objective and reliable disposition criteria, and diagnosis-based risk prediction is not feasible at early time points. The aim was to derive a risk score (TRIAL) based on routinely collected baseline (TRIage level and Age) and Laboratory data—supporting disposition decisions by risk stratification based on mortality. We prospectively included consecutive patients presenting to the emergency department over 18 weeks. Data sets of routinely collected baseline (triage level and age) and laboratory data were used for multivariable logistic regression to develop the TRIAL risk score predicting mortality. Routine laboratory variables and disposition cut-offs were chosen beforehand by expert consensus. Risk stratification was based on low risk (<1%), intermediate risk (1–10%), and high risk (>10%) of in-hospital mortality. In total, 8687 data sets were analyzed. Variables identified to develop the TRIAL risk score were triage level (Emergency Severity Index), age, lactate dehydrogenase, creatinine, albumin, bilirubin, and leukocyte count. The area under the ROC curve for in-hospital mortality was 0.93. Stratification according to the TRIAL score showed that 67.5% of all patients were in the low-risk category. Mortality was 0.1% in low-risk, 3.5% in intermediate-risk, and 26.2% in high-risk patients. The TRIAL risk score based on routinely available baseline and laboratory data provides prognostic information for disposition decisions. TRIAL could be used to minimize admission in low-risk and to maximize observation in high-risk patients.

Validity and Reliability of Emergency Severity Index and Conventional Three-Tier Triage System in the Emergency Department, Hospital Universiti Sains Malaysia

Background

The study aimed to examine the reliability and validity of the existing three-tier triaging system and a new five-level emergency triaging system, emergency severity index (ESI), in the Emergency Department (ED) of Hospital Universiti Sains Malaysia (HUSM).

Methods

This study was conducted in HUSM’s ED over two study periods. In the first three months, 300 patients were triaged under the three-tier triaging system, and, in the subsequent three months, 280 patients were triaged under the ESI. The patients were triaged by junior paramedics and the triage records were retained and later re-triaged by senior paramedics. The inter-rater reliability was evaluated using Cohen's Kappa statistics. The acuity ratings of the junior paramedics were compared with those of the expert panel to determine the sensitivity and specificity of each acuity level for both the ESI and the three-tier triaging system. The over-triage rate, under-triage rate, amount of resources used, admission rate and discharge rate were also determined.

Results

The inter-rater agreement for the three-tier triaging system was 0.81 while that of the ESI was 0.75. The ESI had a higher average sensitivity of 74.3% and a specificity of 94.4% while the three-tier system’s average sensitivity was 68.5% and its specificity 87.0%. The average under-triage and over-triage rates for the ESI were 10.7% and 6.2%, respectively, which were lower than the three-tier system’s average under-triage rate of 13.1% and over-triage rate of 17.1%. The urgency levels of both the ESI and the three-tier system were associated with increased admission rates and resources used in the ED.

Conclusion

The ESI’s inter-rater reliability was comparable to the three-tier triaging system and it demonstrated better validity than the existing three-tier system.

Physicians' Disease Severity Ratings are Non-Inferior to the Emergency Severity Index

Our objective was to compare informal physicians’ disease severity ratings (PDSR) at presentation with the well-established Emergency Severity Index (ESI) in order to test for non-inferiority of the discriminatory ability regarding hospitalization, intensive care, and mortality. We made a prospective observational study with consecutive enrollment. At presentation, the PDSR and subsequently Emergency Severity Index (ESI) levels were recorded. The primary outcome was the non-inferiority of the discriminatory ability (PDSR versus ESI) regarding hospitalization, intensive care, and mortality. The secondary outcomes were the reliability, the predictive validity, and the safety of PDSR. We included 6859 patients. The median age was 51 years (interquartile range (IQR) = 33 to 72 years); 51.4% were males. There were 159 non-survivors (2.4%) at the 30 day follow-up. The PDSR’s discriminatory ability was non-inferior to the ESI’s discriminatory ability. The safety assessment showed mortality of <0.5% in low-acuity patients in both tools. The predictive validity increased by 3.5 to 7 times if adding high-acuity PDSR to ESI in all categories with mortality of >1%. Our data showed the non-inferiority of PDSR compared with ESI regarding discriminatory ability, a moderate reliability, and an acceptable safety of both tools.

Validation of the Emergency Severity Index (Version 4) for the Triage of Adult Emergency Department Patients With Active Cancer

BACKGROUND

Patients with active cancer account for a growing percentage of all emergency department (ED) visits and have a unique set of risks related to their disease and its treatments. Effective triage for this population is fundamental to facilitating their emergency care.

OBJECTIVES

We evaluated the validity of the Emergency Severity Index (ESI; version 4) triage tool to predict ED-relevant outcomes among adult patients with active cancer.

METHODS

We conducted a prespecified analysis of the observational cohort established by the National Cancer Institute-supported Comprehensive Oncologic Emergencies Research Network's multicenter (18 sites) study of ED visits by patients with active cancer (N = 1075). We used a series of χ² tests for independence to relate ESI scores with 1) disposition, 2) ED resource use, 3) hospital length of stay, and 4) 30-day mortality.

RESULTS

Among the 1008 subjects included in this analysis, the ESI distribution skewed heavily toward high acuity (>95% of subjects had an ESI level of 1, 2, or 3). ESI was significantly associated with patient disposition and ED resource use (p values < 0.05). No significant associations were observed between ESI and the non-ED based outcomes of hospital length of stay or 30-day mortality.

CONCLUSION

ESI scores among ED patients with active cancer indicate higher acuity than the general ED population and are predictive of disposition and ED resource use. These findings show that the ESI is a valid triage tool for use in this population for outcomes directly relevant to ED care.

Reliability of triage systems for paediatric emergency care: a systematic review

OBJECTIVE

To present a systematic review on the reliability of triage systems for paediatric emergency care.

METHODS

A search of MEDLINE, Cochrane Library, Latin American and Caribbean Health Sciences Literature, Scientific Electronic Library Online, Nursing Database Index and Spanish Health Sciences Bibliographic Index for articles in English, French, Portuguese or Spanish was conducted to identify reliability studies of five-level triage systems for patients aged 0-18 years published up to April 2018. Two reviewers performed study selection, data extraction and quality assessment as recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

Twenty studies on nine triage systems were selected: the National Triage System (n=1); the Australasian Triage Scale (n=3); the paediatric Canadian Triage and Acuity Scale (PedCTAS) (n=5); the Manchester Triage System (MTS) (n=1); the Emergency Severity Index (ESI) (n=5); an adaptation of the South African Triage Scale for the Princess Marina Hospital in Botswana (n=1); the Soterion Rapid Triage System (n=1); the Rapid Emergency Triage and Treatment System-paediatric version (n=2); the Paediatric Risk Classification Protocol (n=1). Ten studies were performed with actual patients, while the others used hypothetical scenarios. The studies were rated low (n=14) or moderate (n=6) quality. Kappa was the most used statistic, although many studies did not specify the weighting. PedCTAS, MTS and ESI V.4 exhibited substantial to almost perfect agreement in moderate quality studies.

CONCLUSIONS

There is some evidence on the reliability of the PedCTAS, MTS and ESI V.4, but most studies are limited to the countries where they were developed. Efforts are needed to improve the quality of the studies, and cross-cultural adaptation of those tools is recommended in countries with different professional qualification and sociocultural contexts.

Inter-rater reliability in the Paediatric Observation Priority Score (POPS)

OBJECTIVE

The primary objective of this study was to determine the level of inter-rater reliability between nursing staff for the Paediatric Observation Priority Score (POPS).

DESIGN

Retrospective observational study.

SETTING

Single-centre paediatric emergency department.

PARTICIPANTS

12 participants from a convenience sample of 21 nursing staff.

INTERVENTIONS

Participants were shown video footage of three pre-recorded paediatric assessments and asked to record their own POPS for each child. The participants were blinded to the original, in-person POPS. Further data were gathered in the form of a questionnaire to determine the level of training and experience the candidate had using the POPS score prior to undertaking this study.

MAIN OUTCOME MEASURES

Inter-rater reliability among participants scoring of the POPS.

RESULTS

Overall kappa value for case 1 was 0.74 (95% CI 0.605 to 0.865), case 2 was 1 (perfect agreement) and case 3 was 0.66 (95% CI 0.58 to 0.744).

CONCLUSION

This study suggests there is good inter-rater reliability between different nurses' use of POPS in assessing sick children in the emergency department.

A Web-Based Tool for Patient Triage in Emergency Department Settings: Validation Using the Emergency Severity Index

Background

We evaluated the concordance between triage scores generated by a novel Internet clinical decision support tool, Clinical GPS (cGPS) (Lumiata Inc, San Mateo, CA), and the Emergency Severity Index (ESI), a well-established and clinically validated patient severity scale in use today. Although the ESI and cGPS use different underlying algorithms to calculate patient severity, both utilize a five-point integer scale with level 1 representing the highest severity.

Objective

The objective of this study was to compare cGPS results with an established gold standard in emergency triage.

Methods

We conducted a blinded trial comparing triage scores from the ESI: A Triage Tool for Emergency Department Care, Version 4, Implementation Handbook to those generated by cGPS from the text of 73 sample case vignettes. A weighted, quadratic kappa statistic was used to assess agreement between cGPS derived severity scores and those published in the ESI handbook for all 73 cases. Weighted kappa concordance was defined a priori as almost perfect (kappa > 0.8), substantial (0.6 < kappa < 0.8), moderate (0.4 < kappa < 0.6), fair (0.2 < kappa< 0.4), or slight (kappa < 0.2).

Results

Of the 73 case vignettes, the cGPS severity score matched the ESI handbook score in 95% of cases (69/73 cases), in addition, the weighted, quadratic kappa statistic showed almost perfect agreement (kappa = 0.93, 95% CI 0.854-0.996). In the subanalysis of 41 case vignettes assigned ESI scores of level 1 or 2, the cGPS and ESI severity scores matched in 95% of cases (39/41 cases).

Conclusions

These results indicate that the cGPS is a reliable indicator of triage severity, based on its comparison to a standardized index, the ESI. Future studies are needed to determine whether the cGPS can accurately assess the triage of patients in real clinical environments.