Development of the Canadian Emergency Department Diagnosis Shortlist

Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario-based cross-sectional study

OBJECTIVES

Artificial intelligence companies have been increasing their initiatives recently to improve the results of chatbots, which are software programs that can converse with a human in natural language. The role of chatbots in health care is deemed worthy of research. OpenAI's ChatGPT is a supervised and empowered machine learning-based chatbot. The aim of this study was to determine the performance of ChatGPT in emergency medicine (EM) triage prediction.

METHODS

This was a preliminary, cross-sectional study conducted with case scenarios generated by the researchers based on the emergency severity index (ESI) handbook v4 cases. Two independent EM specialists who were experts in the ESI triage scale determined the triage categories for each case. A third independent EM specialist was consulted as arbiter, if necessary. Consensus results for each case scenario were assumed as the reference triage category. Subsequently, each case scenario was queried with ChatGPT and the answer was recorded as the index triage category. Inconsistent classifications between the ChatGPT and reference category were defined as over-triage (false positive) or under-triage (false negative).

RESULTS

Fifty case scenarios were assessed in the study. Reliability analysis showed a fair agreement between EM specialists and ChatGPT (Cohen's Kappa: 0.341). Eleven cases (22%) were over triaged and 9 (18%) cases were under triaged by ChatGPT. In 9 cases (18%), ChatGPT reported two consecutive triage categories, one of which matched the expert consensus. It had an overall sensitivity of 57.1% (95% confidence interval [CI]: 34-78.2), specificity of 34.5% (95% CI: 17.9-54.3), positive predictive value (PPV) of 38.7% (95% CI: 21.8-57.8), negative predictive value (NPV) of 52.6 (95% CI: 28.9-75.6), and an F1 score of 0.461. In high acuity cases (ESI-1 and ESI-2), ChatGPT showed a sensitivity of 76.2% (95% CI: 52.8-91.8), specificity of 93.1% (95% CI: 77.2-99.2), PPV of 88.9% (95% CI: 65.3-98.6), NPV of 84.4 (95% CI: 67.2-94.7), and an F1 score of 0.821. The receiver operating characteristic curve showed an area under the curve of 0.846 (95% CI: 0.724-0.969, P < 0.001) for high acuity cases.

CONCLUSION

The performance of ChatGPT was best when predicting high acuity cases (ESI-1 and ESI-2). It may be useful when determining the cases requiring critical care. When trained with more medical knowledge, ChatGPT may be more accurate for other triage category predictions.

A Simple Admission Order-set Improves Adherence to Canadian Guidelines for Hospitalized Patients With Severe Ulcerative Colitis

Background

Individuals hospitalized with severe ulcerative colitis represent a complex group of patients. Variation exists in the quality of care of admitted patients with inflammatory bowel disease. We hypothesized that implementation of a standardized admission order set could result in improved adherence to current best practice guidelines (Toronto Consensus Statements) for the management of this patient population.

Methods

A retrospective cohort study of patients admitted with severe ulcerative colitis to a Montreal tertiary center was conducted. Two cohorts were defined based on pre- and post-implementation of a standardized order set. Adherence to 11 quality indicators was assessed before and after implementation of the intervention. These included: Clostridioides difficile and stool cultures testing, ordering an abdominal X-ray and CRP, organizing a flexible sigmoidoscopy, documenting latent tuberculosis, initiating thromboprophylaxis, use of intravenous steroids, prescribing infliximab if refractory to steroids, limiting narcotics, and surgical consultation if refractory to medical therapy.

Results

Adherence to 6 of the 11 quality indicators was improved in the post-intervention cohort. Significant increases were noted in adherence to C difficile testing (75.5% versus 91.9%, P &lt; 0.05), CRP testing (71.4% versus 94.6%, P &lt; 0.01), testing for latent tuberculosis (38.1% versus 84.6%, P &lt; 0.01), thromboprophylaxis (28.6% versus 94.6%, P &lt; 0.01), adequate corticosteroids prescription (72.9% versus 94.6%, P &lt; 0.01), and limitation of narcotics prescribed (68.8% versus 38.9%, P &lt; 0.01).

Conclusions

Implementation of a standardized order set, focused on pre-defined quality indicators for hospitalized patients with severe UC, was associated with meaningful improvements to most quality indicators defined by the Toronto Consensus Statements.

Calibrating a chief complaint list for low resource settings: a methodologic case study

BACKGROUND

The chief or presenting complaint is the reason for seeking health care, often in the patient's own words. In limited resource settings, a diagnosis-based approach to quantifying burden of disease is not possible, partly due to limited availability of an established lexicon or coding system. Our group worked with colleagues from the African Federation of Emergency Medicine building on the existing literature to create a pilot symptom list representing an attempt to standardize undifferentiated chief complaints in emergency and acute care settings. An ideal list for any setting is one that strikes a balance between ease of use and length, while covering the vast majority of diseases with enough detail to permit epidemiologic surveillance and make informed decisions about resource needs.

METHODS

This study was incorporated as a part of a larger prospective observational study on human immunodeficiency virus testing in Emergency Departments in South Africa. The pilot symptom list was used for chief complaint coding in three Emergency Departments. Data was collected on 3357 patients using paper case report forms. Chief complaint terms were reviewed by two study team members to determine the frequency of concordance between the coded chief complaint term and the selected symptom(s) from the pilot symptom list.

RESULTS

Overall, 3537 patients' chief complaints were reviewed, of which 640 were identified as 'potential mismatches.' When considering the 191 confirmed mismatches (29.8%), the Delphi process identified 6 (3.1%) false mismatches and 185 (96.9%) true mismatches. Significant chief-complaint clustering was identified with 9 sets of complaints frequently selected together for the same patient. "Pain" was used 2076 times for 58.7% of all patients. A combination of user feedback and expert-panel modified Delphi analysis of mismatched complaints and clustered complaints resulted in several substantial changes to the pilot symptom list.

CONCLUSIONS

This study presented a systematic methodology for calibrating a chief complaint list for the local context. Our revised list removed/reworded symptoms that frequently clustered together or were misinterpreted by health professionals. Recommendations for additions, modifications, and/or deletions from the pilot chief complaint list we believe will improve the functionality of the list in low resource environments.

Classification of hospital admissions into emergency and elective care: a machine learning approach

Rising admissions from emergency departments (EDs) to hospitals are a primary concern for many healthcare systems. The issue of how to differentiate urgent admissions from non-urgent or even elective admissions is crucial. We aim to develop a model for classifying inpatient admissions based on a patient's primary diagnosis as either emergency care or elective care and predicting urgency as a numerical value. We use supervised machine learning techniques and train the model with physician-expert judgments. Our model is accurate (96%) and has a high area under the ROC curve (>.99). We provide the first comprehensive classification and urgency categorization for inpatient emergency and elective care. This model assigns urgency values to every relevant diagnosis in the ICD catalog, and these values are easily applicable to existing hospital data. Our findings may provide a basis for policy makers to create incentives for hospitals to reduce the number of inappropriate ED admissions.

Emergency Department Return Visits Within a Large Geographic Area

BACKGROUND

Return visits to the emergency department (RTED) contribute to overcrowding and may be a quality of care indicator. Previous studies focused on factors predicting returns to and from the same center. Little is known about RTEDs across a range of community and specialty hospitals within a large geographic area.

OBJECTIVE

We sought to measure the frequency of pediatric RTEDs and describe their directional pattern across centers in a large catchment area.

METHODS

We conducted a multicenter, retrospective cross-sectional study of pediatric emergency visits in the Vancouver lower mainland within 1 year. Visits were linked across study sites, including one pediatric quaternary care referral center and 17 sites ranging from large regional centers to smaller community emergency departments (EDs). Returns were defined as subsequent visits to any site with a compatible diagnosis within 7 days of an index visit.

RESULTS

Among a total of 139,278 index ED visits by children, 12,133 (8.7% [95% confidence interval 8.6-8.9%]) were associated with 14,645 return visits to an ED. Three quarters of all index visits occurred at a general ED center, of which 8.9% had at least one RTED and 22% of these returns occurred at the pediatric ED (PED). Among PED index visits, 8.2% had at least one RTED and 13.6% of these returned to a general center. Overall, 38.9% of all RTEDs occurred at the PED. Multivariate regression did not identify any statistically significant association between ED crowding measures and likelihood of RTEDs.

CONCLUSIONS

Compared to single-center studies, this study linking hospitals within a large geographic area identified a higher proportion of RTEDs with a disproportionate burden on the PED.

Predictors of Emergency Department Use by Persons with Inflammatory Bowel Diseases: A Population-based Study

BACKGROUND

To describe the patterns and predictors of emergency department (ED) attendance and post-ED hospitalization by persons with inflammatory bowel disease (IBD).

METHODS

We linked the University of Manitoba IBD Epidemiology Database with the Emergency Department Information System of the Winnipeg Regional Health Authority to determine the rates of presentation to the ED by persons with IBD from January 01, 2009 to March 31, 2012. Incident cases were diagnosed during the study period and all others were considered prevalent cases. Multivariate logistic regression was used to determine predictors of attendance in the ED and for hospitalization within 2 days of ED attendance.

RESULTS

The study population included 300 incident and 3394 prevalent IBD cases, of whom 76% and 49%, respectively, attended the ED at least once during the study period. Incident cases with Crohn's disease or with a history of opioid use were more likely to attend the ED. Those who had seen a gastroenterologist within the year before diagnosis were less likely to visit the ED. Among prevalent cases, higher comorbidity, opioid or corticosteroid use, and recent hospital admission were predictive of ED attendance and those who saw only 1 physician in the preceding year had lower ED attendance. Presenting to the ED with a primary gastrointestinal complaint was the strongest predictor of subsequent hospital admission.

CONCLUSIONS

ED attendance by both incident and prevalent cases of IBD is high. Identified predictors of ED attendance and post-ED hospitalization could guide the optimization of outpatient IBD care to limit ED attendance and potentially post-ED hospitalization.

[Prevalence of non-traumatic musculoskeletal pathology as main complaint and its impact in a emergency department]

BACKGROUND

Non-traumatic musculoskeletal pathology (NTMP) generates a high healthcare demand in primary care, however, European studies designed to assess its real impact in Emergency Departments are scarce. The present study aims to determine the prevalence of NTMP and its impact in Emergency Department of a university hospital in Madrid.

MATERIAL AND METHOD

Two thousand randomized medical registries were reviewed from 2008 to 2011. The epidemiological data collected were, main complaints, time consumed, image test requests, and need of further assessment within a month.

RESULTS

Prevalence of NTMP was 13.8% (95% CI; 12.1%-15.4%) of all patients. The most frequent musculoskeletal complaint was lumbar pain. An imaging test was requested in 79.1% of all the NTMP cases assessed. Patients with NTMP consumed an average of 79 minutes, with 17% of them requesting a new urgent assessment within the first month.

CONCLUSIONS

The results of this study show that NTMP is the leading cause for emergency department visits in our area, producing the highest consumption of time and the highest frequency of new queries for the same reason within a month. The overuse of the emergency services and the lack of medical training in the management of this type of pathology can cause this phenomenon. During the design of strategies to optimize patients care in emergency departments, the importance of this type of pathology should be taken into account.

Making recording and analysis of chief complaint a priority for global emergency care research in low-income countries

The chief complaint is a patient's self-reported primary reason for presenting for medical care. The clinical utility and analytical importance of recording chief complaints have been widely accepted in highly developed emergency care systems, but this practice is far from universal in global emergency care, especially in limited-resource areas. It is precisely in these settings, however, that the use of chief complaints may have particular benefit. Chief complaints may be used to quantify, analyze, and plan for emergency care and provide valuable information on acute care needs where there are crucial data gaps. Globally, much work has been done to establish local practices around chief complaint collection and use, but no standards have been established and little work has been done to identify minimum effective sets of chief complaints that may be used in limited-resource settings. As part of the Academic Emergency Medicine consensus conference, "Global Health and Emergency Care: A Research Agenda," the breakout group on data management identified the lack of research on emergency chief complaints globally-especially in low-income countries where the highest proportion of the world's population resides-as a major gap in global emergency care research. This article reviews global research on emergency chief complaints in high-income countries with developed emergency care systems and sets forth an agenda for future research on chief complaints in limited-resource settings.

Emergency department length of stay for patients requiring mechanical ventilation: a prospective observational study

BACKGROUND

Recommendations for acceptable emergency department (ED) length of stay (LOS) vary internationally with ≤ 8 h generally considered acceptable. Protracted ED LOS may place critically ill patients requiring mechanical ventilation at increased risk of adverse events as most EDs are not resourced for longitudinal delivery of critical care. Our objective was to quantify the ED LOS for mechanically ventilated patients (invasive and/or non-invasive ventilation [NIV]) and to explore patient and system level predictors of prolonged ED LOS. Additionally, we aimed to describe delivery and monitoring of ventilation in the ED.

METHODS

Prospective observational study of ED LOS for all patients receiving mechanical ventilation at four metropolitan EDs in Toronto, Canada over two six-month periods in 2009 and 2010.

RESULTS

We identified 618 mechanically ventilated patients which represented 0.5% (95% CI 0.4%-0.5%) of all ED visits. Of these, 484 (78.3%) received invasive ventilation, 118 (19.1%) received NIV; 16 received both during the ED stay. Median Kaplan-Meier estimated duration of ED stay for all patients was 6.4 h (IQR 2.8-14.6). Patients with trauma diagnoses had a shorter median (IQR) LOS, 2.5 h (1.3-5.1), compared to ventilated patients with non-trauma diagnoses, 8.5 h (3.3-14.0) (p <0.001). Patients requiring NIV had a longer ED stay (16.6 h, 8.2-27.9) compared to those receiving invasive ventilation exclusively (4.6 h, 2.2-11.1) and patients receiving both (15.4 h, 6.4-32.6) (p <0.001). Longer ED LOS was associated with ED site and lower priority triage scores. Shorter ED LOS was associated with intubation at another ED prior to transfer.

CONCLUSIONS

While patients requiring mechanical ventilation represent a small proportion of overall ED visits these critically ill patients frequently experienced prolonged ED stay especially those treated with NIV, assigned lower priority triage scores at ED presentation, and non-trauma patients.