A Web-Based Tool for Patient Triage in Emergency Department Settings: Validation Using the Emergency Severity Index

Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study

Background

Computerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care.

Objective

The purpose of this study was to assess the performance, safety, and usability of a CDSS in a university hospital emergency department setting in Kuopio, Finland.

Methods

Patients entering the emergency department were asked to voluntarily participate in this study. Patients aged 17 years or younger, patients with cognitive impairments, and patients who entered the unit in an ambulance or with the need for immediate care were excluded. Patients completed the CDSS web-based form and usability questionnaire when waiting for the triage nurse's evaluation. The CDSS data were anonymized and did not affect the patients' usual evaluation or treatment. Retrospectively, 2 medical doctors evaluated the urgency of each patient's condition by using the triage nurse's information, and urgent and nonurgent groups were created. The International Statistical Classification of Diseases, Tenth Revision diagnoses were collected from the electronic health records. Usability was assessed by using a positive version of the System Usability Scale questionnaire.

Results

In total, our analyses included 248 patients. Regarding urgency, the mean sensitivities were 85% and 19%, respectively, for urgent and nonurgent cases when assessing the performance of CDSS evaluations in comparison to that of physicians. The mean sensitivities were 85% and 35%, respectively, when comparing the evaluations between the two physicians. Our CDSS did not miss any cases that were evaluated to be emergencies by physicians; thus, all emergency cases evaluated by physicians were evaluated as either urgent cases or emergency cases by the CDSS. In differential diagnosis, the CDSS had an exact match accuracy of 45.5% (97/213). The usability was good, with a mean System Usability Scale score of 78.2 (SD 16.8).

Conclusions

In a university hospital emergency department setting with a large real-world population, our CDSS was found to be equally as sensitive in urgent patient cases as physicians and was found to have an acceptable differential diagnosis accuracy, with good usability. These results suggest that this CDSS can be safely assessed further in a real-world setting. A CDSS could accelerate triage by providing patient-provided data in advance of patients' initial consultations and categorize patient cases as urgent and nonurgent cases upon patients' arrival to the emergency department.

Effect of Emergency Severity Index Annual Competency Assessment on Mistriage

BACKGROUND

There is limited evidence identifying best practices to promote competency of accurate assignment of the Emergency Severity Index (ESI) acuity level to patients who present to the emergency department (ED) triage.

LOCAL PROBLEM

Triage-trained nurses do not receive competency training in an ESI triage tool.

METHODS

A retrospective chart review of 150 patients was completed to evaluate mistriage rates before and after triage-trained nurses completed an ESI competency assessment.

RESULTS

The retrospective chart review showed no statistically significant difference in mistriage from pre- to postintervention (P = .8535).

CONCLUSIONS

Implementation of an ESI annual competency assessment aligns well with an emerging theme in the literature that ED nurses should be provided with ongoing education that reinforces knowledge and implementation of ESI triaging.

Human- Versus Machine Learning-Based Triage Using Digitalized Patient Histories in Primary Care: Comparative Study

BACKGROUND

Smartphones have made it possible for patients to digitally report symptoms before physical primary care visits. Using machine learning (ML), these data offer an opportunity to support decisions about the appropriate level of care (triage).

OBJECTIVE

The purpose of this study was to explore the interrater reliability between human physicians and an automated ML-based triage method.

METHODS

After testing several models, a naïve Bayes triage model was created using data from digital medical histories, capable of classifying digital medical history reports as either in need of urgent physical examination or not in need of urgent physical examination. The model was tested on 300 digital medical history reports and classification was compared with the majority vote of an expert panel of 5 primary care physicians (PCPs). Reliability between raters was measured using both Cohen κ (adjusted for chance agreement) and percentage agreement (not adjusted for chance agreement).

RESULTS

Interrater reliability as measured by Cohen κ was 0.17 when comparing the majority vote of the reference group with the model. Agreement was 74% (138/186) for cases judged not in need of urgent physical examination and 42% (38/90) for cases judged to be in need of urgent physical examination. No specific features linked to the model's triage decision could be identified. Between physicians within the panel, Cohen κ was 0.2. Intrarater reliability when 1 physician retriaged 50 reports resulted in Cohen κ of 0.55.

CONCLUSIONS

Low interrater and intrarater agreement in triage decisions among PCPs limits the possibility to use human decisions as a reference for ML to automate triage in primary care.

Five-level emergency triage systems: variation in assessment of validity

INTRODUCTION

Triage systems are scales developed to rate the degree of urgency among patients who arrive at EDs. A number of different scales are in use; however, the way in which they have been validated is inconsistent. Also, it is difficult to define a surrogate that accurately predicts urgency. This systematic review described reference standards and measures used in previous validation studies of five-level triage systems.

METHODS

We searched PubMed, EMBASE and CINAHL to identify studies that had assessed the validity of five-level triage systems and described the reference standards and measures applied in these studies. Studies were divided into those using criterion validity (reference standards developed by expert panels or triage systems already in use) and those using construct validity (prognosis, costs and resource use).

RESULTS

A total of 57 studies examined criterion and construct validity of 14 five-level triage systems. Criterion validity was examined by evaluating (1) agreement between the assigned degree of urgency with objective standard criteria (12 studies), (2) overtriage and undertriage (9 studies) and (3) sensitivity and specificity of triage systems (7 studies). Construct validity was examined by looking at (4) the associations between the assigned degree of urgency and measures gauged in EDs (48 studies) and (5) the associations between the assigned degree of urgency and measures gauged after hospitalisation (13 studies). Particularly, among 46 validation studies of the most commonly used triages (Canadian Triage and Acuity Scale, Emergency Severity Index and Manchester Triage System), 13 and 39 studies examined criterion and construct validity, respectively.

CONCLUSION

Previous studies applied various reference standards and measures to validate five-level triage systems. They either created their own reference standard or used a combination of severity/resource measures.

How to improve vital sign data quality for use in clinical decision support systems? A qualitative study in nine Swedish emergency departments

Background

Vital sign data are important for clinical decision making in emergency care. Clinical Decision Support Systems (CDSS) have been advocated to increase patient safety and quality of care. However, the efficiency of CDSS depends on the quality of the underlying vital sign data. Therefore, possible factors affecting vital sign data quality need to be understood.

This study aims to explore the factors affecting vital sign data quality in Swedish emergency departments and to determine in how far clinicians perceive vital sign data to be fit for use in clinical decision support systems. A further aim of the study is to provide recommendations on how to improve vital sign data quality in emergency departments.

Methods

Semi-structured interviews were conducted with sixteen physicians and nurses from nine hospitals and vital sign documentation templates were collected and analysed. Follow-up interviews and process observations were done at three of the hospitals to verify the results. Content analysis with constant comparison of the data was used to analyse and categorize the collected data.

Results

Factors related to care process and information technology were perceived to affect vital sign data quality. Despite electronic health records (EHRs) being available in all hospitals, these were not always used for vital sign documentation. Only four out of nine sites had a completely digitalized vital sign documentation flow and paper-based triage records were perceived to provide a better mobile workflow support than EHRs. Observed documentation practices resulted in low currency, completeness, and interoperability of the vital signs. To improve vital sign data quality, we propose to standardize the care process, improve the digital documentation support, provide workflow support, ensure interoperability and perform quality control.

Conclusions

Vital sign data quality in Swedish emergency departments is currently not fit for use by CDSS. To address both technical and organisational challenges, we propose five steps for vital sign data quality improvement to be implemented in emergency care settings.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-016-0305-4) contains supplementary material, which is available to authorized users.