Consistency of triage scores by presenting complaint pre- and post-implementation of a real-time electronic triage decision support tool

Evaluating the Reliability of a Remote Acuity Prediction Tool in a Canadian Academic Emergency Department

STUDY OBJECTIVE

There is increasing interest in harnessing artificial intelligence to virtually triage patients seeking care. The objective was to examine the reliability of a virtual machine learning algorithm to remotely predict acuity scores for patients seeking emergency department (ED) care by applying the algorithm to retrospective ED data.

METHODS

This was a retrospective review of adult patients conducted at an academic tertiary care ED (annual census 65,000) from January 2021 to August 2022. Data including ED visit date and time, patient age, sex, reason for visit, presenting complaint and patient-reported pain score were used by the machine learning algorithm to predict acuity scores. The algorithm was designed to up-triage high-risk complaints to promote safety for remote use. The predicted scores were then compared to nurse-led triage scores previously derived in real time using the electronic Canadian Triage and Acuity Scale (eCTAS), an electronic triage decision-support tool used in the ED. Interrater reliability was estimated using kappa statistics with 95% confidence intervals (CIs).

RESULTS

In total, 21,469 unique ED patient encounters were included. Exact modal agreement was achieved for 10,396 (48.4%) patient encounters. Interrater reliability ranged from poor to fair, as estimated using unweighted kappa (0.18, 95% CI 0.17 to 0.19), linear-weighted kappa (0.25, 95% CI 0.24 to 0.26), and quadratic-weighted kappa (0.36, 95% CI 0.35 to 0.37) statistics. Using the nurse-led eCTAS score as the reference, the machine learning algorithm overtriaged 9,897 (46.1%) and undertriaged 1,176 (5.5%) cases. Some of the presenting complaints under-triaged were conditions generally requiring further probing to delineate their nature, including abnormal lab/imaging results, visual disturbance, and fever.

CONCLUSION

This machine learning algorithm needs further refinement before being safely implemented for patient use.

Influence of electronic triage decision-support on hospital admission, left without being seen and time to physician initial assessment in the emergency department

OBJECTIVE

To explore the impact of the implementation of eCTAS, a real-time electronic decision-support tool, on hospital admission, rate of left without being seen, and time from triage to physician initial assessment.

METHODS

We conducted a cohort study using population-based administrative data from all Ontario emergency departments (EDs) that had implemented eCTAS for 9 months. We compared 6 months post-eCTAS data to the same 6 months the previous year (pre-eCTAS). We included triage encounters of adult (≥ 18 years) patients if they had one of 16 pre-specified, high-volume presenting complaints. Multivariable logistic regression and quantile regression models informed the effect of eCTAS on outcomes.

RESULTS

We included data from 354,176 triage encounters from 31 EDs. There was a change in the distribution of triage scores post-eCTAS, with fewer patients classified as CTAS 2 and CTAS 3, and more patients classified as CTAS 1 and CTAS 4. Overall, hospital admission decreased post-eCTAS (adjusted OR: 0.98; 95% CI: 0.97 to 1.00), with fewer CTAS 2 and more CTAS 3 and CTAS 4 patients admitted post-eCTAS. The rate of left without being seen increased (2.8% vs. 3.0%; adjusted OR: 1.07; 95% CI: 1.03 to 1.11) post-eCTAS, while time to physician initial assessment proved similar pre and post-eCTAS.

CONCLUSIONS

eCTAS implementation had little impact on admission, rate of left without being seen and time to physician initial assessment. eCTAS appears to reclassify patients from higher to lower acuity scores, resulting in higher admission rates for CTAS 3 and CTAS 4 patients. It remains unknown if this reclassification is appropriate.