Validity of the Manchester Triage System in emergency patients receiving life-saving intervention or acute medical treatment-A prospective observational study in the emergency department

External validation of the TFC (triage frailty and comorbidity) tool: a prospective observational study

Assessing patient frailty during triage evaluations has become increasingly relevant in Emergency Departments (ED). This study aimed to externally validating the Triage Frailty and Comorbidity (TFC) tool. This prospective study was conducted from June 1 to December 31, 2023. During this period, 12 triage nurses applied the TFC tool during triage evaluation of ED patients. We used receiver operating characteristic (ROC) curves and Decision Curve Analysis to assess the predictive ability of the TFC tool for a 90-day mortality (the same endpoint used during tool development) and a 30-day mortality. 1270 patients were included and 56 of them died within 90 days. The TFC tool had an AUROC of 0.894 (0.858-0.929) for 90-day mortality and 0.885 (0.834-0.938) for 30-day mortality. In Decision Curve Analysis, it yielded higher net benefits up to a threshold probability of 0.30. The externally validated TFC tool appears very effective at identifying patients with increased risk of 90-day mortality after ED attendance. It could be implemented in clinical practice and enhance the predictive ability of standard triage systems.

Enhancing triage accuracy: The influence of nursing education on risk prediction

AIM

This study aimed to compare the performance in risk prediction of various outcomes between specially trained triage nurses and the Manchester Triage System (MTS).

DESIGN

Prospective observational study.

METHODS

The study was conducted from June 1st to December 31st, 2023, at the Emergency Department of Merano Hospital. Triage nurses underwent continuous training through dedicated courses and daily audits. We compared the risk stratification performed by expert nurses with that of MTS on various outcomes such as mortality, hospitalisation, and urgency defined by the physicians. Comparisons were made using the Areas Under the Receiver Operating Characteristic curve (AUROC).

RESULTS

The agreement in code classification between the MTS and the expert nurse was very low. The AUROC curve analysis showed that the expert nurse outperformed the MTS in all outcomes. The triage nurse's experience led to statistically significant better stratification in admission rates, ICU admissions, and all outcomes based on the physician's assessment.

CONCLUSIONS

The continuous training of nurses enables them to achieve better risk prediction compared to standardized triage systems like MTS, emphasizing the utility and necessity of implementing continuous training pathways for these highly specialised personnel.

Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study

BACKGROUND

Large language models (LLMs) have demonstrated impressive performances in various medical domains, prompting an exploration of their potential utility within the high-demand setting of emergency department (ED) triage. This study evaluated the triage proficiency of different LLMs and ChatGPT, an LLM-based chatbot, compared to professionally trained ED staff and untrained personnel. We further explored whether LLM responses could guide untrained staff in effective triage.

OBJECTIVE

This study aimed to assess the efficacy of LLMs and the associated product ChatGPT in ED triage compared to personnel of varying training status and to investigate if the models' responses can enhance the triage proficiency of untrained personnel.

METHODS

A total of 124 anonymized case vignettes were triaged by untrained doctors; different versions of currently available LLMs; ChatGPT; and professionally trained raters, who subsequently agreed on a consensus set according to the Manchester Triage System (MTS). The prototypical vignettes were adapted from cases at a tertiary ED in Germany. The main outcome was the level of agreement between raters' MTS level assignments, measured via quadratic-weighted Cohen κ. The extent of over- and undertriage was also determined. Notably, instances of ChatGPT were prompted using zero-shot approaches without extensive background information on the MTS. The tested LLMs included raw GPT-4, Llama 3 70B, Gemini 1.5, and Mixtral 8x7b.

RESULTS

GPT-4-based ChatGPT and untrained doctors showed substantial agreement with the consensus triage of professional raters (κ=mean 0.67, SD 0.037 and κ=mean 0.68, SD 0.056, respectively), significantly exceeding the performance of GPT-3.5-based ChatGPT (κ=mean 0.54, SD 0.024; P<.001). When untrained doctors used this LLM for second-opinion triage, there was a slight but statistically insignificant performance increase (κ=mean 0.70, SD 0.047; P=.97). Other tested LLMs performed similar to or worse than GPT-4-based ChatGPT or showed odd triaging behavior with the used parameters. LLMs and ChatGPT models tended toward overtriage, whereas untrained doctors undertriaged.

CONCLUSIONS

While LLMs and the LLM-based product ChatGPT do not yet match professionally trained raters, their best models' triage proficiency equals that of untrained ED doctors. In its current form, LLMs or ChatGPT thus did not demonstrate gold-standard performance in ED triage and, in the setting of this study, failed to significantly improve untrained doctors' triage when used as decision support. Notable performance enhancements in newer LLM versions over older ones hint at future improvements with further technological development and specific training.

Human intelligence versus Chat-GPT: who performs better in correctly classifying patients in triage?

INTRODUCTION

Chat-GPT is rapidly emerging as a promising and potentially revolutionary tool in medicine. One of its possible applications is the stratification of patients according to the severity of clinical conditions and prognosis during the triage evaluation in the emergency department (ED).

METHODS

Using a randomly selected sample of 30 vignettes recreated from real clinical cases, we compared the concordance in risk stratification of ED patients between healthcare personnel and Chat-GPT. The concordance was assessed with Cohen's kappa, and the performance was evaluated with the area under the receiver operating characteristic curve (AUROC) curves. Among the outcomes, we considered mortality within 72 h, the need for hospitalization, and the presence of a severe or time-dependent condition.

RESULTS

The concordance in triage code assignment between triage nurses and Chat-GPT was 0.278 (unweighted Cohen's kappa; 95% confidence intervals: 0.231-0.388). For all outcomes, the ROC values were higher for the triage nurses. The most relevant difference was found in 72-h mortality, where triage nurses showed an AUROC of 0.910 (0.757-1.000) compared to only 0.669 (0.153-1.000) for Chat-GPT.

CONCLUSIONS

The current level of Chat-GPT reliability is insufficient to make it a valid substitute for the expertise of triage nurses in prioritizing ED patients. Further developments are required to enhance the safety and effectiveness of AI for risk stratification of ED patients.

Machine learning models predicting undertriage in telephone triage

BACKGROUND

Undertriaged patients have worse outcomes than appropriately triaged patients. Machine learning provides better triage prediction than conventional triage in emergency departments, but no machine learning-based undertriage prediction models have yet been developed for prehospital telephone triage. We developed and validated machine learning models for telephone triage.

MATERIALS AND METHODS

We conducted a retrospective cohort study with the largest after-hour house-call (AHHC) service dataset in Japan. Participants were ≥16 years and used the AHHC service between 1 November 2018 and 31 January 2021. We developed five prediction models based on support vector machine (SVM), lasso regression (LR), random forest (RF), gradient-boosted decision tree (XGB), and deep neural network (DNN). The primary outcome was undertriage, and predictors were telephone triage level and routinely available telephone-based data, including age, sex, 80 chief complaint categories and 10 comorbidities. We measured the area under the receiver operating characteristic curve (AUROC) for all the models.

RESULTS

We identified 15,442 eligible patients (age: 38.4 ± 16.6, male: 57.2%), including 298 (1.9%; age: 58.2 ± 23.9, male: 55.0%) undertriaged patients. RF and XGB outperformed the other models, with the AUROC values (95% confidence interval; 95% CI) of the SVM, LR, RF, XGB and DNN for undertriage being 0.62 (0.55-0.69), 0.79 (0.74-0.83), 0.81 (0.76-0.86), 0.80 (0.75-0.84) and 0.77 (0.73-0.82), respectively.

CONCLUSIONS

We found that RF and XGB outperformed other models. Our findings suggest that machine learning models can facilitate the early detection of undertriage and early intervention to yield substantially improved patient outcomes.KEY MESSAGESUndertriaged patients experience worse outcomes than appropriately triaged patients; thus, we developed machine learning models for predicting undertriage in the prehospital setting. In addition, we identified the predictors of risk factors associated with undertriage.Random forest and gradient-boosted decision tree models demonstrated better prediction performance, and the models identified the risk factors associated with undertriage.Machine learning models aid in the early detection of undertriage, leading to significantly improved patient outcomes and identifying undertriage-associated risk factors, including chief complaint categories, could help prioritize conventional telephone triage protocol revision.

Prehospital triage of trauma patients before and after implementation of a regional triage guideline

INTRODUCTION

Severely injured trauma patients have a considerable mortality rate. One way to reduce the mortality is to ensure optimal triage. The American College of Surgeons Committee on Trauma has since 1986 made guidelines for the triage of trauma patients. These guidelines formed the basis, when the capital region of Denmark implemented a regional trauma triage guideline on February 15th 2016. It is uncertain how the implementation of the regional trauma triage guideline has influenced the triage of trauma patients. The aim of this study was to investigate the changes in admission pattern of trauma patients in the entire region after the implementation of the regional trauma triage guideline. We hypothesized that there would be a reduction in the proportion of trauma patients admitted to the trauma center after the implementation of the regional trauma triage guideline.

PATIENTS AND METHODS

In this observational cohort study with one-year follow-up, we used a national patient registry in Denmark. We identified trauma patients three years before and three years after the implementation of a new regional trauma triage guideline. The primary outcome was the proportion of trauma patients triaged to the regional trauma center. Secondary outcomes were: 30-day and one-year mortality, overtriage, and undertriage.

RESULTS

We found a significant reduction in the proportion of trauma patients triaged to the trauma center from 2115/5951 (35.5%) to 1970/5857 (33.6%), after the implementation of the regional trauma triage guideline, the difference being 1.9% (95% CI: 0.19 to 3.6%); P = 0.03. Further, a significant reduction of overtriage from 15.4% to 9.5% (difference 5.9% with 95% CI of 3.8 to 7.9%) was found. No significant changes in undertriage, 30-day or one-year mortality were found (1.07% vs 0.97%, 4.3% vs 4.5%, and 15.7% vs 16.6% respectively).

CONCLUSION

A significant decrease in the proportion of trauma patients admitted to the trauma center was found after implementation of a new regional trauma triage guideline. A reduction was seen in overtriage, but no changes were found in undertriage and both short-term and long-term mortality remained unchanged.

Effect of the Emergency Department Assessment of Chest Pain Score on the Triage Performance in Patients With Chest Pain

The sensitivity of triage systems in identifying acute cardiovascular events in patients presented to the emergency department with chest pain is not optimal. Recently, a clinical score, the Emergency Department Assessment of Chest Pain Score (EDACS), has been proposed for a rapid assessment without additional instruments. To evaluate whether the integration of EDACS into triage evaluation of patients with chest pain can improve the triage's predictive validity for an acute cardiovascular event, a single-center prospective observational study was conducted. This study involved all patients who needed a triage admission for chest pain between January 1, 2020, and December 31, 2020. All enrolled patients first underwent a standard triage assessment and then the EDACS was calculated. The primary outcome of the study was the presence of an acute cardiovascular event. The discriminatory ability of EDACS in triage compared with standard triage assessment was evaluated by comparing the areas under the receiver operating characteristic curve, decision curve analysis, and net reclassification improvement. The study involved 1,596 patients, of that 7.3% presented the study outcome. The discriminatory ability of triage presented an area under the receiver operating characteristic curve of 0.688 that increased to 0.818 after the application of EDACS in the triage assessment. EDACS improved the baseline assessment of priority assigned in triage, with a net reclassification improvement of 33.6% (p <0.001), and the decision curve analyses demonstrated that EDACS in triage resulted in a clear net clinical benefit. In conclusion, the results of the study suggest that EDACS has a good discriminatory capacity for acute cardiovascular events and that its implementation in routine triage may improve triage performance in patients with chest pain.

Validation of the National Early Warning Score (NEWS)-2 for adults in the emergency department in a tertiary-level clinic in Colombia: Cohort study

The National Early Warning Score (NEWS)-2 is an early warning scale that is used in emergency departments to identify patients at risk of clinical deterioration and to help establish rapid and timely management. The objective of this study was to determine the validity and prediction of mortality using the NEWS2 scale for adults in the emergency department of a tertiary clinic in Colombia.A prospective observational study was conducted between August 2018 and June 2019 at the Universidad de La Sabana Clinic.The nursing staff in the triage classified the patients admitted to the emergency room according to Emergency Severity Index and NEWS2. Demographic data, physiological variables, admission diagnosis, mortality outcome, and comorbidities were extracted.Three thousand nine hundred eighty-six patients were included in the study. Ninety-two (2%) patients required intensive care unit management, with a mean NEWS2 score of 7. A total of 158 patients died in hospital, of which 63 were women (40%). Of these 65 patients required intensive care unit management. The receiver operating characteristic curve for NEWS2 had an area of 0.90 (CI 95%: 0.87-0.92). A classification and score equivalency analysis was performed between triage and the NEWS2 scale in terms of mortality. Of the patients classified as triage I, 32.3% died, and those who obtained a NEWS2 score greater than or equal to 10 had a mortality of 38.6%.Among our population, NEWS2 was not inferior in its area under the receiver operating characteristic curve when predicting mortality than triage, and the cutoff point for NEWS2 to predict in-hospital mortality was higher.

The Manchester Triage System's performance in clinical risk prioritisation of patients presenting with headache in emergency department: A retrospective observational study

OBJECTIVE

Non-traumatic headache is a frequent reason for visits to the emergency department (ED). We evaluated the performance of the Manchester Triage System (MTS) in prioritising patients presenting to the ED with non-traumatic headache.

METHODS

In this single-centre observational retrospective study, we compared the association of MTS priority classification codes with a final diagnosis of a severe neurological condition requiring timely management (ischaemic or haemorrhagic stroke, subarachnoid haemorrhage, cerebral sinus venous thrombosis, central nervous system infection or brain tumour). The study was conducted and reported according to the STROBE statement. The overall prioritisation accuracy of MTS was estimated by the area under the receiver operating characteristic (ROC) curve. The correctness of triage prediction was estimated based on the "very urgent" MTS grouping. An undertriage was defined as a patient with an urgent and severe neurological who received a low priority/urgency MTS code (green/yellow).

RESULTS

Over 30 months, 3002 triage evaluations of non-traumatic headache occurred (1.7% of ED visits). Of these, 2.3% (68/3002) were eventually diagnosed with an urgent and severe neurological condition. The MTS had an acceptable prioritisation accuracy, with an area under the ROC curve of 0.734 (95% CI 0.668-0.799). The sensitivity of the MTS for urgent codes (yellow, orange and red) was 79.4% (95% CI 74.5-84.3), with a specificity of 54.1% (95% CI 52.9-55.3). The triage prediction was incorrect in only 6.3% (190/3002) of patients with headache.

CONCLUSION

The MTS is a safe and accurate tool for prioritising patients with non-traumatic headache in the ED. However, MTS may need further specific tools for evaluating the more complicated symptoms and for correctly identifying patients with urgent and severe underlying pathologies.

RELEVANCE TO CLINICAL PRACTICE

The triage nurse using MTS may need additional tools to improve the assessment of patients with headache, although MTS provides a good safety profile.

[Assessment of validity of the German version of the Manchester Triage System]

BACKGROUND

The Manchester Triage System (MTS) was first introduced in 1996. Since then, the original English MTS version has undergone multiple evaluations of its validity and reliability. Due to translation and differing algorithms, only contradictory data regarding accuracy are available for the German MTS version. The aim of this study was to assess the validity of the current German version of the MTS in a retrospective observational setting.

MATERIALS AND METHODS

All surgical and medical patients presenting at the emergency department of an Austrian regional hospital between 1 and 30 July 2020 were included in this study. Data of vital parameters at the time of triage, overall hospitalisation rate, length of hospitalisation, mortality rate as well as referral and discharge modalities were collected.

RESULTS

A total of 773 patients were included in the study after primary triage. There was a significant correlation between risk level and both hospitalisation rate (p < 0.001) and the length of hospitalisation (p = 0.001). Interestingly, this correlation was lower in patients over 70 years (r² = 0.101 vs. 0.045). Vital signs and mortality rate did not correlate significantly with the initial risk level. The average time for one triage process was 2.1 ± 3.9 min.

CONCLUSION

The German version of the MTS shows good correlation with central risk surrogates, but this correlation is weaker in the subgroup of elderly patients. As the original English version, it does not predict long-term mortality. In clinical practice, the German version of the MTS is as good as the original version in enabling the urgency of treatment to be rapidly assessed so that resource allocation can be optimised.

Acute abdominal pain in triage: A retrospective observational study of the Manchester triage system's validity

OBJECTIVE

Roughly 5% to 10% of patients admitted to the emergency department suffer from acute abdominal pain. Triage plays a key role in patient stratification, identifying patients who need prompt treatment versus those who can safely wait. In this regard, the aim of this study was to estimate the performance of the Manchester Triage System in classifying patients with acute abdominal pain.

METHODS

A total of 9,851 patients admitted at the Emergency Department of the Merano Hospital with acute abdominal pain were retrospectively enrolled between 1 January 2017 and 30 June 2019. The study was conducted and reported according to the STROBE statement. The sensitivity and specificity of the Manchester Triage System were estimated by verifying the triage classification received by the patients and their survival at seven days or the need for acute surgery within 72 h after emergency department access.

RESULTS

Among the patients with acute abdominal pain (median age 50 years), 0.4% died within seven days and 8.9% required surgery within 72 hours. The sensitivity was 44.7% (29.9-61.5), specificity was 95.4% (94.9-95.8), and negative predictive value was 99.7% (99.2-100) in relation to death at seven days.

CONCLUSIONS

The Manchester Triage System shows good specificity and negative predictive value. However, its sensitivity was low due to the amount of incorrect triage prediction in patients with high-priority codes (red/orange), suggesting overtriage in relation to seven-day mortality. This may be a protective measure for the patient. In contrast, the need for acute surgery within 72 h was affected by under-triage.

RELEVANCE TO CLINICAL PRACTICE

The triage nurse using Manchester Triage System can correctly prioritise the majority of patients with acute abdominal pain, especially in low acuity patients. The Manchester Triage System is safe and does not underestimate the severity of the patients.

Errors in nurse-led triage: An observational study

BACKGROUND

Nurses play a crucial role in correctly prioritizing patients entering emergency departments. However, little is known of the accuracy of nurse-led triage systems.

OBJECTIVES

(1) To determine the frequency of nurse-led triage errors within the Manchester Triage System; (2) to explore patient, work environment and individual nurse factors associated with triage errors; and (3) to explore associations between triage errors and patient outcomes (i.e., length of emergency department stay, hospitalization, and 7- and 30-day mortality).

SETTING

This study was conducted in one emergency department in Northern Italy.

PARTICIPANTS

A random sample of 5% (n = 1,929) of all eligible patients accessing the emergency department over an 18-month period.

METHODS

For this retrospective observational study, electronic health record data on triage errors (i.e., incorrect presentational flowchart, specific discriminator and/or priority level) and triage nurses were combined with routine data on patient characteristics, outcomes and the work environment. To explore relationships between these variables, we performed univariate and multivariate logistic regression analyses.

RESULTS

We observed triage errors in 16.3% of patients (n = 314). These were significantly associated with patients' emergency department and hospital stays. Analyses revealed that when > one patient was triaged every 15 min (OR: 2.112;95%CI: 1.331-3.354), older patients (OR: 1.009; 95%CI: 1.003-1.015) with > than two chronic conditions (OR: 1.506; 95%CI: 1.091-2.081) and orange or red priority codes (OR: 1.314; 95%CI: 1.046-1.651,) whose triage nurse had previous experience with another triage system (OR: 3.189; 95%CI: 2.455-4.14) had higher odds of triage errors.

CONCLUSION

We provided primary evidence on triage errors. Confirming our findings on the prevalence, nature and consequences of such errors will require further prospective multicenter studies. Considering patient factors (e.g., age, polychronicity) as additional discriminators could make the nurse-led triage process using the Manchester Triage System more accurate. Investigating the roles of triage nurses' training and background and the emergency department work environment on their mental models regarding the triage process will require qualitative research.