Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study

Coronary artery calcium (CAC) score for cardiovascular risk stratification in a Thai clinical cohort: A comparison of absolute scores and age-sex-specific percentiles

Purposes

Coronary artery calcium (CAC) score provides a quantification of atherosclerotic plaque within the coronary arteries. This study aimed to examine the prevalence and CAC score distribution and to evaluate the association of each CAC score classifications with major adverse cardiovascular events (MACE) in a Thai clinical cohort.

Methods

This study was a retrospective observational cohort. We included patients aged above 35 years who underwent CAC score testing. The absolute and age-sex specific percentile classifications were categorized as 0, 1 to 10, 11 to 100, 101 to 400, and >400 and 0, <75th, 75th - 90th, and >90th, respectively. The endpoint was MACE, including cardiovascular death, myocardial infarction, heart failure hospitalization, coronary artery revascularization procedure, and stroke. Multivariable Cox regression was used to estimate the hazard ratios. The discriminative performance between classifications were compared using Harrell's C-statistics. The agreement was assessed via Cohen's Kappa.

Results

This study included 440 patients, with approximately 70% of Thai patients exhibiting a CAC score. CAC score distributed higher in male than female and increased with age. Both CAC score classification demonstrated the acceptable predictive performance. However, fair agreement was observed between classifications (Cohen's kappa 0.51, 95%CI 0.42-0.59). Within the absolute classification, a higher CAC score was associated with increased hazard ratios for MACE across stratified age-sex-specific percentile levels. In contrast, the hazard ratios for MACE did not consistently rise with higher age-sex-specific percentile CAC score when stratified by absolute CAC score levels.

Conclusions

Both absolute and age-sex-specific percentile CAC score demonstrated acceptable performance in predicting MACE. However, the absolute CAC score classification may be more suitable for risk stratification within the Thai clinical cohort. Our findings offer supportive information that could inform future recommendations for CAC score testing criteria within national clinical practice guidelines.

An XGBoost-based model for assessment of aortic stiffness from wrist photoplethysmogram

BACKGROUND AND OBJECTIVE

Carotid-femoral pulse wave velocity (cf-PWV) is the gold standard for non-invasive assessment of aortic stiffness. Photoplethysmography used in wearable devices provides an indirect measurement method for cf-PWV. This study aimed to construct a cf-PWV prediction method based on the XGBoost algorithm and wrist photoplethysmogram (wPPG) for the early screening of arteriosclerosis in primary healthcare.

METHODS

Data from 210 subjects were used for modeling, and 100 subjects were used as an external validation set. The wPPG pulse waves were filtered by discrete wavelet transform, and various features were extracted from each waveform, including two original indexes. The extraction rate (ER) and Pearson P were calculated to evaluate the applicability of each feature for model training. The magnitude of cf-PWV was predicted by an XGBoost-based model using the selected features and basic physiological parameters (age, sex, height, weight and BMI). The level of aortic stiffness was classified by a 3-classification strategy according to the standard cf-PWV (measured by the Complior device). Bland-Altman plot, Pearson correlation analysis, and accuracy tested performance from two aspects: predicting the magnitude of cf-PWV and classifying the level of aortic stiffness.

RESULTS

In the external validation set (n = 100, age range 22-79), 97 subjects obtained features (ER = 97%). The predicted cf-PWV was significantly correlated with the standard cf-PWV (r = 0.927, P < 0.001). The accuracy (AC) of the 3-classification was 85.6%. The interrater agreement for assessing aortic stiffness was at least substantial (quadratically weighted Kappa = 0.833).

CONCLUSIONS

The multi-parameter fusion cf-PWV prediction method based on the XGBoost algorithm and wPPG pulse wave analysis proves the feasibility of atherosclerosis screening in wearable devices.

A Cartoon Version of Braden Scale to Assess the Risk of Pressure Injury: Content Validity and Interrater Reliability Study

INTRODUCTION

The Braden Scale is widely used to assess the risk of pressure injury. However, the vague literal description of the items creates difficulties for bedside nurses and limits its sensitivity. To solve this problem, we developed a cartoon version of the Braden scale (CVBS) to improve the pressure injury risk assessment ability of bedside nurses.

METHODS

The CVBS was constructed by two nurses, and the final version was determined through a two-round Delphi consultation. The scale's content validity was calculated based on expert ratings. A total of 265 patients were evaluated simultaneously with the CVBS by 119 bedside nurses and 46 wound care specialists; and 114 bedside nurses and the same 46 wound care specialists evaluated 239 patients with the original Braden scale (OBS). The interrater reliability between the two groups was calculated as Kappa value, and then the Kappa values of the two versions were compared.

RESULTS

The content validity for the draft scale was not good enough. After modification, the indices of all the items in the final CVBS reached 1.00. The Kappa value of the OBS was 0.69 (95% CI 0.61-0.76); for each item, it ranged from 0.60 to 0.80. The interrater reliabilities of the CVBS were higher than those of the OBS, with an overall kappa value of 0.87 (95% CI 0.81-0.92) and a range of 0.77 to 0.93 for each item. The differences between the Kappa values of the CVBS and those of the OBS were all statistically significant.

CONCLUSIONS

The CVBS had good validity and showed higher interrater reliability than the OBS, indicating that it may improve bedside nurses' ability to assess pressure injury risk.

Limited reliability of experts' assessment of telephone triage in primary care patients with chest discomfort

OBJECTIVE

Root cause analyses of serious adverse events (SAE) in out-of-hours primary care (OHS-PC) often point to errors in telephone triage. Such analyses are, however, hampered by hindsight bias. We assessed whether experts, blinded to the outcome, recognize (un)safety of triage of patients with chest discomfort, and we quantified inter-rater reliability.

STUDY DESIGN AND SETTING

This is a case-control study with triage recordings from 2013-2017 at OHS-PC. Cases were missed acute coronary syndromes (ACSs, considered as SAE). These cases were age- and gender-matched 1:8 with the controls, sampled from the remainder of people calling for chest discomfort. Fifteen experts listened to the recordings and rated the safety of triage. We calculated sensitivity and specificity of recognizing an ACS and the intraclass correlation.

RESULTS

In total, 135 calls (15 SAE, 120 matched controls) were relistened. The experts identified ACSs with a sensitivity of 0.86 (95% CI: 0.71-0.95) and a specificity of 0.51 (95% CI: 0.43-0.58). Cases were rated significantly more often as unsafe than the controls (73.3% vs. 22.5%, P < 0.001). The inter-rater reliability for safety was poor: ICC 0.16 (95% CI: 0.00-0.32).

CONCLUSIONS

Blinded experts rated calls of missed ACSs more often as unsafe than matched control calls, but with a low level of agreement among the experts.

Digital versus analogue record systems for mass casualty incidents at sea-Results from an exploratory study

Objective

Mis-triage may have serious consequences for patients in mass casualty incidents (MCI) at sea. The purpose of this study was to assess outcome, reliability and validity of an analogue and a digital recording system for triage of a MCI at sea.

Methods

The study based on a triage exercise conducted with a cross-over-design. Forty-eight volunteers were presented a fictional MCI with 50 cases. The volunteers were randomly assigned to start with the analogue (Group A, starting with the analogue followed by the digital system) or digital system (Group B, starting with the digital followed by the analogue system). Triage score distribution and agreement between the triage methods and a predefined standard were reported. Reliability was analysed using Cronbach’s Alpha and Cohen’s Kappa. Validity was measured through sensitivity, specificity and predictive value. Treatment, period and carry-over-effects were analysed using a linear mixed-effects model.

Results

The number of patients triaged (total: n = 3545) with the analogue system (n = 1914; 79.75%) was significantly higher (p = 0.001) than with the digital system (n = 1631; 67.96%). A trend towards a higher percentage of correct triages with the digital system was observed (p = 0.282). Ratio of under-triage was significantly smaller with the digital system (p = 0.001). Validity measured with Cronbach’s Alpha and Cohen’s Kappa was higher with the digital system. So was sensitivity (category; green: 80.67%, yellow: 73.24%, red: 83.54%; analogue: green: 93.28%, yellow: 82.36%, red: 94.04%) and specificity of the digital system (green: 78.07%, yellow: 63.75%, red: 66.25%; analogue: green: 85.50%, yellow: 79.88%, red: 91.50%). Comparing the predictive values and accuracy, the digital system showed higher scores than the analogue system. No significant patterns of carry-over-effects were observed.

Conclusions

Significant differences were found for the number of triages comparing the analogue and digital recording system. The digital system has a slightly higher reliability and validity than the analogue triage system.

Interrater Reliability, Accuracy, and Triage Time Pre- and Post-implementation of a Real-Time Electronic Triage Decision-Support Tool

STUDY OBJECTIVE

The electronic Canadian Triage and Acuity Scale (eCTAS) is a real-time electronic triage decision-support tool designed to improve patient safety and quality of care by standardizing the application of the Canadian Triage and Acuity Scale (CTAS). The objective of this study is to determine interrater agreement of triage scores pre- and post-implementation of eCTAS.

METHODS

This was a prospective, observational study conducted in 7 emergency departments (EDs), selected to represent a mix of triage documentation practices, hospital types, and patient volumes. A provincial CTAS auditor observed triage nurses in the ED pre- and post-implementation of eCTAS and assigned an independent CTAS score in real time. Research assistants independently recorded triage time. Interrater agreement was estimated with κ statistics with 95% confidence intervals (CIs).

RESULTS

A total of 1,491 individual triage assessments (752 pre-eCTAS, 739 post-implementation) were audited during 42 7-hour triage shifts (21 pre-eCTAS, 21 post-implementation). Exact modal agreement was achieved for 567 patients (75.4%) pre-eCTAS compared with 685 patients (92.7%) triaged with eCTAS. With the auditor's CTAS score as the reference, eCTAS significantly reduced the number of patients over-triaged (12.0% versus 5.1%; Δ 6.9; 95% CI 4.0 to 9.7) and under-triaged (12.6% versus 2.2%; Δ 10.4; 95% CI 7.9 to 13.2). Interrater agreement was higher with eCTAS (unweighted κ 0.89 versus 0.63; quadratic-weighted κ 0.93 versus 0.79). Median triage time was 312 seconds (n=3,808 patients) pre-eCTAS and 347 seconds (n=3,489 patients) with eCTAS (Δ 35 seconds; 95% CI 29 to 40 seconds).

CONCLUSION

A standardized, electronic approach to performing triage assessments improves both interrater agreement and data accuracy without substantially increasing triage time.

Reliability of triage systems for paediatric emergency care: a systematic review

OBJECTIVE

To present a systematic review on the reliability of triage systems for paediatric emergency care.

METHODS

A search of MEDLINE, Cochrane Library, Latin American and Caribbean Health Sciences Literature, Scientific Electronic Library Online, Nursing Database Index and Spanish Health Sciences Bibliographic Index for articles in English, French, Portuguese or Spanish was conducted to identify reliability studies of five-level triage systems for patients aged 0-18 years published up to April 2018. Two reviewers performed study selection, data extraction and quality assessment as recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

Twenty studies on nine triage systems were selected: the National Triage System (n=1); the Australasian Triage Scale (n=3); the paediatric Canadian Triage and Acuity Scale (PedCTAS) (n=5); the Manchester Triage System (MTS) (n=1); the Emergency Severity Index (ESI) (n=5); an adaptation of the South African Triage Scale for the Princess Marina Hospital in Botswana (n=1); the Soterion Rapid Triage System (n=1); the Rapid Emergency Triage and Treatment System-paediatric version (n=2); the Paediatric Risk Classification Protocol (n=1). Ten studies were performed with actual patients, while the others used hypothetical scenarios. The studies were rated low (n=14) or moderate (n=6) quality. Kappa was the most used statistic, although many studies did not specify the weighting. PedCTAS, MTS and ESI V.4 exhibited substantial to almost perfect agreement in moderate quality studies.

CONCLUSIONS

There is some evidence on the reliability of the PedCTAS, MTS and ESI V.4, but most studies are limited to the countries where they were developed. Efforts are needed to improve the quality of the studies, and cross-cultural adaptation of those tools is recommended in countries with different professional qualification and sociocultural contexts.

Do employees benefit from collaborations between out of hours general practitioners and emergency departments?

BACKGROUND

In an attempt to redirect patients who are inappropriately attending hospital emergency departments (ED) and in doing so provide the right care at the right place, out-of-hours GP (General Practitioner) services and EDs increasingly collaborate in Urgent Care Collaborations (UCCs). Work satisfaction is an important factor in analysing the impact of this organisational change. The objective of this study is, firstly, to discover if there is a difference in the employee experiences between those working in UCCs and those in traditional out-of-hours services in which EDs and out-of-hours GP services operate separately (i.e. "usual care"). Secondly, we would like to identify which factors affect employees' experiences in these settings.

METHODS

This study followed a cross-sectional study design, comparing usual care with UCCs. Data regarding employee experiences were collected from physicians, nurses, nurse practitioners, medical assistants and front desk personnel, by means of a questionnaire with scales regarding quality, workload and co-operation between the out-of-hours GP service and ED. Independent samples t-tests were used to determine mean differences between the settings. Multiple linear regression analyses were performed to test which items affected the perceived quality, workload and co-operation.

RESULTS

The results showed that mutual co-operation alone was perceived as significantly better in UCCs compared to usual care. If divided between employers, no differences were found in the employee experiences working in out-of-hours GP services. ED employees in UCCs experienced a significantly better co-operation with their GP colleagues than their peers in the usual care setting, but also a higher workload. Remarkably, ED employees were less satisfied in general. The multiple regression model showed that perceived quality, workload and co-operation were interrelated. Co-operation was the only aspect that was rated higher in the UCC setting.

CONCLUSION

While perceived quality is equal and co-operation between out-of-hours GP service and ED is better, the objective and perceived ED workload was higher in UCCs compared to usual care. Though UCCs relieve the pressure on EDs concerning the number of patients, they seem to aggravate the workload. EDs need to be careful not to excessively adjust staff capacity when responding to lower numbers of patients.

Multiple performance measures are needed to evaluate triage systems in the emergency department

OBJECTIVES

Emergency department triage systems can be considered prediction rules with an ordinal outcome, where different directions of misclassification have different clinical consequences. We evaluated strategies to compare the performance of triage systems and aimed to propose a set of performance measures that should be used in future studies.

STUDY DESIGN AND SETTING

We identified performance measures based on literature review and expert knowledge. Their properties are illustrated in a case study evaluating two triage modifications in a cohort of 14,485 pediatric emergency department visits. Strengths and weaknesses of the performance measures were systematically appraised.

RESULTS

Commonly reported performance measures are measures of statistical association (34/60 studies) and diagnostic accuracy (17/60 studies). The case study illustrates that none of the performance measures fulfills all criteria for triage evaluation. Decision curves are the performance measures with the most attractive features but require dichotomization. In addition, paired diagnostic accuracy measures can be recommended for dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R² for ordinal analyses. Other performance measures provide limited additional information.

CONCLUSION

When comparing modifications of triage systems, decision curves and diagnostic accuracy measures should be used in a dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R² in an ordinal approach.

Inter-rater reliability between nurses for a new paediatric triage system based primarily on vital parameters: the Paediatric Triage Instrument (PETI)

INTRODUCTION

The major paediatric triage systems are primarily based on flow charts involving signs and symptoms for orientation and subjective estimates of the patient's condition. In contrast, the 4-level Paediatric Triage Instrument (PETI) is primarily based on vital parameters and was developed exclusively for paediatric triage in patients with medical symptoms. The aim of this study was to assess the inter-rater reliability of this triage system in children when used by nurses.

METHODS

A design was employed in which triage was performed simultaneously and independently by a research nurse and an emergency department (ED) nurse using the PETI. All patients aged ≤12 years who presented at the ED with a medical symptom were considered eligible for participation.

RESULTS

The 89 participants exhibited a median age of 2 years and were triaged by 28 different nurses. The inter-rater reliability between nurses calculated with the quadratic-weighted κ was 0.78 (95% CI 0.67 to 0.89); the linear-weighted κ was 0.67 (95% CI 0.56 to 0.80) and the unweighted κ was 0.59 (95% CI 0.44 to 0.73). For the patients aged <1, 1-3 and >3 years, the quadratic-weighted κ values were 0.67 (95% CI 0.39 to 0.94), 0.86 (95% CI 0.75 to 0.97) and 0.73 (95% CI 0.49 to 0.97), respectively. The median triage duration was 6 min.

CONCLUSIONS

The PETI exhibited substantial reliability when used in children aged ≤12 years and almost perfect reliability among children aged 1-3 years. Moreover, rapid application of the PETI was demonstrated. This study has some limitations, including sample size and generalisability, but the PETI exhibited promise regarding reliability, and the next step could be either a larger reliability study or a validation study.

One-two-triage: validation and reliability of a novel triage system for low-resource settings

Objectives

To validate and assess reliability of a novel triage system, one-two-triage (OTT), that can be applied by inexperienced providers in low-resource settings.

Methods

This study was a two-phase prospective, comparative study conducted at three hospitals. Phase I assessed criterion validity of OTT on all patients arriving at an American university hospital by comparing agreement among three methods of triage: OTT, Emergency Severity Index (ESI) and physician-defined acuity (the gold standard). Agreement was reported in normalised and raw-weighted Cohen κ using two different scales for weighting, Expert-weighted and triage-weighted κ. Phase II tested reliability, reported in Fleiss κ, of OTT using standardised cases among three groups of providers at an urban and rural Cambodian hospital and the American university hospital.

Results

Normalised for prevalence of patients in each category, OTT and ESI performed similarly well for expert-weighted κ (OTT κ=0.58, 95% CI 0.52 to 0.65; ESI κ=0.47, 95% CI 0.40 to 0.53) and triage-weighted κ (κ=0.54, 95% CI 0.48 to 0.61; ESI κ=0.57, 95% CI 0.51 to 0.64). Without normalising, agreement with gold standard was less for both systems but performance of OTT and ESI remained similar, expert-weighted (OTT κ=0.57, 95% CI 0.52 to 0.62; ESI κ=0.6, 95% CI 0.58 to 0.66) and triage-weighted (OTT κ=0.31, 95% CI 0.25 to 0.38; ESI κ=0.41, 95% CI 0.35 to 0.4). In the reliability phase, all triagers showed fair inter-rater agreement, Fleiss κ (κ=0.308).

Conclusions

OTT can be reliably applied and performs as well as ESI compared with gold standard, but requires fewer resources and less experience.

Measuring agreement between healthcare survey instruments using mutual information

BACKGROUND

Healthcare researchers often use multiple healthcare survey instruments to examine a particular patient symptom. The use of multiple instruments can pose some interesting research questions, such as whether the outcomes produced by the different instruments are in agreement. We tackle this problem using information theory, focusing on mutual information to compare outcomes from multiple healthcare survey instruments.

METHODS

We review existing methods of measuring agreement/disagreement between the instruments and suggest a procedure that utilizes mutual information to quantitatively measure the amount of information shared by outcomes from multiple healthcare survey instruments. We also include worked examples to explain the approach.

RESULTS

As a case study, we employ the suggested procedure to analyze multiple healthcare survey instruments used for detecting delirium superimposed on dementia (DSD) in community-dwelling older adults. In addition, several examples are used to assess the mutual information technique in comparison with other measures, such as odds ratio and Cohen's kappa.

CONCLUSIONS

Analysis of mutual information can be useful in explaining agreement/disagreement between multiple instruments. The suggested approach provides new insights into and potential improvements for the application of healthcare survey instruments.

A Web-Based Tool for Patient Triage in Emergency Department Settings: Validation Using the Emergency Severity Index

Background

We evaluated the concordance between triage scores generated by a novel Internet clinical decision support tool, Clinical GPS (cGPS) (Lumiata Inc, San Mateo, CA), and the Emergency Severity Index (ESI), a well-established and clinically validated patient severity scale in use today. Although the ESI and cGPS use different underlying algorithms to calculate patient severity, both utilize a five-point integer scale with level 1 representing the highest severity.

Objective

The objective of this study was to compare cGPS results with an established gold standard in emergency triage.

Methods

We conducted a blinded trial comparing triage scores from the ESI: A Triage Tool for Emergency Department Care, Version 4, Implementation Handbook to those generated by cGPS from the text of 73 sample case vignettes. A weighted, quadratic kappa statistic was used to assess agreement between cGPS derived severity scores and those published in the ESI handbook for all 73 cases. Weighted kappa concordance was defined a priori as almost perfect (kappa > 0.8), substantial (0.6 < kappa < 0.8), moderate (0.4 < kappa < 0.6), fair (0.2 < kappa< 0.4), or slight (kappa < 0.2).

Results

Of the 73 case vignettes, the cGPS severity score matched the ESI handbook score in 95% of cases (69/73 cases), in addition, the weighted, quadratic kappa statistic showed almost perfect agreement (kappa = 0.93, 95% CI 0.854-0.996). In the subanalysis of 41 case vignettes assigned ESI scores of level 1 or 2, the cGPS and ESI severity scores matched in 95% of cases (39/41 cases).

Conclusions

These results indicate that the cGPS is a reliable indicator of triage severity, based on its comparison to a standardized index, the ESI. Future studies are needed to determine whether the cGPS can accurately assess the triage of patients in real clinical environments.

Analysis of emergency medical services triage and dispatch errors by registered nurses in Italy

INTRODUCTION

The major elements of an effective emergency medical services (EMS) system include a single telephone access number, accurate assessment of the urgency of the health problem, and timely dispatch of appropriate personnel and equipment. In Italy, EMS calls are managed by emergency operations centers by registered nurses who have received specialized education in this function. The nurses determine the criticality of the situations and assign an EMS response priority level identified by a color code, ranging from red (very critical) to green (not critical). At times, the severity of a situation may be underestimated, resulting in assignment of a lower EMS response priority and the potential for patient death (code black). The purpose of this study was to analyze factors associated with registered nurse under-triage of EMS calls subsequently found to be associated with deaths, termed "green-black code" cases.

METHODS

We carried out a retrospective qualitative analysis of EMS telephone conversations using Fele's conversation analysis method. The characteristics of green-black code calls were compared with the characteristics of the population of all EMS calls during the study period.

RESULTS

The study patients were older, with a mean age of 81.6 years. The callers were individuals calling on behalf of the patients, rather than the patients themselves. The callers reported symptoms that were not life-threatening. Nurse operators did not always inquire about the patients' vital signs as required by the Medical Priority Dispatch System protocol. The phone conversations were shorter than normal (54.26 seconds vs 65 seconds).

DISCUSSION

Although the importance of dispatch system protocols is wellknown, it is also important that nurse triage operators have proper training to ensure that major parameters such as vital signs and symptomatology are obtained and to reduce caller stress level.

Evaluating implementation of the emergency severity index in a Belgian hospital

INTRODUCTION

Triage aims to categorize patients based on their clinical need and the available departmental resources. To accomplish this goal, one needs to ensure that the implemented triage system is reliable and that staff use it correctly. Therefore this study assessed the ability of Belgium nurses to apply the Emergency Severity Index (ESI), version 4, to hypothetical case scenarios after an educational intervention.

METHODS

An ESI educational intervention was implemented in accordance with the ESI manual. Using paper case scenarios, nurses' interrater agreement was assessed by comparing triage nurse ESI levels with the reference answers noted in the implementation manual. Interrater agreement was measured by the percentage of agreement and Cohen's κ coefficient using different weighting schemes.

RESULTS

Overall, 77.5% of the scenario cases were coded according the ESI guidelines, resulting in a good interrater agreement (κ = 0.72, linear weighted κ = 0.84, quadratic weighted κ = 0.92, and triage-weighted scheme = 0.79). Interrater agreement varied when evaluating each ESI level separately. Undertriage was more common than overtriage. The highest misclassification range (37.8%) occurred in ESI level 2 scenarios, with 99.2% of the misclassifications being undertriaged.

DISCUSSION

Implementation of the ESI into a novel setting guided by a locally developed training program resulted in suboptimal interrater agreement. Existing weighted κ schemes overestimated the interrater agreement between the triage nurse-assigned ESI level and the reference standard. By providing an aggregated measure of agreement, which allows partial agreement, clinically significant misclassification was masked by a misleading "good" interrater agreement.

A review of the quality assurance processes for the Australasian Triage Scale (ATS) and implications for future practice

Triage is a critical component in the delivery of time-sensitive emergency care. Decision-making and the activity of the triage nurse affect patient access to care and allocation of emergency department (ED) resources. It is important to be able to evaluate decision-making at triage to ensure patient safety, and to identify opportunities for professional and service development. At present, there is no standard approach to retrospective examination of triage decisions using the Australasian Triage Scale. The aim of this literature review is to identify the quality control strategies used in triage and the factors that appear to influence triage decisions in relation to the Australasian Triage Scale.

Emergency Severity Index version 4: a valid and reliable tool in pediatric emergency department triage

OBJECTIVES

The Emergency Severity Index version 4 (ESI v.4) is the most recently implemented 5-level triage system. The validity and reliability of this triage tool in the pediatric population have not been extensively established. The goals of this study were to assess the validity of ESI v.4 in predicting hospital admission, emergency department (ED) length of stay (LOS), and number of resources utilized, as well as its reliability in a prospective cohort of pediatric patients.

METHODS

The first arm of the study was a retrospective chart review of 780 pediatric patients presenting to a pediatric ED to determine the validity of ESI v.4. Abstracted data included acuity level assigned by the triage nurse using ESI v.4 algorithm, disposition (admission vs discharge), LOS, and number of resources utilized in the ED. To analyze the validity of ESI v.4, patients were divided into 2 groups for comparison: higher-acuity patients (ESI levels 1, 2, and 3) and lower-acuity patients (ESI levels 4 and 5). Pearson χ analysis was performed for categorical variables. For continuous variables, we conducted a comparison of means based on parametric distribution of variables. The second arm was a prospective cohort study to determine the interrater reliability of ESI v.4 among and between pediatric triage (PT) nurses and pediatric emergency medicine (PEM) physicians. Three raters (2 PT nurses and 1 PEM physician) independently assigned triage scores to 100 patients; k and interclass correlation coefficient were calculated among PT nurses and between the primary PT nurses and physicians.

RESULTS

In the validity arm, the distribution of ESI score levels among the 780 cases are as follows: ESI 1: 2 (0.25%); ESI 2: 73 (9.4%); ESI 3: 289 (37%); ESI 4: 251 (32%); and ESI 5: 165 (21%). Hospital admission rates by ESI level were 1: 100%, 2: 42%, 3: 14.9%, 4: 1.2%, and 5: 0.6%. The admission rate of the higher-acuity group (76/364, 21%) was significantly greater than the lower-acuity group (4/415, 0.96%), P < 0.001. The mean ED LOS (in minutes) for the higher-acuity group was 257 (SD, 132) versus 143 (SD, 81) in the lower-acuity group, P < 0.001. The higher-acuity group also had significantly greater use of resources than the lower-acuity group, P < 0.001. The percentage of low-acuity patients receiving no resources was 54%, compared with only 26% in the higher-acuity group. Conversely, a greater percentage of higher-acuity patients utilized 2 or more resources than the lower-acuity cohorts, 43% vs 12%, respectively, P < 0.001. In the prospective reliability arm of the study, 15 PT nurses and 8 PEM attending physicians participated in the study; k among nurses was 0.92 and between the primary triage nurses and physicians was 0.78, P < 0.001. The intraclass correlation coefficient was 0.96 for PT nurses and 0.91 between the primary triage nurse and physicians, P < 0.001.

CONCLUSIONS

Emergency Severity Index v.4 is a valid predictor of hospital admission, ED LOS, and resource utilization in the pediatric ED population. It is a reliable pediatric triage instrument with high agreement among PT nurses and between PT nurses and PEM physicians.

The South African Triage Scale (adult version) provides reliable acuity ratings

OBJECTIVE

To estimate the inter- and intra-rater reliability of triage ratings on Emergency Centre patients by South African nurses and doctors using the South African Triage Scale (SATS).

METHODS

A cross-sectional reliability study was performed. Five emergency physicians and ten enrolled nursing assistants independently assigned triage categories to 100 written vignettes unaware of the ratings given by others. Four different quantitative reliability measures were calculated and compared. Graphical displays portrayed rating distributions for vignettes with mean ratings at different acuity categories.

RESULTS

The estimated quadratically weighted kappa for the group of emergency physicians was 0.76 (95% CI: 0.67-0.84) and for the group of nurses 0.66 (95% CI: 0.58-0.74). These values were close to the estimated intra-class correlation coefficients. For intra-rater reliability, the average exact agreement was 84%. The graphical displays showed that the least variability was evident in the vignettes that had a mean rating of 'emergency', 'very urgent' or 'routine'.

CONCLUSION

This study indicates good inter- and intra-rater reliability among nurses and doctors using the SATS. It suggests that the SATS is reliably applied, and supports the feasibility of further implementation of the SATS in similar settings.

Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study

OBJECTIVES

It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.

STUDY DESIGN AND SETTING

From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.

RESULTS

The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.

CONCLUSION

When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.

Reliability and validity of triage systems in paediatric emergency care

BACKGROUND

Triage in paediatric emergency care is an important tool to prioritize seriously ill children. Triage can also be used to identify patients who do not need urgent care and who can safely wait. The aim of this review was to provide an overview of the literature on reliability and validity of current triage systems in paediatric emergency care

METHODS

We performed a search in Pubmed and Cochrane on studies on reliability and validity of triage systems in children

RESULTS

The Manchester Triage System (MTS), the Emergency Severity Index (ESI), the Paediatric Canadian Triage and Acuity Score (paedCTAS) and the Australasian Triage Scale (ATS) are common used triage systems and contain specific parts for children. The reliability of the MTS is good and reliability of the ESI is moderate to good. Reliability of the paedCTAS is moderate and is poor to moderate for the ATS.The internal validity is moderate for the MTS and confirmed for the CTAS, but not studied for the most recent version of the ESI, which contains specific fever criteria for children.

CONCLUSION

The MTS and paedCTAS both seem valid to triage children in paediatric emergency care. Reliability of the MTS is good, moderate to good for the ESI and moderate for the paedCTAS. More studies are necessary to evaluate if one triage system is superior over other systems when applied in emergency care.