Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study

Limited reliability of experts' assessment of telephone triage in primary care patients with chest discomfort

OBJECTIVE

Root cause analyses of serious adverse events (SAE) in out-of-hours primary care (OHS-PC) often point to errors in telephone triage. Such analyses are, however, hampered by hindsight bias. We assessed whether experts, blinded to the outcome, recognize (un)safety of triage of patients with chest discomfort, and we quantified inter-rater reliability.

STUDY DESIGN AND SETTING

This is a case-control study with triage recordings from 2013-2017 at OHS-PC. Cases were missed acute coronary syndromes (ACSs, considered as SAE). These cases were age- and gender-matched 1:8 with the controls, sampled from the remainder of people calling for chest discomfort. Fifteen experts listened to the recordings and rated the safety of triage. We calculated sensitivity and specificity of recognizing an ACS and the intraclass correlation.

RESULTS

In total, 135 calls (15 SAE, 120 matched controls) were relistened. The experts identified ACSs with a sensitivity of 0.86 (95% CI: 0.71-0.95) and a specificity of 0.51 (95% CI: 0.43-0.58). Cases were rated significantly more often as unsafe than the controls (73.3% vs. 22.5%, P < 0.001). The inter-rater reliability for safety was poor: ICC 0.16 (95% CI: 0.00-0.32).

CONCLUSIONS

Blinded experts rated calls of missed ACSs more often as unsafe than matched control calls, but with a low level of agreement among the experts.

Reliability of prehospital patient classification in helicopter emergency medical service missions

Background

Several scores and codes are used in prehospital clinical quality registries but little is known of their reliability. The aim of this study is to evaluate the inter-rater reliability of the American Society of Anesthesiologists physical status (ASA-PS) classification system, HEMS benefit score (HBS), International Classification of Primary Care, second edition (ICPC-2) and Eastern Cooperative Oncology Group (ECOG) performance status in a helicopter emergency medical service (HEMS) clinical quality registry (CQR).

Methods

All physicians and paramedics working in HEMS in Finland and responsible for patient registration were asked to participate in this study. The participants entered data of six written fictional missions in the national CQR. The inter-rater reliability of the ASA-PS, HBS, ICPC-2 and ECOG were evaluated using an overall agreement and free-marginal multi-rater kappa (Κ_free).

Results

All 59 Finnish HEMS physicians and paramedics were invited to participate in this study, of which 43 responded and 16 did not answer. One participant was excluded due to unfinished data entering. ASA-PS had an overall agreement of 40.2% and Κ_free of 0.28 in this study. HBS had an overall agreement of 44.7% and Κ_free of 0.39. ICPC-2 coding had an overall agreement of 51.5% and Κ_free of 0.47. ECOG had an overall agreement of 49.6% and Κ_free of 0.40.

Conclusion

This study suggests a marked inter-rater unreliability in prehospital patient scoring and coding even in a relatively uniform group of practitioners working in a highly focused environment. This indicates that the scores and codes should be specifically designed or adapted for prehospital use, and the users should be provided with clear and thorough instructions on how to use them.

Performance of a new guideline for telephone triage in out-of-hours services in Belgium: A pilot study using simulated patients

BACKGROUND

Patients in Belgium needing out-of-hours care have two options: the emergency department or the general practitioner on call often organised in a general practitioner cooperative. Currently, there is no triage system in Belgium so patients do not know where to go.

METHODS

Our primary objective was to examine the ability of a newly developed telephone guideline, called 1733, to adequately estimate the urgency of health problems presented by simulated patients. Ten clinical vignettes were presented to 12 operators in a simulated phone call. The operators had to assign a protocol, urgency level and resource to dispatch (ambulance, general practitioner house visit, etc.) to each case.

RESULTS

A total of 120 phone calls were analysed. The operators chose the right protocol in 69% and the correct urgency level in 35% of the cases. The proportion of under- and over-triage was 26% and 39%, respectively. There was important variation in between the operators. The sensitivity for detecting highly urgent cases was 0.42, the specificity 0.92.

CONCLUSION

Using the new Belgian 1733 guideline for telephone triage, operators mostly chose the appropriate protocol but only chose the correct urgency in one out of three cases. In this phase of development, the studied telephone guideline is not ready for implementation.

The reliability and accuracy of operational system data in a nationwide helicopter emergency medical services mission database

AIM

The aim of this study was to evaluate the reliability and accuracy of documentation in FinnHEMS database, which is a nationwide helicopter emergency service (HEMS) clinical quality registry.

METHODS

This is a nationwide study based on written fictional clinical scenarios. Study subjects were HEMS physicians and paramedics, who filled in the clinical quality registry based on the clinical scenarios. The inter-rater -reliability of the collected data was analyzed with percent agreement and free-marginal multi-rater kappa.

RESULTS

Dispatch coding had a percent agreement of 91% and free-marginal multi-rater kappa value of 0.83. Coding for transportation or mission cancellation resulted in an agreement of 84% and free-marginal kappa value of 0.68. An agreement of 82% and a kappa value of 0.73 for dispatcher coding was found. Mission end, arrival at hospital and HEMS unit dispatch -times had agreements from 80 to 85% and kappa values from 0.61 to 0.73. The emergency call to dispatch centre time had an agreement of 71% and kappa value of 0.56. The documentation of pain had an agreement of 73% on both the first and second measurements. All other vital parameters had less than 70% agreement and 0.40 kappa value in the first measurement. The documentation of secondary vital parameter measurements resulted in agreements from 72 to 91% and kappa values from 0.43 to 0.64.

CONCLUSION

Data from HEMS operations can be gathered reliably in a national clinical quality registry. This study revealed some inaccuracies in data registration and data quality, which are important to detect to improve the overall reliability and validity of the HEMS clinical quality register.

Comparison between Emergency Severity Index and Heart Failure Triage Scale in heart failure patients: A randomized clinical trial

BACKGROUND

It is not clear whether Emergency Severity Index (ESI) is valid to triage heart failure (HF) patients and if HF patients benefit more from a customized triage scale or not. The aim of study is to compare the effect of Heart Failure Triage Scale (HFTS) and ESI on mistriage among patients with HF who present to the emergency department (ED).

METHODS

A randomized clinical trial was conducted from April to June 2017. HF patients with dyspnea were randomly assigned to HFTS or ESI groups. Triage level, used resources and time to electrocardiogram (ECG) were compared between both groups among HF patients who were admitted to coronary care unit (CCU), cardiac unit (CU) and discharged patients from the ED. Content validity was examined using Kappa designating agreement on relevance (K*). Reliability of both scale was evaluated using inter-observer agreement (Kappa).

RESULTS

Seventy-three and 74 HF patients were assigned to HFTS and ESI groups respectively. Time to ECG in HFTS group was significantly shorter than that of ESI group (2.05 vs. 16.82 minutes). Triage level between HFTS and ESI groups was significantly different among patients admitted to CCU (1.0 vs. 2.8), cardiac unit (2.26 vs. 3.06) and discharged patients from the ED (3.53 vs. 2.86). Used resources in HFTS group was significantly different among triage levels (H=25.89; df=3; P<0.001).

CONCLUSION

HFTS is associated with less mistriage than ESI for triaging HF patients. It is recommended to make use of HFTS to triage HF patients in the ED.

Reliability of triage systems for paediatric emergency care: a systematic review

OBJECTIVE

To present a systematic review on the reliability of triage systems for paediatric emergency care.

METHODS

A search of MEDLINE, Cochrane Library, Latin American and Caribbean Health Sciences Literature, Scientific Electronic Library Online, Nursing Database Index and Spanish Health Sciences Bibliographic Index for articles in English, French, Portuguese or Spanish was conducted to identify reliability studies of five-level triage systems for patients aged 0-18 years published up to April 2018. Two reviewers performed study selection, data extraction and quality assessment as recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

Twenty studies on nine triage systems were selected: the National Triage System (n=1); the Australasian Triage Scale (n=3); the paediatric Canadian Triage and Acuity Scale (PedCTAS) (n=5); the Manchester Triage System (MTS) (n=1); the Emergency Severity Index (ESI) (n=5); an adaptation of the South African Triage Scale for the Princess Marina Hospital in Botswana (n=1); the Soterion Rapid Triage System (n=1); the Rapid Emergency Triage and Treatment System-paediatric version (n=2); the Paediatric Risk Classification Protocol (n=1). Ten studies were performed with actual patients, while the others used hypothetical scenarios. The studies were rated low (n=14) or moderate (n=6) quality. Kappa was the most used statistic, although many studies did not specify the weighting. PedCTAS, MTS and ESI V.4 exhibited substantial to almost perfect agreement in moderate quality studies.

CONCLUSIONS

There is some evidence on the reliability of the PedCTAS, MTS and ESI V.4, but most studies are limited to the countries where they were developed. Efforts are needed to improve the quality of the studies, and cross-cultural adaptation of those tools is recommended in countries with different professional qualification and sociocultural contexts.

[A new method for agreement evaluation based on AC1]

Medical studies use various methods for assessing agreement among different raters or measurement methods. Many of these coefficients have limitations, and among them the paradoxes of kappa are the best known. To achieve a higher accuracy and reliability, we propose an alternative statistic method based on AC1, known as CEA, which adjusts the chance agreement. We explored the influences of the prevalence rate and chance agreement probability on the total agreement and compared the accuracy and stability of kappa, AC1 and CEA coefficient through simulations and real data analysis. The proposed method offers a stable and reliable option for assessing agreement of binary data.

The reliability of the Manchester Triage System (MTS): a meta-analysis

OBJECTIVE

Although the Manchester Triage System (MTS) was first developed two decades ago, the reliability of the MTS has not been questioned through comparison with a moderating variable; therefore, the aim of this study is to determine the extent of the reliability of MTS using a meta-analytic review.

METHOD

Electronic databases were searched up to 1 March 2014. Studies were only included if they had reported sample sizes, reliability coefficients, and adequate description of the reliability assessment. The Guidelines for Reporting Reliability and Agreement Studies was used. Two reviewers independently examined abstracts and extracted data. The effect size was obtained by the z-transformation of reliability coefficients. Data were pooled with random-effects models, and meta-regression was performed based on the method-of-moments estimator.

RESULTS

Seven studies were included. The pooled coefficient for the MTS was substantial at 0.751 (CI 95%: 0.677 to 0.810); the incidence of mistriage is greater than 50%. Agreement is higher for the latest version of MTS (for adults) among nurse-experts and in countries in closer proximity to the country of MTS origin (the UK, in Manchester) than for the oldest (pediatric) version, nurse-nurse raters, and countries at a greater distance from the UK.

CONCLUSION

The MTS showed an acceptable level of overall reliability in the emergency department, but more development is required to attain almost perfect agreement.

Inter-rater reliability between nurses for a new paediatric triage system based primarily on vital parameters: the Paediatric Triage Instrument (PETI)

INTRODUCTION

The major paediatric triage systems are primarily based on flow charts involving signs and symptoms for orientation and subjective estimates of the patient's condition. In contrast, the 4-level Paediatric Triage Instrument (PETI) is primarily based on vital parameters and was developed exclusively for paediatric triage in patients with medical symptoms. The aim of this study was to assess the inter-rater reliability of this triage system in children when used by nurses.

METHODS

A design was employed in which triage was performed simultaneously and independently by a research nurse and an emergency department (ED) nurse using the PETI. All patients aged ≤12 years who presented at the ED with a medical symptom were considered eligible for participation.

RESULTS

The 89 participants exhibited a median age of 2 years and were triaged by 28 different nurses. The inter-rater reliability between nurses calculated with the quadratic-weighted κ was 0.78 (95% CI 0.67 to 0.89); the linear-weighted κ was 0.67 (95% CI 0.56 to 0.80) and the unweighted κ was 0.59 (95% CI 0.44 to 0.73). For the patients aged <1, 1-3 and >3 years, the quadratic-weighted κ values were 0.67 (95% CI 0.39 to 0.94), 0.86 (95% CI 0.75 to 0.97) and 0.73 (95% CI 0.49 to 0.97), respectively. The median triage duration was 6 min.

CONCLUSIONS

The PETI exhibited substantial reliability when used in children aged ≤12 years and almost perfect reliability among children aged 1-3 years. Moreover, rapid application of the PETI was demonstrated. This study has some limitations, including sample size and generalisability, but the PETI exhibited promise regarding reliability, and the next step could be either a larger reliability study or a validation study.

The Reliability of the Canadian Triage and Acuity Scale: Meta-analysis

Background:

Although the Canadian Triage and Acuity Scale (CTAS) have been developed since two decades ago, the reliability of the CTAS has not been questioned comparing to moderating variable.

Aims:

The study was to provide a meta-analytic review of the reliability of the CTAS in order to reveal to what extent the CTAS is reliable.

Materials and Methods:

Electronic databases were searched to March 2014. Only studies were included that had reported samples size, reliability coefficients, adequate description of the CTAS reliability assessment. The guidelines for reporting reliability and agreement studies (GRRAS) were used. Two reviewers independently examined abstracts and extracted data. The effect size was obtained by the z-transformation of reliability coefficients. Data were pooled with random-effects models and meta-regression was done based on method of moments estimator.

Results:

Fourteen studies were included. Pooled coefficient for the CTAS was substantial 0.672 (CI 95%: 0.599-0.735). Mistriage is less than 50%. Agreement upon the adult version, among nurse-physician and near countries is higher than pediatrics version, other raters and farther countries, respectively.

Conclusion:

The CTAS showed acceptable level of overall reliability in the emergency department but need more development to reach almost perfect agreement.

A Web-Based Tool for Patient Triage in Emergency Department Settings: Validation Using the Emergency Severity Index

Background

We evaluated the concordance between triage scores generated by a novel Internet clinical decision support tool, Clinical GPS (cGPS) (Lumiata Inc, San Mateo, CA), and the Emergency Severity Index (ESI), a well-established and clinically validated patient severity scale in use today. Although the ESI and cGPS use different underlying algorithms to calculate patient severity, both utilize a five-point integer scale with level 1 representing the highest severity.

Objective

The objective of this study was to compare cGPS results with an established gold standard in emergency triage.

Methods

We conducted a blinded trial comparing triage scores from the ESI: A Triage Tool for Emergency Department Care, Version 4, Implementation Handbook to those generated by cGPS from the text of 73 sample case vignettes. A weighted, quadratic kappa statistic was used to assess agreement between cGPS derived severity scores and those published in the ESI handbook for all 73 cases. Weighted kappa concordance was defined a priori as almost perfect (kappa > 0.8), substantial (0.6 < kappa < 0.8), moderate (0.4 < kappa < 0.6), fair (0.2 < kappa< 0.4), or slight (kappa < 0.2).

Results

Of the 73 case vignettes, the cGPS severity score matched the ESI handbook score in 95% of cases (69/73 cases), in addition, the weighted, quadratic kappa statistic showed almost perfect agreement (kappa = 0.93, 95% CI 0.854-0.996). In the subanalysis of 41 case vignettes assigned ESI scores of level 1 or 2, the cGPS and ESI severity scores matched in 95% of cases (39/41 cases).

Conclusions

These results indicate that the cGPS is a reliable indicator of triage severity, based on its comparison to a standardized index, the ESI. Future studies are needed to determine whether the cGPS can accurately assess the triage of patients in real clinical environments.

The reliability of the Australasian Triage Scale: a meta-analysis

BACKGROUND

Although the Australasian Triage Scale (ATS) has been developed two decades ago, its reliability has not been defined; therefore, we present a meta-analyis of the reliability of the ATS in order to reveal to what extent the ATS is reliable.

DATA SOURCES

Electronic databases were searched to March 2014. The included studies were those that reported samples size, reliability coefficients, and adequate description of the ATS reliability assessment. The guidelines for reporting reliability and agreement studies (GRRAS) were used. Two reviewers independently examined abstracts and extracted data. The effect size was obtained by the z-transformation of reliability coefficients. Data were pooled with random-effects models, and meta-regression was done based on the method of moment's estimator.

RESULTS

Six studies were included in this study at last. Pooled coefficient for the ATS was substantial 0.428 (95%CI 0.340-0.509). The rate of mis-triage was less than fifty percent. The agreement upon the adult version is higher than the pediatric version.

CONCLUSION

The ATS has shown an acceptable level of overall reliability in the emergency department, but it needs more development to reach an almost perfect agreement.

Reliability of the Emergency Severity Index: Meta-analysis

OBJECTIVES

Although triage systems based on the Emergency Severity Index (ESI) have many advantages in terms of simplicity and clarity, previous research has questioned their reliability in practice. Therefore, the aim of this meta-analysis was to determine the reliability of ESI triage scales.

METHODS

This meta-analysis was performed in March 2014. Electronic research databases were searched and articles conforming to the Guidelines for Reporting Reliability and Agreement Studies were selected. Two researchers independently examined selected abstracts. Data were extracted in the following categories: version of scale (latest/older), participants (adult/paediatric), raters (nurse, physician or expert), method of reliability (intra/inter-rater), reliability statistics (weighted/unweighted kappa) and the origin and publication year of the study. The effect size was obtained by the Z-transformation of reliability coefficients. Data were pooled with random-effects models and a meta-regression was performed based on the method of moments estimator.

RESULTS

A total of 19 studies from six countries were included in the analysis. The pooled coefficient for the ESI triage scales was substantial at 0.791 (95% confidence interval: 0.787-0.795). Agreement was higher with the latest and adult versions of the scale and among expert raters, compared to agreement with older and paediatric versions of the scales and with other groups of raters, respectively.

CONCLUSION

ESI triage scales showed an acceptable level of overall reliability. However, ESI scales require more development in order to see full agreement from all rater groups. Further studies concentrating on other aspects of reliability assessment are needed.

Efficacy and task structure of bimanual training post stroke: a systematic review

BACKGROUND

Bimanual training has been shown to be as effective as, but not superior to, unimanual paretic upper extremity (UE) training interventions in improving paretic UE function and use post stroke. However, it is still unclear whether different training interventions or task structures within bimanual interventions may differentially affect the outcomes.

OBJECTIVE

The objectives of this review were to (1) systematically determine the efficacy of bimanual training in relation to the International Classification of Functioning, Disability and Health model components and (2) explore the structure of current bimanual training interventions.

METHOD

A systematic review was conducted using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Eleven studies were accepted for review.

RESULTS

Three main types of bimanual training emerged: functional task training (FTT), bilateral training with rhythmic auditory cues (BATRAC), and robot-assisted training (RAT). Bimanual training is generally efficacious overall in improving paretic UE movement in individuals with subacute and/or chronic stroke as compared with other interventions. FTT, BATRAC, and RAT showed no significant differences compared with conventional therapy. Bimanual training may have greater proximal control benefits but fewer benefits in terms of subjects' perceived amount and quality of use as compared with constraint-induced movement therapy.

CONCLUSION

There were not enough data to draw any conclusions about the effects of bimanual task symmetry or commonality of goal.

Evaluating implementation of the emergency severity index in a Belgian hospital

INTRODUCTION

Triage aims to categorize patients based on their clinical need and the available departmental resources. To accomplish this goal, one needs to ensure that the implemented triage system is reliable and that staff use it correctly. Therefore this study assessed the ability of Belgium nurses to apply the Emergency Severity Index (ESI), version 4, to hypothetical case scenarios after an educational intervention.

METHODS

An ESI educational intervention was implemented in accordance with the ESI manual. Using paper case scenarios, nurses' interrater agreement was assessed by comparing triage nurse ESI levels with the reference answers noted in the implementation manual. Interrater agreement was measured by the percentage of agreement and Cohen's κ coefficient using different weighting schemes.

RESULTS

Overall, 77.5% of the scenario cases were coded according the ESI guidelines, resulting in a good interrater agreement (κ = 0.72, linear weighted κ = 0.84, quadratic weighted κ = 0.92, and triage-weighted scheme = 0.79). Interrater agreement varied when evaluating each ESI level separately. Undertriage was more common than overtriage. The highest misclassification range (37.8%) occurred in ESI level 2 scenarios, with 99.2% of the misclassifications being undertriaged.

DISCUSSION

Implementation of the ESI into a novel setting guided by a locally developed training program resulted in suboptimal interrater agreement. Existing weighted κ schemes overestimated the interrater agreement between the triage nurse-assigned ESI level and the reference standard. By providing an aggregated measure of agreement, which allows partial agreement, clinically significant misclassification was masked by a misleading "good" interrater agreement.