Interobserver agreement in emergency department triage

Reliability of pediatric Rome IV criteria for the diagnosis of disorders of gut-brain interaction

BACKGROUND

The diagnosis of disorders of gut-brain interaction (DGBI) in children is exclusively based on clinical criteria called the Rome criteria. The inter-rater reliability (IRR) measures how well two raters agree with a diagnosis using the same diagnostic tool. Previous versions of the Rome criteria showed only fair to moderate IRR. There have been no studies assessing the IRR of the current edition of the pediatric Rome criteria (Rome IV). This study sought to investigate the IRR of the pediatric Rome IV criteria and compare its reliability with the previous versions of the Rome criteria. We hypothesized that changes made to Rome IV would result in higher IRR than previous versions.

METHODS

This study used the same methodology as the previous studies on Rome II and III, including identical clinical vignettes, number of raters, and levels of expertise. Participants included 10 pediatric gastroenterology fellows and 10 pediatric gastroenterology specialists. IRR was assessed using the percentage of agreement and Cohen's kappa coefficient to account for possible agreement by chance.

RESULTS

The average IRR percentage of agreement using the Rome IV criteria was 55% for pediatric gastroenterologists and 48.5% for fellows, indicating moderate agreement (k = 0.54 for specialists, k = 0.47 for fellows). The results demonstrated higher percentages of agreement and kappa coefficients compared to the Rome II and III criteria.

CONCLUSIONS

The findings demonstrate improved reliability in Rome IV compared to Rome II and III, suggesting that the changes incorporated into the Rome IV criteria have enhanced diagnostic consistency. Despite the advancements, the reliability is still moderate, indicating the need for further refinement of future versions of the Rome criteria.

Improving the wait time to triage at the emergency department

Triaging of patients at the emergency department (ED) is one of the key steps prior to initiation of doctor consult. To improve the overall wait time to consultation, we have identified the need to reduce the wait time to triage for ED patients. We seek to determine if the implementation of a series of plan, do, study, act (PDSA) cycles would improve the wait time to triage within 1 year. The interventions related to the PDSA cycles include the refining of triage criteria, ‘eyeball’ triage by senior nurses to facilitate direct bedding of patients, formation of a triage nurse clinician role, and a needs analysis of required nursing manpower. The baseline period for this study was from January 2017 to April 2017, with the results following implementation of the respective PDSA cycles sequentially tracked from May 2017 to March 2019. There was an improvement in the wait time to triage from a baseline duration of 18 min to the postimplementation period duration of 13 min, with a 25% decrease in variance from 16 to 12 min. The improvements were sustained. Strategies to further reduce wait time to triage at the ED are discussed. We also highlight the importance of adequate triage manpower, data-driven decision making and continued engagement of stakeholders in enabling positive outcomes from this quality improvement effort.

Reliability of the revised Swiss Emergency Triage Scale: a computer simulation study

Background

The Swiss Emergency Triage Scale (SETS) is a four-level emergency scale that previously showed moderate reliability and high rates of undertriage due to a lack of standardization. It was revised to better standardize the measurement and interpretation of vital signs during the triage process.

Objective

The aim of this study was to explore the inter-rater and test–retest reliability, and the rate of correct triage of the revised SETS.

Patients and methods

Thirty clinical scenarios were evaluated twice at a 3-month interval using an interactive computerized triage simulator by 58 triage nurses at an urban teaching emergency department admitting 60 000 patients a year. Inter-rater and test–retest reliabilities were determined using κ statistics. Triage decisions were compared with a gold standard attributed by an expert panel. Rates of correct triage, undertriage, and overtriage were computed. A logistic regression model was used to identify the predictors of correct triage.

Results

A total of 3387 triage situations were analyzed. Inter-rater reliability showed substantial agreement [mean κ: 0.68; 95% confidence interval (CI): 0.60–0.78] and test–retest almost perfect agreement (mean κ: 0.86; 95% CI: 0.84–0.88). The rate of correct triage was 84.1%, and rates of undertriage and overtriage were 7.2 and 8.7%, respectively. Vital sign measurement was an independent predictor of correct triage (odds ratios for correct triage: 1.29 for each additional vital sign measured, 95% CI: 1.20–1.39).

Conclusion

The revised SETS incorporating standardized vital sign measurement and interpretation during the triage process resulted in high reliability and low rates of mistriage.

Validity and Reliability of the Emergency Severity Index and Australasian Triage System in Pediatric Emergency Care of Mofid Children's Hospital in Iran

Objective:

To evaluate the validity, reliability, sensitivity, and specificity of the Emergency Severity Index (ESI) and Australasian Triage System (ATS) for children visiting admitted to the emergency department (ED).

Methods:

This was a prospective study occurred in the Mofid children's Hospital in Iran from August 2017 to November 2018 and children had aged ≤14 years and presented at the ED with a medical symptom were considered eligible for participation. This study was divided into two phases: in the first phase, we determined the inter-rater reliability of ESI version 4 and ATS by triage nurses and pediatric residents. In the second phase, to analyze the validity, sensitivity, and specificity of each triage system. Reliability and agreement rates were measured using kappa statistics.

Results:

ESI showed inter-rater reliability with kappa of 0.65–0.92 (P<0.001) and ATS showed inter-rater reliability with kappa of 0.51–0.87 ESI had sensitivity ranged from 81% to 95% and specificity ranged from 73% to 86%. In addition, sensitivity ranged of the ATS were 80% to 95% and specificity ranged from 74% to 87%. Under triage and over triage occurred in 12% and 15% of patients respectively in ESI and 13% and 15% of patients respectively in ATS.

Conclusion:

The ESI and ATS both valid to triage children in the ED section of Mofid children's Hospital paediatric. Reliability of the ESI is good, moderate to good for the ATS.

Standardized food challenges are subject to variability in interpretation of clinical symptoms

Background

Food challenge tests are the gold standard in diagnosing food allergy. Guidelines provide scoring systems to classify symptoms during challenge and typically recommend that challenges are considered positive when objective symptoms occur. However, currently no standard criteria for the definition of a positive challenge outcome exists and interpretation of food challenges mainly depends on clinical judgment. This study aims to assess inter- and intra-observer variability in outcomes of routinely performed peanut challenges in children.

Methods

All complete food challenge score sheets of double blind placebo controlled peanut challenges performed in 2008-2010 in an academic hospital were included. Score sheets were reassessed independently by three clinical experts including double reassessment in a subset of score sheets. Inter- and intra-observer variability was evaluated using kappa statistics.

Results

We included 191 food challenge score sheets. Inter-observer agreement on overall challenge outcome was moderate (κ = 0.59-0.65) and was fair (κ = 0.31-0.46) on challenges with symptoms. Intra-observer agreement on overall challenge outcome was good (κ = 0.63-0.77) but was moderate (κ = 0.50-0.60) on challenges with symptoms. Subjective symptoms (oral symptoms, abdominal complaints, food aversion) were significantly associated with disagreement between observers.

Conclusions

We demonstrate that, despite strict adherence to guidelines, there is a considerable amount of variability in reassessment of symptoms recorded on food challenges sheets between and within well trained clinicians, especially when subjective symptoms occur.

First evaluation of the paediatric version of the Swedish rapid emergency triage and treatment system shows good reliability

AIM

To investigate the reliability of Retts-p, Rapid emergency triage and treatment system-paediatric, with regard to inter-rater and intra-rater agreement.

METHOD

Twenty nurses responsible for triaging both children and adults at the Emergency Department, Östersund County Hospital, Sweden, were randomly selected to take part in the study. The nurses were asked to use the Retts-p triage system to retrospectively assess the written case reports on 40 paediatric cases, aged from 6 month to 17.5 years, who attended the Emergency Department in 2010 with surgical, orthopaedic and medical symptoms. Using the information provided regarding appearance, symptoms, previous medical history and vital signs, the nurses selected the most appropriate Emergency Symptoms and Signs algorithm and placed the child in one of the five triage categories. Two test rounds were performed, 3 months apart, using the same cases, to study both the inter-rater and intra-rater agreement for the priority level and the triage algorithm chosen by the triage nurses.

RESULTS

Good to very good agreement were shown for both inter-rater (quadratic κw 0.86, 95% CI 0.85-0.87) and intra-rater testing (quadratic κw 0.92, 95% CI 0.88-0.96).

CONCLUSION

Retts-p provided good to very good reliability in this first evaluation study of the triage system.

Suitability of emergency department attenders to be assessed in primary care: survey of general practitioner agreement in a random sample of triage records analysed in a service evaluation project

OBJECTIVES

To assess the proportion of emergency department (ED) attendances that would be suitable for primary care and the inter-rater reliability of general practitioner (GP) assessment of primary care suitability.

DESIGN OF STUDY

Survey of GPs' agreement of suitability for primary care on a random anonymised sample of all ED patients attending over a 1-month period.

SETTING

ED of a UK Hospital serving a population of 600 000.

METHOD

Four GPs independently used data extracted from clinical notes to rate the appropriateness for management in primary care as well as need for investigations, specialist review or admission. Agreement was assessed using Cohen's κ.

RESULTS

The mean percentage of patients that GPs considered suitable for primary care management was 43% (range 38-47%). The κ for agreement was 0.54 (95% CI 0.44 to 0.64) and 0.47(95% CI 0.38 to 0.59). In patients deemed not suitable for primary care, GPs were more likely to determine the need for specialist review (relative risks (RR)=3.5, 95% CI 3.0 to 4.2, p<0.001) and admission (RR=3.9, 95% CI 3.2 to 4.7, p<0.001). In patients assessed as suitable for primary care, GPs would initiate investigations in 51% of cases. Consensus over primary care appropriateness was higher for paediatric than for adult attenders.

CONCLUSIONS

A significant number of patients attending ED could be managed by GPs, including those requiring investigations at triage. A stronger agreement among GPs over place of care may be seen for paediatric than for adult attenders. More effective signposting of patients presenting with acute or urgent problems and supporting a greater role for primary care in relieving the severe workflow pressures in ED in the UK are potential solutions.

How community members and health professionals conceptualize medical emergencies: implications for primary care promotion

Access to continuous care through a primary care provider is associated with improved health outcomes, but many communities rely on emergency departments (EDs) for both emergent and non-emergent health problems. This article describes one portion of a community-based participatory research project and investigates the type of education that might be needed as part of a larger intervention to encourage use of a local primary care clinic. In this article we examine how people who live in a low-income urban community and the healthcare workers who serve them conceptualize 'emergency medical condition'. We conducted forum and focus group discussions with 52 community members and individual interviews with 32 healthcare workers. Our findings indicate that while community members share a common general definition of what constitutes a medical emergency, they also desire better guidelines for how to assess health problems as requiring emergency versus primary care. Pain, uncertainty and anxiety tend to influence their choice to use EDs rather than availability of primary care. Implications for increasing primary care use are discussed.

Effectiveness of a five-level Paediatric Triage System: an analysis of resource utilisation in the emergency department in Taiwan

Objectives

To examine the effectiveness of a five-level Paediatric Triage and Acuity System (Ped-TTAS) by comparing the reliability of patient prioritisation and resource utilisation with the four-level Paediatric Taiwan Triage System (Ped-TTS) among non-trauma paediatric patients in the emergency department (ED).

Methods

The study design used was a retrospective longitudinal analysis based on medical chart review and a computer database. Except for a shorter list of complaints and some abnormal vital sign criteria modifications, the structure and triage process for applying Ped-TTAS was similar to that of the Paediatric Canadian Emergency Triage and Acuity Scale. Non-trauma paediatric patients presenting to the ED were triaged by well-trained triage nurses using the four-level Ped-TTS in 2008 and five-level Ped-TTAS in 2010. Hospitalisation rates and medical resource utilisation were analysed by acuity levels between the contrasting study groups.

Results

There was a significant difference in patient prioritisation between the four-level Ped-TTS and five-level Ped-TTAS. Improved differentiation was observed with the five-level Ped-TTAS in predicting hospitalisation rates and medical costs.

Conclusions

The five-level Ped-TTAS is better able to discriminate paediatric patients by triage acuity in the ED and is also more precise in predicting resource utilisation. The introduction of a more accurate acuity and triage system for use in paediatric emergency care should provide greater patient safety and more timely utilisation of appropriate ED resources.

Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study

OBJECTIVES

It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.

STUDY DESIGN AND SETTING

From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.

RESULTS

The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.

CONCLUSION

When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.

Does nonmedical hospital admission staff accurately triage emergency department patients?

OBJECTIVES

Utilization of hospital emergency departments (EDs) is continuously increasing. Though nurses and physicians are ultimately responsible for the definite triage decisions, initial ED patient triage is frequently performed by hospital admission staff. This study analyzes the quality of assessment of the severity of emergencies and the choice of treatment unit made by hospital admission staff.

METHODS

One thousand fifty-nine consecutive surgical and medical patients entering the ED of the University Hospital Basel during an 11-day period were independently assessed by hospital admission staff without formal medical training, ED nursing staff, and ED physicians. Emergencies were classified by severity (intervention within minutes/hours/days) or by severity and resource utilization (immediate intervention with/without life-threatening condition, delayed intervention with high/low/no demand of resources). Emergency assessment and triage decision (surgical/medical, outpatient/inpatient treatment) were documented independently by all three ED staff groups.

RESULTS

In 64% of the cases, initial assessment by admission staff corresponded with the final assessment by the ED physician. Concordance was, however, poor (kappa=0.23). Underestimation of the severity occurred in 7.5% of cases without severe or lethal consequences. Ninety-four percent of patients were treated in the unit to which they were originally triaged by the admission staff.

CONCLUSION

Triage quality regarding the choice of treatment unit was found to be excellent, whereas the quality of the assessment of the severity of the emergency by nonmedical ED admission staff was acceptable. ED patients have to be assessed by medical staff early after admission to ensure adequate and timely interventions.

Influence of Ambulance Arrival on Emergency Department Time to Be Seen

OBJECTIVES

For a limited number of presenting complaints, arrival by ambulance has been shown in some emergency departments to decrease time to be seen by a physician. We sought to determine if this time advantage could be demonstrated as a national trend over a variety of presenting complaints.

METHODS

A secondary analysis was performed on the National Hospital Ambulatory Medical Care Survey, a national probability sample of emergency department visits. To compare waiting times between patients arriving by ambulance and those arriving by walk-in, a survival analysis was performed using univariate and multivariate Cox proportional hazards models. Primary variables of interest were mode of arrival, waiting time to see physician, and immediacy to be seen (triage category). The weighted values were utilized to produce national estimates. Patients who left without being seen were treated as right censored data.

RESULTS

A total of 61,130 records, weighted to represent 268.3 million emergency department visits from 1997 to 2000, were included in the analysis. Patients arrived by ambulance in 14.4% of these cases. Median wait time for patients arriving by ambulance was 14.1 minutes (95% confidence interval [CI], 4.3 to 34.2) as compared with 26.0 minutes (95% CI, 11.5 to 55.1) for patients who arrived by walk-in. In the multivariate analysis, arrival by ambulance offered a 25.0% (95% CI, 19.0% to 31.6%) time advantage over walk-in and a 40.8% (95% CI, 23.5% to 58.7%) time advantage over arrival by public service.

CONCLUSIONS

Arrival by ambulance offered a time to be seen advantage for a broad range of presenting complaints in the National Hospital Ambulatory Medical Care Survey across all triage categories.

Interactive triage simulator revealed important variability in both process and outcome of emergency triage

BACKGROUND AND OBJECTIVES

(1) to evaluate the performance of emergency department triage; (2) to explore the variability of the triage process; and (3) to examine the reliability of a four-level triage scale, using an interactive triage simulator.

METHODS

We developed 22 interactive computerized vignettes describing patients presenting at the Emergency Department. Each vignette displayed the presenting complaint and offered the possibility to ask questions and obtain vital signs before deciding on the triage severity rating. The vignettes were rated twice by 45 nurses and 8 physicians.

RESULTS

(1) The concordance between the observed triage decision and an expert-attributed emergency level was perfect in 58% of the situations. Triage acuity was overestimated in 11%, and underestimated in 31%. (2) There was a wide variability in the triage process across observers and vignettes. The mean number of questions varied from 1.77 to 18.95 across individuals, and from 3.96 to 11.60 across vignettes. (3) Finally, the test-retest reliability of our instrument was good (weighted kappa = 0.82) but the interrater reliability was moderate (weighted kappa = 0.41).

CONCLUSIONS

The computerized triage simulator is an innovative tool to evaluate the process and the performance of triage and to evaluate the reliability of a triage instrument.

Development of an accelerated MSCT protocol (Triage MSCT) for mass casualty incidents: comparison to MSCT for single-trauma patients

During multiple casualty incidents (MCI) emergency radiology departments have to deal with a large number of patients with suspected severe trauma within a short period of time. The aim of this study was to develop a suitable accelerated multislice computed tomography (MSCT) protocol to increase patient throughput for this kind of emergency situation. We presumed a scenario of 15 patients being admitted to the trauma service with suspicion of severe injuries after a MCI over a period of 2 h. An accelerated Triage MSCT protocol was developed and evaluated for MSCT scanner productivity (patients per hour) and time (minutes) needed for a total MSCT body workup using an anthropomorphic phantom. In addition, time (minutes) for transfer and preparation was measured. These timeframes were compared to a control group consisting of 144 single patients with multiple trauma undergoing standard MSCT according to our trauma room protocol. All MSCT studies were conducted using a 4-detector row scanner. (1) For the study group (Triage MSCT), average time for patient transfer and preparation was 2.9 min (2.5-4.3 min), mean CT examination time was 2.1 min (1.7-2.4 min); image reconstruction took 4.0 min (3.3-4.3 min). Total time in scanner room was 8.9 min (7.7-11.3 min), resulting in a maximal productivity of 6.7 patients per hour. Image transfer to the digital picture archive and communication system archive was completed after an average 9.5 min (8.9-10.8 min). (2) For the control group (single casualty MSCT), the mean time for patient transfer and preparation was 20.4 min (9.0-39.2 min), mean examination time was 6.0 min (3.1-11.3 min). Times for image reconstructions were not recorded in the patient series. Mean total time in scanner room was 25.3 min (11.0-72.4 min), resulting in a patient throughput of 2.4 patients per hour. MSCT has potential to serve as a powerful tool in triage of multiple casualty patients. The introduction of a Triage MSCT scanning protocol resulted in an increase of patient throughput per hour by a factor of almost 3.

Comparison of the Emergency Severity Index (ESI) and the Taiwan Triage System in Predicting Resource Utilization

BACKGROUND/PURPOSE

The importance of accurate triage in Taiwan is becoming more apparent with the increasing number of emergency department (ED) patients, and resources for the National Health Insurance becoming constrained. This study compared the ability of the Taiwan triage system (TTS) and the standardized 5-level Emergency Severity Index (ESI) triage system to predict ED resource utilization.

METHODS

Patients arriving at the ED were triaged by both TTS and by using a two-page checklist of ESI criteria during the 3-month study period. The ESI triage level was calculated independently to avoid bias. Disease category (trauma vs. nontrauma), length of stay (LOS) and hospitalization data were evaluated.

RESULTS

A total of 3172 patients with both ESI and TWN evaluation were included. The distributions of ESI ratings within TTS level 1 were: ESI 1, 21.1%; ESI 2, 68.1%; ESI 3, 7.4%; ESI 4, 3.4%; ESI 5, 0%. For TTS level 3, they were: ESI 1, 0.1%; ESI 2, 26.2%; ESI 3, 39.5%; ESI 4, 27.5%; ESI 5, 6.8%. Hospitalization rates were 74.5%, 40.9% and 22.2% in TTS levels 1, 2 and 3, respectively; and were 96.2%, 47.0%, 30.9%, 6.7%and 6.6% in ESI levels 1, 2, 3, 4 and 5, respectively. TTS triaged more trauma patients as life-threatening/emergent condition than nontrauma patients (68.8% vs. 48.4%, p < 0.001). Triage by ESI, however, showed no significant difference in the percentage of trauma and nontrauma patients with highly acute conditions (44.2% vs. 46.6%, p = 0.230). Patients with ESI level 4 or 5 have significantly shorter ED LOS than those with ESI level 3.

CONCLUSION

ESI produces more accurate discriminating patient acuity, ED LOS and hospitalization rate than TTS. Adopting a standardized 5-level triage tool might improve resource utilization planning of ED practice.

A Bayesian model for triage decision support

OBJECTIVE

To compare triage decisions of an automated emergency department triage system with decisions made by an emergency specialist.

METHODS

In a retrospective setting, data extracted from charts of 90 patients with chief complaint of non-traumatic abdominal pain were used as input for triage system and emergency medicine specialist. The final disposition and diagnoses of the physicians who visited the patient in Emergency Department (ED) as reflected in the medical records were considered as control. Results were compared by chi(2)-test and a binary logistic regression model.

RESULTS

Compared to emergency medicine specialist, triage system had higher sensitivity (90% versus 64%) and lower specificity (25% versus 48%) for patients who required hospitalization. The triage system successfully predicted the Admit decisions made in the ED whereas the emergency medicine specialist decisions could not predict the ED disposition. Both triage system and emergency medicine specialist properly disposed 56% of cases, however, the emergency medicine specialist in this study under-disposed more patients than the triage system considering Admit disposition (p=0.004) while he appropriately discharged more patients compared to the triage system (p=0.017).

CONCLUSION

The triage system studied here shows promise as a triage decision support tool to be used for telephone triage and triage in the emergency departments. This technology may also be useful to the patients as a self-triage tool. However, the efficiency of this particular application of this technology is unclear.

Interobserver and intraobserver reliability of the Rome II criteria in children

BACKGROUND

Functional gastrointestinal disorders are common in children. It has been suggested that the diagnosis of these conditions should be based on the "pediatric Rome II" criteria. The interobserver reliability for the DSM-IV, another symptom-based criteria is considered almost perfect in multiple studies. There are no studies assessing the reliability of the Rome II criteria in children.

OBJECTIVES

To evaluate the reliability of the pediatric Rome II criteria.

METHODS

Interobserver reliability-Ten pediatric gastroenterologists and 10 fellows in pediatric gastroenterology were provided with 20 clinical vignettes, the Rome II criteria, and a list of 15 possible diagnoses. Each of the raters was instructed to select one or more diagnoses for each vignette. Intraobserver reliability-The specialists were provided with the same set of vignettes 4 months later.

RESULTS

Average percentage of agreement coefficient: 45% (specialists), 47% (fellows). In order to correct for possible agreement by chance, we calculated the kappa coefficient, a measure of pairwise agreement corrected for chance. Specialists: k = 0.37 (p < 0.0001), trainees: k = 0.41, (p < 0.0001). Physicians with a special interest in functional gastrointestinal disorders (k = 0.37, p < 0.0001), and other specialists (k = 0.38, p < 0.0001). Analysis of data in pain and constipation diagnosis subgroups revealed even lower kappa (constipation: k = 0.2, p < 0.0001; pain: k = 0.3, p < 0.0001). Intraobserver agreement: k = 0.63 (p < 0.0001).

CONCLUSION

The interobserver reliability of the Rome II criteria among pediatric gastroenterologists and fellows is low. Further validation of the criteria is necessary.