The effectiveness of physiologically based early warning or track and trigger systems after triage in adult patients presenting to emergency departments: a systematic review

An economic evaluation of the Prioritising Responses Of Nurses To deteriorating patient Observations (PRONTO) clinical trial

BACKGROUND

Early recognition and response to clinical deterioration reduce the frequency of in-hospital cardiac arrests, mortality, and unplanned intensive care unit (ICU) admissions. This study aimed to investigate the impact of the Prioritising Responses Of Nurses To deteriorating patient Observations (PRONTO) intervention on hospital costs and patient length of stay (LOS).

METHOD

The PRONTO cluster randomised control trial was conducted to improve nurses' responses to patients with abnormal vital signs. Hospital data were collected pre-intervention (T0) at 6 months (T1) and 12 months (T2) post-intervention. The economic evaluation involved a cost-consequence analysis from the hospital's perspective. Generalised estimating equations were used to estimate the parameters for regression models of the difference in costs and LOS between study groups and time points.

RESULTS

Hospital admission data for 6065 patients (intervention group, 3102; control group, 2963) were collected from four hospitals for T0, T1 and T2. The intervention cost was 69.61 A$ per admitted patient, including the additional intervention training for nurses and associated labour costs. The results showed cost savings and a shorter LOS in the intervention group between T0 - T1 and T0 - T2 (cost differences T0 - T1: -364 (95% CI -3,782; 3049) A$ and T0 - T2: -1,710 (95% CI -5,162; 1,742) A$; and LOS differences T0 - T1: -1.10 (95% CI -2.44; 0.24) days and T0 & T2: -2.18 (95% CI -3.53; -0.82) days).

CONCLUSION

The results of the economic analysis demonstrated that the PRONTO intervention improved nurses' responses to patients with abnormal vital signs and significantly reduced hospital LOS by two days at 12 months in the intervention group compared to baseline. From the hospital's perspective, savings from reduced hospitalisations offset the costs of implementing PRONTO.

CARDS, a Novel Prognostic Index for Risk Stratification and In-Hospital Monitoring

Background: Sodium fluctuation is independently associated with clinical deterioration. We developed and validated a prognostic index based on sodium fluctuation for risk stratification and in-hospital monitoring. Methods: This study included 33,323 adult patients hospitalized at a tertiary care hospital in 2014. The first 28,279 hospitalizations were analyzed to develop the model and then the validity of the model was tested using data from 5044 subsequent hospitalizations. We predict in-hospital mortality using age, comorbidity, range of sodium fluctuation, and duration of sodium fluctuation, abbreviated as CARDS. Results: In-hospital mortality was similar in the derivation (0.6%) and validation (0.4%) cohorts. In the derivation cohort, four independent risk factors for mortality were identified using logistic regression: age (66-75, 2 points; >75, 3 points); Charlson comorbidity index (>2, 5 points); range of sodium fluctuation (7-10, 4 points; >10, 10 points); and duration of fluctuation (≤3, 3 points). The AUC was 0.907 (95% CI: 0.885-0.928) in the derivation cohort and 0.932 (95% CI: 0.895-0.970) in the validation cohort. In the derivation cohort, in-hospital mortality was 0.106% in the low-risk group (0-7 points), 1.076% in the intermediate-risk group (8-14 points), and 8.463% in the high-risk group (15-21 points). In the validation cohort, in-hospital mortality was 0.049% in the low-risk group, 1.064% in the intermediate-risk group, and 8.403% in the high-risk group. Conclusions: These results suggest that patients at low, intermediate, and high risk for in-hospital mortality may be identified by CARDS mainly based on sodium fluctuation.

Developing a consensus-based definition of out-of-hospital clinical deterioration: A Delphi study

BACKGROUND

Clinical deterioration is a time-critical medical emergency requiring rapid recognition and intervention. Deteriorating patients are seen across various healthcare settings, including the out-of-hospital (OOH) environment. OOH care is an evolving area of medicine where decisions are made regarding priority and timing of clinical interventions, ongoing management, and transport to appropriate care. To date, the literature lacks a standardised definition of OOH clinical deterioration.

OBJECTIVE

The objective of this study was to create a consensus-based definition of OOH clinical deterioration informed by emergency medicine health professionals.

METHODS

A Delphi study consisting three rounds was conducted electronically between June 2020 and January 2021. The expert panel consisted of 30 clinicians, including emergency physicians and paramedics.

RESULTS

A consensus-based definition of OOH clinical deterioration was identified as changes from a patient's baseline physiological status resulting in their condition worsening. These changes primarily take the form of measurable vital signs and assessable symptoms but should be evaluated in conjunction with the history of events and pertinent risk factors. Clinicians should be suspicious that a patient could deteriorate when changes occur in one or more of the following vital signs: respiratory rate, heart rate, blood pressure, Glasgow Coma Scale, oxygen saturation, electrocardiogram, and skin colour. Almost all participants (92%) indicated an early warning system would be helpful to assist timely recognition of deteriorating patients.

CONCLUSION

The creation of a consensus-based definition of OOH clinical deterioration can serve as a starting point for the development and validation of OOH-specific early warning systems. Moreover, a standardised definition allows meaningful comparisons to be made across health services and ensures consistency in future research. This study has shown recognition of OOH clinical deterioration to be a complex issue requiring further research. Improving our understanding of key factors contributing to deterioration can assist timely recognition and intervention, potentially reducing unnecessary morbidity and mortality.

Development and validation of the early warning system scores ontology

BACKGROUND

Clinical early warning scoring systems, have improved patient outcomes in a range of specializations and global contexts. These systems are used to predict patient deterioration. A multitude of patient-level physiological decompensation data has been made available through the widespread integration of early warning scoring systems within EHRs across national and international health care organizations. These data can be used to promote secondary research. The diversity of early warning scoring systems and various EHR systems is one barrier to secondary analysis of early warning score data. Given that early warning score parameters are varied, this makes it difficult to query across providers and EHR systems. Moreover, mapping and merging the parameters is challenging. We develop and validate the Early Warning System Scores Ontology (EWSSO), representing three commonly used early warning scores: the National Early Warning Score (NEWS), the six-item modified Early Warning Score (MEWS), and the quick Sequential Organ Failure Assessment (qSOFA) to overcome these problems.

METHODS

We apply the Software Development Lifecycle Framework-conceived by Winston Boyce in 1970-to model the activities involved in organizing, producing, and evaluating the EWSSO. We also follow OBO Foundry Principles and the principles of best practice for domain ontology design, terms, definitions, and classifications to meet BFO requirements for ontology building.

RESULTS

We developed twenty-nine new classes, reused four classes and four object properties to create the EWSSO. When we queried the data our ontology-based process could differentiate between necessary and unnecessary features for score calculation 100% of the time. Further, our process applied the proper temperature conversions for the early warning score calculator 100% of the time.

CONCLUSIONS

Using synthetic datasets, we demonstrate the EWSSO can be used to generate and query health system data on vital signs and provide input to calculate the NEWS, six-item MEWS, and qSOFA. Future work includes extending the EWSSO by introducing additional early warning scores for adult and pediatric patient populations and creating patient profiles that contain clinical, demographic, and outcomes data regarding the patient.

Developing a real-time detection tool and an early warning score using a continuous wearable multi-parameter monitor

Background: Currently-used tools for early recognition of clinical deterioration have high sensitivity, but with low specificity and are based on infrequent measurements. We aimed to develop a pre-symptomatic and real-time detection and warning tool for potential patients’ deterioration based on multi-parameter real-time warning score (MPRT-WS).

Methods: A total of more than 2 million measurements were collected, pooled, and analyzed from 521 participants, of which 361 were patients in general wards defined at high-risk for deterioration and 160 were healthy participants allocation as controls. The risk score stratification was based on cutoffs of multiple physiological parameters predefined by a panel of specialists, and included heart rate, blood oxygen saturation (SpO2), respiratory rate, cuffless systolic and diastolic blood pressure (SBP and DBP), body temperature, stroke volume (SV), cardiac output, and systemic vascular resistance (SVR), recorded every 5 min for a period of up to 72 h. The data was used to define the various risk levels of a real-time detection and warning tool, comparing it with the clinically-used National Early Warning Score (NEWS).

Results: When comparing risk levels among patients using both tools, 92.6%, 6.1%, and 1.3% of the readings were defined as “Low”, “Medium”, and “High” risk with NEWS, and 92.9%, 6.4%, and 0.7%, respectively, with MPRT-WS (p = 0.863 between tools). Among the 39 patients that deteriorated, 30 patients received ‘High’ or ‘Urgent’ using the MPRT-WS (42.7 ± 49.1 h before they deteriorated), and only 6 received ‘High’ score using the NEWS. The main abnormal vitals for the MPRT-WS were SpO2, SBP, and SV for the “Urgent” risk level, DBP, SVR, and SBP for the “High” risk level, and DBP, SpO2, and SVR for the “Medium” risk level.

Conclusion: As the new detection and warning tool is based on highly-frequent monitoring capabilities, it provides medical teams with timely alerts of pre-symptomatic and real-time deterioration.

Tracking the National Early Warning Score 2 from Prehospital Care to the Emergency Department: A Prospective, Ambulance-Based, Observational Study

Aim of the study: To assess the prognostic ability of the National Early Warning Score 2 (NEWS2) at three time points of care -at the emergency scene (NEWS2-1), just before starting the transfer by ambulance to the hospital (NEWS2- 2), and at the hospital triage box (NEWS2-3)- to estimate in-hospital mortality after two days since the index event.Methods: Prospective, multicenter, ambulance-based, cohort ongoing study in adults (>18 years) consecutively attended by advanced life support (ALS) and evacuated with high-priority to the emergency departments (ED) between October 2018 and May 2021. Vital sign measures were used to calculate the NEWS2 score at each time point, then this score was entered in a logistic regression model as the single predictor. Two outcomes were considered: first, all-cause mortality of the patients within 2 days of presentation to EMS, and second, unplanned ICU admission. The calibration and scores comparison was performed by representing the predicted vs the observed risk curves according to NEWS score value.Results: 4943 patients were enrolled. Median age was 69 years (interquartile range 53- 81). The NEWS2-3 presented the better performance for all-cause two-day in-hospital mortality with an AUC of 0.941 (95% CI: 0.917-0.964), showing statistical differences with both the NEWS2-1 (0.872 (95% CI: 0.833-0.911); p < 0.003) and with the NEWS2- 2 (0.895 (95% CI: 0.866-0.925; p < 0.05). The calibration and scores comparison results showed that the NEWS2-3 was the best predictive score followed by the NEWS2-2 and the NEWS2-1, respectively.Conclusions: The NEWS2 has an excellent predictive performance. The score showed a very consistent response over time with the difference between "at the emergency scene" and "pre-evacuation" presenting the sharpest change with decreased threshold values, thus displaying a drop in the risk of acute clinical impairment.

Artificial Intelligence for the Prediction of In-Hospital Clinical Deterioration: A Systematic Review

To analyze the available literature on the performance of artificial intelligence-generated clinical models for the prediction of serious life-threatening events in non-ICU adult patients and evaluate their potential clinical usage.

The use of early warning system scores in prehospital and emergency department settings to predict clinical deterioration: A systematic review and meta-analysis

Background

It is unclear which Early Warning System (EWS) score best predicts in-hospital deterioration of patients when applied in the Emergency Department (ED) or prehospital setting.

Methods

This systematic review (SR) and meta-analysis assessed the predictive abilities of five commonly used EWS scores (National Early Warning Score (NEWS) and its updated version NEWS2, Modified Early Warning Score (MEWS), Rapid Acute Physiological Score (RAPS), and Cardiac Arrest Risk Triage (CART)). Outcomes of interest included admission to intensive care unit (ICU), and 3-to-30-day mortality following hospital admission. Using DerSimonian and Laird random-effects models, pooled estimates were calculated according to the EWS score cut-off points, outcomes, and study setting. Risk of bias was evaluated using the Newcastle-Ottawa scale. Meta-regressions investigated between-study heterogeneity. Funnel plots tested for publication bias. The SR is registered in PROSPERO (CRD42020191254).

Results

Overall, 11,565 articles were identified, of which 20 were included. In the ED setting, MEWS, and NEWS at cut-off points of 3, 4, or 6 had similar pooled diagnostic odds ratios (DOR) to predict 30-day mortality, ranging from 4.05 (95% Confidence Interval (CI) 2.35–6.99) to 6.48 (95% CI 1.83–22.89), p = 0.757. MEWS at a cut-off point ≥3 had a similar DOR when predicting ICU admission (5.54 (95% CI 2.02–15.21)). MEWS ≥5 and NEWS ≥7 had DORs of 3.05 (95% CI 2.00–4.65) and 4.74 (95% CI 4.08–5.50), respectively, when predicting 30-day mortality in patients presenting with sepsis in the ED. In the prehospital setting, the EWS scores significantly predicted 3-day mortality but failed to predict 30-day mortality.

Conclusion

EWS scores’ predictability of clinical deterioration is improved when the score is applied to patients treated in the hospital setting. However, the high thresholds used and the failure of the scores to predict 30-day mortality make them less suited for use in the prehospital setting.

Evaluation of NEWS2 response thresholds in a retrospective observational study from a UK acute hospital

OBJECTIVE

Use of National Early Warning Score 2 (NEWS2) has been mandated in adults admitted to acute hospitals in England. Urgent clinical review is recommended at NEWS2 ≥5. This policy is recognised as requiring ongoing evaluation. We assessed NEWS2 acquisition, alerting at key thresholds and patient outcomes, to understand how response recommendations would affect clinical resource allocation.

SETTING

Adult acute hospital in England.

DESIGN

Retrospective observational cohort study.

PARTICIPANTS

100 362 consecutive admissions between November 2018 and July 2019.

OUTCOME

Death or admission to intensive care unit within 24 hours of a score.

METHODS

NEWS2 were assembled as single scores from consecutive 24-hour time frames, (the first NEWS2 termed 'Index-NEWS2'), or as all scores from the admission (termed All-NEWS2). Scores were excluded when a patient was in intensive care, in the presence of a decision not to attempt cardiopulmonary resuscitation, or on day 1 of elective admission.

RESULTS

A mean of 4.5 NEWS2 were acquired per patient per day. The outcome rate following an Index-NEWS2 was 0.22/100 patient-days. The sensitivity of outcome prediction at Index-NEWS2 ≥5=0.46, and number needed to evaluate (NNE)=52. At this threshold, a mean of 37.6 alerts/100 patient-days would be generated, occurring in 12.3% of patients on any single day. Threshold changes to increase sensitivity by 0.1, would result in a twofold increase in alert rate and 1.5-fold increase in NNE. Overall, NEWS2 classification performance was significantly worse on Index-scores than All-scores (c-statistic=0.78 vs 0.85; p<0.001).

CONCLUSIONS

The combination of low event-rate, high alert-rate and low sensitivity, in patients for cardiopulmonary resuscitation, means that at current NEWS2 thresholds, resource demand would be sufficient to meaningfully compete with other pathways to clinical evaluation. In analyses that epitomise in-patient screening, NEWS2 performance suggests a need for re-evaluation of current response recommendations in this population.

Wavelet Transform Artificial Intelligence Algorithm-Based Data Mining Technology for Norovirus Monitoring and Early Warning

Norovirus monitoring and early warning can be used for diagnosis without etiological testing, and the treatment of this disease does not require the antibiotics. It often occurs in preschool children and affects their growth and development, so the coping measures for this disease are more prevention than treatment. In this study, the clinical data of 2133 children with diarrhea were collected. Based on the artificial intelligence (AI) algorithm of wavelet transform, a related model for data mining and processing of children's intestinal ultrasound images and stool specimens was constructed. Then, the norovirus infection trend was warned based on the wavelet analysis algorithm model. The results showed that the intestinal ultrasound image processed by the wavelet transform algorithm was clearer. The positive detection rate of norovirus in children with clinical diarrhea was as high as 59%, and the children had different degrees of body damage, of which the probability of compensatory metabolic acidosis was the highest. The epidemiological analysis found that children with norovirus infection were mainly concentrated in the age group under 2 years old and over 5 years old and showed a peak of infection in December. In summary, the intelligent algorithm based on wavelet transform can realize the noise reduction of intestinal ultrasound, and it should protect children with susceptible age and susceptible seasons to reduce the clinical infection rate of norovirus.

Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review

OBJECTIVE

To assess predictive performance of universal early warning scores (EWS) in disease subgroups and clinical settings.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, Embase and Cochrane database of systematic reviews from 1997 to 2019.

INCLUSION CRITERIA

Randomised trials and observational studies of internal or external validation of EWS to predict deterioration (mortality, intensive care unit (ICU) transfer and cardiac arrest) in disease subgroups or clinical settings.

RESULTS

We identified 770 studies, of which 103 were included. Study designs and methods were inconsistent, with significant risk of bias (high: n=16 and unclear: n=64 and low risk: n=28). There were only two randomised trials. There was a high degree of heterogeneity in all subgroups and in national early warning score (I2=72%-99%). Predictive accuracy (mean area under the curve; 95% CI) was highest in medical (0.74; 0.74 to 0.75) and surgical (0.77; 0.75 to 0.80) settings and respiratory diseases (0.77; 0.75 to 0.80). Few studies evaluated EWS in specific diseases, for example, cardiology (n=1) and respiratory (n=7). Mortality and ICU transfer were most frequently studied outcomes, and cardiac arrest was least examined (n=8). Integration with electronic health records was uncommon (n=9).

CONCLUSION

Methodology and quality of validation studies of EWS are insufficient to recommend their use in all diseases and all clinical settings despite good performance of EWS in some subgroups. There is urgent need for consistency in methods and study design, following consensus guidelines for predictive risk scores. Further research should consider specific diseases and settings, using electronic health record data, prior to large-scale implementation.

PROSPERO REGISTRATION NUMBER

PROSPERO CRD42019143141.

Consensus on predictors of clinical deterioration in emergency departments: A Delphi process study

AIM

The study aim was to determine relevance and applicability of generic predictors of clinical deterioration in emergency departments based on consensus among clinicians.

METHODS

Thirty-three predictors of clinical deterioration identified from literature were assessed in a modified two-stage Delphi-process. Sixty-eight clinicians (physicians and nurses) participated in the first round and 48 in the second round; all treating hospitalized patients in Danish emergency departments, some with pre-hospital experience. The panel rated the predictors for relevance (relevant marker of clinical deterioration) and applicability (change in clinical presentation over time, generic in nature and possible to detect bedside). They rated their level of agreement on a 9-point Likert scale and were also invited to propose additional generic predictors between the rounds. New predictors suggested by more than one clinician were included in the second round along with non-consensus predictors from the first round. Final decisions of non-consensus predictors after second round were made by a research group and an impartial physician.

RESULTS

The Delphi-process resulted in 19 clinically relevant and applicable predictors based on vital signs and parameters (respiratory rate, saturation, dyspnoea, systolic blood pressure, pulse rate, abnormal electrocardiogram, altered mental state and temperature), biochemical tests (serum c-reactive protein, serum bicarbonate, serum lactate, serum pH, serum potassium, glucose, leucocyte counts and serum haemoglobin), objective clinical observations (skin conditions) and subjective clinical observations (pain reported as new or escalating, and relatives' concerns).

CONCLUSION

The Delphi-process led to consensus of 19 potential predictors of clinical deterioration widely accepted as relevant and applicable in emergency departments.

Translation of evidence into policy to improve clinical practice: the development of an emergency department rapid response system

BACKGROUND

Undetected clinical deterioration is a major cause of high mortality events in Emergency Department (ED) patients. Yet, there is no known model to guide the recognition and response to clinical deterioration in the ED, integrating internal and external resources.

METHODS

An integrative review was firstly conducted to identify the critical components of recognising and responding to clinical deterioration in the ED. Components identified from the review were analysed by clinical experts and informed the development of an ED Clinical Emergency Response System (EDCERS).

RESULTS

Twenty four eligible studies were included in the review. Eight core components were identified: 1) vital sign monitoring; 2) track and trigger system; 3) communication plan; 4) response time; 5) emergency nurse response; 6) emergency physician response; 7) critical care team response; and 8) specialty team response. These components informed the development of the EDCERS protocol, integrating responses from staff internal and external to the ED.

CONCLUSIONS

EDCERS was based on the best available evidence and considered the cultural context of care. Future research is needed to determine the useability and impact of EDCERS on patient and health outcomes.

Addressing overcrowding in an emergency department: an approach for identifying and treating influential factors and a real-life application

Background

Overcrowding in hospital emergency departments that arises from long length-of-stay is an unfortunate common occurrence. While some factors affecting length-of-stay are well known, there may be additional factors that have not yet been properly addressed. This research offers a method for emergency department managers to use available data from their departments to identify new factors that significantly influence emergency departments crowding and patient length-of-stay.

Methods

We propose an algorithm that can assist emergency department managers in determining which of these factors to address, given budgetary constraints. We implemented it in a case study which takes into account factors that are known to be influential, e.g., reason for arrival, occupancy in the emergency department, and arrival time, as well as factors that are explored for the first time in this paper, such as patient heart rate, the number of accompanying escorts, and the number of tests assigned to patients (e.g., blood tests and urinalysis).

Results

All the implemented and new factors are shown to have a significant influence on the length-of-stay and crowding. We also obtained additional support for our results by interviewing emergency departments physicians and nurses from various hospitals.

Conclusions

It is expected that, by taking all the above factors into consideration, emergency departments efficiency can be improved. The algorithm constructed here allows the choice of the most cost-effective factors to be improved, subject to a given budget. We have been able to derive practical recommendations that emergency departments managers might use to limit crowding and patient length-of-stay.

Failure of vital sign normalization is more strongly associated than single measures with mortality and outcomes

BACKGROUND

Modified Early Warning Systems (MEWS) scores offer proxies for morbidity and mortality that are easily acquired, but there are limited data on what changing MEWS scores within the ED indicate. We examined the correlation of changing MEWS scores during resuscitation in the ED and in-hospital morbidity and mortality.

METHODS

We conducted a retrospective analysis on medical ED patients with simplified MEWS scores (without urine output or mental status) admitted to a single academic tertiary care center over one year. Triage-to-Last delta MEWS score and Triage-to-Max delta MEWS scores were calculated and correlated to in-hospital mortality, ICU admission, length of stay (LOS) and diagnosis of sepsis.

RESULTS

Our analysis included 8322 ED patients with an ICU admission rate of 17% and a mortality rate of 2%. Every point of worsened MEWS after triage was more strongly associated with all-cause mortality (OR 2.41, 95% CI 1.96-2.97) than triage MEWS alone (OR 1.33, 95% CI 1.23-1.44; p < 0.001). Likewise, each point of worsened MEWS was associated with increased odds of ICU admission (Triage-to-Last: OR 2.12, 95% CI 1.92-2.33 and Triage-to-Max: OR 1.52, 95% CI 1.45-1.60, respectively). Among patients with suspected infection, similar associations are found.

CONCLUSIONS

Dynamic vital signs in the emergency department, as categorized by delta MEWS, and failure to normalize abnormalities, were associated with increased mortality, ICU admission, LOS, and the diagnosis of sepsis. Our results suggest that MEWS scores that do not normalize, from triage onward, are more strongly associated with outcome than any single score.

Related factors of the nursing diagnosis ineffective breathing pattern in an intensive care unit

OBJECTIVE

to identify the predicting factors and sensitivity, specificity, positive and negative related value of nursing diagnosis Ineffective Breathing Pattern among patients of an intensive care unit.

METHOD

cross-sectional study. A logistic regression was fitted to assess the simultaneous effects of related factors.

RESULTS

among the 120 patients, 67.5% presented Ineffective Breathing Pattern. In the univariate analysis, the related factors were: group of diseases, fatigue, obesity and presence of bronchial secretion, and the defining characteristics were: changes in respiratory depth, auscultation with adventitious sounds, dyspnea, reduced vesicular murmurs, tachypnea, cough and use of the accessory musculature to breathe. The mean age of patients with was higher than those without this diagnosis. The defining characteristics reduced murmurs had high sensitivity (92.6%), specificity (97.4%), negative related value (86.4%) and positive related value (98.7%). The related factors of Ineffective Breathing Pattern were the related factors fatigue, age and group of diseases.

CONCLUSION

fatigue, age and patients with a group of diseases were related factors of Ineffective Breathing Pattern in this study. Reduced vesicular murmurs, auscultation with adventitious sounds and cough may be defining characteristics to be added in the international classification, as well as the related factors bronchial secretion and group of diseases.

Developing outcome, process and balancing measures for an emergency department longitudinal patient monitoring system using a modified Delphi

BACKGROUND

Early warning score systems have been widely recommended for use to detect clinical deterioration in patients. The Irish National Emergency Medicine Programme has developed and piloted an emergency department specific early warning score system. The objective of this study was to develop a consensus among frontline healthcare staff, quality and safety staff and health systems researchers regarding evaluation measures for an early warning score system in the Emergency Department.

METHODS

Participatory action research including a modified Delphi consensus building technique with frontline hospital staff, quality and safety staff, health systems researchers, local and national emergency medicine stakeholders was the method employed in this study. In Stage One, a workshop was held with the participatory action research team including frontline hospital staff, quality and safety staff and health systems researchers to gather suggestions regarding the evaluation measures. In Stage Two, an electronic modified-Delphi study was undertaken with a panel consisting of the workshop participants, key local and national emergency medicine stakeholders. Descriptive statistics were used to summarise the characteristics of the panellists who completed the questionnaires in each round. The mean Likert rating, standard deviation and 95% bias-corrected bootstrapped confidence interval for each variable was calculated. Bonferroni corrections were applied to take account of multiple testing. Data were analysed using Stata 14.0 SE.

RESULTS

Using the Institute for Healthcare Improvement framework, 12 process, outcome and balancing metrics for measuring the effectiveness of an ED-specific early warning score system were developed.

CONCLUSION

There are currently no published measures for evaluating the effectiveness of an ED early warning score system. It was possible in this study to develop a suite of evaluation measures using a modified Delphi consensus approach. Using the collective expertise of frontline hospital staff, quality and safety staff and health systems researchers to develop and categorise the initial set of potential measures was an innovative and unique element of this study.