Measuring adverse events in helicopter emergency medical services: establishing content validity

Trigger tool-based description of adverse events in helicopter emergency medical services in Qatar

INTRODUCTION

Adverse events (AEs) in helicopter emergency medical services (HEMS) remain poorly reported, despite the potential for harm to occur. The trigger tool (TT) represents a novel approach to AE detection in healthcare. The aim of this study was to retrospectively describe the frequency of AEs and their proximal causes (PCs) in Qatar HEMS.

METHODS

Using the Pittsburgh Adverse Event Tool to identify AEs in HEMS, we retrospectively analysed 804 records within an existing AE TT database (21-month period). We calculated outcome measures for triggers, AEs and harm per 100 patient encounters, plotted measures on statistical process control charts, and conducted a multivariate analysis to report harm associations.

RESULTS

We identified 883 triggers in 536 patients, with a rate of 1.1 triggers per patient encounter, where 81.2% had documentation errors (n=436). An AE and harm rate of 27.7% and 3.5%, respectively, was realised. The leading PC was actions by HEMS Crew (81.6%; n=182). The majority of harm (57.1%) stemmed from the intervention and medication triggers (n=16), where deviation from standard of care was common (37.9%; n=11). Age and diagnosis-adjusted odds were significant in the patient condition (6.50; 95% CI 1.71 to 24.67; p=0.01) and interventional (11.85; 95% CI 1.36 to 102.92; p=0.03) trigger groupings, while age and diagnosis had no effect on harm.

CONCLUSION

The TT methodology is a robust, reliable and valid means of AE detection in the HEMS domain. While an AE rate of 27.7% is high, more research is required to understand prehospital clinical decision-making and reasons for guideline deviance. Furthermore, focused quality improvement initiatives to reduce AEs and documentation errors should also be addressed in future research.

Retrospective identification of medication related adverse events in the emergency medical services through the analysis of a patient safety register

Adverse drug events encompass a wide range of potential unintended and harmful events, from adverse drug reactions to medication errors, many of which in retrospect, are considered preventable. However, the primary challenge towards reducing their burden lies in consistently identifying and monitoring these occurrences, a challenge faced across the spectrum of healthcare, including the emergency medical services. The aim of this study was to identify and describe medication related adverse events (AEs) in the out-of-hospital setting. The medication components of a dedicated patient safety register were analysed and described for the period Jan 2017–Sept 2020. Univariate descriptive analysis was used to summarize and report on basic case and patient demographics, intervention related AEs, medication related AEs, and AE severity. Multivariable logistic regression was used to assess the odds of AE severity, by AE type. A total of 3475 patient records were assessed where 161 individual medication AEs were found in 150 (4.32%), 12 of which were categorised as harmful. Failure to provide a required medication was found to be the most common error (1.67%), followed by the administration of medications outside of prescribed practice guidelines (1.18%). There was evidence to suggest a 63% increase in crude odds of any AE severity [OR 1.63 (95% CI 1.03–2.6), p = 0.035] with the medication only AEs when compared to the intervention only AEs. Prehospital medication related adverse events remain a significant threat to patient safety in this setting and warrant greater widespread attention and future identification of strategies aimed at their reduction.

The status of prehospital care delivery for COVID-19 patients in Addis Ababa, Ethiopia: The study emphasizing adverse events occurring in prehospital transport and associated factors

BACKGROUND

COVID-19 patients may require emergency medical services for emergent treatment and/or transport to a hospital for further treatment. However, it is common for the patients to experience adverse events during transport, even the shortest transport may cause life-threatening conditions. Most of the studies that have been done on prehospital care of COVID-19 patients were conducted in developed countries. Differences in population demographics and economy may limit the generalizability of available studies. So, this study was aimed at investigating the status of prehospital care delivery for COVID-19 patients in Addis Ababa focusing on adverse events that occurred during transport and associated factors.

METHODS

A total of 233 patients consecutively transported to Saint Paul's Hospital Millennium Medical College from November 6 to December 31, 2020, were included in the study. A team of physicians and nurses collected the data using a structured questionnaire. Descriptive statistics were used to summarize data, and ordinal logistic regression was carried out to assess the association between explanatory variables and the outcome variable. Results are presented using frequency, percentage, chi-square, crude and adjusted odds ratios (OR) with 95% confidence intervals.

RESULTS

The overall level of adverse events in prehospital setting was 44.2%. Having history of at least one chronic medical illness, [AOR3.2 (95%; CI; 1.11-9.53)]; distance traveled to reach destination facility, [AOR 0.11(95%; CI; 0.02-0.54)]; failure to recognize and administer oxygen to the patient in need of oxygen, [AOR 15.0(95%; CI; 4.0-55.7)]; absent or malfunctioned suctioning device, [AOR 4.0(95%; CI; 1.2-13.0)]; patients handling mishaps, [AOR 12.7(95%; CI; 2.9-56.8)] were the factors associated with adverse events in prehospital transport of COVID-19 patients.

CONCLUSIONS

There were a significant proportion of adverse events in prehospital care among COVID-19 patients. Most of the adverse events were preventable. There is an urgent need to strengthen prehospital emergency care in Ethiopia by equipping the ambulances with essential and properly functioning equipment and trained manpower. Awareness creation and training of transport staff in identifying potential hazards, at-risk patients, adequate documentation, and patient handling during transport could help to prevent or minimize adverse events in prehospital care.

Measurement and monitoring patient safety in prehospital care: a systematic review

BACKGROUND

Prehospital care is potentially hazardous with the possibility for patients to experience an adverse event. However, as compared to secondary care, little is known about how patient safety is managed in prehospital care settings.

OBJECTIVES

The objectives of this systematic review were to identify and classify the methods of measuring and monitoring patient safety that have been used in prehospital care using the five dimensions of the Measuring and Monitoring Safety (MMS) framework and use this classification to identify where there are safety 'blind spots' and make recommendations for how these deficits could be addressed.

METHODS

Searches were conducted in January 2020, with no limit on publication year, using Medline, PsycInfo, CINAHL, Web of Science and Academic Search. Reference lists of included studies and existing related reviews were also screened. English-language, peer-reviewed studies concerned with measuring and monitoring safety in prehospital care were included. Two researchers independently extracted data from studies and applied a quality appraisal tool (the Quality Assessment Tool for Studies with Diverse Designs).

RESULTS

A total of 5301 studies were screened, with 52 included in the review. A total of 73% (38/52) of the studies assessed past harm, 25% (13/52) the reliability of safety critical processes, 1.9% (1/52) sensitivity to operations, 38.5% (20/52) anticipation and preparedness and 5.8% (3/52) integration and learning. A total of 67 methods for measuring and monitoring safety were used across the included studies. Of these methods, 38.8% (26/67) were surveys, 29.9% (20/67) were patient records reviews, 14.9% (10/67) were incident reporting systems, 11.9% (8/67) were interviews or focus groups and 4.5% (3/67) were checklists.

CONCLUSIONS

There is no single method of measuring and monitoring safety in prehospital care. Arguably, most safety monitoring systems have evolved, rather than been designed. This leads to safety blind spots in which information is lacking, as well as to redundancy and duplication of effort. It is suggested that the findings from this systematic review, informed by the MMS framework, can provide a structure for critically thinking about how safety is being measured and monitored in prehospital care. This will support the design of a safety surveillance system that provides a comprehensive understanding of what is being done well, where improvements should be made and whether safety interventions have had the desired effect.

Understanding quality systems in the South African prehospital emergency medical services: a multiple exploratory case study

Introduction

In South Africa (SA), prehospital emergency care is delivered by emergency medical services (EMS) across the country. Within these services, quality systems are in their infancy, and issues regarding transparency, reliability and contextual relevance have been cited as common concerns, exacerbated by poor communication, and ineffective leadership. As a result, we undertook a study to assess the current state of quality systems in EMS in SA, so as to determine priorities for initial focus regarding their development.

Methods

A multiple exploratory case study design was used that employed the Institute for Healthcare Improvement’s 18-point Quality Program Assessment Tool as both a formative assessment and semistructured interview guide using four provincial government EMS and one national private service.

Results

Services generally scored higher for structure and planning. Measurement and improvement were found to be more dependent on utilisation and perceived mandate. There was a relatively strong focus on clinical quality assessment within the private service, whereas in the provincial systems, measures were exclusively restricted to call times with little focus on clinical care. Staff engagement and programme evaluation were generally among the lowest scores. A multitude of contextual factors were identified that affected the effectiveness of quality systems, centred around leadership, vision and mission, and quality system infrastructure and capacity, guided by the need for comprehensive yet pragmatic strategic policies and standards.

Conclusion

Understanding and accounting for these factors will be key to ensuring both successful implementation and ongoing utilisation of healthcare quality systems in emergency care. The result will not only provide a more efficient and effective service, but also positively impact patient safety and quality of care of the services delivered.

Indicators to measure prehospital care quality: a scoping review

OBJECTIVE

The purpose of this scoping review was to locate, examine and describe the literature on indicators used to measure prehospital care quality.

INTRODUCTION

The performance of ambulance services and quality of prehospital care has traditionally been measured using simple indicators, such as response time intervals, based on low-level evidence. The discipline of paramedicine has evolved significantly over the last few decades. Consequently, the validity of utilizing such measures as holistic prehospital care quality indicators (QIs) has been challenged. There is growing interest in finding new and more significant ways to measure prehospital care quality.

INCLUSION CRITERIA

This scoping review examined the concepts of prehospital care quality and QIs developed for ambulance services. This review considered primary and secondary research in any paradigm and utilizing any methods, as well as text and opinion research.

METHODS

Joanna Briggs Institute methodology for conducting scoping reviews was employed. Separate searches were conducted for two review questions; review question 1 addressed the definition of prehospital care quality and review question 2 addressed characteristics of QIs in the context of prehospital care. The following databases were searched: PubMed, CINAHL, Embase, Scopus, Cochrane Library and Web of Science. The searches were limited to publications from January 1, 2000 to the day of the search (April 16, 2017). Non-English articles were excluded. To supplement the above, searches for gray literature were performed, experts in the field of study were consulted and applicable websites were perused.

RESULTS

Review question 1: Nine articles were included. These originated mostly from England (n = 3, 33.3%) and the USA (n = 3, 33.3%). Only one study specifically aimed at defining prehospital care quality. Five articles (55.5%) described attributes specific to prehospital care quality and four (44.4%) articles considered generic healthcare quality attributes to be applicable to the prehospital context. A total of 17 attributes were identified. The most common attributes were Clinical effectiveness (n = 17, 100%), Efficiency (n = 7, 77.8%), Equitability (n = 7, 77.8%) and Safety (n = 6, 66.7%). Timeliness and Accessibility were referred to by four and three (44.4% and 33.3%) articles, respectively.Review question 2: Thirty articles were included. The predominant source of articles was research literature (n = 23; 76.7%) originating mostly from the USA (n = 13; 43.3%). The most frequently applied QI development method was a form of consensus process (n = 15; 50%). A total of 526 QIs were identified. Of these, 283 (53.8%) were categorized as Clinical and 243 (46.2%) as System/Organizational QIs. Within these categories respectively, QIs related to Out-of-hospital cardiac arrest (n = 57; 10.8%) and Time intervals (n = 75; 14.3%) contributed the most. The most commonly addressed prehospital care quality attributes were Appropriateness (n = 250, 47.5%), Clinical effectiveness (n = 174, 33.1%) and Accessibility (n = 124, 23.6%). Most QIs were process indicators (n = 386, 73.4%).

CONCLUSION

Whilst there is paucity in research aiming to specifically define prehospital care quality, the attributes of generic healthcare quality definitions appear to be accepted and applicable to the prehospital context. There is growing interest in developing prehospital care QIs. However, there is a need for validation of existing QIs and de novo development addressing broader aspects of prehospital care.

Adverse events in prehospital emergency care: a trigger tool study

Background

Prehospital emergency care has developed rapidly during the past decades. The care is given in a complex context which makes prehospital care a potential high-risk activity when it comes to patient safety. Patient safety in the prehospital setting has been only sparsely investigated. The aims of the present study were 1) To investigate the incidence of adverse events (AEs) in prehospital care and 2) To investigate the factors contributing to AEs in prehospital care.

Methods

We used a retrospective study design where 30 randomly selected prehospital medical records were screened for AEs each month in three prehospital organizations in Sweden during a period of one year. A total of 1080 prehospital medical records were included. The record review was based on the use of 11 screening criteria.

Results

The reviewers identified 46 AEs in 46 of 1080 (4.3%) prehospital medical records. Of the 46 AEs, 43 were classified as potential for harm (AE1) (4.0, 95% CI = 2.9–5.4) and three as harm identified (AE2) (0.3, 95% CI = 0.1–0.9). However, among patients with a life-threatening condition (priority 1), the risk of AE was higher (16.5%). The most common factors contributing to AEs were deviations from standard of care and missing, incomplete, or unclear documentation. The most common cause of AEs was the result of action(s) or inaction(s) by the emergency medical service (EMS) crew.

Conclusions

There were 4.3 AEs per 100 ambulance missions in Swedish prehospital care. The majority of AEs originated from deviations from standard of care and incomplete documentation. There was an increase in the risk of AE among patients who the EMS team assessed as having a life-threatening condition. Most AEs were possible to avoid.

Electronic supplementary material

The online version of this article (10.1186/s12873-019-0228-3) contains supplementary material, which is available to authorized users.

Détection d’événements indésirables par la méthode des trigger tools à partir des dossiers de régulation au sein d’un CRRA 15

Objectif : La méthode dite des trigger tools permet une analyse objective, rapide et fiable des risques portant sur l’examen de dossiers de patients ciblés afin de mettre en évidence des d’événements indésirables (EI) évitables ou porteurs de risques (EPR). L’intérêt de cette méthode n’a jamais été étudié dans le cadre d’un centre de réception et de régulation des appels 15 (CRRA 15). L’objectif était d’évaluer la pertinence de triggers choisis au sein d’un CRRA 15 afin de mettre en place un système de détection d’EI par la méthode des trigger tools.

Matériel et méthodes : Au sein du SAMU 31, au CHU de Toulouse, les dossiers de régulation (DR) étaient extraits en cas de déclenchement de triggers prédéfinis : délais de départ SMUR, délais d’arrivée, temps de prise en charge sur place, délais de décision d’envoi d’une équipe SMUR après le premier appel, patients décédés et réorientations des patients dans les 48 heures. Le critère de jugement principal était le pourcentage d’EI évitables et d’EPR, parmi les patients dont le dossier a déclenché un trigger donné, c’est-à-dire la prévalence du trigger. Les critères de jugement secondaires étaient la gravité de chaque EI évitable selon l’AHRQ severity scale et les facteurs favorisant leur survenue.

Résultats : Du 9 juillet 2011 au 13 avril 2012, 174 536 DR ont été traités, 4 404 (2,5 %) ont déclenché un trigger (soit 2 882 patients). On retrouvait 203 DR avec un EI, qui concernaient 153 patients ; 73 présentaient un EI évitable, 16 un EPR, 30 un EI non évitable. La prévalence de chaque trigger était de 17,6 % (IC 95 % : 4,8–20,4 %) pour celui « délai de départ », 6,7 % (IC 95 % : 3,7–9,7 %) pour le trigger « réorientation précoce », 6,4 % (IC 95 % : 4,6– 8,2 %) pour celui « délai de décision SMUR », 3,4 % (IC 95 % : 0,1–7,2 %) pour le trigger « temps sur place », 2,9 % (IC 95 % : 0,1–8,5 %) pour celui « délai d’arrivée » et 3 % (IC 95 % : 1,5–4,5 %) pour le trigger « décès ».

Conclusion : Les triggers qui semblaient les plus pertinents étaient donc « délai de décision SMUR », « réorientation précoce » et « décès ». La méthode des trigger tools permet une analyse rapide et efficace des DR, essentielle pour réduire le risque et améliorer la qualité de prise en charge des patients.

Quality Indicators for Evaluating Prehospital Emergency Care: A Scoping Review

Introduction Historically, the quality and performance of prehospital emergency care (PEC) has been assessed largely based on surrogate, non-clinical endpoints such as response time intervals or other crude measures of care (eg, stakeholder satisfaction). However, advances in Emergency Medical Services (EMS) systems and services world-wide have seen their scope and reach continue to expand. This has dictated that novel measures of performance be implemented to compliment this growth. Significant progress has been made in this area, largely in the form of the development of evidence-informed quality indicators (QIs) of PEC. Problem Quality indicators represent an increasingly popular component of health care quality and performance measurement. However, little is known about the development of QIs in the PEC environment. The purpose of this study was to assess the development and characteristics of PEC-specific QIs in the literature.

METHODS

A scoping review was conducted through a search of PubMed (National Center for Biotechnology Information, National Institutes of Health; Bethesda, Maryland USA); EMBase (Elsevier; Amsterdam, Netherlands); CINAHL (EBSCO Information Services; Ipswich, Massachusetts USA); Web of Science (Thomson Reuters; New York, New York USA); and the Cochrane Library (The Cochrane Collaboration; Oxford, United Kingdom). To increase the sensitivity of the literature, a search of the grey literature and review of select websites was additionally conducted. Articles were selected that proposed at least one PEC QI and whose aim was to discuss, analyze, or promote quality measurement in the PEC environment.

RESULTS

The majority of research (n=25 articles) was published within the last decade (68.0%) and largely originated within the USA (68.0%). Delphi and observational methodologies were the most commonly employed for QI development (28.0%). A total of 331 QIs were identified via the article review, with an additional 15 QIs identified via the website review. Of all, 42.8% were categorized as primarily Clinical, with Out-of-Hospital Cardiac Arrest contributing the highest number within this domain (30.4%). Of the QIs categorized as Non-Clinical (57.2%), Time-Based Intervals contributed the greatest number (28.8%). Population on Whom the Data Collection was Constructed made up the most commonly reported QI component (79.8%), followed by a Descriptive Statement (63.6%). Least reported were Timing of Data Collection (12.1%) and Timing of Reporting (12.1%). Pilot testing of the QIs was reported on 34.7% of QIs identified in the review.

CONCLUSION

Overall, there is considerable interest in the understanding and development of PEC quality measurement. However, closer attention to the details and reporting of QIs is required for research of this type to be more easily extrapolated and generalized. Howard I , Cameron P , Wallis L , Castren M , Lindstrom V . Quality indicators for evaluating prehospital emergency care: a scoping review. Prehosp Disaster Med. 2018;33(1):43-52.

Development of a trigger tool to identify adverse events and harm in Emergency Medical Services

BACKGROUND

Adverse event(AE) detection in healthcare has traditionally relied upon several methods including: patient care documentation review, mortality and morbidity review, voluntary reporting, direct observation and complaint systems. A novel sampling strategy, known as the trigger tool (TT) methodology, has been shown to provide a more robust and valid method of detection. The aim of this research was to develop and assess a TT specific to ground-based Emergency Medical Services, to identify cases with the potential risk for adverse events and harm.

METHODS

The study was conducted between March and December 2015. A literature review identified 57 potential triggers, which were grouped together by experts using an affinity process. Triggers for other areas of potential AE/harm were additionally considered for inclusion. An interim TT consisting of nine triggers underwent five iterative rounds of derivation tests of 20 random patient care records (n=100) in two emergency medical services. A final eight-item trigger list underwent a large sample (n=9836) assessment of test characteristics.

RESULTS

The final eight-item TT consisted of triggers divided amongst four categories: Clinical, Medication, Procedural and Return-Call. The TT demonstrated an AE identification rate of 41.5% (sensitivity 79.8% (95% CI, 69.9% to 87.6%); specificity 58.5% (95% CI, 52% to 64.8%)). When identifying potential risk for harm, the TT demonstrated a harm identification rate of 19.3% (sensitivity 97.1% (95% CI, 84.7% to 99.9%); specificity 53.5% (95% CI, 47.7% to 59.3%)).

DISCUSSION

The Emergency Medical Services Trigger Tool (EMSTT) may be used as a sampling strategy similar to the Global Trigger Tool, to identify and measure AE and harm over time, and monitor the success of improvement initiatives within the Emergency Medical Services setting.

Patient safety and patient assessment in pre-hospital care: a study protocol

Background

Patient safety issues in pre-hospital care are poorly investigated. The aim of the planned study is to survey patient safety problems in pre-hospital care in Sweden.

Methods/Design

The study is a retro-perspective structured medical record review based on the use of 11 screening criteria. Two instruments for structured medical record review are used: a trigger tool instrument designed for pre-hospital care and a newly development instrument designed to compare the pre-hospital assessment with the final hospital assessment. Three different ambulance organisations are participating in the study. Every month, one rater in each organisation randomly collects 30 medical records for review. With guidance from the review instrument, he/she independently reviews the record. Every month, the review team meet for a discussion of problematic reviews. The results will be analysed with descriptive statistics and logistic regression.

Discussion

The findings will make an important contribution to knowledge about patient safety issues in pre-hospital care.

Improving the governance of patient safety in emergency care: a systematic review of interventions

OBJECTIVES

To systematically review interventions that aim to improve the governance of patient safety within emergency care on effectiveness, reliability, validity and feasibility.

DESIGN

A systematic review of the literature.

METHODS

PubMed, EMBASE, Cumulative Index to Nursing and Allied Health Literature, the Cochrane Database of Systematic Reviews and PsychInfo were searched for studies published between January 1990 and July 2014. We included studies evaluating interventions relevant for higher management to oversee and manage patient safety, in prehospital emergency medical service (EMS) organisations and hospital-based emergency departments (EDs). Two reviewers independently selected candidate studies, extracted data and assessed study quality. Studies were categorised according to study quality, setting, sample, intervention characteristics and findings.

RESULTS

Of the 18 included studies, 13 (72%) were non-experimental. Nine studies (50%) reported data on the reliability and/or validity of the intervention. Eight studies (44%) reported on the feasibility of the intervention. Only 4 studies (22%) reported statistically significant effects. The use of a simulation-based training programme and well-designed incident reporting systems led to a statistically significant improvement of safety knowledge and attitudes by ED staff and an increase of incident reports within EDs, respectively.

CONCLUSIONS

Characteristics of the interventions included in this review (eg, anonymous incident reporting and validation of incident reports by an independent party) could provide useful input for the design of an effective tool to govern patient safety in EMS organisations and EDs. However, executives cannot rely on a robust set of evidence-based and feasible tools to govern patient safety within their emergency care organisation and in the chain of emergency care. Established strategies from other high-risk sectors need to be evaluated in emergency care settings, using an experimental design with valid outcome measures to strengthen the evidence base.

Simulation-based training and assessment of non-technical skills in the Norwegian Helicopter Emergency Medical Services: a cross-sectional survey

BACKGROUND

Human error and deficient non-technical skills (NTSs) among providers of ALS in helicopter emergency medical services (HEMS) is a threat to patient and operational safety. Skills can be improved through simulation-based training and assessment.

OBJECTIVE

To document the current level of simulation-based training and assessment of seven generic NTSs in crew members in the Norwegian HEMS.

METHODS

A cross-sectional survey, either electronic or paper-based, of all 207 physicians, HEMS crew members (HCMs) and pilots working in the civilian Norwegian HEMS (11 bases), between 8 May and 25 July 2012.

RESULTS

The response rate was 82% (n=193). A large proportion of each of the professional groups lacked simulation-based training and assessment of their NTSs. Compared with pilots and HCMs, physicians undergo statistically significantly less frequent simulation-based training and assessment of their NTSs. Fifty out of 82 (61%) physicians were on call for more than 72 consecutive hours on a regular basis. Of these, 79% did not have any training in coping with fatigue. In contrast, 72 out of 73 (99%) pilots and HCMs were on call for more than 3 days in a row. Of these, 54% did not have any training in coping with fatigue.

CONCLUSIONS

Our study indicates a lack of simulation-based training and assessment. Pilots and HCMs train and are assessed more frequently than physicians. All professional groups are on call for extended hours, but receive limited training in how to cope with fatigue.

A comparative assessment of adverse event classification in the out-of-hospital setting

OBJECTIVES

We sought to test reliability of two approaches to classify adverse events (AEs) associated with helicopter EMS (HEMS) transport.

METHODS

The first approach for AE classification involved flight nurses and paramedics (RN/Medics) and mid-career emergency physicians (MC-EMPs) independently reviewing 50 randomly selected HEMS medical records. The second approach involved RN/Medics and MC-EMPs meeting as a group to openly discuss 20 additional medical records and reach consensus-based AE decision. We compared all AE decisions to a reference criterion based on the decision of three senior emergency physicians (Sr-EMPs). We designed a study to detect an improvement in agreement (reliability) from fair (kappa = 0.2) to moderate (kappa = 0.5). We calculated sensitivity, specificity, percent agreement, and positive and negative predictive values (PPV/NPV).

RESULTS

For the independent reviews, the Sr-EMP group identified 26 AEs while individual clinician reviewers identified between 19 and 50 AEs. Agreement on the presence/absence of an AE between Sr-EMPs and three MC-EMPs ranged from κ = 0.20 to κ = 0.25. Agreement between Sr-EMPs and three RN/Medics ranged from κ = 0.11 to κ = 0.19. For the consensus/open-discussion approach, the Sr-EMPs identified 13 AEs, the MC-EMP group identified 18 AEs, and RN/medic group identified 36 AEs. Agreement between Sr-EMPs and MC-EMP group was (κ = 0.30 95%CI -0.12, 0.72), whereas agreement between Sr-EMPs and RN/medic group was (κ = 0.40 95%CI 0.01, 0.79). Agreement between all three groups was fair (κ = 0.33, 95%CI 0.06, 0.66). Percent agreement (58-68%) and NPV (63-76%) was moderately dissimilar between clinicians, while sensitivity (25-80%), specificity (43-97%), and PPV (48-83%) varied.

CONCLUSIONS

We identified a higher level of agreement/reliability in AE decisions utilizing a consensus-based approach for review rather than independent reviews.