The ability of triggers to retrospectively predict potentially preventable adverse events in a sample of deceased patients

Variation in detected adverse events using trigger tools: A systematic review and meta-analysis

Background

Adverse event (AE) detection is a major patient safety priority. However, despite extensive research on AEs, reported incidence rates vary widely.

Objective

This study aimed: (1) to synthesize available evidence on AE incidence in acute care inpatient settings using Trigger Tool methodology; and (2) to explore whether study characteristics and study quality explain variations in reported AE incidence.

Design

Systematic review and meta-analysis.

Methods

To identify relevant studies, we queried PubMed, EMBASE, CINAHL, Cochrane Library and three journals in the patient safety field (last update search 25.05.2022). Eligible publications fulfilled the following criteria: adult inpatient samples; acute care hospital settings; Trigger Tool methodology; focus on specialty of internal medicine, surgery or oncology; published in English, French, German, Italian or Spanish. Systematic reviews and studies addressing adverse drug events or exclusively deceased patients were excluded. Risk of bias was assessed using an adapted version of the Quality Assessment Tool for Diagnostic Accuracy Studies 2. Our main outcome of interest was AEs per 100 admissions. We assessed nine study characteristics plus study quality as potential sources of variation using random regression models. We received no funding and did not register this review.

Results

Screening 6,685 publications yielded 54 eligible studies covering 194,470 admissions. The cumulative AE incidence was 30.0 per 100 admissions (95% CI 23.9–37.5; I² = 99.7%) and between study heterogeneity was high with a prediction interval of 5.4–164.7. Overall studies’ risk of bias and applicability-related concerns were rated as low. Eight out of nine methodological study characteristics did explain some variation of reported AE rates, such as patient age and type of hospital. Also, study quality did explain variation.

Conclusion

Estimates of AE studies using trigger tool methodology vary while explaining variation is seriously hampered by the low standards of reporting such as the timeframe of AE detection. Specific reporting guidelines for studies using retrospective medical record review methodology are necessary to strengthen the current evidence base and to help explain between study variation.

A Systematic Review of Methods for Medical Record Analysis to Detect Adverse Events in Hospitalized Patients

OBJECTIVE

In this systematic review, we evaluate 2 of the most used trigger tools according to the criteria of the World Health Organization for evaluating methods.

METHODS

We searched Embase, PubMed, and Cochrane databases for studies (2000-2017). Studies were included if medical record review (MRR) was performed with either the Global Trigger Tool or the Harvard Medical Practice Study in a hospital population. Quality assessment was performed in duplicate. Fifty studies were included, and results were reported for every criterion separately.

RESULTS

Medical record review reveals more adverse events (AEs) than any other method. However, at the same time, it detects different AEs. The costs of an AE were on average €4296. Considerable efforts have been made worldwide in health care to improve safety and to reduce errors. These have resulted in some positive effects. The literature showed that MRR is focused on several domains of quality of care and seems suitable for both small and large cohorts. Furthermore, we found a moderate to substantial agreement for the presence of a trigger and a moderate to good agreement for the presence of an AE.

CONCLUSIONS

Medical record review with a trigger tool is a reasonably well-researched method for the evaluation of the medical records for AEs. However, looking at the World Health Organization criteria, much research is still lacking or of moderate quality. Especially for the cost of detecting AEs, valuable information is missing. Moreover, knowledge of how MRR changes quality and safety of care should be evaluated.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

Rate of Preventable Mortality in Hospitalized Patients: a Systematic Review and Meta-analysis

BACKGROUND

The number of preventable inpatient deaths in the USA is commonly estimated as between 44,000 and 98,000 deaths annually. Because many inpatient deaths are believed to be preventable, mortality rates are used for quality measures and reimbursement. We aimed to estimate the proportion of inpatient deaths that are preventable.

METHODS

A systematic literature search of Medline, Embase, Web of Science, and the Cochrane Library through April 8, 2019, was conducted. We included case series of adult patients who died in the hospital and were reviewed by physicians to determine if the death was preventable. Two reviewers independently performed data extraction and study quality assessment. The proportion of preventable deaths from individual studies was pooled using a random-effects model.

RESULTS

Sixteen studies met inclusion criteria. Eight studies of consecutive or randomly selected cohorts including 12,503 deaths were pooled. The pooled rate of preventable mortality was 3.1% (95% CI 2.2-4.1%). Two studies also reported rates of preventable mortality limited to patients expected to live longer than 3 months, ranging from 0.5 to 1.0%. In the USA, these estimates correspond to approximately 22,165 preventable deaths annually and 7150 deaths for patients with greater than 3-month life expectancy.

DISCUSSION

The number of deaths due to medical error is lower than previously reported and the majority occur in patients with less than 3-month life expectancy. The vast majority of hospital deaths are due to underlying disease. Our results have implications for the use of hospital mortality rates for quality reporting and reimbursement.

STUDY REGISTRATION

PROSPERO registration number CRD42018095140.

The Harvard medical practice study trigger system performance in deceased patients

Background

To detect possible threats to quality and safety, multiple systems have been developed. One of them is retrospective chart review. A team of experts scrutinizes medical records, selected by trigger systems, to detect possible adverse events (AEs). The most important AEs and more hints for possible improvement of care appear in deceased patients. Using triggers in a sample of these patients might increase the performance and lower the burden of scrutinizing records without possible preventable AEs. The aim of this study was therefore to determine the performance of the trigger system in a sample of deceased patients and to calculate the specificity and the sensitivity of this trigger system for predicting AEs.

Methods

We performed a study in which the records of deceased patients were screened for triggers by a team of trained nurses. A sample of 100 medical records was randomly selected out of records which had been screened between 2012 and 2015 for the first time, prior to the study in 2016. For the determination of significant differences between the first and second screening, McNemar’s test of symmetry was used. Also, observed agreement, Cohen’s Kappa and prevalence-adjusted and-bias-adjusted-kappa (PABAK) statistics were calculated. This was done for the two trigger rounds on both any trigger present and for every trigger separately.

Results

The observed agreement for any given trigger was 75% with a Kappa and PABAK of 0.5. For the individual triggers, the observed agreement was on average 90%. The corresponding Kappa was on average 0.42 (range: − 0.03-0.78) and the average PABAK was 0.8 (range: 0.44–0.92). Two adverse events were found in cases without triggers previously. The recalculated specificity and sensitivity for the original population were 58 and 92% respectively.

Conclusions

For the reproducibility of triggers it seems that some perform better than others, but on average this is to our opinion suboptimal. The low specificity implies that many records are selected without AEs. This leads to a high false-positive rate making this labour-intensive record review process costly. Therefore, research for better and more expedient systems is required.

Adverse event detection by medical record review is reproducible, but the assessment of their preventability is not

Objective

To assess the reproducibility of adverse event evaluation by a medical record review committee.

Design

Cross-sectional reanalysis of medical records.

Intervention

Reviewers re-examined fifty medical records of deceased patients regarding the presence of adverse events, their potential preventability and their possible contribution to death. Also we investigated the root causes of the preventable AEs. Differences between the first and second assessment were calculated.

Results

The Kappa on the presence of an adverse event was 0.64 and 0.32 for the potential preventability. The intrarater agreement showed a Kappa of 0.61 on the adverse event presence and 0.64 for the potential preventability. Interrater agreement showed a Kappa of 0.66 for the adverse event presence and 0.03 for the potential preventability.

Conclusion

We found a fair reproducibility for the detection of adverse events, but a poor reproducibility for the potential preventability. Possibly this was caused by lack of a definition for the preventability of adverse events. We think giving feedback to professionals using the results of medical record review remains valuable, but an improvement of its reproducibility is essential. To our opinion an international consensus on what exactly constitutes preventability of adverse events and agreement on a definition is necessary. This would result in more comparable studies in this field and could then be more informative on the ideal procedure to avoid certain potentially preventable adverse events in the future.