Predictive value of alert triggers for identification of developing adverse drug events. J Patient Saf. 2009;5:223-228. [PMID: 22130215 DOI: 10.1097/pts.0b013e3181bc05e5]

Automated Detection of Patient Harm: Implementation and Prospective Evaluation of a Real-Time Broad-Spectrum Surveillance Application in a Hospital With Limited Resources

OBJECTIVES

This study aimed to prospectively validate an application that automates the detection of broad categories of hospital adverse events (AEs) extracted from a basic hospital information system, and to efficiently mobilize resources to reduce the level of acquired patient harm.

METHODS

Data were collected from an internally designed software, extracting results from 14 triggers indicative of patient harm, querying clinical and administrative databases including all inpatient admissions (n = 8760) from October 2019 to June 2020. Representative samples of the triggered cases were clinically validated using chart review by a consensus expert panel. The positive predictive value (PPV) of each trigger was evaluated, and the detection sensitivity of the surveillance system was estimated relative to incidence ranges in the literature.

RESULTS

The system identified 394 AEs among 946 triggered cases, associated with 291 patients, yielding an overall PPV of 42%. Variability was observed among the trigger PPVs and among the estimated detection sensitivities across the harm categories, the highest being for the healthcare-associated infections. The median length of stay of patients with an AE showed to be significantly higher than the median for the overall patient population.

CONCLUSIONS

This application was able to identify AEs across a broad spectrum of harm categories, in a real-time manner, while reducing the use of resources required by other harm detection methods. Such a system could serve as a promising patient safety tool for AE surveillance, allowing for timely, targeted, and resource-efficient interventions, even for hospitals with limited resources.

Detection of Drug-Induced Thrombocytopenia Signals in Children Using Routine Electronic Medical Records

Background: Drug-induced thrombocytopenia (DITP) is a severe adverse reaction and a significantly under-recognized clinical problem in children. However, for post-marketing pharmacovigilance purposes, detection of DITP signals is crucial. This study aimed to develop a signal detection model for DITP using the pediatric electronic medical records (EMR) data.

Methods: This study used the electronic medical records collected at Beijing Children’s Hospital between 2009 and 2020. A two-stage modeling method was developed to detect the signal of DITP. In the first stage, we calculated the crude incidence by mining cases of thrombocytopenia to select the potential suspected drugs. In the second stage, we constructed propensity score–matched retrospective cohorts of specific screened drugs from the first stage and estimated the odds ratio (OR) and 95% confidence interval (CI) using conditional logistic regression models. The novelty of the signal was assessed by current evidence.

Results: In the study, from a total of 839 drugs, 21 drugs were initially screened as potentially inducing thrombocytopenia. In total, we identified 18 positive DITP associations. Of these, potential DITP risk of nystatin (OR: 1.75, 95% CI: 1.37–2.22) and latamoxef sodium (OR: 1.61, 95% CI: 1.38–1.88) were two new DITP signals in both children and adults. Six associations between thrombocytopenia and drugs including imipenem (OR: 1.69, 95% CI: 1.16–2.45), teicoplanin (OR: 4.75, 95% CI: 3.33–6.78), fusidic acid (OR: 2.81, 95% CI: 2.06–3.86), ceftizoxime sodium (OR: 1.83, 95% CI: 1.36–2.45), ceftazidime (OR: 2.16, 95% CI: 1.58–2.95), and cefepime (OR: 5.06, 95% CI: 3.77–6.78) were considered as new signals in children.

Conclusion: This study developed a two-stage algorithm to detect safety signals of DITP and found eighteen positive signals of DITP, including six new signals in a pediatric population. This method is a promising tool for pharmacovigilance based on EMR data.

Technical considerations for evaluating clinical prediction indices: a case study for predicting code blue events with MEWS

Objective.There have been many efforts to develop tools predictive of health deterioration in hospitalized patients, but comprehensive evaluation of their predictive ability is often lacking to guide implementation in clinical practice. In this work, we propose new techniques and metrics for evaluating the performance of predictive alert algorithms and illustrate the advantage of capturing the timeliness and the clinical burden of alerts through the example of the modified early warning score (MEWS) applied to the prediction of in-hospital code blue events.Approach. Different implementations of MEWS were calculated from available physiological parameter measurements collected from the electronic health records of ICU adult patients. The performance of MEWS was evaluated using conventional and a set of non-conventional metrics and approaches that take into account the timeliness and practicality of alarms as well as the false alarm burden.Main results. MEWS calculated using the worst-case measurement (i.e. values scoring 3 points in the MEWS definition) over 2 h intervals significantly reduced the false alarm rate by over 50% (from 0.19/h to 0.08/h) while maintaining similar sensitivity levels as MEWS calculated from raw measurements (∼80%). By considering a prediction horizon of 12 h preceding a code blue event, a significant improvement in the specificity (∼60%), the precision (∼155%), and the work-up to detection ratio (∼50%) could be achieved, at the cost of a relatively marginal decrease in sensitivity (∼10%).Significance. Performance aspects pertaining to the timeliness and burden of alarms can aid in understanding the potential utility of a predictive alarm algorithm in clinical settings.

Does Skilled Nursing Facility Trigger Tool used with Global Trigger Tool increase its value for adverse events evaluation?

OBJECTIVE

To assess the additional value in the evaluation of incidents and adverse events by adding the IHI Skilled Nursing Facility Trigger Tool (SNFTT) to the Institute for Healthcare Improvement's Global Trigger Tool (GTT) in an acute geriatric hospital.

MATERIAL AND METHODS

A one-year retrospective study reviewing 240 electronic clinical records using the general GTT, either alone or combined with SNFTT.

MAIN OUTCOME MEASURES

Number of triggers and identified adverse events (AEs), categories of severity and preventability of AEs, GTT incidence rates, and the number needed to alert (NNA).

RESULTS

One hundred and thirty-seven AEs were identified in 107 patients (57.1 AEs per 100 admissions). Of these, 127 (92.7%) occurred 3 or more days after admissions; 49.6% of the harm events were preventable. The NNA for GTT plus SNFTT was 8.6. No significant difference was found using the general GTT alone versus the general GTT plus SNFTT in terms of the main outcome measures. Eleven categories of triggers were better identified when using GTT plus SNFTT because with GTT alone they were allocated to a category of "Other": 9 from the care module (C15) and 2 from the medication module (M13).

CONCLUSIONS

The study demonstrates that adding the SNFTT to the GTT did not increase its effectiveness as regards the evaluation of AEs. However, some triggers are better described in SNFTT and now have now been added into the general GTT method in our hospital.

Trigger Tool-Based Automated Adverse Event Detection in Electronic Health Records: Systematic Review

BACKGROUND

Adverse events in health care entail substantial burdens to health care systems, institutions, and patients. Retrospective trigger tools are often manually applied to detect AEs, although automated approaches using electronic health records may offer real-time adverse event detection, allowing timely corrective interventions.

OBJECTIVE

The aim of this systematic review was to describe current study methods and challenges regarding the use of automatic trigger tool-based adverse event detection methods in electronic health records. In addition, we aimed to appraise the applied studies' designs and to synthesize estimates of adverse event prevalence and diagnostic test accuracy of automatic detection methods using manual trigger tool as a reference standard.

METHODS

PubMed, EMBASE, CINAHL, and the Cochrane Library were queried. We included observational studies, applying trigger tools in acute care settings, and excluded studies using nonhospital and outpatient settings. Eligible articles were divided into diagnostic test accuracy studies and prevalence studies. We derived the study prevalence and estimates for the positive predictive value. We assessed bias risks and applicability concerns using Quality Assessment tool for Diagnostic Accuracy Studies-2 (QUADAS-2) for diagnostic test accuracy studies and an in-house developed tool for prevalence studies.

RESULTS

A total of 11 studies met all criteria: 2 concerned diagnostic test accuracy and 9 prevalence. We judged several studies to be at high bias risks for their automated detection method, definition of outcomes, and type of statistical analyses. Across all the 11 studies, adverse event prevalence ranged from 0% to 17.9%, with a median of 0.8%. The positive predictive value of all triggers to detect adverse events ranged from 0% to 100% across studies, with a median of 40%. Some triggers had wide ranging positive predictive value values: (1) in 6 studies, hypoglycemia had a positive predictive value ranging from 15.8% to 60%; (2) in 5 studies, naloxone had a positive predictive value ranging from 20% to 91%; (3) in 4 studies, flumazenil had a positive predictive value ranging from 38.9% to 83.3%; and (4) in 4 studies, protamine had a positive predictive value ranging from 0% to 60%. We were unable to determine the adverse event prevalence, positive predictive value, preventability, and severity in 40.4%, 10.5%, 71.1%, and 68.4% of the studies, respectively. These studies did not report the overall number of records analyzed, triggers, or adverse events; or the studies did not conduct the analysis.

CONCLUSIONS

We observed broad interstudy variation in reported adverse event prevalence and positive predictive value. The lack of sufficiently described methods led to difficulties regarding interpretation. To improve quality, we see the need for a set of recommendations to endorse optimal use of research designs and adequate reporting of future adverse event detection studies.

Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review

Background

Computerized decision support systems have raised a lot of hopes and expectations in the field of order entry. Although there are numerous studies reporting positive impacts, concerns are increasingly high about alert fatigue and effective impacts of these systems. One of the root causes of fatigue alert reported is the low clinical relevance of these alerts.

Objective

The objective of this systematic review was to assess the reported positive predictive value (PPV), as a proxy to clinical relevance, of decision support systems in computerized provider order entry (CPOE).

Methods

A systematic search of the scientific literature published between February 2009 and March 2015 on CPOE, clinical decision support systems, and the predictive value associated with alert fatigue was conducted using PubMed database. Inclusion criteria were as follows: English language, full text available (free or pay for access), assessed medication, direct or indirect level of predictive value, sensitivity, or specificity. When possible with the information provided, PPV was calculated or evaluated.

Results

Additive queries on PubMed retrieved 928 candidate papers. Of these, 376 were eligible based on abstract. Finally, 26 studies qualified for a full-text review, and 17 provided enough information for the study objectives. An additional 4 papers were added from the references of the reviewed papers. The results demonstrate massive variations in PPVs ranging from 8% to 83% according to the object of the decision support, with most results between 20% and 40%. The best results were observed when patients’ characteristics, such as comorbidity or laboratory test results, were taken into account. There was also an important variation in sensitivity, ranging from 38% to 91%.

Conclusions

There is increasing reporting of alerts override in CPOE decision support. Several causes are discussed in the literature, the most important one being the clinical relevance of alerts. In this paper, we tried to assess formally the clinical relevance of alerts, using a near-strong proxy, which is the PPV of alerts, or any way to express it such as the rate of true and false positive alerts. In doing this literature review, three inferences were drawn. First, very few papers report direct or enough indirect elements that support the use or the computation of PPV, which is a gold standard for all diagnostic tools in medicine and should be systematically reported for decision support. Second, the PPV varies a lot according to the typology of decision support, so that overall rates are not useful, but must be reported by the type of alert. Finally, in general, the PPVs are below or near 50%, which can be considered as very low.

[Targeted pharmacist-led medication order review in hospital: Assessment of a selection method for drug prescriptions]

OBJECTIVES

The aim of this study was to assess a selection method for drug prescriptions developed at the hospital level that allows to target pharmacist-led medication order review for at-risk patients and drugs.

METHODS

A one-month study has been conducted on all targeted medication orders in 19 care units. Selection criteria have been identified: biological criteria, alert medications and drug interactions. Pharmacists' interventions proposed during medication order review were listed and the possible links to the selection criteria were determined.

RESULTS

A total of 1612 prescriptions were analysed and 236 pharmacists' interventions were performed (14.6 interventions per 100 prescriptions). Physicians' acceptance rate was 60.6%. The percentage of pharmacists' interventions linked to the selection criteria was 35.6%. The relevance of the biological criteria was identified, particularly the one identifying patients with creatinine clearance below 30ml/min. Six alert medications were also relevant selection criteria: dabigatran, morphine, gentamicin, methotrexate, potassium chloride and trimethoprim sulfamethoxazole. Drug interactions criteria was irrelevant.

CONCLUSIONS

This study allowed a first assessment of the selection criteria used. A largest study seems necessary to continue the analysis of this selection method for prescriptions, especially the assessment of the alert medications list, in order to refine the prescriptions targeting.

Clinical decision support for drug related events: Moving towards better prevention

Clinical decision support (CDS) systems with automated alerts integrated into electronic medical records demonstrate efficacy for detecting medication errors (ME) and adverse drug events (ADEs). Critically ill patients are at increased risk for ME, ADEs and serious negative outcomes related to these events. Capitalizing on CDS to detect ME and prevent adverse drug related events has the potential to improve patient outcomes. The key to an effective medication safety surveillance system incorporating CDS is advancing the signals for alerts by using trajectory analyses to predict clinical events, instead of waiting for these events to occur. Additionally, incorporating cutting-edge biomarkers into alert knowledge in an effort to identify the need to adjust medication therapy portending harm will advance the current state of CDS. CDS can be taken a step further to identify drug related physiological events, which are less commonly included in surveillance systems. Predictive models for adverse events that combine patient factors with laboratory values and biomarkers are being established and these models can be the foundation for individualized CDS alerts to prevent impending ADEs.

Use of a Clinical Decision Support System Alert to Prevent Supratherapeutic Vancomycin Concentrations

Background: Alerts issued by clinical decision support systems (CDSS) may be useful to identify and prevent the occurrence of acute kidney injury among patients on nephrotoxic drugs, particularly vancomycin. Objective: The purpose of this instructive study was to determine the effectiveness of using a pharmacist-run CDSS alert of early serum creatinine increases in patients receiving intravenous vancomycin to decrease the proportion of severely elevated vancomycin concentrations. Methods: This was a retrospective study of a prospectively reviewed CDSS alert that triggered in patients with an increase in serum creatinine by 25% from baseline within 24 hours. Severely elevated vancomycin concentrations were divided into a control group (before alert implementation) and a study group (after alert implementation) and considered for study inclusion. The proportion of severely elevated vancomycin concentrations (ie, >30 mg/L) were collected in the control and study groups. Results: There were 1290 and 1501 vancomycin concentrations in the control group and the study group, respectively. A total of 696 CDSS alerts triggered during the study period. The proportion of severely elevated vancomycin troughs decreased from 5.3% (n = 68, median = 36.6 mg/L, interquartile range = 33.75-43.2 mg/L) in the control group to 3.7% (n = 55, median = 34.7 mg/L, interquartile range = 31.3-39.3 mg/L) in the study group. This reflects a statistically significant decrease in the proportion of severely elevated vancomycin concentrations ( P = .04). Conclusion: Overall, this instructive analysis on a novel use of CDSS software suggests that the implementation of an alert based on early detection of serum creatinine changes led to a significant decrease in the proportion of severely elevated serum vancomycin concentrations.

A targeted method for standardized assessment of adverse drug events in surgical patients

OBJECTIVES

This study demonstrates the development, reliability and outcome of a targeted method for standardized assessment of adverse drug events (ADEs) in surgical patients.

METHODS

Initial practice evaluation of this ADE assessment method was carried out in a prospective single centre cohort study. In total, 262 electively admitted surgical patients were included. The surgical trigger tool was applied to patients' medical records by two independent reviewers, and subsequent assessment of causality, severity and preventability of ADEs was carried out by two independent expert panels consisting of a consultant surgeon and a clinical pharmacologist. The surgical trigger tool and causality assessment method were each tested on reliability in a separate group of 50 randomly selected patients using Fleiss and Cohen's kappa statistics and percentages of agreement. Comparison of this method with an existing trigger tool method for ADEs was performed.

RESULTS

Our surgical trigger tool contains 51 triggers. The inter- and intra-rater calculations showed substantial to almost perfect levels of agreement (kappa range 0.71-0.83), with a 97.8-98.5% percentage of agreement. Fair to substantial levels of agreement were calculated for causality, severity and preventability (kappa range 0.38-0.79). The percentages of inter- and intra-rater agreement were 68.9 and 70.5% for causality, 67.0 and 82.0% for severity, and both 98.4% for preventability, respectively. Compared with the existing trigger tool method for ADEs, we found an additional 363 triggers, 18 ADEs (an extra 20%) and 3 preventable ADEs in our surgical cohort.

CONCLUSIONS

This targeted trigger tool for standardized assessment of ADEs in surgical patients shows excellent agreement between reviewers. The assessment of medication-related harm had acceptable agreement. Compared with an existing ADE trigger tool method, the present method found almost 20% extra ADEs. This method can be a useful alternative to existing trigger tool methods, in particular to assess medication safety in surgical patients.

Utility of an adverse drug event trigger tool in Veterans Affairs nursing facilities

OBJECTIVE

To determine the utility (i.e., positive predictive value [PPV] and time requirement) of an adverse drug event (ADE) trigger tool in Veterans Affairs (VA) nursing facilities and to describe the most common types of potential ADEs detected with the trigger tool.

DESIGN

Retrospective chart review.

SETTING/PATIENTS

Veterans residing in three VA nursing facilities between September 29, 2010, and October 29, 2010.

MEASUREMENT

We used the Institute for Healthcare Improvement-endorsed nursing facility ADE trigger tool, modified to enhance its clinical relevance to detect potential ADEs. Electronic medical records were screened to identify residents with one or more abnormal laboratory values specified in the trigger tool.

MAIN OUTCOME MEASURES

A potential ADE was defined as the concurrent administration of medication that could cause the abnormal laboratory value. An overall PPV, or proportion of residents with an abnormal laboratory value who had a potential ADE, and average time required to complete each trigger tool assessment, were calculated.

RESULTS

Among 321 veterans, 50.5% (n = 162) had at least one abnormal laboratory value contained in the trigger tool. Ninety-nine potential ADEs involving 146 medications were detected in 65 veterans. The overall PPV of the ADE trigger tool was 40.1% (65/162), and the average time to complete resident assessments was 8.8 (standard deviation ± 5.7) minutes. The most common potential ADEs were acute kidney injury (n = 30 residents), hypokalemia (n = 18), hypoglycemia (n = 13), and hyperkalemia (n = 10).

CONCLUSIONS

The modified nursing facility trigger tool was shown to be an effective and efficient method for detecting potential ADEs.

Use of an abnormal laboratory value-drug combination alert to detect drug-induced thrombocytopenia in critically Ill patients

PURPOSE

The aim of this study was to assess the performance of a commercially available clinical decision support system (CDSS) drug-laboratory result alert in detecting drug-induced thrombocytopenia in critically ill patients.

MATERIALS AND METHODS

Adult patients admitted to the medical and cardiac intensive care unit during an 8-week period and identified by 1 of 3 signals in the CDSS, TheraDoc, were eligible. Alerts were generated when the patient had a low platelet count and was ordered a potentially causal drug. Patients were evaluated in real time for the occurrence of an adverse drug reaction using 3 causality instruments. Positive predictive values were calculated for the alert.

RESULTS

Sixty-four patients with a mean age of 54 years met the inclusion criteria, generating 350 alerts. Positive predictive values were 0.36, 0.83, and 0.40 for signals 1, 2, and 3, respectively. Overall, there were 137 adverse drug reactions identified in the 350 alerts, with heparin, vancomycin, and famotidine as the 3 most common potential causes.

CONCLUSIONS

A commercial CDSS drug-laboratory alert is effective at identifying drug-induced thrombocytopenia in the intensive care unit and may improve patient safety. Compared with previous studies, the combination alert performs better than alerts based exclusively on laboratory values and should be considered to reduce alert fatigue.

Computerized detection of adverse drug reactions in the medical intensive care unit

OBJECTIVE

Clinical event monitors are a type of active medication monitoring system that can use signals to alert clinicians to possible adverse drug reactions. The primary goal was to evaluate the positive predictive values of select signals used to automate the detection of ADRs in the medical intensive care unit.

METHOD

This is a prospective, case series of adult patients in the medical intensive care unit during a six-week period who had one of five signals presents: an elevated blood urea nitrogen, vancomycin, or quinidine concentration, or a low sodium or glucose concentration. Alerts were assessed using 3 objective published adverse drug reaction determination instruments. An event was considered an adverse drug reaction when 2 out of 3 instruments had agreement of possible, probable or definite. Positive predictive values were calculated as the proportion of alerts that occurred, divided by the number of times that alerts occurred and adverse drug reactions were confirmed.

RESULTS

145 patients were eligible for evaluation. For the 48 patients (50% male) having an alert, the mean±SD age was 62±19 years. A total of 253 alerts were generated. Positive predictive values were 1.0, 0.55, 0.38 and 0.33 for vancomycin, glucose, sodium, and blood urea nitrogen, respectively. A quinidine alert was not generated during the evaluation.

CONCLUSIONS

Computerized clinical event monitoring systems should be considered when developing methods to detect adverse drug reactions as part of intensive care unit patient safety surveillance systems, since they can automate the detection of these events using signals that have good performance characteristics by processing commonly available laboratory and medication information.