A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases

An innovative method to strengthen evidence for potential drug safety signals using Electronic Health Records

Reports from spontaneous reporting systems (SRS) are hypothesis generating. Additional evidence such as more reports is required to determine whether the generated drug-event associations are in fact safety signals. However, underreporting of adverse drug reactions (ADRs) delays signal detection. Through the use of natural language processing, different sources of real-world data can be used to proactively collect additional evidence for potential safety signals. This study aims to explore the feasibility of using Electronic Health Records (EHRs) to identify additional cases based on initial indications from spontaneous ADR reports, with the goal of strengthening the evidence base for potential safety signals. For two confirmed and two potential signals generated by the SRS of the Netherlands Pharmacovigilance Centre Lareb, targeted searches in the EHR of the Leiden University Medical Centre were performed using a text-mining based tool, CTcue. The search for additional cases was done by constructing and running queries in the structured and free-text fields of the EHRs. We identified at least five additional cases for the confirmed signals and one additional case for each potential safety signal. The majority of the identified cases for the confirmed signals were documented in the EHRs before signal detection by the Dutch Medicines Evaluation Board. The identified cases for the potential signals were reported to Lareb as further evidence for signal detection. Our findings highlight the feasibility of performing targeted searches in the EHR based on an underlying hypothesis to provide further evidence for signal generation.

The State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Uses of real-world data in drug safety and effectiveness studies are often challenged by various sources of bias. We undertook a systematic search of the published literature through September 2020 to evaluate the state of use and utility of negative controls to address bias in pharmacoepidemiologic studies. Two reviewers independently evaluated study eligibility and abstracted data. Our search identified 184 eligible studies for inclusion. Cohort studies (115, 63%) and administrative data (114, 62%) were, respectively, the most common study design and data type used. Most studies used negative control outcomes (91, 50%), and for most studies the target source of bias was unmeasured confounding (93, 51%). We identified 4 utility domains of negative controls: 1) bias detection (149, 81%), 2) bias correction (16, 9%), 3) P-value calibration (8, 4%), and 4) performance assessment of different methods used in drug safety studies (31, 17%). The most popular methodologies used were the 95% confidence interval and P-value calibration. In addition, we identified 2 reference sets with structured steps to check the causality assumption of the negative control. While negative controls are powerful tools in bias detection, we found many studies lacked checking the underlying assumptions. This article is part of a Special Collection on Pharmacoepidemiology.

Early Detection of Adverse Drug Reaction Signals by Association Rule Mining Using Large-Scale Administrative Claims Data

INTRODUCTION

Adverse drug reactions (ADRs) are a leading cause of mortality worldwide and should be detected promptly to reduce health risks to patients. A data-mining approach using large-scale medical records might be a useful method for the early detection of ADRs. Many studies have analyzed medical records to detect ADRs; however, most of them have focused on a narrow range of ADRs, limiting their usefulness.

OBJECTIVE

This study aimed to identify methods for the early detection of a wide range of ADR signals.

METHODS

First, to evaluate the performance in signal detection of ADRs by data-mining, we attempted to create a gold standard based on clinical evidence. Second, association rule mining (ARM) was applied to patient symptoms and medications registered in claims data, followed by evaluating ADR signal detection performance.

RESULTS

We created a new gold standard consisting of 92 positive and 88 negative controls. In the assessment of ARM using claims data, the areas under the receiver-operating characteristic curve and the precision-recall curve were 0.80 and 0.83, respectively. If the detection criteria were defined as lift > 1, conviction > 1, and p-value < 0.05, ARM could identify 156 signals, of which 90 were true positive controls (sensitivity: 0.98, specificity: 0.25). Evaluation of the capability of ARM with short periods of data revealed that ARM could detect a greater number of positive controls than the conventional analysis method.

CONCLUSIONS

ARM of claims data may be effective in the early detection of a wide range of ADR signals.

Methods for drug safety signal detection using routinely collected observational electronic health care data: A systematic review

PURPOSE

Signal detection is a crucial step in the discovery of post-marketing adverse drug reactions. There is a growing interest in using routinely collected data to complement established spontaneous report analyses. This work aims to systematically review the methods for drug safety signal detection using routinely collected healthcare data and their performance, both in general and for specific types of drugs and outcomes.

METHODS

We conducted a systematic review following the PRISMA guidelines, and registered a protocol in PROSPERO. MEDLINE, EMBASE, PubMed, Web of Science, Scopus, and the Cochrane Library were searched until July 13, 2021.

RESULTS

The review included 101 articles, among which there were 39 methodological works, 25 performance assessment papers, and 24 observational studies. Methods included adaptations from those used with spontaneous reports, traditional epidemiological designs, methods specific to signal detection with real-world data. More recently, implementations of machine learning have been studied in the literature. Twenty-five studies evaluated method performances, 16 of them using the area under the curve (AUC) for a range of positive and negative controls as their main measure. Despite the likelihood that performance measurement could vary by drug-event pair, only 10 studies reported performance stratified by drugs and outcomes, in a heterogeneous manner. The replicability of the performance assessment results was limited due to lack of transparency in reporting and the lack of a gold standard reference set.

CONCLUSIONS

A variety of methods have been described in the literature for signal detection with routinely collected data. No method showed superior performance in all papers and across all drugs and outcomes, performance assessment and reporting were heterogeneous. However, there is limited evidence that self-controlled designs, high dimensional propensity scores, and machine learning can achieve higher performances than other methods.

Leveraging Machine Learning to Facilitate Individual Case Causality Assessment of Adverse Drug Reactions

INTRODUCTION

Causality assessment of individual case safety reports (ICSRs) is an important step in pharmacovigilance case-level review and aims to establish a position on whether a patient's exposure to a drug is causally related to the patient experiencing an untoward adverse event. There are many different approaches for case causality adjudication, including the use of expert opinions and algorithmic frameworks; however, a great deal of variability exists between assessment methods, products, therapeutic classes, individual physicians, change of process and conventions over time, and other factors.

OBJECTIVE

The objective of this study was to develop a machine learning-based model that can predict the likelihood of a causal association of an observed drug-reaction combination in an ICSR.

METHODS

In this study, we used a set of annotated solicited ICSRs (50K cases) from a company post-marketing database. These data were enriched with novel supplementary features from external and internal data sources that aim to capture facets such as temporal plausibility, scientific validity, and confoundedness that have been shown to contribute to causality adjudication. Using these features, we constructed a Bayesian network (BN) model to predict drug-event pair causality assessment. BN topology was driven by an internally developed ICSR causality decision support tool. Performance of the model was evaluated through examination of sensitivity, positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC) on an independent set of data from a temporally adjacent interval (20K cases). No external validation was performed because of a lack of publicly available ICSRs with causality assessments for drug-event pairs.

RESULTS

The model demonstrated high performance in predicting the causality assessment of drug-event pairs compared with clinical judgment using global introspection (AUC 0.924; 95% confidence interval [CI] 0.922-0.927). The sensitivity of the model was 0.900 (95% CI 0.896-0.904), and the PPV of the model was 0.778 (95% CI 0.773-0.783).

CONCLUSION

These results show that robust probabilistic modeling of ICSR causality is feasible, and the approach used in the development of the model can serve as a framework for such causality assessments, leading to improvements in safety decision making.

A reference set of clinically relevant adverse drug-drug interactions

The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms.

Measurement(s)	Adverse Event
Technology Type(s)	digital curation
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.16681933

Development of a Data-Driven Reference Standard for Adverse Drug Reaction (RS-ADR) Signal Assessment (Preprint)

Background

Pharmacovigilance using real-world data (RWD), such as multicenter electronic health records (EHRs), yields massively parallel adverse drug reaction (ADR) signals. However, proper validation of computationally detected ADR signals is not possible due to the lack of a reference standard for positive and negative associations.

Objective

This study aimed to develop a reference standard for ADR (RS-ADR) to streamline the systematic detection, assessment, and understanding of almost all drug-ADR associations suggested by RWD analyses.

Methods

We integrated well-known reference sets for drug-ADR pairs, including Side Effect Resource, Observational Medical Outcomes Partnership, and EU-ADR. We created a pharmacovigilance dictionary using controlled vocabularies and systematically annotated EHR data. Drug-ADR associations computed from MetaLAB and MetaNurse analyses of multicenter EHRs and extracted from the Food and Drug Administration Adverse Event Reporting System were integrated as “empirically determined” positive and negative reference sets by means of cross-validation between institutions.

Results

The RS-ADR consisted of 1344 drugs, 4485 ADRs, and 6,027,840 drug-ADR pairs with positive and negative consensus votes as pharmacovigilance reference sets. After the curation of the initial version of RS-ADR, novel ADR signals such as “famotidine–hepatic function abnormal” were detected and reasonably validated by RS-ADR. Although the validation of the entire reference standard is challenging, especially with this initial version, the reference standard will improve as more RWD participate in the consensus voting with advanced pharmacovigilance dictionaries and analytic algorithms. One can check if a drug-ADR pair has been reported by our web-based search interface for RS-ADRs.

Conclusions

RS-ADRs enriched with the pharmacovigilance dictionary, ADR knowledge, and real-world evidence from EHRs may streamline the systematic detection, evaluation, and causality assessment of computationally detected ADR signals.

Adverse Drug Event Prediction Using Noisy Literature-Derived Knowledge Graphs: Algorithm Development and Validation

BACKGROUND

Adverse drug events (ADEs) are unintended side effects of drugs that cause substantial clinical and economic burdens globally. Not all ADEs are discovered during clinical trials; therefore, postmarketing surveillance, called pharmacovigilance, is routinely conducted to find unknown ADEs. A wealth of information, which facilitates ADE discovery, lies in the growing body of biomedical literature. Knowledge graphs (KGs) encode information from the literature, where the vertices and the edges represent clinical concepts and their relations, respectively. The scale and unstructured form of the literature necessitates the use of natural language processing (NLP) to automatically create such KGs. Previous studies have demonstrated the utility of such literature-derived KGs in ADE prediction. Through unsupervised learning of the representations (features) of clinical concepts from the KG, which are used in machine learning models, state-of-the-art results for ADE prediction were obtained on benchmark data sets.

OBJECTIVE

Due to the use of NLP to infer literature-derived KGs, there is noise in the form of false positive (erroneous) and false negative (absent) nodes and edges. Previous representation learning methods do not account for such inaccuracies in the graph. NLP algorithms can quantify the confidence in their inference of extracted concepts and relations from the literature. Our hypothesis, which motivates this work, is that by using such confidence scores during representation learning, the learned embeddings would yield better features for ADE prediction models.

METHODS

We developed methods to use these confidence scores on two well-known representation learning methods-DeepWalk and Translating Embeddings for Modeling Multi-relational Data (TransE)-to develop their weighted versions: Weighted DeepWalk and Weighted TransE. These methods were used to learn representations from a large literature-derived KG, the Semantic MEDLINE Database, which contains more than 93 million clinical relations. They were compared with Embedding of Semantic Predications, which, to our knowledge, is the best reported representation learning method using the Semantic MEDLINE Database with state-of-the-art results for ADE prediction. Representations learned from different methods were used (separately) as features of drugs and diseases to build classification models for ADE prediction using benchmark data sets. The methods were compared rigorously over multiple cross-validation settings.

RESULTS

The weighted versions we designed were able to learn representations that yielded more accurate predictive models than the corresponding unweighted versions of both DeepWalk and TransE, as well as Embedding of Semantic Predications, in our experiments. There were performance improvements of up to 5.75% in the F1-score and 8.4% in the area under the receiver operating characteristic curve value, thus advancing the state of the art in ADE prediction from literature-derived KGs.

CONCLUSIONS

Our classification models can be used to aid pharmacovigilance teams in detecting potentially new ADEs. Our experiments demonstrate the importance of modeling inaccuracies in the inferred KGs for representation learning.

Evaluating Drug Risk Using GAN and SMOTE Based on CFDA's Spontaneous Reporting Data

Adverse drug reactions (ADRs) pose health threats to humans. Therefore, the risk re-evaluation of post-marketing drugs has become an important part of the pharmacovigilance work of various countries. In China, drugs are mainly divided into three categories, from high-risk to low-risk drugs, namely, prescription drugs (Rx), over-the-counter drugs A (OTC-A), and over-the-counter drugs B (OTC-B). Until now, there has been a lack of automated evaluation methods for the three status switch of drugs. Based on China Food and Drug Administration's (CFDA) spontaneous reporting database (CSRD), we proposed a classification model to predict risk level of drugs by using feature enhancement based on Generative Adversarial Networks (GAN) and Synthetic Minority Over-Sampling Technique (SMOTE). A total of 985,960 spontaneous reports from 2011 to 2018 were selected from CSRD in Jiangsu Province as experimental data. After data preprocessing, a class-imbalance data set was obtained, which contained 887 Rx (accounting for 84.72%), 113 OTC-A (10.79%), and 47 OTC-B (4.49%). Taking drugs as the samples, ADRs as the features, and signal detection results obtained by proportional reporting ratio (PRR) method as the feature values, we constructed the original data matrix, where the last column represents the category label of each drug. Our proposed model expands the ADR data from both the sample space and the feature space. In terms of feature space, we use feature selection (FS) to screen ADR symptoms with higher importance scores. Then, we use GAN to generate artificial data, which are added to the feature space to achieve feature enhancement. In terms of sample space, we use SMOTE technology to expand the minority samples to balance three categories of drugs and minimize the classification deviation caused by the gap in the sample size. Finally, we use random forest (RF) algorithm to classify the feature-enhanced and balanced data set. The experimental results show that the accuracy of the proposed classification model reaches 98%. Our proposed model can well evaluate drug risk levels and provide automated methods for status switch of post-marketing drugs.

The impact of regulation changes in the spontaneous reporting system for vaccines on reporting trends and signal detection in Japan

PURPOSE

Spontaneous reporting constitutes one of the most fundamental and important systems for pharmacovigilance. In Japan, important regulation changes in the vaccine spontaneous reporting were implemented between 2009 and 2013; however, no studies had yet assessed the impact of the changes. The objective of this study was to assess the impact on the reporting trends in vaccine reports and on signal detection for vaccines.

METHODS

For assessment of the impact on the reporting trends, we performed the joinpoint trend analysis and descriptively considered number of vaccine reports grouped by the timing of the regulation change. For assessment of the impact on signal detection, we performed signal detection using dataset during the pre or postperiod of the regulation changes, and compared their agreement rates, which was calculated with a reference set for vaccines, created by the Global Research in Paediatrics project.

RESULTS

We retrieved 467 635 spontaneous reports, including 12 287 vaccine reports from April 2004 to March 2019. The average number of vaccine reports per year increased from 231 reports during the preperiod to 1227 during the postperiod. The joinpoint trend analysis found two joinpoints and differentiated three trends, significant increased trend of which was observed when regulations had changed. For signal detection, the agreement rate was improved when using data during the postperiod.

CONCLUSION

We concluded that the regulation changes increased the number of vaccine reports, and could have improved signal detection performance for vaccines by accelerating accumulation of reports, while more spontaneous reports are necessary to optimize signal detection.

The 2011–2020 Trends of Data-Driven Approaches in Medical Informatics for Active Pharmacovigilance

Pharmacovigilance, the scientific discipline pertaining to drug safety, has been studied extensively and is progressing continuously. In this field, medical informatics techniques and interpretation play important roles, and appropriate approaches are required. In this study, we investigated and analyzed the trends of pharmacovigilance systems, especially the data collection, detection, assessment, and monitoring processes. We used PubMed to collect papers on pharmacovigilance published over the past 10 years, and analyzed a total of 40 significant papers to determine the characteristics of the databases and data analysis methods used to identify drug safety indicators. Through systematic reviews, we identified the difficulty of standardizing data and terminology and establishing an adverse drug reactions (ADR) evaluation system in pharmacovigilance, and their corresponding implications. We found that appropriate methods and guidelines for active pharmacovigilance using medical big data are still required and should continue to be developed.

Empirical assessment of case-based methods for identification of drugs associated with acute liver injury in the French National Healthcare System database (SNDS)

PURPOSES

Drug induced acute liver injury (ALI) is a frequent cause of liver failure. Case-based designs were empirically assessed and calibrated in the French National claims database (SNDS), aiming to identify the optimum design for drug safety alert generation associated with ALI.

METHODS

All cases of ALI were extracted from SNDS (2009-2014) using specific and sensitive definitions. Positive and negative drug controls were used to compare 196 self-controlled case series (SCCS), case-control (CC), and case-population (CP) design variants, using area under the receiver operating curve (AUC), mean square error (MSE) and coverage probability. Parameters that had major impacts on results were identified through logistic regression.

RESULTS

Using a specific ALI definition, AUCs ranged from 0.78 to 0.94, 0.64 to 0.92 and 0.48 to 0.85, for SCCS, CC and CP, respectively. MSE ranged from 0.12 to 0.40, 0.22 to 0.39 and 1.03 to 5.29, respectively. Variants adjusting for multiple drug use had higher coverage probabilities. Univariate regressions showed that high AUCs were achieved with SCCS using exposed time as the risk window. The top SCCS variant yielded an AUC = 0.93 and MSE = 0.22 and coverage = 86%, with 1/7 negative and 13/18 positive controls presenting significant estimates.

CONCLUSIONS

SCCS adjusting for multiple drugs and using exposed time as the risk window performed best in generating ALI-related drug safety alert and providing estimates of the magnitude of the risk. This approach may be useful for ad-hoc pharmacoepidemiology studies to support regulatory actions.

Ficus plants: State of the art from a phytochemical, pharmacological, and toxicological perspective

Ficus genus is typically tropical plants and is among the earliest fruit trees cultivated by humans. Ficus carica L. is the common fig, Ficus benjamina L. is the weeping fig, and Ficus pumila L. is the creeping fig. These species are commonly used in traditional medicine for a wide range of diseases and contain rich secondary metabolites that have shown diverse applications. This comprehensive review describes for Ficus genus the phytochemical compounds, traditional uses and contemporary pharmacological activities such as antioxidant, cytotoxic, antimicrobial, anti-inflammatory, antidiabetic, antiulcer, and anticonvulsant. An extended survey of the current literature (Science Direct, Scopus, PubMed) has been carried out as part of the current work. The trends in the phytochemistry, pharmacological mechanisms and activities of Ficus genus are overviewed in this manuscript: antimicrobial, antidiabetic, anti-inflammatory and analgesic activity, antiseizure and anti-Parkinson's diseases, cytotoxic and antioxidant. Health-promoting effects, recent human clinical studies, safety and adverse effects of Ficus plants also are covered. The medical potential and long-term pharmacotherapeutic use of the genus Ficus along with no serious reported adverse events, suggests that it can be considered as being safe.

Combining Social Media and FDA Adverse Event Reporting System to Detect Adverse Drug Reactions

INTRODUCTION

Adverse drug reactions (ADRs) are unintended reactions caused by a drug or combination of drugs taken by a patient. The current safety surveillance system relies on spontaneous reporting systems (SRSs) and more recently on observational health data; however, ADR detection may be delayed and lack geographic diversity. The broad scope of social media conversations, such as those on Twitter, can include health-related topics. Consequently, these data could be used to detect potentially novel ADRs with less latency. Although research regarding ADR detection using social media has made progress, findings are based on single information sources, and no study has yet integrated drug safety evidence from both an SRS and Twitter.

OBJECTIVE

The aim of this study was to combine signals from an SRS and Twitter to facilitate the detection of safety signals and compare the performance of the combined system with signals generated by individual data sources.

METHODS

We extracted potential drug-ADR posts from Twitter, used Monte Carlo expectation maximization to generate drug safety signals from both the US FDA Adverse Event Reporting System and posts from Twitter, and then integrated these signals using a Bayesian hierarchical model. The results from the integrated system and two individual sources were evaluated using a reference standard derived from drug labels. Area under the receiver operating characteristics curve (AUC) was computed to measure performance.

RESULTS

We observed a significant improvement in the AUC of the combined system when comparing it with Twitter alone, and no improvement when comparing with the SRS alone. The AUCs ranged from 0.587 to 0.637 for the combined SRS and Twitter, from 0.525 to 0.534 for Twitter alone, and from 0.612 to 0.642 for the SRS alone. The results varied because different preprocessing procedures were applied to Twitter.

CONCLUSION

The accuracy of signal detection using social media can be improved by combining signals with those from SRSs. However, the combined system cannot achieve better AUC performance than data from FAERS alone, which may indicate that Twitter data are not ready to be integrated into a purely data-driven combination system.

Empirical assessment of case-based methods for drug safety alert identification in the French National Healthcare System database (SNDS): Methodology of the ALCAPONE project

OBJECTIVES

To introduce the methodology of the ALCAPONE project.

BACKGROUND

The French National Healthcare System Database (SNDS), covering 99% of the French population, provides a potentially valuable opportunity for drug safety alert generation. ALCAPONE aimed to assess empirically in the SNDS case-based designs for alert generation related to four health outcomes of interest.

METHODS

ALCAPONE used a reference set adapted from observational medical outcomes partnership (OMOP) and Exploring and Understanding Adverse Drug Reactions (EU-ADR) project, with four outcomes-acute liver injury (ALI), myocardial infarction (MI), acute kidney injury (AKI), and upper gastrointestinal bleeding (UGIB)-and positive and negative drug controls. ALCAPONE consisted of four main phases: (1) data preparation to fit the OMOP Common Data Model and select the drug controls; (2) detection of the selected controls via three case-based designs: case-population, case-control, and self-controlled case series, including design variants (varying risk window, adjustment strategy, etc.); (3) comparison of design variant performance (area under the ROC curve, mean square error, etc.); and (4) selection of the optimal design variants and their calibration for each outcome.

RESULTS

Over 2009-2014, 5225 cases of ALI, 354 109 MI, 12 633 AKI, and 156 057 UGIB were identified using specific definitions. The number of detectable drugs ranged from 61 for MI to 25 for ALI. Design variants generated more than 50 000 points estimates. Results by outcome will be published in forthcoming papers.

CONCLUSIONS

ALCAPONE has shown the interest of the empirical assessment of pharmacoepidemiological approaches for drug safety alert generation and may encourage other researchers to do the same in other databases.

How Confident Are We about Observational Findings in Healthcare: A Benchmark Study

Healthcare professionals increasingly rely on observational healthcare data, such as administrative claims and electronic health records, to estimate the causal effects of interventions. However, limited prior studies raise concerns about the real-world performance of the statistical and epidemiological methods that are used. We present the "OHDSI Methods Benchmark" that aims to evaluate the performance of effect estimation methods on real data. The benchmark comprises a gold standard, a set of metrics, and a set of open source software tools. The gold standard is a collection of real negative controls (drug-outcome pairs where no causal effect appears to exist) and synthetic positive controls (drug-outcome pairs that augment negative controls with simulated causal effects). We apply the benchmark using four large healthcare databases to evaluate methods commonly used in practice: the new-user cohort, self-controlled cohort, case-control, case-crossover, and self-controlled case series designs. The results confirm the concerns about these methods, showing that for most methods the operating characteristics deviate considerably from nominal levels. For example, in most contexts, only half of the 95% confidence intervals we calculated contain the corresponding true effect size. We previously developed an "empirical calibration" procedure to restore these characteristics and we also evaluate this procedure. While no one method dominates, self-controlled methods such as the empirically calibrated self-controlled case series perform well across a wide range of scenarios.

Adverse Drug Reaction Risk Measures: A Comparison of Estimates from Drug Surveillance and Randomised Trials

BACKGROUND

Most drug regulatory agencies and pharmaceutical companies hold databases of spontaneous reports of suspected adverse drug reactions (ADRs). Detection systems for ADR signals have been created by specialists to analyse such reports, based on the concept of disproportionality, in order to support safety decision making. However, these measures are often misinterpreted by non-specialists in pharmacovigilance.

OBJECTIVES

Our aim was to assess agreement between estimates of risk from spontaneous reports of suspected ADRs and estimates of risks of ADRs from randomised controlled trials (RCTs).

METHODS

From 150 drugs randomly selected from the US Food and Drug Administration's Adverse Event Reporting System (FAERS), we identified drugs where FAERS provided reporting odds ratios (RORs) and corresponding systematic reviews from the Cochrane database gave (pooled) odds ratios (ORs) for the same drugs and adverse reactions. We assessed agreement between (ln) RORs and (ln) ORs using the Pearson correlation coefficient and the Bland-Altman agreement method, and performed sensitivity analyses.

RESULTS

We identified 6 drugs and 125 ADRs. Overall, there was a weak correlation (r = 0.20) between RORs (FAERS) and ORs (RCTs). However, we observed a stronger correlation (r = 0.78) between RORs and ORs for one drug (roflumilast) that received market approval relatively recently (2011).

CONCLUSIONS

Spontaneous reporting of suspected ADRs is an important tool for regulatory agencies and pharmaceutical companies in making decisions and detecting drug safety signals. Although there was moderate-to-strong agreement between ADR risk estimates from drug surveillance and RCTs for one drug, this study illustrates the current recommendations not to use disproportionality measures as valid proxies for risk estimates.

Complementing Observational Signals with Literature-Derived Distributed Representations for Post-Marketing Drug Surveillance

INTRODUCTION

As a result of the well documented limitations of data collected by spontaneous reporting systems (SRS), such as bias and under-reporting, a number of authors have evaluated the utility of other data sources for the purpose of pharmacovigilance, including the biomedical literature. Previous work has demonstrated the utility of literature-derived distributed representations (concept embeddings) with machine learning for the purpose of drug side-effect prediction. In terms of data sources, these methods are complementary, observing drug safety from two different perspectives (knowledge extracted from the literature and statistics from SRS data). However, the combined utility of these pharmacovigilance methods has yet to be evaluated.

OBJECTIVE

This research investigates the utility of directly or indirectly combining an observational signal from SRS with literature-derived distributed representations into a single feature vector or in an ensemble approach for downstream machine learning (logistic regression).

METHODS

Leveraging a recently developed representation scheme, concept embeddings were generated from relational connections extracted from the literature and composed to represent drug and associated adverse reactions, as defined by two reference standards of positive (likely causal) and negative (no causal evidence) pairs. Embeddings were presented with and without common measures of observational signal from SRS sources to logistic regressors, and performance was evaluated with the receiver operating characteristic (ROC) area under the curve (AUC) metric.

RESULTS

ROC AUC performance with these composite models improves up to ≈ 20% over SRS-based disproportionality metrics alone and exceeds the best prior results reported in the literature when models leverage both sources of information.

CONCLUSIONS

Results from this study support the hypothesis that knowledge extracted from the literature can enhance the performance of SRS-based methods (and vice versa). Across reference sets, using literature and SRS information together performed better than using either source alone, providing strong support for the complementary nature of these approaches to post-marketing drug surveillance.

Learning predictive models of drug side-effect relationships from distributed representations of literature-derived semantic predications

Objective

The aim of this work is to leverage relational information extracted from biomedical literature using a novel synthesis of unsupervised pretraining, representational composition, and supervised machine learning for drug safety monitoring.

Methods

Using ≈80 million concept-relationship-concept triples extracted from the literature using the SemRep Natural Language Processing system, distributed vector representations (embeddings) were generated for concepts as functions of their relationships utilizing two unsupervised representational approaches. Embeddings for drugs and side effects of interest from two widely used reference standards were then composed to generate embeddings of drug/side-effect pairs, which were used as input for supervised machine learning. This methodology was developed and evaluated using cross-validation strategies and compared to contemporary approaches. To qualitatively assess generalization, models trained on the Observational Medical Outcomes Partnership (OMOP) drug/side-effect reference set were evaluated against a list of ≈1100 drugs from an online database.

Results

The employed method improved performance over previous approaches. Cross-validation results advance the state of the art (AUC 0.96; F1 0.90 and AUC 0.95; F1 0.84 across the two sets), outperforming methods utilizing literature and/or spontaneous reporting system data. Examination of predictions for unseen drug/side-effect pairs indicates the ability of these methods to generalize, with over tenfold label support enrichment in the top 100 predictions versus the bottom 100 predictions.

Discussion and Conclusion

Our methods can assist the pharmacovigilance process using information from the biomedical literature. Unsupervised pretraining generates a rich relationship-based representational foundation for machine learning techniques to classify drugs in the context of a putative side effect, given known examples.

Agreement Among Different Scales for Causality Assessment in Drug-Induced Liver Injury

BACKGROUND AND OBJECTIVE

The causality assessment of drug-induced liver injury (DILI) remains a challenge and eagerly awaits the development of reliable hepatotoxicity biomarkers. None of the different available algorithms used for the causality assessment of DILI has been universally accepted as the gold standard. This study was conducted to examine the agreement among different causality assessment scales in reporting DILI.

METHODS

The World Health Organization-Uppsala Monitoring Center (WHO-UMC), Naranjo, Roussel Uclaf Causality Assessment Method (RUCAM), Maria & Victorino (M & V) and Digestive Disease Week-Japan (DDW-J) assessment scales were used to compare the causalities in all the reported cases of DILI in our adverse drug reaction (ADR) monitoring centre from January 2014 to June 2017. The probability of the causality assessment was classified as 'definite', 'probable', 'possible' and 'unlikely'. The agreement obtained among the causality assessments was analysed using the weighted kappa (κ _w) test.

RESULTS

A total of 33 cases of DILI were included in our analyses. Anti-tubercular therapy (ATT) and methotrexate were the drugs that most commonly caused DILI. The overall agreement among the different scales was poor. The best agreement was found between RUCAM and DDW-J scales (κ _w: 0.685).

CONCLUSION

There were discrepancies among the different causality scales in assessing DILI. This might be due to the different definitions of causality criteria and subjective variability during assessment. A personalised assessment scale incorporating the latest information on specific risk factors and evidence-based criteria for DILI is warranted.

Screening of anticancer drugs to detect drug-induced interstitial pneumonia using the accumulated data in the electronic medical record

Because drug-induced interstitial pneumonia (DIP) is a serious adverse drug reaction, its quantitative risk with individual medications should be taken into due consideration when selecting a medicine. We developed an algorithm to detect DIP using medical record data accumulated in a hospital. Chest computed tomography (CT) is mainly used for the diagnosis of IP, and chest X-ray reports, KL-6, and SP-D values are used to support the diagnosis. The presence of IP in the reports was assessed by a method using natural language-processing, in which IP was estimated according to the product of the likelihood ratio of characteristic keywords in each report. The sensitivity and the specificity of the method for chest CT reports were 0.92 and 0.97, while those for chest X-ray reports were 0.83 and 1, respectively. The occurrence of DIP was estimated by the patterns of presence of IP before, during, and after the administration of the target medicine. The occurrence rate of DIP in cases administered Gefitinib; Methotrexate (MTX); Tegafur, Gimeracil, and Oteracil potassium (TS-1); and Tegafur and Uracil (UTF) was 6.0%, 2.3%, 1.4%, and 0.7%, respectively. The estimated DIP cases were checked by having the medical records independently reviewed by medical doctors. By chart review, the positive predictive values of DIP against Gefitinib, MTX, TS-1, and UFT were 69.2%, 44.4%, 58.6%, and 77.8%, respectively. Although the cases extracted by this method included some that did not have DIP, this method can estimate the relative risk of DIP between medicines.

Standard-based comprehensive detection of adverse drug reaction signals from nursing statements and laboratory results in electronic health records

Objective. We propose 2 Medical Dictionary for Regulatory Activities–enabled pharmacovigilance algorithms, MetaLAB and MetaNurse, powered by a per-year meta-analysis technique and improved subject sampling strategy.

Matrials and methods. This study developed 2 novel algorithms, MetaLAB for laboratory abnormalities and MetaNurse for standard nursing statements, as significantly improved versions of our previous electronic health record (EHR)–based pharmacovigilance method, called CLEAR. Adverse drug reaction (ADR) signals from 117 laboratory abnormalities and 1357 standard nursing statements for all precautionary drugs (n  = 101) were comprehensively detected and validated against SIDER (Side Effect Resource) by MetaLAB and MetaNurse against 11 817 and 76 457 drug-ADR pairs, respectively.

Results. We demonstrate that MetaLAB (area under the curve, AUC = 0.61 ± 0.18) outperformed CLEAR (AUC = 0.55 ± 0.06) when we applied the same 470 drug-event pairs as the gold standard, as in our previous research. Receiver operating characteristic curves for 101 precautionary terms in the Medical Dictionary for Regulatory Activities Preferred Terms were obtained for MetaLAB and MetaNurse (0.69 ± 0.11; 0.62 ± 0.07), which complemented each other in terms of ADR signal coverage. Novel ADR signals discovered by MetaLAB and MetaNurse were successfully validated against spontaneous reports in the US Food and Drug Administration Adverse Event Reporting System database.

Discussion. The present study demonstrates the symbiosis of laboratory test results and nursing statements for ADR signal detection in terms of their system organ class coverage and performance profiles.

Conclusion. Systematic discovery and evaluation of the wide spectrum of ADR signals using standard-based observational electronic health record data across many institutions will affect drug development and use, as well as postmarketing surveillance and regulation.

Imputation of adverse drug reactions: Causality assessment in hospitals

BACKGROUND & OBJECTIVES

Different algorithms have been developed to standardize the causality assessment of adverse drug reactions (ADR). Although most share common characteristics, the results of the causality assessment are variable depending on the algorithm used. Therefore, using 10 different algorithms, the study aimed to compare inter-rater and multi-rater agreement for ADR causality assessment and identify the most consistent to hospitals.

METHODS

Using ten causality algorithms, four judges independently assessed the first 44 cases of ADRs reported during the first year of implementation of a risk management service in a medium complexity hospital in the state of Sao Paulo (Brazil). Owing to variations in the terminology used for causality, the equivalent imputation terms were grouped into four categories: definite, probable, possible and unlikely. Inter-rater and multi-rater agreement analysis was performed by calculating the Cohen´s and Light´s kappa coefficients, respectively.

RESULTS

None of the algorithms showed 100% reproducibility in the causal imputation. Fair inter-rater and multi-rater agreement was found. Emanuele (1984) and WHO-UMC (2010) algorithms showed a fair rate of agreement between the judges (k = 0.36).

INTERPRETATION & CONCLUSIONS

Although the ADR causality assessment algorithms were poorly reproducible, our data suggest that WHO-UMC algorithm is the most consistent for imputation in hospitals, since it allows evaluating the quality of the report. However, to improve the ability of assessing the causality using algorithms, it is necessary to include criteria for the evaluation of drug-related problems, which may be related to confounding variables that underestimate the causal association.

Accuracy of an automated knowledge base for identifying drug adverse reactions

INTRODUCTION

Drug safety researchers seek to know the degree of certainty with which a particular drug is associated with an adverse drug reaction. There are different sources of information used in pharmacovigilance to identify, evaluate, and disseminate medical product safety evidence including spontaneous reports, published peer-reviewed literature, and product labels. Automated data processing and classification using these evidence sources can greatly reduce the manual curation currently required to develop reference sets of positive and negative controls (i.e. drugs that cause adverse drug events and those that do not) to be used in drug safety research.

METHODS

In this paper we explore a method for automatically aggregating disparate sources of information together into a single repository, developing a predictive model to classify drug-adverse event relationships, and applying those predictions to a real world problem of identifying negative controls for statistical method calibration.

RESULTS

Our results showed high predictive accuracy for the models combining all available evidence, with an area under the receiver-operator curve of ⩾0.92 when tested on three manually generated lists of drugs and conditions that are known to either have or not have an association with an adverse drug event.

CONCLUSIONS

Results from a pilot implementation of the method suggests that it is feasible to develop a scalable alternative to the time-and-resource-intensive, manual curation exercise previously applied to develop reference sets of positive and negative controls to be used in drug safety research.

A time-indexed reference standard of adverse drug reactions

Undetected adverse drug reactions (ADRs) pose a major burden on the health system. Data mining methodologies designed to identify signals of novel ADRs are of deep importance for drug safety surveillance. The development and evaluation of these methodologies requires proper reference benchmarks. While progress has recently been made in developing such benchmarks, our understanding of the performance characteristics of the data mining methodologies is limited because existing benchmarks do not support prospective performance evaluations. We address this shortcoming by providing a reference standard to support prospective performance evaluations. The reference standard was systematically curated from drug labeling revisions, such as new warnings, which were issued and communicated by the US Food and Drug Administration in 2013. The reference standard includes 62 positive test cases and 75 negative controls, and covers 44 drugs and 38 events. We provide usage guidance and empirical support for the reference standard by applying it to analyze two data sources commonly mined for drug safety surveillance.

Dramatyping: a generic algorithm for detecting reasonable temporal correlations between drug administration and lab value alterations

According to the World Health Organization, one of the criteria for the standardized assessment of case causality in adverse drug reactions is the temporal relationship between the intake of a drug and the occurrence of a reaction or a laboratory test abnormality. This article presents and describes an algorithm for the detection of a reasonable temporal correlation between the administration of a drug and the alteration of a laboratory value course. The algorithm is designed to process normalized lab values and is therefore universally applicable. It has a sensitivity of 0.932 for the detection of lab value courses that show changes in temporal correlation with the administration of a drug and it has a specificity of 0.967 for the detection of lab value courses that show no changes. Therefore, the algorithm is appropriate to screen the data of electronic health records and to support human experts in revealing adverse drug reactions. A reference implementation in Python programming language is available.

Reference set for performance testing of pediatric vaccine safety signal detection methods and systems

BACKGROUND

Safety signal detection in spontaneous reporting system databases and electronic healthcare records is key to detection of previously unknown adverse events following immunization. Various statistical methods for signal detection in these different datasources have been developed, however none are geared to the pediatric population and none specifically to vaccines. A reference set comprising pediatric vaccine-adverse event pairs is required for reliable performance testing of statistical methods within and across data sources.

METHODS

The study was conducted within the context of the Global Research in Paediatrics (GRiP) project, as part of the seventh framework programme (FP7) of the European Commission. Criteria for the selection of vaccines considered in the reference set were routine and global use in the pediatric population. Adverse events were primarily selected based on importance. Outcome based systematic literature searches were performed for all identified vaccine-adverse event pairs and complemented by expert committee reports, evidence based decision support systems (e.g. Micromedex), and summaries of product characteristics. Classification into positive (PC) and negative control (NC) pairs was performed by two independent reviewers according to a pre-defined algorithm and discussed for consensus in case of disagreement.

RESULTS

We selected 13 vaccines and 14 adverse events to be included in the reference set. From a total of 182 vaccine-adverse event pairs, we classified 18 as PC, 113 as NC and 51 as unclassifiable. Most classifications (91) were based on literature review, 45 were based on expert committee reports, and for 46 vaccine-adverse event pairs, an underlying pathomechanism was not plausible classifying the association as NC.

CONCLUSION

A reference set of vaccine-adverse event pairs was developed. We propose its use for comparing signal detection methods and systems in the pediatric population.

Structured assessment for prospective identification of safety signals in electronic medical records: evaluation in the health improvement network

Background

Pharmacovigilance signal detection largely relies on individual case reports, but longitudinal health data are being explored as complementary information sources. Research to date has focused on the ability of epidemiological methods to distinguish established adverse drug reactions (ADRs) from unrelated adverse events.

Objective

The aim of this study was to evaluate a process for structured clinical and epidemiological assessment of temporally associated drugs and medical events in electronic medical records.

Methods

Pairs of drugs and medical events were selected for review on the basis of their temporal association according to a calibrated self-controlled cohort analysis in The Health Improvement Network. Six assessors trained in pharmacovigilance and/or epidemiology evaluated seven drugs each, with up to 20 medical events per drug. A pre-specified questionnaire considered aspects related to the nature of the temporal pattern, demographic features of the cohort, concomitant medicines, earlier signs and symptoms, and possible confounding by underlying disease. This informed a classification of drug–event pairs as known ADRs, meriting further evaluation, or dismissed.

Results

The number of temporally associated medical events per drug ranged from 11 to 307 (median 50) for the 42 selected drugs. Out of the 509 relevant drug–event combinations subjected to the assessment, 127 (25 %) were classified as known ADRs. Ninety-one (24 %) of the remaining pairs were classified as potential signals meriting further evaluation and 291 (76 %) were dismissed. Suggestive temporal patterns and lack of clear alternative explanations were the most common reasons that drug–event pairs were classified as meriting further evaluation. Earlier signs and symptoms and confounding by the underlying disease were the most common reasons that drug–event pairs were dismissed.

Conclusions

Exploratory analysis of electronic medical records can detect important potential safety signals. However, effective signal detection requires that statistical signal detection be combined with clinical and epidemiological review to achieve an acceptable false positive rate.

Text mining for adverse drug events: the promise, challenges, and state of the art

Text mining is the computational process of extracting meaningful information from large amounts of unstructured text. It is emerging as a tool to leverage underutilized data sources that can improve pharmacovigilance, including the objective of adverse drug event (ADE) detection and assessment. This article provides an overview of recent advances in pharmacovigilance driven by the application of text mining, and discusses several data sources-such as biomedical literature, clinical narratives, product labeling, social media, and Web search logs-that are amenable to text mining for pharmacovigilance. Given the state of the art, it appears text mining can be applied to extract useful ADE-related information from multiple textual sources. Nonetheless, further research is required to address remaining technical challenges associated with the text mining methodologies, and to conclusively determine the relative contribution of each textual source to improving pharmacovigilance.

Proton Pump Inhibitor Usage and the Risk of Myocardial Infarction in the General Population

Background and Aims

Proton pump inhibitors (PPIs) have been associated with adverse clinical outcomes amongst clopidogrel users after an acute coronary syndrome. Recent pre-clinical results suggest that this risk might extend to subjects without any prior history of cardiovascular disease. We explore this potential risk in the general population via data-mining approaches.

Methods

Using a novel approach for mining clinical data for pharmacovigilance, we queried over 16 million clinical documents on 2.9 million individuals to examine whether PPI usage was associated with cardiovascular risk in the general population.

Results

In multiple data sources, we found gastroesophageal reflux disease (GERD) patients exposed to PPIs to have a 1.16 fold increased association (95% CI 1.09–1.24) with myocardial infarction (MI). Survival analysis in a prospective cohort found a two-fold (HR = 2.00; 95% CI 1.07–3.78; P = 0.031) increase in association with cardiovascular mortality. We found that this association exists regardless of clopidogrel use. We also found that H₂ blockers, an alternate treatment for GERD, were not associated with increased cardiovascular risk; had they been in place, such pharmacovigilance algorithms could have flagged this risk as early as the year 2000.

Conclusions

Consistent with our pre-clinical findings that PPIs may adversely impact vascular function, our data-mining study supports the association of PPI exposure with risk for MI in the general population. These data provide an example of how a combination of experimental studies and data-mining approaches can be applied to prioritize drug safety signals for further investigation.

Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance

Pharmacovigilance seeks to detect and describe adverse drug reactions early. Ideally, we would like to see objective evidence that a chosen signal detection approach can be expected to be effective. The development and evaluation of evidence-based methods require benchmarks for signal detection performance, and recent years have seen unprecedented efforts to build such reference sets. Here, we argue that evaluation should be made against emerging and not established adverse drug reactions, and we present real-world examples that illustrate the relevance of this to pharmacovigilance methods development for both individual case reports and longitudinal health records. The establishment of broader reference sets of emerging safety signals must be made a top priority to achieve more effective pharmacovigilance methods development and evaluation.

Bridging islands of information to establish an integrated knowledge base of drugs and health outcomes of interest

The entire drug safety enterprise has a need to search, retrieve, evaluate, and synthesize scientific evidence more efficiently. This discovery and synthesis process would be greatly accelerated through access to a common framework that brings all relevant information sources together within a standardized structure. This presents an opportunity to establish an open-source community effort to develop a global knowledge base, one that brings together and standardizes all available information for all drugs and all health outcomes of interest (HOIs) from all electronic sources pertinent to drug safety. To make this vision a reality, we have established a workgroup within the Observational Health Data Sciences and Informatics (OHDSI, http://ohdsi.org) collaborative. The workgroup’s mission is to develop an open-source standardized knowledge base for the effects of medical products and an efficient procedure for maintaining and expanding it. The knowledge base will make it simpler for practitioners to access, retrieve, and synthesize evidence so that they can reach a rigorous and accurate assessment of causal relationships between a given drug and HOI. Development of the knowledge base will proceed with the measureable goal of supporting an efficient and thorough evidence-based assessment of the effects of 1,000 active ingredients across 100 HOIs. This non-trivial task will result in a high-quality and generally applicable drug safety knowledge base. It will also yield a reference standard of drug–HOI pairs that will enable more advanced methodological research that empirically evaluates the performance of drug safety analysis methods.

Dose-specific adverse drug reaction identification in electronic patient records: temporal data mining in an inpatient psychiatric population

Background

Data collected for medical, filing and administrative purposes in electronic patient records (EPRs) represent a rich source of individualised clinical data, which has great potential for improved detection of patients experiencing adverse drug reactions (ADRs), across all approved drugs and across all indication areas.

Objectives

The aim of this study was to take advantage of techniques for temporal data mining of EPRs in order to detect ADRs in a patient- and dose-specific manner.

Methods

We used a psychiatric hospital’s EPR system to investigate undesired drug effects. Within one workflow the method identified patient-specific adverse events (AEs) and links these to specific drugs and dosages in a temporal manner, based on integration of text mining results and structured data. The structured data contained precise information on drug identity, dosage and strength.

Results

When applying the method to the 3,394 patients in the cohort, we identified AEs linked with a drug in 2,402 patients (70.8 %). Of the 43,528 patient-specific drug substances prescribed, 14,736 (33.9 %) were linked with AEs. From these links we identified multiple ADRs (p < 0.05) and found them to occur at similar frequencies, as stated by the manufacturer and in the literature. We showed that drugs displaying similar ADR profiles share targets, and we compared submitted spontaneous AE reports with our findings. For nine of the ten most prescribed antipsychotics in the patient population, larger doses were prescribed to sedated patients than non-sedated patients; five patients exhibited a significant difference (p < 0.05). Finally, we present two cases (p < 0.05) identified by the workflow. The method identified the potentially fatal AE QT prolongation caused by methadone, and a non-described likely ADR between levomepromazine and nightmares found among the hundreds of identified novel links between drugs and AEs (p < 0.05).

Conclusions

The developed method can be used to extract dose-dependent ADR information from already collected EPR data. Large-scale AE extraction from EPRs may complement or even replace current drug safety monitoring methods in the future, reducing or eliminating manual reporting and enabling much faster ADR detection.

Electronic supplementary material

The online version of this article (doi:10.1007/s40264-014-0145-z) contains supplementary material, which is available to authorised users.

Pediatric drug safety signal detection: a new drug-event reference set for performance testing of data-mining methods and systems

BACKGROUND

Better evidence regarding drug safety in the pediatric population might be generated from existing data sources such as spontaneous reporting systems and electronic healthcare records. The Global Research in Paediatrics (GRiP)-Network of Excellence aims to develop pediatric-specific methods that can be applied to these data sources. A reference set of positive and negative drug-event associations is required.

OBJECTIVE

The aim of this study was to develop a pediatric-specific reference set of positive and negative drug-event associations.

METHODS

Considering user patterns and expert opinion, 16 drugs that are used in individuals aged 0-18 years were selected and evaluated against 16 events, regarded as important safety outcomes. A cross-table of unique drug-event pairs was created. Each pair was classified as potential positive or negative control based on information from the drug's Summary of Product Characteristics and Micromedex. If both information sources consistently listed the event as an adverse event, the combination was reviewed as potential positive control. If both did not, the combination was evaluated as potential negative control. Further evaluation was based on published literature.

RESULTS

Selected drugs include ibuprofen, flucloxacillin, domperidone, methylphenidate, montelukast, quinine, and cyproterone/ethinylestradiol. Selected events include bullous eruption, aplastic anemia, ventricular arrhythmia, sudden death, acute kidney injury, psychosis, and seizure. Altogether, 256 unique combinations were reviewed, yielding 37 positive (17 with evidence from the pediatric population and 20 with evidence from adults only) and 90 negative control pairs, with the remainder being unclassifiable.

CONCLUSION

We propose a drug-event reference set that can be used to compare different signal detection methods in the pediatric population.

Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection

BACKGROUND

Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice.

OBJECTIVES

To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using "positive" and "negative" drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action.

METHODS

Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated.

RESULTS

The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45-73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82-96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78-91 %) and accuracy (78-72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy.

CONCLUSION

Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.

Signal detection of potentially drug-induced acute liver injury in children using a multi-country healthcare database network

BACKGROUND

Data mining in spontaneous reporting databases has shown that drug-induced liver injury is infrequently reported in children.

OBJECTIVES

Our objectives were to (i) identify drugs potentially associated with acute liver injury (ALI) in children and adolescents using electronic healthcare record (EHR) data; and (ii) to evaluate the significance and novelty of these associations.

METHODS

We identified potential cases of ALI during exposure to any prescribed/dispensed drug for individuals <18 years old from the EU-ADR network, which includes seven databases from three countries, covering the years 1996-2010. Several new methods for signal detection were applied to identify all statistically significant associations between drugs and ALI. A drug was considered statistically significantly associated with ALI, using all other time as a reference category, if the 95% CI lower band of the relative risk was >1 and in the presence of at least three exposed cases of ALI. Potentially new signals were distinguished from already known associations concerning ALI (whether in adults and/or in the paediatric population) through manual review of published literature and drug product labels.

RESULTS

The study population comprised 4,838,146 individuals aged <18 years, who contributed an overall 25,575,132 person-years of follow-up. Within this population, we identified 1,015 potential cases of ALI. Overall, 20 positive drug-ALI associations were detected. The associations between ALI and domperidone, flunisolide and human insulin were considered as potentially new signals. Citalopram and cetirizine have been previously described as hepatotoxic in adults but not in children, while all remaining associations were already known in both adults and children.

CONCLUSIONS

Data mining of multiple EHR databases for signal detection confirmed known associations between ALI and several drugs, and identified some potentially new signals in children that require further investigation through formal epidemiologic studies. This study shows that EHRs may complement traditional spontaneous reporting systems for signal detection and strengthening.

Assessing liver injury associated with antimycotics: Concise literature review and clues from data mining of the FAERS database

AIM: To inform clinicians on the level of hepatotoxic risk among antimycotics in the post-marketing setting, following the marketing suspension of oral ketoconazole for drug-induced liver injury (DILI).

METHODS: The publicly available international FAERS database (2004-2011) was used to extract DILI cases (including acute liver failure events), where antimycotics with systemic use or potential systemic absorption were reported as suspect or interacting agents. The reporting pattern was analyzed by calculating the reporting odds ratio and corresponding 95%CI, a measure of disproportionality, with time-trend analysis where appropriate.

RESULTS: From 1687284 reports submitted over the 8-year period, 68115 regarded liver injury. Of these, 2.9% are related to antimycotics (1964 cases, of which 112 of acute liver failure). Eleven systemic antimycotics (including ketoconazole and the newer triazole derivatives voriconazole and posaconazole) and terbinafine (used systemically to treat onychomicosis) generated a significant disproportionality, indicating a post-marketing signal of risk.

CONCLUSION: Virtually all antimycotics with systemic action or absorption are commonly reported in clinically significant cases of DILI. Clinicians must be aware of this aspect and monitor patients in case switch is considered, especially in critical poly-treated patients under chronic treatment.

Assessing liver injury associated with antimycotics: Concise literature review and clues from data mining of the FAERS database

AIM

To inform clinicians on the level of hepatotoxic risk among antimycotics in the post-marketing setting, following the marketing suspension of oral ketoconazole for drug-induced liver injury (DILI).

METHODS

The publicly available international FAERS database (2004-2011) was used to extract DILI cases (including acute liver failure events), where antimycotics with systemic use or potential systemic absorption were reported as suspect or interacting agents. The reporting pattern was analyzed by calculating the reporting odds ratio and corresponding 95%CI, a measure of disproportionality, with time-trend analysis where appropriate.

RESULTS

From 1687284 reports submitted over the 8-year period, 68115 regarded liver injury. Of these, 2.9% are related to antimycotics (1964 cases, of which 112 of acute liver failure). Eleven systemic antimycotics (including ketoconazole and the newer triazole derivatives voriconazole and posaconazole) and terbinafine (used systemically to treat onychomicosis) generated a significant disproportionality, indicating a post-marketing signal of risk.

CONCLUSION

Virtually all antimycotics with systemic action or absorption are commonly reported in clinically significant cases of DILI. Clinicians must be aware of this aspect and monitor patients in case switch is considered, especially in critical poly-treated patients under chronic treatment.

Defining a reference set to support methodological research in drug safety

BACKGROUND

Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards.

OBJECTIVES

To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues.

RESEARCH DESIGN

Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding.

RESULTS

Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes.

CONCLUSIONS

A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.

Validation study in four health-care databases: upper gastrointestinal bleeding misclassification affects precision but not magnitude of drug-related upper gastrointestinal bleeding risk

OBJECTIVE

To evaluate the accuracy of disease codes and free text in identifying upper gastrointestinal bleeding (UGIB) from electronic health-care records (EHRs).

STUDY DESIGN AND SETTING

We conducted a validation study in four European electronic health-care record (EHR) databases such as Integrated Primary Care Information (IPCI), Health Search/CSD Patient Database (HSD), ARS, and Aarhus, in which we identified UGIB cases using free text or disease codes: (1) International Classification of Disease (ICD)-9 (HSD, ARS); (2) ICD-10 (Aarhus); and (3) International Classification of Primary Care (ICPC) (IPCI). From each database, we randomly selected and manually reviewed 200 cases to calculate positive predictive values (PPVs). We employed different case definitions to assess the effect of outcome misclassification on estimation of risk of drug-related UGIB.

RESULTS

PPV was 22% [95% confidence interval (CI): 16, 28] and 21% (95% CI: 16, 28) in IPCI for free text and ICPC codes, respectively. PPV was 91% (95% CI: 86, 95) for ICD-9 codes and 47% (95% CI: 35, 59) for free text in HSD. PPV for ICD-9 codes in ARS was 72% (95% CI: 65, 78) and 77% (95% CI: 69, 83) for ICD-10 codes (Aarhus). More specific definitions did not have significant impact on risk estimation of drug-related UGIB, except for wider CIs.

CONCLUSIONS

ICD-9-CM and ICD-10 disease codes have good PPV in identifying UGIB from EHR; less granular terminology (ICPC) may require additional strategies. Use of more specific UGIB definitions affects precision, but not magnitude, of risk estimates.