Defining a reference set to support methodological research in drug safety

The State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Uses of real-world data in drug safety and effectiveness studies are often challenged by various sources of bias. We undertook a systematic search of the published literature through September 2020 to evaluate the state of use and utility of negative controls to address bias in pharmacoepidemiologic studies. Two reviewers independently evaluated study eligibility and abstracted data. Our search identified 184 eligible studies for inclusion. Cohort studies (115, 63%) and administrative data (114, 62%) were, respectively, the most common study design and data type used. Most studies used negative control outcomes (91, 50%), and for most studies the target source of bias was unmeasured confounding (93, 51%). We identified 4 utility domains of negative controls: 1) bias detection (149, 81%), 2) bias correction (16, 9%), 3) P-value calibration (8, 4%), and 4) performance assessment of different methods used in drug safety studies (31, 17%). The most popular methodologies used were the 95% confidence interval and P-value calibration. In addition, we identified 2 reference sets with structured steps to check the causality assumption of the negative control. While negative controls are powerful tools in bias detection, we found many studies lacked checking the underlying assumptions. This article is part of a Special Collection on Pharmacoepidemiology.

Evaluation of four machine learning models for signal detection

Background

Logistic regression-based signal detection algorithms have benefits over disproportionality analysis due to their ability to handle potential confounders and masking factors. Feature exploration and developing alternative machine learning algorithms can further strengthen signal detection.

Objectives

Our objective was to compare the signal detection performance of logistic regression, gradient-boosted trees, random forest and support vector machine models utilizing Food and Drug Administration adverse event reporting system data.

Design

Cross-sectional study.

Methods

The quarterly data extract files from 1 October 2017 through 31 December 2020 were downloaded. Due to an imbalanced outcome, two training sets were used: one stratified on the outcome variable and another using Synthetic Minority Oversampling Technique (SMOTE). A crude model and a model with tuned hyperparameters were developed for each algorithm. Model performance was compared against a reference set using accuracy, precision, F1 score, recall, the receiver operating characteristic area under the curve (ROCAUC), and the precision-recall curve area under the curve (PRCAUC).

Results

Models trained on the balanced training set had higher accuracy, F1 score and recall compared to models trained on the SMOTE training set. When using the balanced training set, logistic regression, gradient-boosted trees, random forest and support vector machine models obtained similar performance evaluation metrics. The gradient-boosted trees hyperparameter tuned model had the highest ROCAUC (0.646) and the random forest crude model had the highest PRCAUC (0.839) when using the balanced training set.

Conclusion

All models trained on the balanced training set performed similarly. Logistic regression models had higher accuracy, precision and recall. Logistic regression, random forest and gradient-boosted trees hyperparameter tuned models had a PRCAUC ⩾ 0.8. All models had an ROCAUC ⩾ 0.5. Including both disproportionality analysis results and additional case report information in models resulted in higher performance evaluation metrics than disproportionality analysis alone.

Using public clinical trial reports to probe non-experimental causal inference methods

BACKGROUND

Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study.

METHODS

We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo "ground truths" about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset.

RESULTS

From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline.

CONCLUSIONS

We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data.

The Coming of Age of AI/ML in Drug Discovery, Development, Clinical Testing, and Manufacturing: The FDA Perspectives

Artificial intelligence (AI) and machine learning (ML) represent significant advancements in computing, building on technologies that humanity has developed over millions of years-from the abacus to quantum computers. These tools have reached a pivotal moment in their development. In 2021 alone, the U.S. Food and Drug Administration (FDA) received over 100 product registration submissions that heavily relied on AI/ML for applications such as monitoring and improving human performance in compiling dossiers. To ensure the safe and effective use of AI/ML in drug discovery and manufacturing, the FDA and numerous other U.S. federal agencies have issued continuously updated, stringent guidelines. Intriguingly, these guidelines are often generated or updated with the aid of AI/ML tools themselves. The overarching goal is to expedite drug discovery, enhance the safety profiles of existing drugs, introduce novel treatment modalities, and improve manufacturing compliance and robustness. Recent FDA publications offer an encouraging outlook on the potential of these tools, emphasizing the need for their careful deployment. This has expanded market opportunities for retraining personnel handling these technologies and enabled innovative applications in emerging therapies such as gene editing, CRISPR-Cas9, CAR-T cells, mRNA-based treatments, and personalized medicine. In summary, the maturation of AI/ML technologies is a testament to human ingenuity. Far from being autonomous entities, these are tools created by and for humans designed to solve complex problems now and in the future. This paper aims to present the status of these technologies, along with examples of their present and future applications.

Sensitivity and specificity in signal detection with the reporting odds ratio and the information component

PURPOSE

As measures of association between an adverse drug reaction (ADR) and exposure to a drug the reporting odds ratio (ROR) and the information component (IC) can be used. We sought to test the reliability of signal detection with these.

METHODS

We simulated ADR counts as binomially distributed random numbers for different expected ADR frequencies and theoretical reporting odds ratios (RORs). We then calculated the empirical IC and the empirical ROR and their confidence intervals. The rate of signals that was detected despite a theoretical ROR of 1 represented the false positive rate, and represented the sensitivity if the ROR was >1.

RESULTS

For expected case counts below 1 the false positive rate oscillates from 0.01 to 0.1 even though 0.025 were intended. Even beyond expected case counts of 5 oscillations can cover a range of 0.018 to 0.035. The first n oscillations with the largest amplitude are eliminated if a minimum case count of n is required. To detect an ROR of 2 with a sensitivity of 0.8, a minimum of 12 expected ADRs are required. In contrast, 2 expected ADRs suffice to detect an ROR of 4.

CONCLUSION

Summaries of measures for disproportionality should include the expected number of cases in the group of interest if a signal was detected. If no signal was detected the sensitivity for the detection of a representative ROR or the minimum ROR that could be detected with probability 0.8 should be reported.

Will the future of pharmacovigilance be more automated?

INTRODUCTION

Artificial intelligence (AI) based tools offer new opportunities for pharmacovigilance (PV) activities. Nevertheless, their contribution to PV needs to be tailored to preserve and strengthen medical and pharmacological expertise in drug safety.

AREAS COVERED

This work aims to describe PV tasks in which the contribution of AI and intelligent automation (IA) tools is required, in the context of a continuous increase of spontaneous reporting cases and regulatory tasks. A narrative review with expert selection of pertinent references was performed through Medline. Two areas were covered, management of spontaneous reporting cases and signal detection.

PERSPECTIVE

The use of AI and IA tools will assist a large spectrum of PV activities, both in public and private PV systems, in particular for tasks of low added value (e.g. initial quality check, verification of essential regulatory information, search for duplicates). Testing, validating, and integrating these tools in the PV routine are the actual challenges for modern PV systems, to guarantee high-quality standards in terms of case management and signal detection.

Performance of subgrouped proportional reporting ratios in the US Food and Drug Administration (FDA) adverse event reporting system

BACKGROUND

Many signal detection algorithms give the same weight to information from all products and patients, which may result in signals being masked or false positives being flagged as potential signals. Subgrouped analysis can be used to help correct for this.

RESEARCH DESIGN AND METHODS

The publicly available US Food and Drug Administration Adverse Event Reporting System quarterly data extract files from 1 January 2015 through 30 September 2017 were utilized. A proportional reporting ratio (PRR) analysis subgrouped by either age, sex, ADE report type, seriousness of ADE, or reporter was compared to the crude PRR analysis using sensitivity, specificity, precision, and c-statistic.

RESULTS

Subgrouping by age (n = 78, 34.5% increase), sex (n = 67, 15.5% increase), and reporter (n = 64, 10.3% increase) identified more signals than the crude analysis. Subgrouping by either age or sex increased both the sensitivity and precision. Subgrouping by report type or seriousness resulted in fewer signals (n = 50, -13.8% for both). Subgrouped analyses had higher c-statistic values, with age having the highest (0.468).

CONCLUSIONS

Subgrouping by either age or sex produced more signals with higher sensitivity and precision than the crude PRR analysis. Subgrouping by these variables can unmask potentially important associations.

Cardiovascular outcomes among patients with castration-resistant prostate cancer: A comparative safety study using US administrative claims data

BACKGROUND

Cardiovascular conditions are the most prevalent comorbidity among patients with prostate cancer, regardless of treatment. Additionally, cardiovascular risk has been shown to increase following exposure to certain treatments for advanced prostate cancer. There is conflicting evidence on risk of overall and specific cardiovascular outcomes among men treated for metastatic castrate resistant prostate cancer (CRPC). We, therefore, sought to compare incidence of serious cardiovascular events among CRPC patients treated with abiraterone acetate plus predniso(lo)ne (AAP) and enzalutamide (ENZ), the two most widely used CRPC therapies.

METHODS

Using US administrative claims data, we selected CRPC patients newly exposed to either treatment after August 31, 2012, with prior androgen deprivation therapy (ADT). We assessed incidence of hospitalization for heart failure (HHF), ischemic stroke, and acute myocardial infarction (AMI) during the period 30-days after AAP or ENZ initiation to discontinuation, outcome occurrence, death, or disenrollment. We matched treatment groups on propensity-scores (PSs) to control for observed confounding to estimate the average treatment effect among the treated (AAP) using conditional Cox proportional hazards models. To account for residual bias, we calibrated our estimates against a distribution of effect estimates from 124 negative-control outcomes.

RESULTS

The HHF analysis included 2322 (45.1%) AAP initiators and 2827 (54.9%) ENZ initiators. In this analysis, the median follow-up times among AAP and ENZ initiators (after PS matching) were 144 and 122 days, respectively. The empirically calibrated hazard ratio (HR) estimate for HHF was 2.56 (95% confidence interval [CI]: 1.32, 4.94). Corresponding HRs for AMI and ischemic stroke were 1.94 (95% CI: 0.90, 4.18) and 1.25 (95% CI: 0.54, 2.85), respectively.

CONCLUSIONS

Our study sought to quantify risk of HHF, AMI and ischemic stroke among CRPC patients initiating AAP relative to ENZ within a national administrative claims database. Increased risk for HHF among AAP compared to ENZ users was observed. The difference in myocardial infarction did not attain statistical significance after controlling for residual bias, and no differences were noted in ischemic stroke between the two treatments. These findings confirm labeled warnings and precautions for AAP for HHF and contribute to the comparative real-world evidence on AAP relative to ENZ.

Early Detection of Adverse Drug Reaction Signals by Association Rule Mining Using Large-Scale Administrative Claims Data

INTRODUCTION

Adverse drug reactions (ADRs) are a leading cause of mortality worldwide and should be detected promptly to reduce health risks to patients. A data-mining approach using large-scale medical records might be a useful method for the early detection of ADRs. Many studies have analyzed medical records to detect ADRs; however, most of them have focused on a narrow range of ADRs, limiting their usefulness.

OBJECTIVE

This study aimed to identify methods for the early detection of a wide range of ADR signals.

METHODS

First, to evaluate the performance in signal detection of ADRs by data-mining, we attempted to create a gold standard based on clinical evidence. Second, association rule mining (ARM) was applied to patient symptoms and medications registered in claims data, followed by evaluating ADR signal detection performance.

RESULTS

We created a new gold standard consisting of 92 positive and 88 negative controls. In the assessment of ARM using claims data, the areas under the receiver-operating characteristic curve and the precision-recall curve were 0.80 and 0.83, respectively. If the detection criteria were defined as lift > 1, conviction > 1, and p-value < 0.05, ARM could identify 156 signals, of which 90 were true positive controls (sensitivity: 0.98, specificity: 0.25). Evaluation of the capability of ARM with short periods of data revealed that ARM could detect a greater number of positive controls than the conventional analysis method.

CONCLUSIONS

ARM of claims data may be effective in the early detection of a wide range of ADR signals.

Using a data-driven approach for the development and evaluation of phenotype algorithms for systemic lupus erythematosus

BACKGROUND

Systemic lupus erythematosus (SLE) is a chronic autoimmune disease of unknown origin. The objective of this research was to develop phenotype algorithms for SLE suitable for use in epidemiological studies using empirical evidence from observational databases.

METHODS

We used a process for empirically determining and evaluating phenotype algorithms for health conditions to be analyzed in observational research. The process started with a literature search to discover prior algorithms used for SLE. We then used a set of Observational Health Data Sciences and Informatics (OHDSI) open-source tools to refine and validate the algorithms. These included tools to discover codes for SLE that may have been missed in prior studies and to determine possible low specificity and index date misclassification in algorithms for correction.

RESULTS

We developed four algorithms using our process: two algorithms for prevalent SLE and two for incident SLE. The algorithms for both incident and prevalent cases are comprised of a more specific version and a more sensitive version. Each of the algorithms corrects for possible index date misclassification. After validation, we found the highest positive predictive value estimate for the prevalent, specific algorithm (89%). The highest sensitivity estimate was found for the sensitive, prevalent algorithm (77%).

CONCLUSION

We developed phenotype algorithms for SLE using a data-driven approach. The four final algorithms may be used directly in observational studies. The validation of these algorithms provides researchers an added measure of confidence that the algorithms are selecting subjects correctly and allows for the application of quantitative bias analysis.

Leveraging Machine Learning to Facilitate Individual Case Causality Assessment of Adverse Drug Reactions

INTRODUCTION

Causality assessment of individual case safety reports (ICSRs) is an important step in pharmacovigilance case-level review and aims to establish a position on whether a patient's exposure to a drug is causally related to the patient experiencing an untoward adverse event. There are many different approaches for case causality adjudication, including the use of expert opinions and algorithmic frameworks; however, a great deal of variability exists between assessment methods, products, therapeutic classes, individual physicians, change of process and conventions over time, and other factors.

OBJECTIVE

The objective of this study was to develop a machine learning-based model that can predict the likelihood of a causal association of an observed drug-reaction combination in an ICSR.

METHODS

In this study, we used a set of annotated solicited ICSRs (50K cases) from a company post-marketing database. These data were enriched with novel supplementary features from external and internal data sources that aim to capture facets such as temporal plausibility, scientific validity, and confoundedness that have been shown to contribute to causality adjudication. Using these features, we constructed a Bayesian network (BN) model to predict drug-event pair causality assessment. BN topology was driven by an internally developed ICSR causality decision support tool. Performance of the model was evaluated through examination of sensitivity, positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC) on an independent set of data from a temporally adjacent interval (20K cases). No external validation was performed because of a lack of publicly available ICSRs with causality assessments for drug-event pairs.

RESULTS

The model demonstrated high performance in predicting the causality assessment of drug-event pairs compared with clinical judgment using global introspection (AUC 0.924; 95% confidence interval [CI] 0.922-0.927). The sensitivity of the model was 0.900 (95% CI 0.896-0.904), and the PPV of the model was 0.778 (95% CI 0.773-0.783).

CONCLUSION

These results show that robust probabilistic modeling of ICSR causality is feasible, and the approach used in the development of the model can serve as a framework for such causality assessments, leading to improvements in safety decision making.

A reference set of clinically relevant adverse drug-drug interactions

The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms.

Measurement(s)	Adverse Event
Technology Type(s)	digital curation
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.16681933

Combining a Pharmacological Network Model with a Bayesian Signal Detection Algorithm to Improve the Detection of Adverse Drug Events

Introduction: Improving adverse drug event (ADE) detection is important for post-marketing drug safety surveillance. Existing statistical approaches can be further optimized owing to their high efficiency and low cost.

Objective: The objective of this study was to evaluate the proposed approach for use in pharmacovigilance, the early detection of potential ADEs, and the improvement of drug safety.

Methods: We developed a novel integrated approach, the Bayesian signal detection algorithm, based on the pharmacological network model (IC_PNM) using the FDA Adverse Event Reporting System (FAERS) data published from 2004 to 2009 and from 2014 to 2019Q2, PubChem, and DrugBank database. First, we used a pharmacological network model to generate the probabilities for drug-ADE associations, which comprised the proper prior information component (IC). We then defined the probability of the propensity score adjustment based on a logistic regression model to control for the confounding bias. Finally, we chose the Side Effect Resource (SIDER) and the Observational Medical Outcomes Partnership (OMOP) data to evaluate the detection performance and robustness of the IC_PNM compared with the statistical approaches [disproportionality analysis (DPA)] by using the area under the receiver operator characteristics curve (AUC) and Youden’s index.

Results: Of the statistical approaches implemented, the IC_PNM showed the best performance (AUC, 0.8291; Youden’s index, 0.5836). Meanwhile, the AUCs of the IC, EBGM, ROR, and PRR were 0.7343, 0.7231, 0.6828, and 0.6721, respectively.

Conclusion: The proposed IC_PNM combined the strengths of the pharmacological network model and the Bayesian signal detection algorithm and performed better in detecting true drug-ADE associations. It also detected newer ADE signals than a DPA and may be complementary to the existing statistical approaches.

Development of a Data-Driven Reference Standard for Adverse Drug Reaction (RS-ADR) Signal Assessment (Preprint)

Background

Pharmacovigilance using real-world data (RWD), such as multicenter electronic health records (EHRs), yields massively parallel adverse drug reaction (ADR) signals. However, proper validation of computationally detected ADR signals is not possible due to the lack of a reference standard for positive and negative associations.

Objective

This study aimed to develop a reference standard for ADR (RS-ADR) to streamline the systematic detection, assessment, and understanding of almost all drug-ADR associations suggested by RWD analyses.

Methods

We integrated well-known reference sets for drug-ADR pairs, including Side Effect Resource, Observational Medical Outcomes Partnership, and EU-ADR. We created a pharmacovigilance dictionary using controlled vocabularies and systematically annotated EHR data. Drug-ADR associations computed from MetaLAB and MetaNurse analyses of multicenter EHRs and extracted from the Food and Drug Administration Adverse Event Reporting System were integrated as “empirically determined” positive and negative reference sets by means of cross-validation between institutions.

Results

The RS-ADR consisted of 1344 drugs, 4485 ADRs, and 6,027,840 drug-ADR pairs with positive and negative consensus votes as pharmacovigilance reference sets. After the curation of the initial version of RS-ADR, novel ADR signals such as “famotidine–hepatic function abnormal” were detected and reasonably validated by RS-ADR. Although the validation of the entire reference standard is challenging, especially with this initial version, the reference standard will improve as more RWD participate in the consensus voting with advanced pharmacovigilance dictionaries and analytic algorithms. One can check if a drug-ADR pair has been reported by our web-based search interface for RS-ADRs.

Conclusions

RS-ADRs enriched with the pharmacovigilance dictionary, ADR knowledge, and real-world evidence from EHRs may streamline the systematic detection, evaluation, and causality assessment of computationally detected ADR signals.

Adverse Drug Event Prediction Using Noisy Literature-Derived Knowledge Graphs: Algorithm Development and Validation

BACKGROUND

Adverse drug events (ADEs) are unintended side effects of drugs that cause substantial clinical and economic burdens globally. Not all ADEs are discovered during clinical trials; therefore, postmarketing surveillance, called pharmacovigilance, is routinely conducted to find unknown ADEs. A wealth of information, which facilitates ADE discovery, lies in the growing body of biomedical literature. Knowledge graphs (KGs) encode information from the literature, where the vertices and the edges represent clinical concepts and their relations, respectively. The scale and unstructured form of the literature necessitates the use of natural language processing (NLP) to automatically create such KGs. Previous studies have demonstrated the utility of such literature-derived KGs in ADE prediction. Through unsupervised learning of the representations (features) of clinical concepts from the KG, which are used in machine learning models, state-of-the-art results for ADE prediction were obtained on benchmark data sets.

OBJECTIVE

Due to the use of NLP to infer literature-derived KGs, there is noise in the form of false positive (erroneous) and false negative (absent) nodes and edges. Previous representation learning methods do not account for such inaccuracies in the graph. NLP algorithms can quantify the confidence in their inference of extracted concepts and relations from the literature. Our hypothesis, which motivates this work, is that by using such confidence scores during representation learning, the learned embeddings would yield better features for ADE prediction models.

METHODS

We developed methods to use these confidence scores on two well-known representation learning methods-DeepWalk and Translating Embeddings for Modeling Multi-relational Data (TransE)-to develop their weighted versions: Weighted DeepWalk and Weighted TransE. These methods were used to learn representations from a large literature-derived KG, the Semantic MEDLINE Database, which contains more than 93 million clinical relations. They were compared with Embedding of Semantic Predications, which, to our knowledge, is the best reported representation learning method using the Semantic MEDLINE Database with state-of-the-art results for ADE prediction. Representations learned from different methods were used (separately) as features of drugs and diseases to build classification models for ADE prediction using benchmark data sets. The methods were compared rigorously over multiple cross-validation settings.

RESULTS

The weighted versions we designed were able to learn representations that yielded more accurate predictive models than the corresponding unweighted versions of both DeepWalk and TransE, as well as Embedding of Semantic Predications, in our experiments. There were performance improvements of up to 5.75% in the F1-score and 8.4% in the area under the receiver operating characteristic curve value, thus advancing the state of the art in ADE prediction from literature-derived KGs.

CONCLUSIONS

Our classification models can be used to aid pharmacovigilance teams in detecting potentially new ADEs. Our experiments demonstrate the importance of modeling inaccuracies in the inferred KGs for representation learning.

Augmenting aer2vec: Enriching distributed representations of adverse event report data with orthographic and lexical information

Adverse Drug Events (ADEs) are prevalent, costly, and sometimes preventable. Post-marketing drug surveillance aims to monitor ADEs that occur after a drug is released to market. Reports of such ADEs are aggregated by reporting systems, such as the Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS). In this paper, we consider the topic of how best to represent data derived from reports in FAERS for the purpose of detecting post-marketing surveillance signals, in order to inform regulatory decision making. In our previous work, we developed aer2vec, a method for deriving distributed representations (concept embeddings) of drugs and side effects from ADE reports, establishing the utility of distributional information for pharmacovigilance signal detection. In this paper, we advance this line of research further by evaluating the utility of encoding orthographic and lexical information. We do so by adapting two Natural Language Processing methods, subword embedding and vector retrofitting, which were developed to encode such information into word embeddings. Models were compared for their ability to distinguish between positive and negative examples in a set of manually curated drug/ADE relationships, with both aer2vec enhancements offering advantages in performances over baseline models, and best performance obtained when retrofitting and subword embeddings were applied in concert. In addition, this work demonstrates that models leveraging distributed representations do not require extensive manual preprocessing to perform well on this pharmacovigilance signal detection task, and may even benefit from information that would otherwise be lost during the normalization and standardization process.

Leveraging the Variability of Pharmacovigilance Disproportionality Analyses to Improve Signal Detection Performances

Background: A plethora of methods and models of disproportionality analyses for safety surveillance have been developed to date without consensus nor a gold standard, leading to methodological heterogeneity and substantial variability in results. We hypothesized that this variability is inversely correlated to the robustness of a signal of disproportionate reporting (SDR) and could be used to improve signal detection performances.

Methods: We used a validated reference set containing 399 true and false drug-event pairs and performed, with a frequentist and a Bayesian disproportionality method, seven types of analyses (model) for which the results were very unlikely to be related to actual differences in absolute risks of ADR. We calculated sensitivity, specificity and plotted ROC curves for each model. We then evaluated the predictive capacities of all models and assessed the impact of combining such models with the number of positive SDR for a given drug-event pair through binomial regression models.

Results: We found considerable variability in disproportionality analysis results, both positive and negative SDR could be generated for 60% of all drug-event pairs depending on the model used whatever their truthfulness. Furthermore, using the number of positive SDR for a given drug-event pair largely improved the signal detection performances of all models.

Conclusion: We therefore advocate for the pre-registration of protocols and the presentation of a set of secondary and sensitivity analyses instead of a unique result to avoid selective outcome reporting and because variability in the results may reflect the likelihood of a signal being a true adverse drug reaction.

Using computable knowledge mined from the literature to elucidate confounders for EHR-based pharmacovigilance

INTRODUCTION

Drug safety research asks causal questions but relies on observational data. Confounding bias threatens the reliability of studies using such data. The successful control of confounding requires knowledge of variables called confounders affecting both the exposure and outcome of interest. However, causal knowledge of dynamic biological systems is complex and challenging. Fortunately, computable knowledge mined from the literature may hold clues about confounders. In this paper, we tested the hypothesis that incorporating literature-derived confounders can improve causal inference from observational data.

METHODS

We introduce two methods (semantic vector-based and string-based confounder search) that query literature-derived information for confounder candidates to control, using SemMedDB, a database of computable knowledge mined from the biomedical literature. These methods search SemMedDB for confounders by applying semantic constraint search for indications treated by the drug (exposure) and that are also known to cause the adverse event (outcome). We then include the literature-derived confounder candidates in statistical and causal models derived from free-text clinical notes. For evaluation, we use a reference dataset widely used in drug safety containing labeled pairwise relationships between drugs and adverse events and attempt to rediscover these relationships from a corpus of 2.2 M NLP-processed free-text clinical notes. We employ standard adjustment and causal inference procedures to predict and estimate causal effects by informing the models with varying numbers of literature-derived confounders and instantiating the exposure, outcome, and confounder variables in the models with dichotomous EHR-derived data. Finally, we compare the results from applying these procedures with naive measures of association (χ2 and reporting odds ratio) and with each other.

RESULTS AND CONCLUSIONS

We found semantic vector-based search to be superior to string-based search at reducing confounding bias. However, the effect of including more rather than fewer literature-derived confounders was inconclusive. We recommend using targeted learning estimation methods that can address treatment-confounder feedback, where confounders also behave as intermediate variables, and engaging subject-matter experts to adjudicate the handling of problematic covariates.

Empirical assessment of case-based methods for identification of drugs associated with acute liver injury in the French National Healthcare System database (SNDS)

PURPOSES

Drug induced acute liver injury (ALI) is a frequent cause of liver failure. Case-based designs were empirically assessed and calibrated in the French National claims database (SNDS), aiming to identify the optimum design for drug safety alert generation associated with ALI.

METHODS

All cases of ALI were extracted from SNDS (2009-2014) using specific and sensitive definitions. Positive and negative drug controls were used to compare 196 self-controlled case series (SCCS), case-control (CC), and case-population (CP) design variants, using area under the receiver operating curve (AUC), mean square error (MSE) and coverage probability. Parameters that had major impacts on results were identified through logistic regression.

RESULTS

Using a specific ALI definition, AUCs ranged from 0.78 to 0.94, 0.64 to 0.92 and 0.48 to 0.85, for SCCS, CC and CP, respectively. MSE ranged from 0.12 to 0.40, 0.22 to 0.39 and 1.03 to 5.29, respectively. Variants adjusting for multiple drug use had higher coverage probabilities. Univariate regressions showed that high AUCs were achieved with SCCS using exposed time as the risk window. The top SCCS variant yielded an AUC = 0.93 and MSE = 0.22 and coverage = 86%, with 1/7 negative and 13/18 positive controls presenting significant estimates.

CONCLUSIONS

SCCS adjusting for multiple drugs and using exposed time as the risk window performed best in generating ALI-related drug safety alert and providing estimates of the magnitude of the risk. This approach may be useful for ad-hoc pharmacoepidemiology studies to support regulatory actions.

Combining Social Media and FDA Adverse Event Reporting System to Detect Adverse Drug Reactions

INTRODUCTION

Adverse drug reactions (ADRs) are unintended reactions caused by a drug or combination of drugs taken by a patient. The current safety surveillance system relies on spontaneous reporting systems (SRSs) and more recently on observational health data; however, ADR detection may be delayed and lack geographic diversity. The broad scope of social media conversations, such as those on Twitter, can include health-related topics. Consequently, these data could be used to detect potentially novel ADRs with less latency. Although research regarding ADR detection using social media has made progress, findings are based on single information sources, and no study has yet integrated drug safety evidence from both an SRS and Twitter.

OBJECTIVE

The aim of this study was to combine signals from an SRS and Twitter to facilitate the detection of safety signals and compare the performance of the combined system with signals generated by individual data sources.

METHODS

We extracted potential drug-ADR posts from Twitter, used Monte Carlo expectation maximization to generate drug safety signals from both the US FDA Adverse Event Reporting System and posts from Twitter, and then integrated these signals using a Bayesian hierarchical model. The results from the integrated system and two individual sources were evaluated using a reference standard derived from drug labels. Area under the receiver operating characteristics curve (AUC) was computed to measure performance.

RESULTS

We observed a significant improvement in the AUC of the combined system when comparing it with Twitter alone, and no improvement when comparing with the SRS alone. The AUCs ranged from 0.587 to 0.637 for the combined SRS and Twitter, from 0.525 to 0.534 for Twitter alone, and from 0.612 to 0.642 for the SRS alone. The results varied because different preprocessing procedures were applied to Twitter.

CONCLUSION

The accuracy of signal detection using social media can be improved by combining signals with those from SRSs. However, the combined system cannot achieve better AUC performance than data from FAERS alone, which may indicate that Twitter data are not ready to be integrated into a purely data-driven combination system.

Acute pancreatitis risk in type 2 diabetes patients treated with canagliflozin versus other antihyperglycemic agents: an observational claims database study

Objective: Observational evidence suggests that patients with type 2 diabetes mellitus (T2DM) are at increased risk for acute pancreatitis (AP) versus those without T2DM. A small number of AP events were reported in clinical trials of the sodium glucose co-transporter 2 inhibitor canagliflozin, though no imbalances were observed between treatment groups. This observational study evaluated risk of AP among new users of canagliflozin compared with new users of six classes of other antihyperglycemic agents (AHAs).Methods: Three US claims databases were analyzed based on a prespecified protocol approved by the European Medicines Agency. Propensity score adjustment controlled for imbalances in baseline covariates. Cox regression models estimated the hazard ratio of AP with canagliflozin compared with other AHAs using on-treatment (primary) and intent-to-treat approaches. Sensitivity analyses assessed robustness of findings.Results: Across the three databases, there were between 12,023-80,986 new users of canagliflozin; the unadjusted incidence rates of AP (per 1000 person-years) were between 1.5-2.2 for canagliflozin and 1.1-6.6 for other AHAs. The risk of AP was generally similar for new users of canagliflozin compared with new users of glucagon-like peptide-1 receptor agonists, dipeptidyl peptidase-4 inhibitors, sulfonylureas, thiazolidinediones, insulin, and other AHAs, with no consistent between-treatment differences observed across databases. Intent-to-treat and sensitivity analysis findings were qualitatively consistent with on-treatment findings.Conclusions: In this large observational study, incidence rates of AP in patients with T2DM treated with canagliflozin or other AHAs were generally similar, with no evidence suggesting that canagliflozin is associated with increased risk of AP compared with other AHAs.

Evaluation of disability in patients exposed to fluoroquinolones

BACKGROUND

Fluoroquinolones are used for conditions including sinusitis, bronchitis, and urinary tract infections. It has been suggested that exposure to fluoroquinolones for these conditions is associated with disability resulting from adverse events in 2 or more organ systems. The objectives were to: describe: 1) fluoroquinolone, azithromycin, and sulfamethoxazole / trimethoprim utilization for these infections; 2) the rate of disability associated with exposure to each of these antibiotic classes and adverse events in 2 or more system organ classes, and 3) compare outcome rates for each of the antibiotic classes.

METHODS

This study was conducted using administrative data to mitigate the limitations of spontaneous reports. The sampling frame was a U.S. population with both medical and disability insurance, including patients with the above uncomplicated infections who were prescribed the antibiotics of interest. The primary outcome was an incident short-term disability claim associated with adverse events in 2 different organ systems within 120 days of exposure. A matched analysis was used to compare the outcome for patients receiving each of the drug classes.

RESULTS

After propensity score matching, there were 119,653 individuals in each of the exposure groups. There were 264 fluoroquinolone associated disability events and 243 azithromycin/ sulfamethoxazole associated disability events (relative risk =1.09 (95% CI: 0.92-1.30; calibrated p = 0.84)). The results were not significantly different from the null hypothesis of no difference between groups.

CONCLUSION

Comparative assessments are difficult to conduct in spontaneous reports. This examination of disability associated with adverse events in different system organ classes showed no difference between fluoroquinolones and azithromycin or sulfamethoxazole in administrative data.

Development and validation of data quality rules in administrative health data using association rule mining

Background

Data quality assessment presents a challenge for research using coded administrative health data. The objective of this study is to develop and validate a set of coding association rules for coded diagnostic data.

Methods

We used the Canadian re-abstracted hospital discharge abstract data coded in International Classification of Disease, 10th revision (ICD-10) codes. Association rule mining was conducted on the re-abstracted data in four age groups (0–4, 20–44, 45–64; ≥ 65) to extract ICD-10 coding association rules at the three-digit (category of diagnosis) and four-digit levels (category of diagnosis with etiology, anatomy, or severity). The rules were reviewed by a panel of 5 physicians and 2 classification specialists using a modified Delphi rating process. We proposed and defined the variance and bias to assess data quality using the rules.

Results

After the rule mining process and the panel review, 388 rules at the three-digit level and 275 rules at the four-digit level were developed. Half of the rules were from the age group of ≥65. Rules captured meaningful age-specific clinical associations, with rules at the age group of ≥65 being more complex and comprehensive than other age groups. The variance and bias can identify rules with high bias and variance in Alberta data and provides directions for quality improvement.

Conclusions

A set of ICD-10 data quality rules were developed and validated by a clinical and classification expert panel. The rules can be used as a tool to assess ICD-coded data, enabling the monitoring and comparison of data quality across institutions, provinces, and countries.

Empirical assessment of case-based methods for drug safety alert identification in the French National Healthcare System database (SNDS): Methodology of the ALCAPONE project

OBJECTIVES

To introduce the methodology of the ALCAPONE project.

BACKGROUND

The French National Healthcare System Database (SNDS), covering 99% of the French population, provides a potentially valuable opportunity for drug safety alert generation. ALCAPONE aimed to assess empirically in the SNDS case-based designs for alert generation related to four health outcomes of interest.

METHODS

ALCAPONE used a reference set adapted from observational medical outcomes partnership (OMOP) and Exploring and Understanding Adverse Drug Reactions (EU-ADR) project, with four outcomes-acute liver injury (ALI), myocardial infarction (MI), acute kidney injury (AKI), and upper gastrointestinal bleeding (UGIB)-and positive and negative drug controls. ALCAPONE consisted of four main phases: (1) data preparation to fit the OMOP Common Data Model and select the drug controls; (2) detection of the selected controls via three case-based designs: case-population, case-control, and self-controlled case series, including design variants (varying risk window, adjustment strategy, etc.); (3) comparison of design variant performance (area under the ROC curve, mean square error, etc.); and (4) selection of the optimal design variants and their calibration for each outcome.

RESULTS

Over 2009-2014, 5225 cases of ALI, 354 109 MI, 12 633 AKI, and 156 057 UGIB were identified using specific definitions. The number of detectable drugs ranged from 61 for MI to 25 for ALI. Design variants generated more than 50 000 points estimates. Results by outcome will be published in forthcoming papers.

CONCLUSIONS

ALCAPONE has shown the interest of the empirical assessment of pharmacoepidemiological approaches for drug safety alert generation and may encourage other researchers to do the same in other databases.

How Confident Are We about Observational Findings in Healthcare: A Benchmark Study

Healthcare professionals increasingly rely on observational healthcare data, such as administrative claims and electronic health records, to estimate the causal effects of interventions. However, limited prior studies raise concerns about the real-world performance of the statistical and epidemiological methods that are used. We present the "OHDSI Methods Benchmark" that aims to evaluate the performance of effect estimation methods on real data. The benchmark comprises a gold standard, a set of metrics, and a set of open source software tools. The gold standard is a collection of real negative controls (drug-outcome pairs where no causal effect appears to exist) and synthetic positive controls (drug-outcome pairs that augment negative controls with simulated causal effects). We apply the benchmark using four large healthcare databases to evaluate methods commonly used in practice: the new-user cohort, self-controlled cohort, case-control, case-crossover, and self-controlled case series designs. The results confirm the concerns about these methods, showing that for most methods the operating characteristics deviate considerably from nominal levels. For example, in most contexts, only half of the 95% confidence intervals we calculated contain the corresponding true effect size. We previously developed an "empirical calibration" procedure to restore these characteristics and we also evaluate this procedure. While no one method dominates, self-controlled methods such as the empirically calibrated self-controlled case series perform well across a wide range of scenarios.

Inferring new relations between medical entities using literature curated term co-occurrences

Objectives

Identifying new relations between medical entities, such as drugs, diseases, and side effects, is typically a resource-intensive task, involving experimentation and clinical trials. The increased availability of related data and curated knowledge enables a computational approach to this task, notably by training models to predict likely relations. Such models rely on meaningful representations of the medical entities being studied. We propose a generic features vector representation that leverages co-occurrences of medical terms, linked with PubMed citations.

Materials and Methods

We demonstrate the usefulness of the proposed representation by inferring two types of relations: a drug causes a side effect and a drug treats an indication. To predict these relations and assess their effectiveness, we applied 2 modeling approaches: multi-task modeling using neural networks and single-task modeling based on gradient boosting machines and logistic regression.

Results

These trained models, which predict either side effects or indications, obtained significantly better results than baseline models that use a single direct co-occurrence feature. The results demonstrate the advantage of a comprehensive representation.

Discussion

Selecting the appropriate representation has an immense impact on the predictive performance of machine learning models. Our proposed representation is powerful, as it spans multiple medical domains and can be used to predict a wide range of relation types.

Conclusion

The discovery of new relations between various medical entities can be translated into meaningful insights, for example, related to drug development or disease understanding. Our representation of medical entities can be used to train models that predict such relations, thus accelerating healthcare-related discoveries.

A Comparison Study of Algorithms to Detect Drug-Adverse Event Associations: Frequentist, Bayesian, and Machine-Learning Approaches

INTRODUCTION

It is important to monitor the safety profile of drugs, and mining for strong associations between drugs and adverse events is an effective and inexpensive method of post-marketing safety surveillance.

OBJECTIVE

The objective of our work was to compare the accuracy of both common and innovative methods of data mining for pharmacovigilance purposes.

METHODS

We used the reference standard provided by the Observational Medical Outcomes Partnership, which contains 398 drug-adverse event pairs (165 positive controls, 233 negative controls). Ten methods and algorithms were applied to the US FDA Adverse Event Reporting System data to investigate the 398 pairs. The ten methods include popular methods in the pharmacovigilance literature, newly developed pharmacovigilance methods as at 2018, and popular methods in the genome-wide association study literature. We compared their performance using the receiver operating characteristic (ROC) plot, area under the curve (AUC), and Youden's index.

RESULTS

The Bayesian confidence propagation neural network had the highest AUC overall. Monte Carlo expectation maximization, a method developed in 2018, had the second highest AUC and the highest Youden's index, and performed very well in terms of high specificity. The regression-adjusted gamma Poisson shrinkage model performed best under high-sensitivity requirements.

CONCLUSION

Our results will be useful to help choose a method for a given desired level of specificity. Methods popular in the genome-wide association study literature did not perform well because of the sparsity of data and will need modification before their properties can be used in the drug-adverse event association problem.

Complementing Observational Signals with Literature-Derived Distributed Representations for Post-Marketing Drug Surveillance

INTRODUCTION

As a result of the well documented limitations of data collected by spontaneous reporting systems (SRS), such as bias and under-reporting, a number of authors have evaluated the utility of other data sources for the purpose of pharmacovigilance, including the biomedical literature. Previous work has demonstrated the utility of literature-derived distributed representations (concept embeddings) with machine learning for the purpose of drug side-effect prediction. In terms of data sources, these methods are complementary, observing drug safety from two different perspectives (knowledge extracted from the literature and statistics from SRS data). However, the combined utility of these pharmacovigilance methods has yet to be evaluated.

OBJECTIVE

This research investigates the utility of directly or indirectly combining an observational signal from SRS with literature-derived distributed representations into a single feature vector or in an ensemble approach for downstream machine learning (logistic regression).

METHODS

Leveraging a recently developed representation scheme, concept embeddings were generated from relational connections extracted from the literature and composed to represent drug and associated adverse reactions, as defined by two reference standards of positive (likely causal) and negative (no causal evidence) pairs. Embeddings were presented with and without common measures of observational signal from SRS sources to logistic regressors, and performance was evaluated with the receiver operating characteristic (ROC) area under the curve (AUC) metric.

RESULTS

ROC AUC performance with these composite models improves up to ≈ 20% over SRS-based disproportionality metrics alone and exceeds the best prior results reported in the literature when models leverage both sources of information.

CONCLUSIONS

Results from this study support the hypothesis that knowledge extracted from the literature can enhance the performance of SRS-based methods (and vice versa). Across reference sets, using literature and SRS information together performed better than using either source alone, providing strong support for the complementary nature of these approaches to post-marketing drug surveillance.

A systematic review on literature-based discovery workflow

As scientific publication rates increase, knowledge acquisition and the research development process have become more complex and time-consuming. Literature-Based Discovery (LBD), supporting automated knowledge discovery, helps facilitate this process by eliciting novel knowledge by analysing existing scientific literature. This systematic review provides a comprehensive overview of the LBD workflow by answering nine research questions related to the major components of the LBD workflow (i.e., input, process, output, and evaluation). With regards to the input component, we discuss the data types and data sources used in the literature. The process component presents filtering techniques, ranking/thresholding techniques, domains, generalisability levels, and resources. Subsequently, the output component focuses on the visualisation techniques used in LBD discipline. As for the evaluation component, we outline the evaluation techniques, their generalisability, and the quantitative measures used to validate results. To conclude, we summarise the findings of the review for each component by highlighting the possible future research directions.

Comparative effectiveness of canagliflozin, SGLT2 inhibitors and non-SGLT2 inhibitors on the risk of hospitalization for heart failure and amputation in patients with type 2 diabetes mellitus: A real-world meta-analysis of 4 observational databases (OBSERVE-4D)

AIMS

Sodium glucose co-transporter 2 inhibitors (SGLT2i) are indicated for treatment of type 2 diabetes mellitus (T2DM); some SGLT2i have reported cardiovascular benefit, and some have reported risk of below-knee lower extremity (BKLE) amputation. This study examined the real-world comparative effectiveness within the SGLT2i class and compared with non-SGLT2i antihyperglycaemic agents.

MATERIALS AND METHODS

Data from 4 large US administrative claims databases were used to characterize risk and provide population-level estimates of canagliflozin's effects on hospitalization for heart failure (HHF) and BKLE amputation vs other SGLT2i and non-SGLT2i in T2DM patients. Comparative analyses using a propensity score-adjusted new-user cohort design examined relative hazards of outcomes across all new users and a subpopulation with established cardiovascular disease.

RESULTS

Across the 4 databases (142 800 new users of canagliflozin, 110 897 new users of other SGLT2i, 460 885 new users of non-SGLT2i), the meta-analytic hazard ratio estimate for HHF with canagliflozin vs non-SGLT2i was 0.39 (95% CI, 0.26-0.60) in the on-treatment analysis. The estimate for BKLE amputation with canagliflozin vs non-SGLT2i was 0.75 (95% CI, 0.40-1.41) in the on-treatment analysis and 1.01 (95% CI, 0.93-1.10) in the intent-to-treat analysis. Effects in the subpopulation with established cardiovascular disease were similar for both outcomes. No consistent differences were observed between canagliflozin and other SGLT2i.

CONCLUSIONS

In this large comprehensive analysis, canagliflozin and other SGLT2i demonstrated HHF benefits consistent with clinical trial data, but showed no increased risk of BKLE amputation vs non-SGLT2i. HHF and BKLE amputation results were similar in the subpopulation with established cardiovascular disease. This study helps further characterize the potential benefits and harms of SGLT2i in routine clinical practice to complement evidence from clinical trials and prior observational studies.

Signal Detection for Recently Approved Products: Adapting and Evaluating Self-Controlled Case Series Method Using a US Claims and UK Electronic Medical Records Database

INTRODUCTION

The Self-Controlled Case Series (SCCS) method has been widely used for hypothesis testing, but there is limited evidence of its performance for safety signal detection.

OBJECTIVE

The objective of this study was to evaluate SCCS for signal detection on recently approved products.

METHODS

A retrospective study covered the period after three recently marketed drugs were launched through to 31 December 2010 using The Health Improvement Network, a UK primary care database, and Optum, a US claims database. The SCCS method was applied to examine five heterogenous outcomes with desvenlafaxine and escitalopram and six outcomes with adalimumab for Signals of Disproportional Recording (SDRs); a positive finding was determined to be when the lower bound of 95% Confidence Interval of the incidence rate ratio (IRR) estimate was >  1. Multiple design choices were tested and the trend in IRR estimates over calendar time for one drug event pair was examined.

RESULTS

All six outcomes with adalimumab, three of five outcomes with desvenlafaxine, and four of five outcomes with escitalopram had SDRs. SCCS highlighted all acute events in the primary analysis but was less successful with slower-onset outcomes. Performance varied by risk period definition. Changes in IRR estimates over quarterly intervals for adalimumab with herpes zoster showed marked higher SDR within 9 months of drug launch.

CONCLUSION

SCCS shows promise for signal detection: it may highlight known associations for recent marketed products and has potential for early signal identification. SCCS performance varied by design choice and the nature of both exposure and event pair. Future work is needed to determine how effective the approach is in prospective testing and determining the performance characteristics of the approach.

Learning predictive models of drug side-effect relationships from distributed representations of literature-derived semantic predications

Objective

The aim of this work is to leverage relational information extracted from biomedical literature using a novel synthesis of unsupervised pretraining, representational composition, and supervised machine learning for drug safety monitoring.

Methods

Using ≈80 million concept-relationship-concept triples extracted from the literature using the SemRep Natural Language Processing system, distributed vector representations (embeddings) were generated for concepts as functions of their relationships utilizing two unsupervised representational approaches. Embeddings for drugs and side effects of interest from two widely used reference standards were then composed to generate embeddings of drug/side-effect pairs, which were used as input for supervised machine learning. This methodology was developed and evaluated using cross-validation strategies and compared to contemporary approaches. To qualitatively assess generalization, models trained on the Observational Medical Outcomes Partnership (OMOP) drug/side-effect reference set were evaluated against a list of ≈1100 drugs from an online database.

Results

The employed method improved performance over previous approaches. Cross-validation results advance the state of the art (AUC 0.96; F1 0.90 and AUC 0.95; F1 0.84 across the two sets), outperforming methods utilizing literature and/or spontaneous reporting system data. Examination of predictions for unseen drug/side-effect pairs indicates the ability of these methods to generalize, with over tenfold label support enrichment in the top 100 predictions versus the bottom 100 predictions.

Discussion and Conclusion

Our methods can assist the pharmacovigilance process using information from the biomedical literature. Unsupervised pretraining generates a rich relationship-based representational foundation for machine learning techniques to classify drugs in the context of a putative side effect, given known examples.

Risk of hospital admission for liver injury in users of NSAIDs and nonoverdose paracetamol: Preliminary results from the EPIHAM study

PURPOSE

The SALT study found similar per-user risks of acute liver failure (ALF) leading to transplantation (ALFT) between NSAIDs and a threefold higher risk in nonoverdose paracetamol (NOP) users. The objective of EPIHAM was to identify the risks of hospital admission for acute liver injury (ALI) associated with NSAIDs and NOP.

METHODS

Case-population study in the 1/97 sample of the French population claims database. Acute liver injury was identified from hospital discharge summaries, from 2009 to 2013. Exposure for cases was dispensing of NSAID or NOP resulting in exposure within 30 days before admission. Population exposure was the number of patients using the drugs over the study timeframe and total number of DDD dispensed.

RESULTS

Of 63 cases of ALI, 13 had been exposed to NSAIDs and 24 to NOP. Events per million DDD (95% CI) ranged from 0.46 (0.09-1.34) (ketoprofen) to 1.43 (0.04-7.97) (diclofenac combinations), 0.43 (0.23-0.73) all NSAIDs combined, 0.58 (0.37-0.86) for NOP. There was no association with average duration of treatment. Per patient risk ranged from 19.5 (5.31-49.9) (ibuprofen) per million users to 37.2 (19.8-63.6) all NSAIDs combined, 58.0 (37.2-86.3) for NOP. There was a linear relationship between average treatment duration and per-user risk (R²  = 0.51, P < .05 for NSAIDs, R²  = 0.97, P < .01 for NOP).

CONCLUSIONS

Risk of hospital admission for ALI with NSAIDs and NOP was similar and indicative of a dose and duration-related effect (pharmacological) effect. Acute liver injury rates were not predictive of ALFT risk.

Three-Component Mixture Model-Based Adverse Drug Event Signal Detection for the Adverse Event Reporting System

The US Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) is an important source for detecting adverse drug event (ADE) signals. In this article, we propose a three-component mixture model (3CMM) for FAERS signal detection. In 3CMM, a drug-ADE pair is assumed to have either a zero relative risk (RR), or a background RR (mean RR = 1), or an increased RR (mean RR >1). By clearly defining the second component (mean RR = 1) as the null distribution, 3CMM estimates local false discovery rates (FDRs) for ADE signals under the empirical Bayes framework. Compared with existing approaches, the local FDR's top signals have noninferior or better sensitivities to detect true signals in both FAERS analysis and simulation studies. Additionally, we identify that the top signals of different approaches have different patterns, and they are complementary to each other.

Benefits of combining change-point analysis with disproportionality analysis in pharmacovigilance signal detection

BACKGROUND

Change-point analysis (CPA) is a powerful method to analyse pharmacovigilance data but it has never been used on the disproportionality metric.

OBJECTIVES

To optimize signal detection investigating the interest of time-series analysis in pharmacovigilance and the benefits of combining CPA with the proportional reporting ratio (PRR).

METHODS

We investigated the couple benfluorex and aortic valve incompetence (AVI) using the French National Pharmacovigilance and EudraVigilance databases: CPA was applied on monthly counts of reports and the lower bound of monthly computed PRR (PRR-). We stated a CPA hypothesis that the substance-event combination is more likely to be a signal when the 2 following criteria are fulfilled: PRR- is greater than 1 with at least 5 cases, and CPA method detects at least 2 successive change points of PRR- which made consecutively increasing segments. We tested this hypothesis by 95 test cases identified from a drug safety reference set and 2 validated signals from EudraVigilance database: CPA was applied on PRR-.

RESULTS

For benfluorex and AVI, change points detected by CPA on PRR- were more meaningful compared with monthly counts of reports: More change points detected and detected earlier. In the reference set, 14 positive controls satisfied CPA hypothesis, 6 positive controls only met first requirements, 3 negative controls only met first requirement, and 2 validated signals satisfied CPA hypothesis.

CONCLUSIONS

The combination of CPA and PRR represents a significant advantage in detecting earlier signals and reducing false-positive signals. This approach should be confirmed in further studies.

Channeling in the Use of Nonprescription Paracetamol and Ibuprofen in an Electronic Medical Records Database: Evidence and Implications

Introduction

Over-the-counter analgesics such as paracetamol and ibuprofen are among the most widely used, and having a good understanding of their safety profile is important to public health. Prior observational studies estimating the risks associated with paracetamol use acknowledge the inherent limitations of these studies. One threat to the validity of observational studies is channeling bias, i.e. the notion that patients are systematically exposed to one drug or the other, based on current and past comorbidities, in a manner that affects estimated relative risk.

Objectives

The aim of this study was to examine whether evidence of channeling bias exists in observational studies that compare paracetamol with ibuprofen, and, if so, the extent to which confounding adjustment can mitigate this bias.

Study Design and Setting

In a cohort of 140,770 patients, we examined whether those who received any paracetamol (including concomitant users) were more likely to have prior diagnoses of gastrointestinal (GI) bleeding, myocardial infarction (MI), stroke, or renal disease than those who received ibuprofen alone. We compared propensity score distributions between drugs, and examined the degree to which channeling bias could be controlled using a combination of negative control disease outcome models and large-scale propensity score matching. Analyses were conducted using the Clinical Practice Research Datalink.

Results

The proportions of prior MI, GI bleeding, renal disease, and stroke were significantly higher in those prescribed any paracetamol versus ibuprofen alone, after adjusting for sex and age. We were not able to adequately remove selection bias using a selected set of covariates for propensity score adjustment; however, when we fit the propensity score model using a substantially larger number of covariates, evidence of residual bias was attenuated.

Conclusions

Although using selected covariates for propensity score adjustment may not sufficiently reduce bias, large-scale propensity score matching offers a novel approach to consider to mitigate the effects of channeling bias.

Electronic supplementary material

The online version of this article (doi:10.1007/s40264-017-0581-7) contains supplementary material, which is available to authorized users.

Exploration of association rule mining for coding consistency and completeness assessment in inpatient administrative health data

OBJECTIVE

Data quality assessment is a challenging facet for research using coded administrative health data. Current assessment approaches are time and resource intensive. We explored whether association rule mining (ARM) can be used to develop rules for assessing data quality.

MATERIALS AND METHODS

We extracted 2013 and 2014 records from the hospital discharge abstract database (DAD) for patients between the ages of 55 and 65 from five acute care hospitals in Alberta, Canada. The ARM was conducted using the 2013 DAD to extract rules with support ≥0.0019 and confidence ≥0.5 using the bootstrap technique, and tested in the 2014 DAD. The rules were compared against the method of coding frequency and assessed for their ability to detect error introduced by two kinds of data manipulation: random permutation and random deletion.

RESULTS

The association rules generally had clear clinical meanings. Comparing 2014 data to 2013 data (both original), there were 3 rules with a confidence difference >0.1, while coding frequency difference of codes in the right hand of rules was less than 0.004. After random permutation of 50% of codes in the 2014 data, average rule confidence dropped from 0.72 to 0.27 while coding frequency remained unchanged. Rule confidence decreased with the increase of coding deletion, as expected. Rule confidence was more sensitive to code deletion compared to coding frequency, with slope of change ranging from 1.7 to 184.9 with a median of 9.1.

CONCLUSION

The ARM is a promising technique to assess data quality. It offers a systematic way to derive coding association rules hidden in data, and potentially provides a sensitive and efficient method of assessing data quality compared to standard methods.

An MCEM Framework for Drug Safety Signal Detection and Combination from Heterogeneous Real World Evidence

Delayed drug safety insights can impact patients, pharmaceutical companies, and the whole society. Post-market drug safety surveillance plays a critical role in providing drug safety insights, where real world evidence such as spontaneous reporting systems (SRS) and a series of disproportional analysis serve as a cornerstone of proactive and predictive drug safety surveillance. However, they still face several challenges including concomitant drugs confounders, rare adverse drug reaction (ADR) detection, data bias, and the under-reporting issue. In this paper, we are developing a new framework that detects improved drug safety signals from multiple data sources via Monte Carlo Expectation-Maximization (MCEM) and signal combination. In MCEM procedure, we propose a new sampling approach to generate more accurate SRS signals for each ADR through iteratively down-weighting their associations with irrelevant drugs in case reports. While in signal combination step, we adopt Bayesian hierarchical model and propose a new summary statistic such that SRS signals can be combined with signals derived from other observational health data allowing for related signals to borrow statistical support with adjustment of data reliability. They combined effectively alleviate the concomitant confounders, data bias, rare ADR and under-reporting issues. Experimental results demonstrated the effectiveness and usefulness of the proposed framework.

Toward multimodal signal detection of adverse drug reactions

OBJECTIVE

Improving mechanisms to detect adverse drug reactions (ADRs) is key to strengthening post-marketing drug safety surveillance. Signal detection is presently unimodal, relying on a single information source. Multimodal signal detection is based on jointly analyzing multiple information sources. Building on, and expanding the work done in prior studies, the aim of the article is to further research on multimodal signal detection, explore its potential benefits, and propose methods for its construction and evaluation.

MATERIAL AND METHODS

Four data sources are investigated; FDA's adverse event reporting system, insurance claims, the MEDLINE citation database, and the logs of major Web search engines. Published methods are used to generate and combine signals from each data source. Two distinct reference benchmarks corresponding to well-established and recently labeled ADRs respectively are used to evaluate the performance of multimodal signal detection in terms of area under the ROC curve (AUC) and lead-time-to-detection, with the latter relative to labeling revision dates.

RESULTS

Limited to our reference benchmarks, multimodal signal detection provides AUC improvements ranging from 0.04 to 0.09 based on a widely used evaluation benchmark, and a comparative added lead-time of 7-22 months relative to labeling revision dates from a time-indexed benchmark.

CONCLUSIONS

The results support the notion that utilizing and jointly analyzing multiple data sources may lead to improved signal detection. Given certain data and benchmark limitations, the early stage of development, and the complexity of ADRs, it is currently not possible to make definitive statements about the ultimate utility of the concept. Continued development of multimodal signal detection requires a deeper understanding the data sources used, additional benchmarks, and further research on methods to generate and synthesize signals.

Assessing performance of sequential analysis methods for active drug safety surveillance using observational data

The routine use of sequential methods is well established in clinical studies. Recently, there has been increasing interest in applying these methods to prospectively monitor the safety of newly approved drugs through accrual of real-world data. However, the application to marketed drugs using real-world data has been limited and work is needed to determine which sequential approaches are most suited to such data. In this study, the conditional sequential sampling procedure (CSSP), a group sequential method, was compared with a log-linear model with Poisson distribution (LLMP) through a SAS procedure (PROC GENMOD) combined with an alpha-spending function on two large longitudinal US administrative health claims databases. Relative performance in identifying known drug-outcome associations was examined using a set of 50 well-studied drug-outcome pairs. The study finds that neither method correctly identified all pairs but that LLMP often provides better ability and shorter time for identifying the known drug-outcome associations with superior computational performance when compared with CSSP, albeit with more false positives. With the features of flexible confounding control and ease of implementation, LLMP may be a good alternative or complement to CSSP.

Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing

Objectives

We seek to develop a prototype software analytical tool to augment FDA regulatory reviewers’ capacity to harness scientific literature reports in PubMed/MEDLINE for pharmacovigilance and adverse drug event (ADE) safety signal detection. We also aim to gather feedback through usability testing to assess design, performance, and user satisfaction with the tool.

Methods

A prototype, open source, web-based, software analytical tool generated statistical disproportionality data mining signal scores and dynamic visual analytics for ADE safety signal detection and management. We leveraged Medical Subject Heading (MeSH) indexing terms assigned to published citations in PubMed/MEDLINE to generate candidate drug-adverse event pairs for quantitative data mining. Six FDA regulatory reviewers participated in usability testing by employing the tool as part of their ongoing real-life pharmacovigilance activities to provide subjective feedback on its practical impact, added value, and fitness for use.

Results

All usability test participants cited the tool’s ease of learning, ease of use, and generation of quantitative ADE safety signals, some of which corresponded to known established adverse drug reactions. Potential concerns included the comparability of the tool’s automated literature search relative to a manual ‘all fields’ PubMed search, missing drugs and adverse event terms, interpretation of signal scores, and integration with existing computer-based analytical tools.

Conclusions

Usability testing demonstrated that this novel tool can automate the detection of ADE safety signals from published literature reports. Various mitigation strategies are described to foster improvements in design, productivity, and end user satisfaction.

Embedding of semantic predications

This paper concerns the generation of distributed vector representations of biomedical concepts from structured knowledge, in the form of subject-relation-object triplets known as semantic predications. Specifically, we evaluate the extent to which a representational approach we have developed for this purpose previously, known as Predication-based Semantic Indexing (PSI), might benefit from insights gleaned from neural-probabilistic language models, which have enjoyed a surge in popularity in recent years as a means to generate distributed vector representations of terms from free text. To do so, we develop a novel neural-probabilistic approach to encoding predications, called Embedding of Semantic Predications (ESP), by adapting aspects of the Skipgram with Negative Sampling (SGNS) algorithm to this purpose. We compare ESP and PSI across a number of tasks including recovery of encoded information, estimation of semantic similarity and relatedness, and identification of potentially therapeutic and harmful relationships using both analogical retrieval and supervised learning. We find advantages for ESP in some, but not all of these tasks, revealing the contexts in which the additional computational work of neural-probabilistic modeling is justified.

Accuracy of an automated knowledge base for identifying drug adverse reactions

INTRODUCTION

Drug safety researchers seek to know the degree of certainty with which a particular drug is associated with an adverse drug reaction. There are different sources of information used in pharmacovigilance to identify, evaluate, and disseminate medical product safety evidence including spontaneous reports, published peer-reviewed literature, and product labels. Automated data processing and classification using these evidence sources can greatly reduce the manual curation currently required to develop reference sets of positive and negative controls (i.e. drugs that cause adverse drug events and those that do not) to be used in drug safety research.

METHODS

In this paper we explore a method for automatically aggregating disparate sources of information together into a single repository, developing a predictive model to classify drug-adverse event relationships, and applying those predictions to a real world problem of identifying negative controls for statistical method calibration.

RESULTS

Our results showed high predictive accuracy for the models combining all available evidence, with an area under the receiver-operator curve of ⩾0.92 when tested on three manually generated lists of drugs and conditions that are known to either have or not have an association with an adverse drug event.

CONCLUSIONS

Results from a pilot implementation of the method suggests that it is feasible to develop a scalable alternative to the time-and-resource-intensive, manual curation exercise previously applied to develop reference sets of positive and negative controls to be used in drug safety research.

A Method for the Minimization of Competition Bias in Signal Detection from Spontaneous Reporting Databases

INTRODUCTION

The two methods for minimizing competition bias in signal of disproportionate reporting (SDR) detection--masking factor (MF) and masking ratio (MR)--have focused on the strength of disproportionality for identifying competitors and have been tested using competitors at the drug level.

OBJECTIVES

The aim of this study was to develop a method that relies on identifying competitors by considering the proportion of reports of adverse events (AEs) that mention the drug class at an adequate level of drug grouping to increase sensitivity (Se) for SDR unmasking, and its comparison with MF and MR.

METHODS

Reports in the French spontaneous reporting database between 2000 and 2005 were selected. Five AEs were considered: myocardial infarction, pancreatitis, aplastic anemia, convulsions, and gastrointestinal bleeding; related reports were retrieved using standardized Medical Dictionary for Regulatory Activities (MedDRA(®)) queries. Potential competitors of AEs were identified using the developed method, i.e. Competition Index (ComIn), as well as MF and MR. All three methods were tested according to Anatomical Therapeutic Chemical (ATC) classification levels 2-5. For each AE, SDR detection was performed, first in the complete database, and second after removing reports mentioning competitors; SDRs only detected after the removal were unmasked. All unmasked SDRs were validated using the Summary of Product Characteristics, and constituted the reference dataset used for computing the performance for SDR unmasking (area under the curve [AUC], Se).

RESULTS

Performance of the ComIn was highest when considering competitors at ATC level 3 (AUC: 62 %; Se: 52 %); similar results were obtained with MF and MR.

CONCLUSION

The ComIn could greatly minimize the competition bias in SDR detection. Further study using a larger dataset is needed.

Using Rich Data on Comorbidities in Case-Control Study Design with Electronic Health Record Data Improves Control of Confounding in the Detection of Adverse Drug Reactions

Recent research has suggested that the case-control study design, unlike the self-controlled study design, performs poorly in controlling confounding in the detection of adverse drug reactions (ADRs) from administrative claims and electronic health record (EHR) data, resulting in biased estimates of the causal effects of drugs on health outcomes of interest (HOI) and inaccurate confidence intervals. Here we show that using rich data on comorbidities and automatic variable selection strategies for selecting confounders can better control confounding within a case-control study design and provide a more solid basis for inference regarding the causal effects of drugs on HOIs. Four HOIs are examined: acute kidney injury, acute liver injury, acute myocardial infarction and gastrointestinal ulcer hospitalization. For each of these HOIs we use a previously published reference set of positive and negative control drugs to evaluate the performance of our methods. Our methods have AUCs that are often substantially higher than the AUCs of a baseline method that only uses demographic characteristics for confounding control. Our methods also give confidence intervals for causal effect parameters that cover the expected no effect value substantially more often than this baseline method. The case-control study design, unlike the self-controlled study design, can be used in the fairly typical setting of EHR databases without longitudinal information on patients. With our variable selection method, these databases can be more effectively used for the detection of ADRs.

A time-indexed reference standard of adverse drug reactions

Undetected adverse drug reactions (ADRs) pose a major burden on the health system. Data mining methodologies designed to identify signals of novel ADRs are of deep importance for drug safety surveillance. The development and evaluation of these methodologies requires proper reference benchmarks. While progress has recently been made in developing such benchmarks, our understanding of the performance characteristics of the data mining methodologies is limited because existing benchmarks do not support prospective performance evaluations. We address this shortcoming by providing a reference standard to support prospective performance evaluations. The reference standard was systematically curated from drug labeling revisions, such as new warnings, which were issued and communicated by the US Food and Drug Administration in 2013. The reference standard includes 62 positive test cases and 75 negative controls, and covers 44 drugs and 38 events. We provide usage guidance and empirical support for the reference standard by applying it to analyze two data sources commonly mined for drug safety surveillance.