A time-indexed reference standard of adverse drug reactions

Preclinical side effect prediction through pathway engineering of protein interaction network models

Modeling tools aim to predict potential drug side effects, although they suffer from imperfect performance. Specifically, protein-protein interaction models predict drug effects from proteins surrounding drug targets, but they tend to overpredict drug phenotypes and require well-defined pathway phenotypes. In this study, we used PathFX, a protein-protein interaction tool, to predict side effects for active ingredient-side effect pairs extracted from drug labels. We observed limited performance and defined new pathway phenotypes using pathway engineering strategies. We defined new pathway phenotypes using a network-based and gene expression-based approach. Overall, we discovered a trade-off between sensitivity and specificity values and demonstrated a way to limit overprediction for side effects with sufficient true positive examples. We compared our predictions to animal models and demonstrated similar performance metrics, suggesting that protein-protein interaction models do not need perfect evaluation metrics to be useful. Pathway engineering, through the inclusion of true positive examples and omics measurements, emerges as a promising approach to enhance the utility of protein interaction network models for drug effect prediction.

Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug-drug interactions

PURPOSE

To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs).

METHODS

Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data.

RESULTS

For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates.

CONCLUSIONS

The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others.

Early Detection of Adverse Drug Reaction Signals by Association Rule Mining Using Large-Scale Administrative Claims Data

INTRODUCTION

Adverse drug reactions (ADRs) are a leading cause of mortality worldwide and should be detected promptly to reduce health risks to patients. A data-mining approach using large-scale medical records might be a useful method for the early detection of ADRs. Many studies have analyzed medical records to detect ADRs; however, most of them have focused on a narrow range of ADRs, limiting their usefulness.

OBJECTIVE

This study aimed to identify methods for the early detection of a wide range of ADR signals.

METHODS

First, to evaluate the performance in signal detection of ADRs by data-mining, we attempted to create a gold standard based on clinical evidence. Second, association rule mining (ARM) was applied to patient symptoms and medications registered in claims data, followed by evaluating ADR signal detection performance.

RESULTS

We created a new gold standard consisting of 92 positive and 88 negative controls. In the assessment of ARM using claims data, the areas under the receiver-operating characteristic curve and the precision-recall curve were 0.80 and 0.83, respectively. If the detection criteria were defined as lift > 1, conviction > 1, and p-value < 0.05, ARM could identify 156 signals, of which 90 were true positive controls (sensitivity: 0.98, specificity: 0.25). Evaluation of the capability of ARM with short periods of data revealed that ARM could detect a greater number of positive controls than the conventional analysis method.

CONCLUSIONS

ARM of claims data may be effective in the early detection of a wide range of ADR signals.

Detecting early safety signals of infliximab using machine learning algorithms in the Korea adverse event reporting system

There has been a growing attention on using machine learning (ML) in pharmacovigilance. This study aimed to investigate the utility of supervised ML algorithms on timely detection of safety signals in the Korea Adverse Event Reporting System (KAERS), using infliximab as a case drug, between 2009 and 2018. Input data set for ML training was constructed based on the drug label information and spontaneous reports in the KAERS. Gold standard dataset containing known AEs was randomly divided into the training and test sets. Two supervised ML algorithms (gradient boosting machine [GBM], random forest [RF]) were fitted with hyperparameters tuned on the training set by using a fivefold validation. Then, we stratified the KAERS data by calendar year to create 10 cumulative yearly datasets, in which ML algorithms were applied to detect five pre-specified AEs of infliximab identified during post-marketing surveillance. Four AEs were detected by both GBM and RF in the first year they appeared in the KAERS and earlier than they were updated in the drug label of infliximab. We further applied our models to data retrieved from the US Food and Drug Administration Adverse Event Reporting System repository and found that they outperformed existing disproportionality methods. Both GBM and RF demonstrated reliable performance in detecting early safety signals and showed promise for applying such approaches to pharmacovigilance.

Artificial Intelligence Based on Machine Learning in Pharmacovigilance: A Scoping Review

INTRODUCTION

Artificial intelligence based on machine learning has made large advancements in many fields of science and medicine but its impact on pharmacovigilance is yet unclear.

OBJECTIVE

The present study conducted a scoping review of the use of artificial intelligence based on machine learning to understand how it is used for pharmacovigilance tasks, characterize differences with other fields, and identify opportunities to improve pharmacovigilance through the use of machine learning.

DESIGN

The PubMed, Embase, Web of Science, and IEEE Xplore databases were searched to identify articles pertaining to the use of machine learning in pharmacovigilance published from the year 2000 to September 2021. After manual screening of 7744 abstracts, a total of 393 papers met the inclusion criteria for further analysis. Extraction of key data on study design, data sources, sample size, and machine learning methodology was performed. Studies with the characteristics of good machine learning practice were defined and manual review focused on identifying studies that fulfilled these criteria and results that showed promise.

RESULTS

The majority of studies (53%) were focused on detecting safety signals using traditional statistical methods. Of the studies that used more recent machine learning methods, 61% used off-the-shelf techniques with minor modifications. Temporal analysis revealed that newer methods such as deep learning have shown increased use in recent years. We found only 42 studies (10%) that reflect current best practices and trends in machine learning. In the subset of 154 papers that focused on data intake and ingestion, 30 (19%) were found to incorporate the same best practices.

CONCLUSION

Advances from artificial intelligence have yet to fully penetrate pharmacovigilance, although recent studies show signs that this may be changing.

From random to predictive: a context-specific interaction framework improves selection of drug protein-protein interactions for unknown drug pathways

With high drug attrition, protein-protein interaction (PPI) network models are attractive as efficient methods for predicting drug outcomes by analyzing proteins downstream of drug targets. Unfortunately, these methods tend to overpredict associations and they have low precision and prediction performance; performance is often no better than random (AUROC ~0.5). Typically, PPI models identify ranked phenotypes associated with downstream proteins, yet methods differ in prioritization of downstream proteins. Most methods apply global approaches for assessing all phenotypes. We hypothesized that a per-phenotype analysis could improve prediction performance. We compared two global approaches-statistical and distance-based-and our novel per-phenotype approach, 'context-specific interaction' (CSI) analysis, on severe side effect prediction. We used a novel dataset of adverse events (or designated medical events, DMEs) and discovered that CSI had a 50% improvement over global approaches (AUROC 0.77 compared to 0.51), and a 76-95% improvement in average precision (0.499 compared to 0.284, 0.256). Our results provide a quantitative rationale for considering downstream proteins on a per-phenotype basis when using PPI network methods to predict drug phenotypes.

A reference set of clinically relevant adverse drug-drug interactions

The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms.

Measurement(s)	Adverse Event
Technology Type(s)	digital curation
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.16681933

Development of a Data-Driven Reference Standard for Adverse Drug Reaction (RS-ADR) Signal Assessment (Preprint)

Background

Pharmacovigilance using real-world data (RWD), such as multicenter electronic health records (EHRs), yields massively parallel adverse drug reaction (ADR) signals. However, proper validation of computationally detected ADR signals is not possible due to the lack of a reference standard for positive and negative associations.

Objective

This study aimed to develop a reference standard for ADR (RS-ADR) to streamline the systematic detection, assessment, and understanding of almost all drug-ADR associations suggested by RWD analyses.

Methods

We integrated well-known reference sets for drug-ADR pairs, including Side Effect Resource, Observational Medical Outcomes Partnership, and EU-ADR. We created a pharmacovigilance dictionary using controlled vocabularies and systematically annotated EHR data. Drug-ADR associations computed from MetaLAB and MetaNurse analyses of multicenter EHRs and extracted from the Food and Drug Administration Adverse Event Reporting System were integrated as “empirically determined” positive and negative reference sets by means of cross-validation between institutions.

Results

The RS-ADR consisted of 1344 drugs, 4485 ADRs, and 6,027,840 drug-ADR pairs with positive and negative consensus votes as pharmacovigilance reference sets. After the curation of the initial version of RS-ADR, novel ADR signals such as “famotidine–hepatic function abnormal” were detected and reasonably validated by RS-ADR. Although the validation of the entire reference standard is challenging, especially with this initial version, the reference standard will improve as more RWD participate in the consensus voting with advanced pharmacovigilance dictionaries and analytic algorithms. One can check if a drug-ADR pair has been reported by our web-based search interface for RS-ADRs.

Conclusions

RS-ADRs enriched with the pharmacovigilance dictionary, ADR knowledge, and real-world evidence from EHRs may streamline the systematic detection, evaluation, and causality assessment of computationally detected ADR signals.

The Use of Social Media in Detecting Drug Safety-Related New Black Box Warnings, Labeling Changes, or Withdrawals: Scoping Review

Background

Social media has become a new source for obtaining real-world data on adverse drug reactions. Many studies have investigated the use of social media to detect early signals of adverse drug reactions. However, the trustworthiness of signals derived from social media is questionable. To confirm this, a confirmatory study with a positive control (eg, new black box warnings, labeling changes, or withdrawals) is required.

Objective

This study aimed to evaluate the use of social media in detecting new black box warnings, labeling changes, or withdrawals in advance.

Methods

This scoping review adhered to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews checklist. A researcher searched PubMed and EMBASE in January 2021. Original studies analyzing black box warnings, labeling changes, or withdrawals from social media were selected, and the results of the studies were summarized.

Results

A total of 14 studies were included in this scoping review. Most studies (8/14, 57.1%%) collected data from a single source, and 10 (71.4%) used specialized health care social networks and forums. The analytical methods used in these studies varied considerably. Three studies (21.4%) manually annotated posts, while 5 (35.7%) adopted machine learning algorithms. Nine studies (64.2%) concluded that social media could detect signals 3 months to 9 years before action from regulatory authorities. Most of these studies (8/9, 88.9%) were conducted on specialized health care social networks and forums. On the contrary, 5 (35.7%) studies yielded modest or negative results. Of these, 2 (40%) used generic social networking sites, 2 (40%) used specialized health care networks and forums, and 1 (20%) used both generic social networking sites and specialized health care social networks and forums. The most recently published study recommends not using social media for pharmacovigilance. Several challenges remain in using social media for pharmacovigilance regarding coverage, data quality, and analytic processing.

Conclusions

Social media, along with conventional pharmacovigilance measures, can be used to detect signals associated with new black box warnings, labeling changes, or withdrawals. Several challenges remain; however, social media will be useful for signal detection of frequently mentioned drugs in specialized health care social networks and forums. Further studies are required to advance natural language processing and mine real-world data on social media.

An Ensembled SVM Based Approach for Predicting Adverse Drug Reactions

Background:

Preventing adverse drug reactions (ADRs) is imperative for the safety of the people. The problem of under-reporting the ADRs has been prevalent across the world, making it difficult to develop the prediction models, which are unbiased. As a result, most of the models are skewed to the negative samples leading to high accuracy but poor performance in other metrics such as precision, recall, F1 score, and AUROC score.

Objective:

In this work, we have proposed a novel way of predicting the ADRs by balancing the dataset.

Method:

The whole data set has been partitioned into balanced smaller data sets. SVMs with optimal kernel have been learned using each of the balanced data sets and the prediction of given ADR for the given drug has been obtained by voting from the ensembled optimal SVMs learned.

Results:

We have found that results are encouraging and comparable with the competing methods in the literature and obtained the average sensitivity of 0.97 for all the ADRs. The model has been interpreted and explained with SHAP values by various plots.

Conclusion:

A novel way of predicting ADRs by balancing the dataset has been proposed thereby reducing the effect of unbalanced datasets.

Cardiovascular Toxicities of Antiangiogenic Tyrosine Kinase Inhibitors: A Retrospective, Pharmacovigilance Study

BACKGROUND

Vascular endothelial growth factor receptor (VEGFR) tyrosine kinase inhibitors (TKIs) are an essential therapeutic option in the management of various solid tumors, particularly renal cell carcinoma (RCC). However, post-marketing data regarding their potential cardiovascular toxicities are scant.

OBJECTIVE

To identify and characterize cardiovascular adverse events (CVAEs) of VEGFR-TKIs indicated for RCC.

PATIENTS AND METHODS

Disproportionality analysis of the US Food and Drug Administration adverse event reporting system (July 2014-December 2019) using the reporting odds ratio (ROR) and the lower bound of the Information component (IC) 95% credibility interval (IC₀₂₅ > 0 is significant).

RESULTS

We identified 51,836 adverse event reports of sunitinib, pazopanib, axitinib, cabozantinib, and lenvatinib in the full database [36% women; median age 65 years (range 57-73)]. CVAEs accounted for 11,784 (23%) of the reports, with hypertension [n = 5548 (11%), ROR = 6.55 (95% CI 6.37-6.74), IC₀₂₅ = 2.48] and hemorrhages [n = 3710 (7.2%), ROR = 1.28 (1.24-1.32), IC₀₂₅ = 0.28] being the most frequent types. Additional CVAEs were over-reported with VEGFR-TKIs treatment, including aortic dissection [n = 61 (0.1%), ROR = 3.50 (2.71-4.51)], pericardial diseases [n = 173 (0.3%), ROR = 1.98 (1.70-2.30)], cardiomyopathy [n = 61 (0.1%), ROR = 1.89 (1.47-2.43)], heart failure [n = 868 (1.7%), ROR = 1.35 (1.26-1.44)], and venous thromboembolism [n = 604 (1.2%), ROR = 1.33 (1.23-1.45), all IC₀₂₅ > 0]. The major pericardial disorder was non-malignant pericardial effusion [n = 134 (77%)]. Aortic dissections were also over-reported in patients without concomitant elevated blood pressure [ROR = 2.68 (1.97-3.63), IC₀₂₅ = 0.91]. Finally, CVAEs were reported more often following lenvatinib and sunitinib treatment compared to other VEGFR-TKIs.

CONCLUSIONS

In post-marketing surveillance data, VEGFR-TKIs are associated with increased reporting of various CVAEs, including pericardial diseases, particularly non-malignant pericardial effusion, and aortic dissections. Moreover, VEGFR-TKIs differ in their CVAE reporting patterns. Clinicians should be conscious of these findings in the care of VEGFR-TKIs recipients.

Retrofitting Vector Representations of Adverse Event Reporting Data to Structured Knowledge to Improve Pharmacovigilance Signal Detection

Adverse drug events (ADE) are prevalent and costly. Clinical trials are constrained in their ability to identify potential ADEs, motivating the development of spontaneous reporting systems for post-market surveillance. Statistical methods provide a convenient way to detect signals from these reports but have limitations in leveraging relationships between drugs and ADEs given their discrete count-based nature. A previously proposed method, aer2vec, generates distributed vector representations of ADE report entities that capture patterns of similarity but cannot utilize lexical knowledge. We address this limitation by retrofitting aer2vec drug embeddings to knowledge from RxNorm and developing a novel retrofitting variant using vector rescaling to preserve magnitude. When evaluated in the context of a pharmacovigilance signal detection task, aer2vec with retrofitting consistently outperforms disproportionality metrics when trained on minimally preprocessed data. Retrofitting with rescaling results in further improvements in the larger and more challenging of two pharmacovigilance reference sets used for evaluation.

Machine Learning for Detection of Safety Signals From Spontaneous Reporting System Data: Example of Nivolumab and Docetaxel

Introduction: Various methods have been implemented to detect adverse drug reaction (ADR) signals. However, the applicability of machine learning methods has not yet been fully evaluated.

Objective: To evaluate the feasibility of machine learning algorithms in detecting ADR signals of nivolumab and docetaxel, new and old anticancer agents.

Methods: We conducted a safety surveillance study of nivolumab and docetaxel using the Korea national spontaneous reporting database from 2009 to 2018. We constructed a novel input dataset for each study drug comprised of known ADRs that were listed in the drug labels and unknown ADRs. Given the known ADRs, we trained machine learning algorithms and evaluated predictive performance in generating safety signals of machine learning algorithms (gradient boosting machine [GBM] and random forest [RF]) compared with traditional disproportionality analysis methods (reporting odds ratio [ROR] and information component [IC]) by using the area under the curve (AUC). Each method then was implemented to detect new safety signals from the unknown ADR datasets.

Results: Of all methods implemented, GBM achieved the best average predictive performance (AUC: 0.97 and 0.93 for nivolumab and docetaxel). The AUC achieved by each method was 0.95 and 0.92 (RF), 0.55 and 0.51 (ROR), and 0.49 and 0.48 (IC) for respective drug. GBM detected additional 24 and nine signals for nivolumab and 82 and 76 for docetaxel compared to ROR and IC, respectively, from the unknown ADR datasets.

Conclusion: Machine learning algorithm based on GBM performed better and detected more new ADR signals than traditional disproportionality analysis methods.

Combining Social Media and FDA Adverse Event Reporting System to Detect Adverse Drug Reactions

INTRODUCTION

Adverse drug reactions (ADRs) are unintended reactions caused by a drug or combination of drugs taken by a patient. The current safety surveillance system relies on spontaneous reporting systems (SRSs) and more recently on observational health data; however, ADR detection may be delayed and lack geographic diversity. The broad scope of social media conversations, such as those on Twitter, can include health-related topics. Consequently, these data could be used to detect potentially novel ADRs with less latency. Although research regarding ADR detection using social media has made progress, findings are based on single information sources, and no study has yet integrated drug safety evidence from both an SRS and Twitter.

OBJECTIVE

The aim of this study was to combine signals from an SRS and Twitter to facilitate the detection of safety signals and compare the performance of the combined system with signals generated by individual data sources.

METHODS

We extracted potential drug-ADR posts from Twitter, used Monte Carlo expectation maximization to generate drug safety signals from both the US FDA Adverse Event Reporting System and posts from Twitter, and then integrated these signals using a Bayesian hierarchical model. The results from the integrated system and two individual sources were evaluated using a reference standard derived from drug labels. Area under the receiver operating characteristics curve (AUC) was computed to measure performance.

RESULTS

We observed a significant improvement in the AUC of the combined system when comparing it with Twitter alone, and no improvement when comparing with the SRS alone. The AUCs ranged from 0.587 to 0.637 for the combined SRS and Twitter, from 0.525 to 0.534 for Twitter alone, and from 0.612 to 0.642 for the SRS alone. The results varied because different preprocessing procedures were applied to Twitter.

CONCLUSION

The accuracy of signal detection using social media can be improved by combining signals with those from SRSs. However, the combined system cannot achieve better AUC performance than data from FAERS alone, which may indicate that Twitter data are not ready to be integrated into a purely data-driven combination system.

Recommendations for the Use of Social Media in Pharmacovigilance: Lessons from IMI WEB-RADR

Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR project has explored the value of social media (i.e., information exchanged through the internet, typically via online social networks) for identifying adverse events as well as for safety signal detection. Many patients and clinicians have taken to social media to discuss their positive and negative experiences of medications, creating a source of publicly available information that has the potential to provide insights into medicinal product safety concerns. The WEB-RADR project has developed a collaborative English language workspace for visualising and analysing social media data for a number of medicinal products. Further, novel text and data mining methods for social media analysis have been developed and evaluated. From this original research, several recommendations are presented with supporting rationale and consideration of the limitations. Recommendations for further research that extend beyond the scope of the current project are also presented.

Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project

INTRODUCTION AND OBJECTIVE

Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events.

METHODS

Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed.

RESULTS

Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64-0.69 and 0.55-0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%).

CONCLUSIONS

Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities.

Learning predictive models of drug side-effect relationships from distributed representations of literature-derived semantic predications

Objective

The aim of this work is to leverage relational information extracted from biomedical literature using a novel synthesis of unsupervised pretraining, representational composition, and supervised machine learning for drug safety monitoring.

Methods

Using ≈80 million concept-relationship-concept triples extracted from the literature using the SemRep Natural Language Processing system, distributed vector representations (embeddings) were generated for concepts as functions of their relationships utilizing two unsupervised representational approaches. Embeddings for drugs and side effects of interest from two widely used reference standards were then composed to generate embeddings of drug/side-effect pairs, which were used as input for supervised machine learning. This methodology was developed and evaluated using cross-validation strategies and compared to contemporary approaches. To qualitatively assess generalization, models trained on the Observational Medical Outcomes Partnership (OMOP) drug/side-effect reference set were evaluated against a list of ≈1100 drugs from an online database.

Results

The employed method improved performance over previous approaches. Cross-validation results advance the state of the art (AUC 0.96; F1 0.90 and AUC 0.95; F1 0.84 across the two sets), outperforming methods utilizing literature and/or spontaneous reporting system data. Examination of predictions for unseen drug/side-effect pairs indicates the ability of these methods to generalize, with over tenfold label support enrichment in the top 100 predictions versus the bottom 100 predictions.

Discussion and Conclusion

Our methods can assist the pharmacovigilance process using information from the biomedical literature. Unsupervised pretraining generates a rich relationship-based representational foundation for machine learning techniques to classify drugs in the context of a putative side effect, given known examples.

Supervised signal detection for adverse drug reactions in medication dispensing data

MOTIVATION

Adverse drug reactions (ADRs) are one of the leading causes of morbidity and mortality and thus should be detected early to reduce consequences on health outcomes. Medication dispensing data are comprehensive sources of information about medicine uses that can be utilized for the signal detection of ADRs. Sequence symmetry analysis (SSA) has been employed in previous studies to detect signals of ADRs from medication dispensing data, but it has a moderate sensitivity and tends to miss some ADR signals. With successful applications in various areas, supervised machine learning (SML) methods are promising in detecting ADR signals. Gold standards of known ADRs and non- ADRs from previous studies create opportunities to take into account additional domain knowledge to improve ADR signal detection with SML.

OBJECTIVE

We assess the utility of SML as a signal detection tool for ADRs in medication dispensing data with the consideration of domain knowledge from DrugBank and MedDRA. We compare the best performing SML method with SSA.

METHODS

We model the ADR signal detection problem as a supervised machine learning problem by linking medication dispensing data with domain knowledge bases. Suspected ADR signals are extracted from the Australian Pharmaceutical Benefit Scheme (PBS) medication dispensing data from 2013 to 2016. We construct predictive features for each signal candidate based on its occurrences in medication dispensing data as well as its pharmacological properties. Pharmaceutical knowledge bases including DrugBank and MedDRA are employed to provide pharmacological features for a signal candidate. Given a gold standard of known ADRs and non-ADRs, SML learns to differentiate between known ADRs and non-ADRs based on their combined predictive features from linked sources, and then predicts whether a new case is a potential ADR signal.

RESULTS

We evaluate the performance of six widely used SML methods with two gold standards of known ADRs and non-ADRs from previous studies. On average, gradient boosting classifier achieves the sensitivity of 77%, specificity of 81%, positive predictive value of 76%, negative predictive value of 82%, area under precision-recall curve of 81%, and area under receiver operating characteristic curve of 82%, most of which are higher than in other SML methods. In particular, gradient boosting classifier has 21% higher sensitivity than and comparable specificity with SSA. Furthermore, gradient boosting classifier detects 10% more unknown potential ADR signals than SSA.

CONCLUSIONS

Our study demonstrates that gradient boosting classifier is a promising supervised signal detection tool for ADRs in medication dispensing data to complement SSA.

Replicability, Reproducibility, and Agent-based Simulation of Interventions

Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.

Toward multimodal signal detection of adverse drug reactions

OBJECTIVE

Improving mechanisms to detect adverse drug reactions (ADRs) is key to strengthening post-marketing drug safety surveillance. Signal detection is presently unimodal, relying on a single information source. Multimodal signal detection is based on jointly analyzing multiple information sources. Building on, and expanding the work done in prior studies, the aim of the article is to further research on multimodal signal detection, explore its potential benefits, and propose methods for its construction and evaluation.

MATERIAL AND METHODS

Four data sources are investigated; FDA's adverse event reporting system, insurance claims, the MEDLINE citation database, and the logs of major Web search engines. Published methods are used to generate and combine signals from each data source. Two distinct reference benchmarks corresponding to well-established and recently labeled ADRs respectively are used to evaluate the performance of multimodal signal detection in terms of area under the ROC curve (AUC) and lead-time-to-detection, with the latter relative to labeling revision dates.

RESULTS

Limited to our reference benchmarks, multimodal signal detection provides AUC improvements ranging from 0.04 to 0.09 based on a widely used evaluation benchmark, and a comparative added lead-time of 7-22 months relative to labeling revision dates from a time-indexed benchmark.

CONCLUSIONS

The results support the notion that utilizing and jointly analyzing multiple data sources may lead to improved signal detection. Given certain data and benchmark limitations, the early stage of development, and the complexity of ADRs, it is currently not possible to make definitive statements about the ultimate utility of the concept. Continued development of multimodal signal detection requires a deeper understanding the data sources used, additional benchmarks, and further research on methods to generate and synthesize signals.

Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing

Objectives

We seek to develop a prototype software analytical tool to augment FDA regulatory reviewers’ capacity to harness scientific literature reports in PubMed/MEDLINE for pharmacovigilance and adverse drug event (ADE) safety signal detection. We also aim to gather feedback through usability testing to assess design, performance, and user satisfaction with the tool.

Methods

A prototype, open source, web-based, software analytical tool generated statistical disproportionality data mining signal scores and dynamic visual analytics for ADE safety signal detection and management. We leveraged Medical Subject Heading (MeSH) indexing terms assigned to published citations in PubMed/MEDLINE to generate candidate drug-adverse event pairs for quantitative data mining. Six FDA regulatory reviewers participated in usability testing by employing the tool as part of their ongoing real-life pharmacovigilance activities to provide subjective feedback on its practical impact, added value, and fitness for use.

Results

All usability test participants cited the tool’s ease of learning, ease of use, and generation of quantitative ADE safety signals, some of which corresponded to known established adverse drug reactions. Potential concerns included the comparability of the tool’s automated literature search relative to a manual ‘all fields’ PubMed search, missing drugs and adverse event terms, interpretation of signal scores, and integration with existing computer-based analytical tools.

Conclusions

Usability testing demonstrated that this novel tool can automate the detection of ADE safety signals from published literature reports. Various mitigation strategies are described to foster improvements in design, productivity, and end user satisfaction.

Generalized enrichment analysis improves the detection of adverse drug events from the biomedical literature

Background

Identification of associations between marketed drugs and adverse events from the biomedical literature assists drug safety monitoring efforts. Assessing the significance of such literature-derived associations and determining the granularity at which they should be captured remains a challenge. Here, we assess how defining a selection of adverse event terms from MeSH, based on information content, can improve the detection of adverse events for drugs and drug classes.

Results

We analyze a set of 105,354 candidate drug adverse event pairs extracted from article indexes in MEDLINE. First, we harmonize extracted adverse event terms by aggregating them into higher-level MeSH terms based on the terms’ information content. Then, we determine statistical enrichment of adverse events associated with drug and drug classes using a conditional hypergeometric test that adjusts for dependencies among associated terms. We compare our results with methods based on disproportionality analysis (proportional reporting ratio, PRR) and quantify the improvement in signal detection with our generalized enrichment analysis (GEA) approach using a gold standard of drug-adverse event associations spanning 174 drugs and four events. For single drugs, the best GEA method (Precision: .92/Recall: .71/F1-measure: .80) outperforms the best PRR based method (.69/.69/.69) on all four adverse event outcomes in our gold standard. For drug classes, our GEA performs similarly (.85/.69/.74) when increasing the level of abstraction for adverse event terms. Finally, on examining the 1609 individual drugs in our MEDLINE set, which map to chemical substances in ATC, we find signals for 1379 drugs (10,122 unique adverse event associations) on applying GEA with p < 0.005.

Conclusions

We present an approach based on generalized enrichment analysis that can be used to detect associations between drugs, drug classes and adverse events at a given level of granularity, at the same time correcting for known dependencies among events. Our study demonstrates the use of GEA, and the importance of choosing appropriate abstraction levels to complement current drug safety methods. We provide an R package for exploration of alternative abstraction levels of adverse event terms based on information content.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-016-1080-z) contains supplementary material, which is available to authorized users.

Leveraging MEDLINE indexing for pharmacovigilance - Inherent limitations and mitigation strategies

BACKGROUND

Traditional approaches to pharmacovigilance center on the signal detection from spontaneous reports, e.g., the U.S. Food and Drug Administration (FDA) adverse event reporting system (FAERS). In order to enrich the scientific evidence and enhance the detection of emerging adverse drug events that can lead to unintended harmful outcomes, pharmacovigilance activities need to evolve to encompass novel complementary data streams, for example the biomedical literature available through MEDLINE.

OBJECTIVES

(1) To review how the characteristics of MEDLINE indexing influence the identification of adverse drug events (ADEs); (2) to leverage this knowledge to inform the design of a system for extracting ADEs from MEDLINE indexing; and (3) to assess the specific contribution of some characteristics of MEDLINE indexing to the performance of this system.

METHODS

We analyze the characteristics of MEDLINE indexing. We integrate three specific characteristics into the design of a system for extracting ADEs from MEDLINE indexing. We experimentally assess the specific contribution of these characteristics over a baseline system based on co-occurrence between drug descriptors qualified by adverse effects and disease descriptors qualified by chemically induced.

RESULTS

Our system extracted 405,300 ADEs from 366,120 MEDLINE articles. The baseline system accounts for 297,093 ADEs (73%). 85,318 ADEs (21%) can be extracted only after integrating specific pre-coordinated MeSH descriptors and additional qualifiers. 22,889 ADEs (6%) can be extracted only after considering indirect links between the drug of interest and the descriptor that bears the ADE context.

CONCLUSIONS

In this paper, we demonstrate significant improvement over a baseline approach to identifying ADEs from MEDLINE indexing, which mitigates some of the inherent limitations of MEDLINE indexing for pharmacovigilance. ADEs extracted from MEDLINE indexing are complementary to, not a replacement for, other sources.

Analyzing Information Seeking and Drug-Safety Alert Response by Health Care Professionals as New Methods for Surveillance

Background

Patterns in general consumer online search logs have been used to monitor health conditions and to predict health-related activities, but the multiple contexts within which consumers perform online searches make significant associations difficult to interpret. Physician information-seeking behavior has typically been analyzed through survey-based approaches and literature reviews. Activity logs from health care professionals using online medical information resources are thus a valuable yet relatively untapped resource for large-scale medical surveillance.

Objective

To analyze health care professionals’ information-seeking behavior and assess the feasibility of measuring drug-safety alert response from the usage logs of an online medical information resource.

Methods

Using two years (2011-2012) of usage logs from UpToDate, we measured the volume of searches related to medical conditions with significant burden in the United States, as well as the seasonal distribution of those searches. We quantified the relationship between searches and resulting page views. Using a large collection of online mainstream media articles and Web log posts we also characterized the uptake of a Food and Drug Administration (FDA) alert via changes in UpToDate search activity compared with general online media activity related to the subject of the alert.

Results

Diseases and symptoms dominate UpToDate searches. Some searches result in page views of only short duration, while others consistently result in longer-than-average page views. The response to an FDA alert for Celexa, characterized by a change in UpToDate search activity, differed considerably from general online media activity. Changes in search activity appeared later and persisted longer in UpToDate logs. The volume of searches and page view durations related to Celexa before the alert also differed from those after the alert.

Conclusions

Understanding the information-seeking behavior associated with online evidence sources can offer insight into the information needs of health professionals and enable large-scale medical surveillance. Our Web log mining approach has the potential to monitor responses to FDA alerts at a national level. Our findings can also inform the design and content of evidence-based medical information resources such as UpToDate.

A method for systematic discovery of adverse drug events from clinical notes

OBJECTIVE

Adverse drug events (ADEs) are undesired harmful effects resulting from use of a medication, and occur in 30% of hospitalized patients. The authors have developed a data-mining method for systematic, automated detection of ADEs from electronic medical records.

MATERIALS AND METHODS

This method uses the text from 9.5 million clinical notes, along with prior knowledge of drug usages and known ADEs, as inputs. These inputs are further processed into statistics used by a discriminative classifier which outputs the probability that a given drug-disorder pair represents a valid ADE association. Putative ADEs identified by the classifier are further filtered for positive support in 2 independent, complementary data sources. The authors evaluate this method by assessing support for the predictions in other curated data sources, including a manually curated, time-indexed reference standard of label change events.

RESULTS

This method uses a classifier that achieves an area under the curve of 0.94 on a held out test set. The classifier is used on 2,362,950 possible drug-disorder pairs comprised of 1602 unique drugs and 1475 unique disorders for which we had data, resulting in 240 high-confidence, well-supported drug-AE associations. Eighty-seven of them (36%) are supported in at least one of the resources that have information that was not available to the classifier.

CONCLUSION

This method demonstrates the feasibility of systematic post-marketing surveillance for ADEs using electronic medical records, a key component of the learning healthcare system.

Pediatric drug safety signal detection: a new drug-event reference set for performance testing of data-mining methods and systems

BACKGROUND

Better evidence regarding drug safety in the pediatric population might be generated from existing data sources such as spontaneous reporting systems and electronic healthcare records. The Global Research in Paediatrics (GRiP)-Network of Excellence aims to develop pediatric-specific methods that can be applied to these data sources. A reference set of positive and negative drug-event associations is required.

OBJECTIVE

The aim of this study was to develop a pediatric-specific reference set of positive and negative drug-event associations.

METHODS

Considering user patterns and expert opinion, 16 drugs that are used in individuals aged 0-18 years were selected and evaluated against 16 events, regarded as important safety outcomes. A cross-table of unique drug-event pairs was created. Each pair was classified as potential positive or negative control based on information from the drug's Summary of Product Characteristics and Micromedex. If both information sources consistently listed the event as an adverse event, the combination was reviewed as potential positive control. If both did not, the combination was evaluated as potential negative control. Further evaluation was based on published literature.

RESULTS

Selected drugs include ibuprofen, flucloxacillin, domperidone, methylphenidate, montelukast, quinine, and cyproterone/ethinylestradiol. Selected events include bullous eruption, aplastic anemia, ventricular arrhythmia, sudden death, acute kidney injury, psychosis, and seizure. Altogether, 256 unique combinations were reviewed, yielding 37 positive (17 with evidence from the pediatric population and 20 with evidence from adults only) and 90 negative control pairs, with the remainder being unclassifiable.

CONCLUSION

We propose a drug-event reference set that can be used to compare different signal detection methods in the pediatric population.