1
|
Alidoost M, Wilson JL. Preclinical side effect prediction through pathway engineering of protein interaction network models. CPT Pharmacometrics Syst Pharmacol 2024; 13:1180-1200. [PMID: 38736280 PMCID: PMC11247120 DOI: 10.1002/psp4.13150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 03/01/2024] [Accepted: 04/08/2024] [Indexed: 05/14/2024] Open
Abstract
Modeling tools aim to predict potential drug side effects, although they suffer from imperfect performance. Specifically, protein-protein interaction models predict drug effects from proteins surrounding drug targets, but they tend to overpredict drug phenotypes and require well-defined pathway phenotypes. In this study, we used PathFX, a protein-protein interaction tool, to predict side effects for active ingredient-side effect pairs extracted from drug labels. We observed limited performance and defined new pathway phenotypes using pathway engineering strategies. We defined new pathway phenotypes using a network-based and gene expression-based approach. Overall, we discovered a trade-off between sensitivity and specificity values and demonstrated a way to limit overprediction for side effects with sufficient true positive examples. We compared our predictions to animal models and demonstrated similar performance metrics, suggesting that protein-protein interaction models do not need perfect evaluation metrics to be useful. Pathway engineering, through the inclusion of true positive examples and omics measurements, emerges as a promising approach to enhance the utility of protein interaction network models for drug effect prediction.
Collapse
Affiliation(s)
- Mohammadali Alidoost
- Department of Bioengineering, University of California, Los Angeles, California, USA
| | - Jennifer L Wilson
- Department of Bioengineering, University of California, Los Angeles, California, USA
| |
Collapse
|
2
|
Kontsioti E, Maskell S, Pirmohamed M. Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug-drug interactions. Pharmacoepidemiol Drug Saf 2023; 32:832-844. [PMID: 36916014 PMCID: PMC10947279 DOI: 10.1002/pds.5609] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 02/13/2023] [Accepted: 03/08/2023] [Indexed: 03/16/2023]
Abstract
PURPOSE To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs). METHODS Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. RESULTS For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. CONCLUSIONS The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others.
Collapse
Affiliation(s)
- Elpida Kontsioti
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Simon Maskell
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Munir Pirmohamed
- The Wolfson Center for Personalized Medicine, Center for Drug Safety Science, Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
| |
Collapse
|
3
|
Early Detection of Adverse Drug Reaction Signals by Association Rule Mining Using Large-Scale Administrative Claims Data. Drug Saf 2023; 46:371-389. [PMID: 36828947 PMCID: PMC10113351 DOI: 10.1007/s40264-023-01278-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/06/2023] [Indexed: 02/26/2023]
Abstract
INTRODUCTION Adverse drug reactions (ADRs) are a leading cause of mortality worldwide and should be detected promptly to reduce health risks to patients. A data-mining approach using large-scale medical records might be a useful method for the early detection of ADRs. Many studies have analyzed medical records to detect ADRs; however, most of them have focused on a narrow range of ADRs, limiting their usefulness. OBJECTIVE This study aimed to identify methods for the early detection of a wide range of ADR signals. METHODS First, to evaluate the performance in signal detection of ADRs by data-mining, we attempted to create a gold standard based on clinical evidence. Second, association rule mining (ARM) was applied to patient symptoms and medications registered in claims data, followed by evaluating ADR signal detection performance. RESULTS We created a new gold standard consisting of 92 positive and 88 negative controls. In the assessment of ARM using claims data, the areas under the receiver-operating characteristic curve and the precision-recall curve were 0.80 and 0.83, respectively. If the detection criteria were defined as lift > 1, conviction > 1, and p-value < 0.05, ARM could identify 156 signals, of which 90 were true positive controls (sensitivity: 0.98, specificity: 0.25). Evaluation of the capability of ARM with short periods of data revealed that ARM could detect a greater number of positive controls than the conventional analysis method. CONCLUSIONS ARM of claims data may be effective in the early detection of a wide range of ADR signals.
Collapse
|
4
|
Detecting early safety signals of infliximab using machine learning algorithms in the Korea adverse event reporting system. Sci Rep 2022; 12:14869. [PMID: 36050484 PMCID: PMC9436954 DOI: 10.1038/s41598-022-18522-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2021] [Accepted: 08/16/2022] [Indexed: 11/26/2022] Open
Abstract
There has been a growing attention on using machine learning (ML) in pharmacovigilance. This study aimed to investigate the utility of supervised ML algorithms on timely detection of safety signals in the Korea Adverse Event Reporting System (KAERS), using infliximab as a case drug, between 2009 and 2018. Input data set for ML training was constructed based on the drug label information and spontaneous reports in the KAERS. Gold standard dataset containing known AEs was randomly divided into the training and test sets. Two supervised ML algorithms (gradient boosting machine [GBM], random forest [RF]) were fitted with hyperparameters tuned on the training set by using a fivefold validation. Then, we stratified the KAERS data by calendar year to create 10 cumulative yearly datasets, in which ML algorithms were applied to detect five pre-specified AEs of infliximab identified during post-marketing surveillance. Four AEs were detected by both GBM and RF in the first year they appeared in the KAERS and earlier than they were updated in the drug label of infliximab. We further applied our models to data retrieved from the US Food and Drug Administration Adverse Event Reporting System repository and found that they outperformed existing disproportionality methods. Both GBM and RF demonstrated reliable performance in detecting early safety signals and showed promise for applying such approaches to pharmacovigilance.
Collapse
|
5
|
Kompa B, Hakim JB, Palepu A, Kompa KG, Smith M, Bain PA, Woloszynek S, Painter JL, Bate A, Beam AL. Artificial Intelligence Based on Machine Learning in Pharmacovigilance: A Scoping Review. Drug Saf 2022; 45:477-491. [PMID: 35579812 PMCID: PMC9883349 DOI: 10.1007/s40264-022-01176-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/13/2022] [Indexed: 02/07/2023]
Abstract
INTRODUCTION Artificial intelligence based on machine learning has made large advancements in many fields of science and medicine but its impact on pharmacovigilance is yet unclear. OBJECTIVE The present study conducted a scoping review of the use of artificial intelligence based on machine learning to understand how it is used for pharmacovigilance tasks, characterize differences with other fields, and identify opportunities to improve pharmacovigilance through the use of machine learning. DESIGN The PubMed, Embase, Web of Science, and IEEE Xplore databases were searched to identify articles pertaining to the use of machine learning in pharmacovigilance published from the year 2000 to September 2021. After manual screening of 7744 abstracts, a total of 393 papers met the inclusion criteria for further analysis. Extraction of key data on study design, data sources, sample size, and machine learning methodology was performed. Studies with the characteristics of good machine learning practice were defined and manual review focused on identifying studies that fulfilled these criteria and results that showed promise. RESULTS The majority of studies (53%) were focused on detecting safety signals using traditional statistical methods. Of the studies that used more recent machine learning methods, 61% used off-the-shelf techniques with minor modifications. Temporal analysis revealed that newer methods such as deep learning have shown increased use in recent years. We found only 42 studies (10%) that reflect current best practices and trends in machine learning. In the subset of 154 papers that focused on data intake and ingestion, 30 (19%) were found to incorporate the same best practices. CONCLUSION Advances from artificial intelligence have yet to fully penetrate pharmacovigilance, although recent studies show signs that this may be changing.
Collapse
Affiliation(s)
- Benjamin Kompa
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Joe B Hakim
- Department of Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Anil Palepu
- Department of Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | | | - Michael Smith
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Paul A Bain
- Countway Library of Medicine, Harvard Medical School, Boston, MA, USA
| | | | | | - Andrew Bate
- GlaxoSmithKline, Brentford, UK
- Department of Non-Communicable Disease Epidemiology, London School of Hygiene and Tropical Medicine, University of London, London, UK
- Department of Medicine, NYU Grossman School of Medicine, New York, NY, USA
| | - Andrew L Beam
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
| |
Collapse
|
6
|
Wilson JL, Gravina A, Grimes K. From random to predictive: a context-specific interaction framework improves selection of drug protein-protein interactions for unknown drug pathways. Integr Biol (Camb) 2022; 14:13-24. [PMID: 35293584 DOI: 10.1093/intbio/zyac002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Revised: 02/01/2022] [Accepted: 02/03/2022] [Indexed: 12/20/2022]
Abstract
With high drug attrition, protein-protein interaction (PPI) network models are attractive as efficient methods for predicting drug outcomes by analyzing proteins downstream of drug targets. Unfortunately, these methods tend to overpredict associations and they have low precision and prediction performance; performance is often no better than random (AUROC ~0.5). Typically, PPI models identify ranked phenotypes associated with downstream proteins, yet methods differ in prioritization of downstream proteins. Most methods apply global approaches for assessing all phenotypes. We hypothesized that a per-phenotype analysis could improve prediction performance. We compared two global approaches-statistical and distance-based-and our novel per-phenotype approach, 'context-specific interaction' (CSI) analysis, on severe side effect prediction. We used a novel dataset of adverse events (or designated medical events, DMEs) and discovered that CSI had a 50% improvement over global approaches (AUROC 0.77 compared to 0.51), and a 76-95% improvement in average precision (0.499 compared to 0.284, 0.256). Our results provide a quantitative rationale for considering downstream proteins on a per-phenotype basis when using PPI network methods to predict drug phenotypes.
Collapse
Affiliation(s)
- Jennifer L Wilson
- Department of Bioengineering, University of California Los Angeles, Los Angeles, CA, USA
| | - Alessio Gravina
- Department of Computer Science, University of Pisa, Pisa, Italy
| | - Kevin Grimes
- Department of Chemical and Systems Biology, Stanford University, Stanford, CA, USA
| |
Collapse
|
7
|
Kontsioti E, Maskell S, Dutta B, Pirmohamed M. A reference set of clinically relevant adverse drug-drug interactions. Sci Data 2022; 9:72. [PMID: 35246559 PMCID: PMC8897500 DOI: 10.1038/s41597-022-01159-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2021] [Accepted: 01/13/2022] [Indexed: 12/03/2022] Open
Abstract
The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms. Measurement(s) | Adverse Event | Technology Type(s) | digital curation | Sample Characteristic - Organism | Homo sapiens |
Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.16681933
Collapse
Affiliation(s)
- Elpida Kontsioti
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK.
| | - Simon Maskell
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Bhaskar Dutta
- Patient Safety Center of Excellence, Chief Medical Office Organization, AstraZeneca Pharmaceuticals, Gaithersburg, MD, USA
| | - Munir Pirmohamed
- The Wolfson Centre for Personalized Medicine, MRC Centre for Drug Safety Science, Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, United Kingdom
| |
Collapse
|
8
|
Lee S, Lee JH, Kim GJ, Kim JY, Shin H, Ko I, Choe S, Kim JH. Development of a Data-Driven Reference Standard for Adverse Drug Reaction (RS-ADR) Signal Assessment (Preprint). J Med Internet Res 2021; 24:e35464. [PMID: 36201386 PMCID: PMC9585444 DOI: 10.2196/35464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Revised: 04/29/2022] [Accepted: 07/14/2022] [Indexed: 11/13/2022] Open
Abstract
Background Pharmacovigilance using real-world data (RWD), such as multicenter electronic health records (EHRs), yields massively parallel adverse drug reaction (ADR) signals. However, proper validation of computationally detected ADR signals is not possible due to the lack of a reference standard for positive and negative associations. Objective This study aimed to develop a reference standard for ADR (RS-ADR) to streamline the systematic detection, assessment, and understanding of almost all drug-ADR associations suggested by RWD analyses. Methods We integrated well-known reference sets for drug-ADR pairs, including Side Effect Resource, Observational Medical Outcomes Partnership, and EU-ADR. We created a pharmacovigilance dictionary using controlled vocabularies and systematically annotated EHR data. Drug-ADR associations computed from MetaLAB and MetaNurse analyses of multicenter EHRs and extracted from the Food and Drug Administration Adverse Event Reporting System were integrated as “empirically determined” positive and negative reference sets by means of cross-validation between institutions. Results The RS-ADR consisted of 1344 drugs, 4485 ADRs, and 6,027,840 drug-ADR pairs with positive and negative consensus votes as pharmacovigilance reference sets. After the curation of the initial version of RS-ADR, novel ADR signals such as “famotidine–hepatic function abnormal” were detected and reasonably validated by RS-ADR. Although the validation of the entire reference standard is challenging, especially with this initial version, the reference standard will improve as more RWD participate in the consensus voting with advanced pharmacovigilance dictionaries and analytic algorithms. One can check if a drug-ADR pair has been reported by our web-based search interface for RS-ADRs. Conclusions RS-ADRs enriched with the pharmacovigilance dictionary, ADR knowledge, and real-world evidence from EHRs may streamline the systematic detection, evaluation, and causality assessment of computationally detected ADR signals.
Collapse
Affiliation(s)
- Suehyun Lee
- Department of Biomedical Informatics, College of Medicine, Konyang University, Daejeon, Republic of Korea
| | - Jeong Hoon Lee
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Grace Juyun Kim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Jong-Yeup Kim
- Healthcare Data Science Center, Konyang University Hospital, Daejeon, Republic of Korea
| | - Hyunah Shin
- Healthcare Data Science Center, Konyang University Hospital, Daejeon, Republic of Korea
| | - Inseok Ko
- Healthcare Data Science Center, Konyang University Hospital, Daejeon, Republic of Korea
| | - Seon Choe
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Ju Han Kim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| |
Collapse
|
9
|
Lee JY, Lee YS, Kim DH, Lee HS, Yang BR, Kim MG. The Use of Social Media in Detecting Drug Safety-Related New Black Box Warnings, Labeling Changes, or Withdrawals: Scoping Review. JMIR Public Health Surveill 2021; 7:e30137. [PMID: 34185021 PMCID: PMC8277336 DOI: 10.2196/30137] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Revised: 05/22/2021] [Accepted: 05/30/2021] [Indexed: 01/05/2023] Open
Abstract
Background Social media has become a new source for obtaining real-world data on adverse drug reactions. Many studies have investigated the use of social media to detect early signals of adverse drug reactions. However, the trustworthiness of signals derived from social media is questionable. To confirm this, a confirmatory study with a positive control (eg, new black box warnings, labeling changes, or withdrawals) is required. Objective This study aimed to evaluate the use of social media in detecting new black box warnings, labeling changes, or withdrawals in advance. Methods This scoping review adhered to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews checklist. A researcher searched PubMed and EMBASE in January 2021. Original studies analyzing black box warnings, labeling changes, or withdrawals from social media were selected, and the results of the studies were summarized. Results A total of 14 studies were included in this scoping review. Most studies (8/14, 57.1%%) collected data from a single source, and 10 (71.4%) used specialized health care social networks and forums. The analytical methods used in these studies varied considerably. Three studies (21.4%) manually annotated posts, while 5 (35.7%) adopted machine learning algorithms. Nine studies (64.2%) concluded that social media could detect signals 3 months to 9 years before action from regulatory authorities. Most of these studies (8/9, 88.9%) were conducted on specialized health care social networks and forums. On the contrary, 5 (35.7%) studies yielded modest or negative results. Of these, 2 (40%) used generic social networking sites, 2 (40%) used specialized health care networks and forums, and 1 (20%) used both generic social networking sites and specialized health care social networks and forums. The most recently published study recommends not using social media for pharmacovigilance. Several challenges remain in using social media for pharmacovigilance regarding coverage, data quality, and analytic processing. Conclusions Social media, along with conventional pharmacovigilance measures, can be used to detect signals associated with new black box warnings, labeling changes, or withdrawals. Several challenges remain; however, social media will be useful for signal detection of frequently mentioned drugs in specialized health care social networks and forums. Further studies are required to advance natural language processing and mine real-world data on social media.
Collapse
Affiliation(s)
- Jae-Young Lee
- College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea
| | - Yae-Seul Lee
- College of Pharmacy, Ewha Womans University, Seoul, Republic of Korea
| | - Dong Hyun Kim
- College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea
| | - Han Sol Lee
- College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea
| | - Bo Ram Yang
- College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea
| | - Myeong Gyu Kim
- College of Pharmacy, Ewha Womans University, Seoul, Republic of Korea.,Graduate School of Pharmaceutical Sciences, Ewha Womans University, Seoul, Republic of Korea
| |
Collapse
|
10
|
Joshi P, Vedhanayagam M, Ramesh R. An Ensembled SVM Based Approach for Predicting Adverse Drug Reactions. Curr Bioinform 2021. [DOI: 10.2174/1574893615999200707141420] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
Abstract
Background:
Preventing adverse drug reactions (ADRs) is imperative for the safety of
the people. The problem of under-reporting the ADRs has been prevalent across the world, making it
difficult to develop the prediction models, which are unbiased. As a result, most of the models are
skewed to the negative samples leading to high accuracy but poor performance in other metrics such
as precision, recall, F1 score, and AUROC score.
Objective:
In this work, we have proposed a novel way of predicting the ADRs by balancing the dataset.
Method:
The whole data set has been partitioned into balanced smaller data sets. SVMs with
optimal kernel have been learned using each of the balanced data sets and the prediction of given
ADR for the given drug has been obtained by voting from the ensembled optimal SVMs learned.
Results:
We have found that results are encouraging and comparable with the competing methods in
the literature and obtained the average sensitivity of 0.97 for all the ADRs. The model has been
interpreted and explained with SHAP values by various plots.
Conclusion:
A novel way of predicting ADRs by balancing the dataset has been proposed thereby
reducing the effect of unbalanced datasets.
Collapse
Affiliation(s)
- Pratik Joshi
- Department of Computer Science and Engineering, IIITDM Kancheepuram, Chennai, India
| | | | | |
Collapse
|
11
|
Goldman A, Bomze D, Dankner R, Fourey D, Boursi B, Arad M, Maor E. Cardiovascular Toxicities of Antiangiogenic Tyrosine Kinase Inhibitors: A Retrospective, Pharmacovigilance Study. Target Oncol 2021; 16:471-483. [PMID: 33970401 DOI: 10.1007/s11523-021-00817-2] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/23/2021] [Indexed: 12/19/2022]
Abstract
BACKGROUND Vascular endothelial growth factor receptor (VEGFR) tyrosine kinase inhibitors (TKIs) are an essential therapeutic option in the management of various solid tumors, particularly renal cell carcinoma (RCC). However, post-marketing data regarding their potential cardiovascular toxicities are scant. OBJECTIVE To identify and characterize cardiovascular adverse events (CVAEs) of VEGFR-TKIs indicated for RCC. PATIENTS AND METHODS Disproportionality analysis of the US Food and Drug Administration adverse event reporting system (July 2014-December 2019) using the reporting odds ratio (ROR) and the lower bound of the Information component (IC) 95% credibility interval (IC025 > 0 is significant). RESULTS We identified 51,836 adverse event reports of sunitinib, pazopanib, axitinib, cabozantinib, and lenvatinib in the full database [36% women; median age 65 years (range 57-73)]. CVAEs accounted for 11,784 (23%) of the reports, with hypertension [n = 5548 (11%), ROR = 6.55 (95% CI 6.37-6.74), IC025 = 2.48] and hemorrhages [n = 3710 (7.2%), ROR = 1.28 (1.24-1.32), IC025 = 0.28] being the most frequent types. Additional CVAEs were over-reported with VEGFR-TKIs treatment, including aortic dissection [n = 61 (0.1%), ROR = 3.50 (2.71-4.51)], pericardial diseases [n = 173 (0.3%), ROR = 1.98 (1.70-2.30)], cardiomyopathy [n = 61 (0.1%), ROR = 1.89 (1.47-2.43)], heart failure [n = 868 (1.7%), ROR = 1.35 (1.26-1.44)], and venous thromboembolism [n = 604 (1.2%), ROR = 1.33 (1.23-1.45), all IC025 > 0]. The major pericardial disorder was non-malignant pericardial effusion [n = 134 (77%)]. Aortic dissections were also over-reported in patients without concomitant elevated blood pressure [ROR = 2.68 (1.97-3.63), IC025 = 0.91]. Finally, CVAEs were reported more often following lenvatinib and sunitinib treatment compared to other VEGFR-TKIs. CONCLUSIONS In post-marketing surveillance data, VEGFR-TKIs are associated with increased reporting of various CVAEs, including pericardial diseases, particularly non-malignant pericardial effusion, and aortic dissections. Moreover, VEGFR-TKIs differ in their CVAE reporting patterns. Clinicians should be conscious of these findings in the care of VEGFR-TKIs recipients.
Collapse
Affiliation(s)
- Adam Goldman
- Leviev Heart Center, Sheba Medical Center, Ramat Gan, Israel
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - David Bomze
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Rachel Dankner
- Unit for Cardiovascular Epidemiology, The Gertner Institute for Epidemiology and Health Policy Research, Sheba Medical Center, Ramat Gan, Israel
- Department of Epidemiology and Preventive Medicine, School of Public Health, Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Dana Fourey
- Leviev Heart Center, Sheba Medical Center, Ramat Gan, Israel
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Ben Boursi
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
- Department of Oncology, Sheba Medical Center, Ramat Gan, Israel
- Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, PA, USA
| | - Michael Arad
- Leviev Heart Center, Sheba Medical Center, Ramat Gan, Israel
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Elad Maor
- Leviev Heart Center, Sheba Medical Center, Ramat Gan, Israel.
- School of Medicine, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel.
| |
Collapse
|
12
|
Ding X, Cohen T. Retrofitting Vector Representations of Adverse Event Reporting Data to Structured Knowledge to Improve Pharmacovigilance Signal Detection. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2021; 2020:383-392. [PMID: 33936411 PMCID: PMC8075473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Adverse drug events (ADE) are prevalent and costly. Clinical trials are constrained in their ability to identify potential ADEs, motivating the development of spontaneous reporting systems for post-market surveillance. Statistical methods provide a convenient way to detect signals from these reports but have limitations in leveraging relationships between drugs and ADEs given their discrete count-based nature. A previously proposed method, aer2vec, generates distributed vector representations of ADE report entities that capture patterns of similarity but cannot utilize lexical knowledge. We address this limitation by retrofitting aer2vec drug embeddings to knowledge from RxNorm and developing a novel retrofitting variant using vector rescaling to preserve magnitude. When evaluated in the context of a pharmacovigilance signal detection task, aer2vec with retrofitting consistently outperforms disproportionality metrics when trained on minimally preprocessed data. Retrofitting with rescaling results in further improvements in the larger and more challenging of two pharmacovigilance reference sets used for evaluation.
Collapse
Affiliation(s)
- Xiruo Ding
- University of Washington, Seattle, WA, USA
| | | |
Collapse
|
13
|
Bae JH, Baek YH, Lee JE, Song I, Lee JH, Shin JY. Machine Learning for Detection of Safety Signals From Spontaneous Reporting System Data: Example of Nivolumab and Docetaxel. Front Pharmacol 2021; 11:602365. [PMID: 33628176 PMCID: PMC7898680 DOI: 10.3389/fphar.2020.602365] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2020] [Accepted: 11/02/2020] [Indexed: 12/14/2022] Open
Abstract
Introduction: Various methods have been implemented to detect adverse drug reaction (ADR) signals. However, the applicability of machine learning methods has not yet been fully evaluated. Objective: To evaluate the feasibility of machine learning algorithms in detecting ADR signals of nivolumab and docetaxel, new and old anticancer agents. Methods: We conducted a safety surveillance study of nivolumab and docetaxel using the Korea national spontaneous reporting database from 2009 to 2018. We constructed a novel input dataset for each study drug comprised of known ADRs that were listed in the drug labels and unknown ADRs. Given the known ADRs, we trained machine learning algorithms and evaluated predictive performance in generating safety signals of machine learning algorithms (gradient boosting machine [GBM] and random forest [RF]) compared with traditional disproportionality analysis methods (reporting odds ratio [ROR] and information component [IC]) by using the area under the curve (AUC). Each method then was implemented to detect new safety signals from the unknown ADR datasets. Results: Of all methods implemented, GBM achieved the best average predictive performance (AUC: 0.97 and 0.93 for nivolumab and docetaxel). The AUC achieved by each method was 0.95 and 0.92 (RF), 0.55 and 0.51 (ROR), and 0.49 and 0.48 (IC) for respective drug. GBM detected additional 24 and nine signals for nivolumab and 82 and 76 for docetaxel compared to ROR and IC, respectively, from the unknown ADR datasets. Conclusion: Machine learning algorithm based on GBM performed better and detected more new ADR signals than traditional disproportionality analysis methods.
Collapse
Affiliation(s)
- Ji-Hwan Bae
- School of Pharmacy, Sungkyunkwan University, Suwon-si, South Korea
| | - Yeon-Hee Baek
- School of Pharmacy, Sungkyunkwan University, Suwon-si, South Korea
| | - Jeong-Eun Lee
- School of Pharmacy, Sungkyunkwan University, Suwon-si, South Korea
| | - Inmyung Song
- Department of Health Administration, College of Nursing and Health, Kongju National University, Gongju-si, South Korea
| | - Jee-Hyong Lee
- Department of Artificial Intelligence, Sungkyunkwan University, Suwon-si, South Korea
| | - Ju-Young Shin
- School of Pharmacy, Sungkyunkwan University, Suwon-si, South Korea.,Samsung Advanced Institute for Health Sciences and Technology, Sungkyunkwan University, Jongno-gu, South Korea
| |
Collapse
|
14
|
Li Y, Jimeno Yepes A, Xiao C. Combining Social Media and FDA Adverse Event Reporting System to Detect Adverse Drug Reactions. Drug Saf 2020; 43:893-903. [PMID: 32385840 PMCID: PMC7434724 DOI: 10.1007/s40264-020-00943-2] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
INTRODUCTION Adverse drug reactions (ADRs) are unintended reactions caused by a drug or combination of drugs taken by a patient. The current safety surveillance system relies on spontaneous reporting systems (SRSs) and more recently on observational health data; however, ADR detection may be delayed and lack geographic diversity. The broad scope of social media conversations, such as those on Twitter, can include health-related topics. Consequently, these data could be used to detect potentially novel ADRs with less latency. Although research regarding ADR detection using social media has made progress, findings are based on single information sources, and no study has yet integrated drug safety evidence from both an SRS and Twitter. OBJECTIVE The aim of this study was to combine signals from an SRS and Twitter to facilitate the detection of safety signals and compare the performance of the combined system with signals generated by individual data sources. METHODS We extracted potential drug-ADR posts from Twitter, used Monte Carlo expectation maximization to generate drug safety signals from both the US FDA Adverse Event Reporting System and posts from Twitter, and then integrated these signals using a Bayesian hierarchical model. The results from the integrated system and two individual sources were evaluated using a reference standard derived from drug labels. Area under the receiver operating characteristics curve (AUC) was computed to measure performance. RESULTS We observed a significant improvement in the AUC of the combined system when comparing it with Twitter alone, and no improvement when comparing with the SRS alone. The AUCs ranged from 0.587 to 0.637 for the combined SRS and Twitter, from 0.525 to 0.534 for Twitter alone, and from 0.612 to 0.642 for the SRS alone. The results varied because different preprocessing procedures were applied to Twitter. CONCLUSION The accuracy of signal detection using social media can be improved by combining signals with those from SRSs. However, the combined system cannot achieve better AUC performance than data from FAERS alone, which may indicate that Twitter data are not ready to be integrated into a purely data-driven combination system.
Collapse
Affiliation(s)
- Ying Li
- Center for Computational Health, IBM Thomas J. Watson Research Center, 1101 Kitchawan Rd, Yorktown Heights, NY, 10598, USA.
| | | | - Cao Xiao
- Analytics Center of Excellence, IQVIA, Cambridge, MA, USA
| |
Collapse
|
15
|
van Stekelenborg J, Ellenius J, Maskell S, Bergvall T, Caster O, Dasgupta N, Dietrich J, Gama S, Lewis D, Newbould V, Brosch S, Pierce CE, Powell G, Ptaszyńska-Neophytou A, Wiśniewski AFZ, Tregunno P, Norén GN, Pirmohamed M. Recommendations for the Use of Social Media in Pharmacovigilance: Lessons from IMI WEB-RADR. Drug Saf 2019; 42:1393-1407. [PMID: 31446567 PMCID: PMC6858385 DOI: 10.1007/s40264-019-00858-7] [Citation(s) in RCA: 45] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022]
Abstract
Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR project has explored the value of social media (i.e., information exchanged through the internet, typically via online social networks) for identifying adverse events as well as for safety signal detection. Many patients and clinicians have taken to social media to discuss their positive and negative experiences of medications, creating a source of publicly available information that has the potential to provide insights into medicinal product safety concerns. The WEB-RADR project has developed a collaborative English language workspace for visualising and analysing social media data for a number of medicinal products. Further, novel text and data mining methods for social media analysis have been developed and evaluated. From this original research, several recommendations are presented with supporting rationale and consideration of the limitations. Recommendations for further research that extend beyond the scope of the current project are also presented.
Collapse
Affiliation(s)
| | | | - Simon Maskell
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
- Department of Computer Science, University of Liverpool, Liverpool, L69 3BX, UK
| | | | - Ola Caster
- Uppsala Monitoring Centre, Uppsala, Sweden
| | - Nabarun Dasgupta
- Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
| | | | - Sara Gama
- Chief Medical Office and Patient Safety, Novartis Global Drug Development, Novartis Pharma Basel, Basel, Switzerland
| | - David Lewis
- Chief Medical Office and Patient Safety, Novartis Global Drug Development, Novartis Pharma Basel, Basel, Switzerland
- Dept of Pharmacy, Pharmacology and Postgraduate Medicine, University of Hertfordshire, Hatfield, UK
| | - Victoria Newbould
- Pharmacovigilance Department, Inspections and Human Medicines Pharmacovigilance Division, European Medicines Agency (EMA), Amsterdam, The Netherlands
| | - Sabine Brosch
- Pharmacovigilance Department, Inspections and Human Medicines Pharmacovigilance Division, European Medicines Agency (EMA), Amsterdam, The Netherlands
| | - Carrie E Pierce
- Booz Allen Hamilton (formerly Epidemico, Inc.), Boston, MA, USA
| | - Gregory Powell
- GlaxoSmithKline, Global Clinical Safety and Pharmacovigilance, RTP, Research Triangle Park, NC, 27709, USA
| | - Alicia Ptaszyńska-Neophytou
- Vigilance, Intelligence and Research Group, Medicines and Healthcare products Regulatory Agency (MHRA), 10 South Colonnade, Canary Wharf, London, E14 4PU, UK
| | - Antoni F Z Wiśniewski
- AstraZeneca, Patient Safety, Office of the Chief Medical Officer, Cambridge, UK, Granta Park, Cambridge, CB21 6GH, UK
| | - Phil Tregunno
- Vigilance, Intelligence and Research Group, Medicines and Healthcare products Regulatory Agency (MHRA), 10 South Colonnade, Canary Wharf, London, E14 4PU, UK
| | | | - Munir Pirmohamed
- Department of Molecular and Clinical Pharmacology, University of Liverpool, Liverpool, L69 3GL, UK
- Royal Liverpool and Broadgreen University Hospital NHS Trust, Liverpool, L7 8XP, UK
| |
Collapse
|
16
|
Caster O, Dietrich J, Kürzinger ML, Lerch M, Maskell S, Norén GN, Tcherny-Lessenot S, Vroman B, Wisniewski A, van Stekelenborg J. Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project. Drug Saf 2018; 41:1355-1369. [PMID: 30043385 PMCID: PMC6223695 DOI: 10.1007/s40264-018-0699-2] [Citation(s) in RCA: 35] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
INTRODUCTION AND OBJECTIVE Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64-0.69 and 0.55-0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities.
Collapse
Affiliation(s)
- Ola Caster
- Uppsala Monitoring Centre, Box 1051, Uppsala, 75140, Sweden.
| | | | | | | | | | - G Niklas Norén
- Uppsala Monitoring Centre, Box 1051, Uppsala, 75140, Sweden
| | | | | | | | | |
Collapse
|
17
|
Comment on "Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project". Drug Saf 2018; 41:1371-1373. [PMID: 30341678 DOI: 10.1007/s40264-018-0747-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
|
18
|
Mower J, Subramanian D, Cohen T. Learning predictive models of drug side-effect relationships from distributed representations of literature-derived semantic predications. J Am Med Inform Assoc 2018; 25:1339-1350. [PMID: 30010902 PMCID: PMC6454491 DOI: 10.1093/jamia/ocy077] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Revised: 04/23/2018] [Accepted: 06/05/2018] [Indexed: 02/01/2023] Open
Abstract
Objective The aim of this work is to leverage relational information extracted from biomedical literature using a novel synthesis of unsupervised pretraining, representational composition, and supervised machine learning for drug safety monitoring. Methods Using ≈80 million concept-relationship-concept triples extracted from the literature using the SemRep Natural Language Processing system, distributed vector representations (embeddings) were generated for concepts as functions of their relationships utilizing two unsupervised representational approaches. Embeddings for drugs and side effects of interest from two widely used reference standards were then composed to generate embeddings of drug/side-effect pairs, which were used as input for supervised machine learning. This methodology was developed and evaluated using cross-validation strategies and compared to contemporary approaches. To qualitatively assess generalization, models trained on the Observational Medical Outcomes Partnership (OMOP) drug/side-effect reference set were evaluated against a list of ≈1100 drugs from an online database. Results The employed method improved performance over previous approaches. Cross-validation results advance the state of the art (AUC 0.96; F1 0.90 and AUC 0.95; F1 0.84 across the two sets), outperforming methods utilizing literature and/or spontaneous reporting system data. Examination of predictions for unseen drug/side-effect pairs indicates the ability of these methods to generalize, with over tenfold label support enrichment in the top 100 predictions versus the bottom 100 predictions. Discussion and Conclusion Our methods can assist the pharmacovigilance process using information from the biomedical literature. Unsupervised pretraining generates a rich relationship-based representational foundation for machine learning techniques to classify drugs in the context of a putative side effect, given known examples.
Collapse
Affiliation(s)
- Justin Mower
- Baylor College of Medicine, Quantitative and Computational Biosciences, Houston, Texas, USA
| | | | - Trevor Cohen
- School of Biomedical Informatics, University of Texas Health Science Center Houston, Texas, USA
| |
Collapse
|
19
|
Hoang T, Liu J, Roughead E, Pratt N, Li J. Supervised signal detection for adverse drug reactions in medication dispensing data. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2018; 161:25-38. [PMID: 29852965 DOI: 10.1016/j.cmpb.2018.03.021] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/24/2017] [Revised: 03/12/2018] [Accepted: 03/20/2018] [Indexed: 06/08/2023]
Abstract
MOTIVATION Adverse drug reactions (ADRs) are one of the leading causes of morbidity and mortality and thus should be detected early to reduce consequences on health outcomes. Medication dispensing data are comprehensive sources of information about medicine uses that can be utilized for the signal detection of ADRs. Sequence symmetry analysis (SSA) has been employed in previous studies to detect signals of ADRs from medication dispensing data, but it has a moderate sensitivity and tends to miss some ADR signals. With successful applications in various areas, supervised machine learning (SML) methods are promising in detecting ADR signals. Gold standards of known ADRs and non- ADRs from previous studies create opportunities to take into account additional domain knowledge to improve ADR signal detection with SML. OBJECTIVE We assess the utility of SML as a signal detection tool for ADRs in medication dispensing data with the consideration of domain knowledge from DrugBank and MedDRA. We compare the best performing SML method with SSA. METHODS We model the ADR signal detection problem as a supervised machine learning problem by linking medication dispensing data with domain knowledge bases. Suspected ADR signals are extracted from the Australian Pharmaceutical Benefit Scheme (PBS) medication dispensing data from 2013 to 2016. We construct predictive features for each signal candidate based on its occurrences in medication dispensing data as well as its pharmacological properties. Pharmaceutical knowledge bases including DrugBank and MedDRA are employed to provide pharmacological features for a signal candidate. Given a gold standard of known ADRs and non-ADRs, SML learns to differentiate between known ADRs and non-ADRs based on their combined predictive features from linked sources, and then predicts whether a new case is a potential ADR signal. RESULTS We evaluate the performance of six widely used SML methods with two gold standards of known ADRs and non-ADRs from previous studies. On average, gradient boosting classifier achieves the sensitivity of 77%, specificity of 81%, positive predictive value of 76%, negative predictive value of 82%, area under precision-recall curve of 81%, and area under receiver operating characteristic curve of 82%, most of which are higher than in other SML methods. In particular, gradient boosting classifier has 21% higher sensitivity than and comparable specificity with SSA. Furthermore, gradient boosting classifier detects 10% more unknown potential ADR signals than SSA. CONCLUSIONS Our study demonstrates that gradient boosting classifier is a promising supervised signal detection tool for ADRs in medication dispensing data to complement SSA.
Collapse
Affiliation(s)
- Tao Hoang
- School of Information Technology and Mathematical Sciences, University of South Australia, Mawson Lakes Boulevard, South Australia 5095, Australia.
| | - Jixue Liu
- School of Information Technology and Mathematical Sciences, University of South Australia, Mawson Lakes Boulevard, South Australia 5095, Australia
| | - Elizabeth Roughead
- School of Pharmacy and Medical Sciences, University of South Australia, City East Campus, North Terrace Adelaide, South Australia 5001, Australia
| | - Nicole Pratt
- School of Pharmacy and Medical Sciences, University of South Australia, City East Campus, North Terrace Adelaide, South Australia 5001, Australia
| | - Jiuyong Li
- School of Information Technology and Mathematical Sciences, University of South Australia, Mawson Lakes Boulevard, South Australia 5095, Australia
| |
Collapse
|
20
|
Hum RS, Kleinberg S. Replicability, Reproducibility, and Agent-based Simulation of Interventions. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2018; 2017:959-968. [PMID: 29854163 PMCID: PMC5977631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.
Collapse
|
21
|
Harpaz R, DuMouchel W, Schuemie M, Bodenreider O, Friedman C, Horvitz E, Ripple A, Sorbello A, White RW, Winnenburg R, Shah NH. Toward multimodal signal detection of adverse drug reactions. J Biomed Inform 2017; 76:41-49. [PMID: 29081385 PMCID: PMC8502488 DOI: 10.1016/j.jbi.2017.10.013] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2017] [Revised: 10/14/2017] [Accepted: 10/24/2017] [Indexed: 11/27/2022]
Abstract
OBJECTIVE Improving mechanisms to detect adverse drug reactions (ADRs) is key to strengthening post-marketing drug safety surveillance. Signal detection is presently unimodal, relying on a single information source. Multimodal signal detection is based on jointly analyzing multiple information sources. Building on, and expanding the work done in prior studies, the aim of the article is to further research on multimodal signal detection, explore its potential benefits, and propose methods for its construction and evaluation. MATERIAL AND METHODS Four data sources are investigated; FDA's adverse event reporting system, insurance claims, the MEDLINE citation database, and the logs of major Web search engines. Published methods are used to generate and combine signals from each data source. Two distinct reference benchmarks corresponding to well-established and recently labeled ADRs respectively are used to evaluate the performance of multimodal signal detection in terms of area under the ROC curve (AUC) and lead-time-to-detection, with the latter relative to labeling revision dates. RESULTS Limited to our reference benchmarks, multimodal signal detection provides AUC improvements ranging from 0.04 to 0.09 based on a widely used evaluation benchmark, and a comparative added lead-time of 7-22 months relative to labeling revision dates from a time-indexed benchmark. CONCLUSIONS The results support the notion that utilizing and jointly analyzing multiple data sources may lead to improved signal detection. Given certain data and benchmark limitations, the early stage of development, and the complexity of ADRs, it is currently not possible to make definitive statements about the ultimate utility of the concept. Continued development of multimodal signal detection requires a deeper understanding the data sources used, additional benchmarks, and further research on methods to generate and synthesize signals.
Collapse
Affiliation(s)
- Rave Harpaz
- Oracle Health Sciences, Bedford, MA, United States.
| | | | | | | | | | | | - Anna Ripple
- National Library of Medicine, NIH, Bethesda, MD, United States
| | | | | | | | - Nigam H Shah
- Stanford University, Stanford, CA, United States
| |
Collapse
|
22
|
Sorbello A, Ripple A, Tonning J, Munoz M, Hasan R, Ly T, Francis H, Bodenreider O. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8:291-305. [PMID: 28326432 PMCID: PMC5373771 DOI: 10.4338/aci-2016-11-ra-0188] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2016] [Accepted: 01/14/2017] [Indexed: 11/23/2022] Open
Abstract
Objectives We seek to develop a prototype software analytical tool to augment FDA regulatory reviewers’ capacity to harness scientific literature reports in PubMed/MEDLINE for pharmacovigilance and adverse drug event (ADE) safety signal detection. We also aim to gather feedback through usability testing to assess design, performance, and user satisfaction with the tool. Methods A prototype, open source, web-based, software analytical tool generated statistical disproportionality data mining signal scores and dynamic visual analytics for ADE safety signal detection and management. We leveraged Medical Subject Heading (MeSH) indexing terms assigned to published citations in PubMed/MEDLINE to generate candidate drug-adverse event pairs for quantitative data mining. Six FDA regulatory reviewers participated in usability testing by employing the tool as part of their ongoing real-life pharmacovigilance activities to provide subjective feedback on its practical impact, added value, and fitness for use. Results All usability test participants cited the tool’s ease of learning, ease of use, and generation of quantitative ADE safety signals, some of which corresponded to known established adverse drug reactions. Potential concerns included the comparability of the tool’s automated literature search relative to a manual ‘all fields’ PubMed search, missing drugs and adverse event terms, interpretation of signal scores, and integration with existing computer-based analytical tools. Conclusions Usability testing demonstrated that this novel tool can automate the detection of ADE safety signals from published literature reports. Various mitigation strategies are described to foster improvements in design, productivity, and end user satisfaction.
Collapse
Affiliation(s)
- Alfred Sorbello
- Alfred Sorbello, DO, MPH, US Food and Drug Administration, Center for Drug Evaluation and Research, Office of Translational Sciences, 10903 New Hampshire Avenue, Silver Spring, MD 20993-0002 USA,
| | | | | | | | | | | | | | | |
Collapse
|
23
|
Generalized enrichment analysis improves the detection of adverse drug events from the biomedical literature. BMC Bioinformatics 2016; 17:250. [PMID: 27333889 PMCID: PMC4918084 DOI: 10.1186/s12859-016-1080-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2015] [Accepted: 05/11/2016] [Indexed: 01/12/2023] Open
Abstract
Background Identification of associations between marketed drugs and adverse events from the biomedical literature assists drug safety monitoring efforts. Assessing the significance of such literature-derived associations and determining the granularity at which they should be captured remains a challenge. Here, we assess how defining a selection of adverse event terms from MeSH, based on information content, can improve the detection of adverse events for drugs and drug classes. Results We analyze a set of 105,354 candidate drug adverse event pairs extracted from article indexes in MEDLINE. First, we harmonize extracted adverse event terms by aggregating them into higher-level MeSH terms based on the terms’ information content. Then, we determine statistical enrichment of adverse events associated with drug and drug classes using a conditional hypergeometric test that adjusts for dependencies among associated terms. We compare our results with methods based on disproportionality analysis (proportional reporting ratio, PRR) and quantify the improvement in signal detection with our generalized enrichment analysis (GEA) approach using a gold standard of drug-adverse event associations spanning 174 drugs and four events. For single drugs, the best GEA method (Precision: .92/Recall: .71/F1-measure: .80) outperforms the best PRR based method (.69/.69/.69) on all four adverse event outcomes in our gold standard. For drug classes, our GEA performs similarly (.85/.69/.74) when increasing the level of abstraction for adverse event terms. Finally, on examining the 1609 individual drugs in our MEDLINE set, which map to chemical substances in ATC, we find signals for 1379 drugs (10,122 unique adverse event associations) on applying GEA with p < 0.005. Conclusions We present an approach based on generalized enrichment analysis that can be used to detect associations between drugs, drug classes and adverse events at a given level of granularity, at the same time correcting for known dependencies among events. Our study demonstrates the use of GEA, and the importance of choosing appropriate abstraction levels to complement current drug safety methods. We provide an R package for exploration of alternative abstraction levels of adverse event terms based on information content. Electronic supplementary material The online version of this article (doi:10.1186/s12859-016-1080-z) contains supplementary material, which is available to authorized users.
Collapse
|
24
|
Hauben M, Aronson JK, Ferner RE. Evidence of Misclassification of Drug–Event Associations Classified as Gold Standard ‘Negative Controls’ by the Observational Medical Outcomes Partnership (OMOP). Drug Saf 2016; 39:421-32. [DOI: 10.1007/s40264-016-0392-2] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
25
|
Leveraging MEDLINE indexing for pharmacovigilance - Inherent limitations and mitigation strategies. J Biomed Inform 2015; 57:425-35. [PMID: 26342964 DOI: 10.1016/j.jbi.2015.08.022] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2014] [Revised: 07/30/2015] [Accepted: 08/25/2015] [Indexed: 11/24/2022]
Abstract
BACKGROUND Traditional approaches to pharmacovigilance center on the signal detection from spontaneous reports, e.g., the U.S. Food and Drug Administration (FDA) adverse event reporting system (FAERS). In order to enrich the scientific evidence and enhance the detection of emerging adverse drug events that can lead to unintended harmful outcomes, pharmacovigilance activities need to evolve to encompass novel complementary data streams, for example the biomedical literature available through MEDLINE. OBJECTIVES (1) To review how the characteristics of MEDLINE indexing influence the identification of adverse drug events (ADEs); (2) to leverage this knowledge to inform the design of a system for extracting ADEs from MEDLINE indexing; and (3) to assess the specific contribution of some characteristics of MEDLINE indexing to the performance of this system. METHODS We analyze the characteristics of MEDLINE indexing. We integrate three specific characteristics into the design of a system for extracting ADEs from MEDLINE indexing. We experimentally assess the specific contribution of these characteristics over a baseline system based on co-occurrence between drug descriptors qualified by adverse effects and disease descriptors qualified by chemically induced. RESULTS Our system extracted 405,300 ADEs from 366,120 MEDLINE articles. The baseline system accounts for 297,093 ADEs (73%). 85,318 ADEs (21%) can be extracted only after integrating specific pre-coordinated MeSH descriptors and additional qualifiers. 22,889 ADEs (6%) can be extracted only after considering indirect links between the drug of interest and the descriptor that bears the ADE context. CONCLUSIONS In this paper, we demonstrate significant improvement over a baseline approach to identifying ADEs from MEDLINE indexing, which mitigates some of the inherent limitations of MEDLINE indexing for pharmacovigilance. ADEs extracted from MEDLINE indexing are complementary to, not a replacement for, other sources.
Collapse
|
26
|
Callahan A, Pernek I, Stiglic G, Leskovec J, Strasberg HR, Shah NH. Analyzing Information Seeking and Drug-Safety Alert Response by Health Care Professionals as New Methods for Surveillance. J Med Internet Res 2015; 17:e204. [PMID: 26293444 PMCID: PMC4642796 DOI: 10.2196/jmir.4427] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2015] [Accepted: 07/24/2015] [Indexed: 11/13/2022] Open
Abstract
Background Patterns in general consumer online search logs have been used to monitor health conditions and to predict health-related activities, but the multiple contexts within which consumers perform online searches make significant associations difficult to interpret. Physician information-seeking behavior has typically been analyzed through survey-based approaches and literature reviews. Activity logs from health care professionals using online medical information resources are thus a valuable yet relatively untapped resource for large-scale medical surveillance. Objective To analyze health care professionals’ information-seeking behavior and assess the feasibility of measuring drug-safety alert response from the usage logs of an online medical information resource. Methods Using two years (2011-2012) of usage logs from UpToDate, we measured the volume of searches related to medical conditions with significant burden in the United States, as well as the seasonal distribution of those searches. We quantified the relationship between searches and resulting page views. Using a large collection of online mainstream media articles and Web log posts we also characterized the uptake of a Food and Drug Administration (FDA) alert via changes in UpToDate search activity compared with general online media activity related to the subject of the alert. Results Diseases and symptoms dominate UpToDate searches. Some searches result in page views of only short duration, while others consistently result in longer-than-average page views. The response to an FDA alert for Celexa, characterized by a change in UpToDate search activity, differed considerably from general online media activity. Changes in search activity appeared later and persisted longer in UpToDate logs. The volume of searches and page view durations related to Celexa before the alert also differed from those after the alert. Conclusions Understanding the information-seeking behavior associated with online evidence sources can offer insight into the information needs of health professionals and enable large-scale medical surveillance. Our Web log mining approach has the potential to monitor responses to FDA alerts at a national level. Our findings can also inform the design and content of evidence-based medical information resources such as UpToDate.
Collapse
Affiliation(s)
- Alison Callahan
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, United States.
| | | | | | | | | | | |
Collapse
|
27
|
Wang G, Jung K, Winnenburg R, Shah NH. A method for systematic discovery of adverse drug events from clinical notes. J Am Med Inform Assoc 2015; 22:1196-204. [PMID: 26232442 DOI: 10.1093/jamia/ocv102] [Citation(s) in RCA: 42] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2015] [Accepted: 06/16/2015] [Indexed: 01/11/2023] Open
Abstract
OBJECTIVE Adverse drug events (ADEs) are undesired harmful effects resulting from use of a medication, and occur in 30% of hospitalized patients. The authors have developed a data-mining method for systematic, automated detection of ADEs from electronic medical records. MATERIALS AND METHODS This method uses the text from 9.5 million clinical notes, along with prior knowledge of drug usages and known ADEs, as inputs. These inputs are further processed into statistics used by a discriminative classifier which outputs the probability that a given drug-disorder pair represents a valid ADE association. Putative ADEs identified by the classifier are further filtered for positive support in 2 independent, complementary data sources. The authors evaluate this method by assessing support for the predictions in other curated data sources, including a manually curated, time-indexed reference standard of label change events. RESULTS This method uses a classifier that achieves an area under the curve of 0.94 on a held out test set. The classifier is used on 2,362,950 possible drug-disorder pairs comprised of 1602 unique drugs and 1475 unique disorders for which we had data, resulting in 240 high-confidence, well-supported drug-AE associations. Eighty-seven of them (36%) are supported in at least one of the resources that have information that was not available to the classifier. CONCLUSION This method demonstrates the feasibility of systematic post-marketing surveillance for ADEs using electronic medical records, a key component of the learning healthcare system.
Collapse
Affiliation(s)
- Guan Wang
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Kenneth Jung
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Rainer Winnenburg
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Nigam H Shah
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| |
Collapse
|
28
|
Comment on: "Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance". Drug Saf 2015; 38:113-4. [PMID: 25432779 DOI: 10.1007/s40264-014-0245-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|
29
|
Osokogu OU, Fregonese F, Ferrajolo C, Verhamme K, de Bie S, 't Jong G, Catapano M, Weibel D, Kaguelidou F, Bramer WM, Hsia Y, Wong ICK, Gazarian M, Bonhoeffer J, Sturkenboom M. Pediatric drug safety signal detection: a new drug-event reference set for performance testing of data-mining methods and systems. Drug Saf 2015; 38:207-17. [PMID: 25663078 PMCID: PMC4328124 DOI: 10.1007/s40264-015-0265-0] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
BACKGROUND Better evidence regarding drug safety in the pediatric population might be generated from existing data sources such as spontaneous reporting systems and electronic healthcare records. The Global Research in Paediatrics (GRiP)-Network of Excellence aims to develop pediatric-specific methods that can be applied to these data sources. A reference set of positive and negative drug-event associations is required. OBJECTIVE The aim of this study was to develop a pediatric-specific reference set of positive and negative drug-event associations. METHODS Considering user patterns and expert opinion, 16 drugs that are used in individuals aged 0-18 years were selected and evaluated against 16 events, regarded as important safety outcomes. A cross-table of unique drug-event pairs was created. Each pair was classified as potential positive or negative control based on information from the drug's Summary of Product Characteristics and Micromedex. If both information sources consistently listed the event as an adverse event, the combination was reviewed as potential positive control. If both did not, the combination was evaluated as potential negative control. Further evaluation was based on published literature. RESULTS Selected drugs include ibuprofen, flucloxacillin, domperidone, methylphenidate, montelukast, quinine, and cyproterone/ethinylestradiol. Selected events include bullous eruption, aplastic anemia, ventricular arrhythmia, sudden death, acute kidney injury, psychosis, and seizure. Altogether, 256 unique combinations were reviewed, yielding 37 positive (17 with evidence from the pediatric population and 20 with evidence from adults only) and 90 negative control pairs, with the remainder being unclassifiable. CONCLUSION We propose a drug-event reference set that can be used to compare different signal detection methods in the pediatric population.
Collapse
Affiliation(s)
- Osemeke U Osokogu
- Department of Medical Informatics, Erasmus University Medical Center, PO Box 2040, 3000 CA, Rotterdam, The Netherlands,
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|