1
|
Kontsioti E, Maskell S, Pirmohamed M. Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug-drug interactions. Pharmacoepidemiol Drug Saf 2023; 32:832-844. [PMID: 36916014 PMCID: PMC10947279 DOI: 10.1002/pds.5609] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 02/13/2023] [Accepted: 03/08/2023] [Indexed: 03/16/2023]
Abstract
PURPOSE To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs). METHODS Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. RESULTS For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. CONCLUSIONS The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others.
Collapse
Affiliation(s)
- Elpida Kontsioti
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Simon Maskell
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Munir Pirmohamed
- The Wolfson Center for Personalized Medicine, Center for Drug Safety Science, Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
| |
Collapse
|
2
|
Kontsioti E, Maskell S, Dutta B, Pirmohamed M. A reference set of clinically relevant adverse drug-drug interactions. Sci Data 2022; 9:72. [PMID: 35246559 PMCID: PMC8897500 DOI: 10.1038/s41597-022-01159-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2021] [Accepted: 01/13/2022] [Indexed: 12/03/2022] Open
Abstract
The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms. Measurement(s) | Adverse Event | Technology Type(s) | digital curation | Sample Characteristic - Organism | Homo sapiens |
Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.16681933
Collapse
Affiliation(s)
- Elpida Kontsioti
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK.
| | - Simon Maskell
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, UK
| | - Bhaskar Dutta
- Patient Safety Center of Excellence, Chief Medical Office Organization, AstraZeneca Pharmaceuticals, Gaithersburg, MD, USA
| | - Munir Pirmohamed
- The Wolfson Centre for Personalized Medicine, MRC Centre for Drug Safety Science, Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, United Kingdom
| |
Collapse
|
3
|
Banda JM, Sarraju A, Abbasi F, Parizo J, Pariani M, Ison H, Briskin E, Wand H, Dubois S, Jung K, Myers SA, Rader DJ, Leader JB, Murray MF, Myers KD, Wilemon K, Shah NH, Knowles JW. Finding missed cases of familial hypercholesterolemia in health systems using machine learning. NPJ Digit Med 2019; 2:23. [PMID: 31304370 PMCID: PMC6550268 DOI: 10.1038/s41746-019-0101-5] [Citation(s) in RCA: 47] [Impact Index Per Article: 9.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2018] [Accepted: 03/13/2019] [Indexed: 01/26/2023] Open
Abstract
Familial hypercholesterolemia (FH) is an underdiagnosed dominant genetic condition affecting approximately 0.4% of the population and has up to a 20-fold increased risk of coronary artery disease if untreated. Simple screening strategies have false positive rates greater than 95%. As part of the FH Foundation's FIND FH initiative, we developed a classifier to identify potential FH patients using electronic health record (EHR) data at Stanford Health Care. We trained a random forest classifier using data from known patients (n = 197) and matched non-cases (n = 6590). Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 on a held-out test-set. We evaluated the accuracy of the classifier's predictions by chart review of 100 patients at risk of FH not included in the original dataset. The classifier correctly flagged 84% of patients at the highest probability threshold, with decreasing performance as the threshold lowers. In external validation on 466 FH patients (236 with genetically proven FH) and 5000 matched non-cases from the Geisinger Healthcare System our FH classifier achieved a PPV of 0.85. Our EHR-derived FH classifier is effective in finding candidate patients for further FH screening. Such machine learning guided strategies can lead to effective identification of the highest risk patients for enhanced management strategies.
Collapse
Affiliation(s)
- Juan M. Banda
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA USA
- Department of Computer Science, Georgia State University, Atlanta, GA USA
| | - Ashish Sarraju
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Fahim Abbasi
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Justin Parizo
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Mitchel Pariani
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Hannah Ison
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Elinor Briskin
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Hannah Wand
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
| | - Sebastien Dubois
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA USA
| | - Kenneth Jung
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA USA
| | | | - Daniel J. Rader
- Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA USA
- The FH Foundation, Pasadena, CA USA
| | - Joseph B. Leader
- Geisinger Health System, Genomic Medicine Institute, Forty Fort, PA USA
| | | | - Kelly D. Myers
- Atomo, Inc, Austin, TX USA
- The FH Foundation, Pasadena, CA USA
| | | | - Nigam H. Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA USA
| | - Joshua W. Knowles
- Cardiovascular Medicine and Cardiovascular Institute, Stanford University, Stanford, CA USA
- The FH Foundation, Pasadena, CA USA
- Stanford Diabetes Research Center, Stanford, CA USA
| |
Collapse
|
4
|
Mower J, Subramanian D, Cohen T. Learning predictive models of drug side-effect relationships from distributed representations of literature-derived semantic predications. J Am Med Inform Assoc 2018; 25:1339-1350. [PMID: 30010902 PMCID: PMC6454491 DOI: 10.1093/jamia/ocy077] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Revised: 04/23/2018] [Accepted: 06/05/2018] [Indexed: 02/01/2023] Open
Abstract
Objective The aim of this work is to leverage relational information extracted from biomedical literature using a novel synthesis of unsupervised pretraining, representational composition, and supervised machine learning for drug safety monitoring. Methods Using ≈80 million concept-relationship-concept triples extracted from the literature using the SemRep Natural Language Processing system, distributed vector representations (embeddings) were generated for concepts as functions of their relationships utilizing two unsupervised representational approaches. Embeddings for drugs and side effects of interest from two widely used reference standards were then composed to generate embeddings of drug/side-effect pairs, which were used as input for supervised machine learning. This methodology was developed and evaluated using cross-validation strategies and compared to contemporary approaches. To qualitatively assess generalization, models trained on the Observational Medical Outcomes Partnership (OMOP) drug/side-effect reference set were evaluated against a list of ≈1100 drugs from an online database. Results The employed method improved performance over previous approaches. Cross-validation results advance the state of the art (AUC 0.96; F1 0.90 and AUC 0.95; F1 0.84 across the two sets), outperforming methods utilizing literature and/or spontaneous reporting system data. Examination of predictions for unseen drug/side-effect pairs indicates the ability of these methods to generalize, with over tenfold label support enrichment in the top 100 predictions versus the bottom 100 predictions. Discussion and Conclusion Our methods can assist the pharmacovigilance process using information from the biomedical literature. Unsupervised pretraining generates a rich relationship-based representational foundation for machine learning techniques to classify drugs in the context of a putative side effect, given known examples.
Collapse
Affiliation(s)
- Justin Mower
- Baylor College of Medicine, Quantitative and Computational Biosciences, Houston, Texas, USA
| | | | - Trevor Cohen
- School of Biomedical Informatics, University of Texas Health Science Center Houston, Texas, USA
| |
Collapse
|
5
|
Generalized enrichment analysis improves the detection of adverse drug events from the biomedical literature. BMC Bioinformatics 2016; 17:250. [PMID: 27333889 PMCID: PMC4918084 DOI: 10.1186/s12859-016-1080-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2015] [Accepted: 05/11/2016] [Indexed: 01/12/2023] Open
Abstract
Background Identification of associations between marketed drugs and adverse events from the biomedical literature assists drug safety monitoring efforts. Assessing the significance of such literature-derived associations and determining the granularity at which they should be captured remains a challenge. Here, we assess how defining a selection of adverse event terms from MeSH, based on information content, can improve the detection of adverse events for drugs and drug classes. Results We analyze a set of 105,354 candidate drug adverse event pairs extracted from article indexes in MEDLINE. First, we harmonize extracted adverse event terms by aggregating them into higher-level MeSH terms based on the terms’ information content. Then, we determine statistical enrichment of adverse events associated with drug and drug classes using a conditional hypergeometric test that adjusts for dependencies among associated terms. We compare our results with methods based on disproportionality analysis (proportional reporting ratio, PRR) and quantify the improvement in signal detection with our generalized enrichment analysis (GEA) approach using a gold standard of drug-adverse event associations spanning 174 drugs and four events. For single drugs, the best GEA method (Precision: .92/Recall: .71/F1-measure: .80) outperforms the best PRR based method (.69/.69/.69) on all four adverse event outcomes in our gold standard. For drug classes, our GEA performs similarly (.85/.69/.74) when increasing the level of abstraction for adverse event terms. Finally, on examining the 1609 individual drugs in our MEDLINE set, which map to chemical substances in ATC, we find signals for 1379 drugs (10,122 unique adverse event associations) on applying GEA with p < 0.005. Conclusions We present an approach based on generalized enrichment analysis that can be used to detect associations between drugs, drug classes and adverse events at a given level of granularity, at the same time correcting for known dependencies among events. Our study demonstrates the use of GEA, and the importance of choosing appropriate abstraction levels to complement current drug safety methods. We provide an R package for exploration of alternative abstraction levels of adverse event terms based on information content. Electronic supplementary material The online version of this article (doi:10.1186/s12859-016-1080-z) contains supplementary material, which is available to authorized users.
Collapse
|
6
|
Hauben M, Aronson JK, Ferner RE. Evidence of Misclassification of Drug–Event Associations Classified as Gold Standard ‘Negative Controls’ by the Observational Medical Outcomes Partnership (OMOP). Drug Saf 2016; 39:421-32. [DOI: 10.1007/s40264-016-0392-2] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
7
|
White RW, Wang S, Pant A, Harpaz R, Shukla P, Sun W, DuMouchel W, Horvitz E. Early identification of adverse drug reactions from search log data. J Biomed Inform 2016; 59:42-8. [DOI: 10.1016/j.jbi.2015.11.005] [Citation(s) in RCA: 31] [Impact Index Per Article: 3.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2015] [Revised: 11/07/2015] [Accepted: 11/12/2015] [Indexed: 01/28/2023]
|
8
|
Wang G, Jung K, Winnenburg R, Shah NH. A method for systematic discovery of adverse drug events from clinical notes. J Am Med Inform Assoc 2015; 22:1196-204. [PMID: 26232442 DOI: 10.1093/jamia/ocv102] [Citation(s) in RCA: 42] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2015] [Accepted: 06/16/2015] [Indexed: 01/11/2023] Open
Abstract
OBJECTIVE Adverse drug events (ADEs) are undesired harmful effects resulting from use of a medication, and occur in 30% of hospitalized patients. The authors have developed a data-mining method for systematic, automated detection of ADEs from electronic medical records. MATERIALS AND METHODS This method uses the text from 9.5 million clinical notes, along with prior knowledge of drug usages and known ADEs, as inputs. These inputs are further processed into statistics used by a discriminative classifier which outputs the probability that a given drug-disorder pair represents a valid ADE association. Putative ADEs identified by the classifier are further filtered for positive support in 2 independent, complementary data sources. The authors evaluate this method by assessing support for the predictions in other curated data sources, including a manually curated, time-indexed reference standard of label change events. RESULTS This method uses a classifier that achieves an area under the curve of 0.94 on a held out test set. The classifier is used on 2,362,950 possible drug-disorder pairs comprised of 1602 unique drugs and 1475 unique disorders for which we had data, resulting in 240 high-confidence, well-supported drug-AE associations. Eighty-seven of them (36%) are supported in at least one of the resources that have information that was not available to the classifier. CONCLUSION This method demonstrates the feasibility of systematic post-marketing surveillance for ADEs using electronic medical records, a key component of the learning healthcare system.
Collapse
Affiliation(s)
- Guan Wang
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Kenneth Jung
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Rainer Winnenburg
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| | - Nigam H Shah
- Stanford University, Center for Biomedical Informatics, Stanford, California, USA
| |
Collapse
|
9
|
Authors' reply to Harpaz et al. comment on: "Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance". Drug Saf 2015; 38:115-6. [PMID: 25432780 DOI: 10.1007/s40264-014-0246-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|