1
|
Ball R, Talal AH, Dang O, Muñoz M, Markatou M. Trust but Verify: Lessons Learned for the Application of AI to Case-Based Clinical Decision-Making From Postmarketing Drug Safety Assessment at the US Food and Drug Administration. J Med Internet Res 2024; 26:e50274. [PMID: 38842929 PMCID: PMC11190620 DOI: 10.2196/50274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2023] [Revised: 12/22/2023] [Accepted: 04/26/2024] [Indexed: 06/07/2024] Open
Abstract
Adverse drug reactions are a common cause of morbidity in health care. The US Food and Drug Administration (FDA) evaluates individual case safety reports of adverse events (AEs) after submission to the FDA Adverse Event Reporting System as part of its surveillance activities. Over the past decade, the FDA has explored the application of artificial intelligence (AI) to evaluate these reports to improve the efficiency and scientific rigor of the process. However, a gap remains between AI algorithm development and deployment. This viewpoint aims to describe the lessons learned from our experience and research needed to address both general issues in case-based reasoning using AI and specific needs for individual case safety report assessment. Beginning with the recognition that the trustworthiness of the AI algorithm is the main determinant of its acceptance by human experts, we apply the Diffusion of Innovations theory to help explain why certain algorithms for evaluating AEs at the FDA were accepted by safety reviewers and others were not. This analysis reveals that the process by which clinicians decide from case reports whether a drug is likely to cause an AE is not well defined beyond general principles. This makes the development of high performing, transparent, and explainable AI algorithms challenging, leading to a lack of trust by the safety reviewers. Even accounting for the introduction of large language models, the pharmacovigilance community needs an improved understanding of causal inference and of the cognitive framework for determining the causal relationship between a drug and an AE. We describe specific future research directions that underpin facilitating implementation and trust in AI for drug safety applications, including improved methods for measuring and controlling of algorithmic uncertainty, computational reproducibility, and clear articulation of a cognitive framework for causal inference in case-based reasoning.
Collapse
Affiliation(s)
- Robert Ball
- Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, United States
| | - Andrew H Talal
- Jacobs School of Medicine and Biomedical Sciences, Buffalo, NY, United States
| | - Oanh Dang
- Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, United States
| | - Monica Muñoz
- Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, United States
| | - Marianthi Markatou
- School of Public Health and Health Professions, University at Buffalo, Buffalo, NY, United States
| |
Collapse
|
2
|
Cocco M, Carnovale C, Clementi E, Barbieri MA, Battini V, Sessa M. Exploring the impact of co-exposure timing on drug-drug interactions in signal detection through spontaneous reporting system databases: a scoping review. Expert Rev Clin Pharmacol 2024; 17:441-453. [PMID: 38619027 DOI: 10.1080/17512433.2024.2343875] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2023] [Accepted: 04/12/2024] [Indexed: 04/16/2024]
Abstract
INTRODUCTION Drug-drug interactions (DDIs) are defined as the pharmacological effects produced by the concomitant administration of two or more drugs. To minimize false positive signals and ensure their validity when analyzing Spontaneous Reporting System (SRS) databases, it has been suggested to incorporate key pharmacological principles, such as temporal plausibility. AREAS COVERED The scoping review of the literature was completed using MEDLINE from inception to March 2023. Included studies had to provide detailed methods for identifying DDIs in SRS databases. Any methodological approach and adverse event were accepted. Descriptive analyzes were excluded as we focused on automatic signal detection methods. The result is an overview of all the available methods for DDI signal detection in SRS databases, with a specific focus on the evaluation of the co-exposure time of the interacting drugs. It is worth noting that only a limited number of studies (n = 3) have attempted to address the issue of overlapping drug administration times. EXPERT OPINION Current guidelines for signal validation focus on factors like the number of reports and temporal association, but they lack guidance on addressing overlapping drug administration times, highlighting a need for further research and method development.
Collapse
Affiliation(s)
- Marianna Cocco
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark
- Department of Drug Sciences, University of Pavia, Pavia, Italy
| | - Carla Carnovale
- Pharmacovigilance & Clinical Research, International Centre for Pesticides and Health Risk Prevention, Department of Biomedical and Clinical Sciences (DIBIC), ASST Fatebenefratelli-Sacco University Hospital, Università Degli Studi di Milano, Milan, Italy
| | - Emilio Clementi
- Pharmacovigilance & Clinical Research, International Centre for Pesticides and Health Risk Prevention, Department of Biomedical and Clinical Sciences (DIBIC), ASST Fatebenefratelli-Sacco University Hospital, Università Degli Studi di Milano, Milan, Italy
- Scientific Institute, IRCCS E. Medea, Bosisio Parini, LC, Italy
| | - Maria Antonietta Barbieri
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark
- Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy
| | - Vera Battini
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark
| | - Maurizio Sessa
- Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
3
|
Kiguba R, Isabirye G, Mayengo J, Owiny J, Tregunno P, Harrison K, Pirmohamed M, Ndagije HB. Navigating duplication in pharmacovigilance databases: a scoping review. BMJ Open 2024; 14:e081990. [PMID: 38684275 PMCID: PMC11086478 DOI: 10.1136/bmjopen-2023-081990] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/11/2023] [Accepted: 04/12/2024] [Indexed: 05/02/2024] Open
Abstract
OBJECTIVES Pharmacovigilance databases play a critical role in monitoring drug safety. The duplication of reports in pharmacovigilance databases, however, undermines their data integrity. This scoping review sought to provide a comprehensive understanding of duplication in pharmacovigilance databases worldwide. DESIGN A scoping review. DATA SOURCES Reviewers comprehensively searched the literature in PubMed, Web of Science, Wiley Online Library, EBSCOhost, Google Scholar and other relevant websites. ELIGIBILITY CRITERIA Peer-reviewed publications and grey literature, without language restriction, describing duplication and/or methods relevant to duplication in pharmacovigilance databases from inception to 1 September 2023. DATA EXTRACTION AND SYNTHESIS We used the Joanna Briggs Institute guidelines for scoping reviews and conformed with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews. Two reviewers independently screened titles, abstracts and full texts. One reviewer extracted the data and performed descriptive analysis, which the second reviewer assessed. Disagreements were resolved by discussion and consensus or in consultation with a third reviewer. RESULTS We screened 22 745 unique titles and 156 were eligible for full-text review. Of the 156 titles, 58 (47 peer-reviewed; 11 grey literature) fulfilled the inclusion criteria for the scoping review. Included titles addressed the extent (5 papers), prevention strategies (15 papers), causes (32 papers), detection methods (25 papers), management strategies (24 papers) and implications (14 papers) of duplication in pharmacovigilance databases. The papers overlapped, discussing more than one field. Advances in artificial intelligence, particularly natural language processing, hold promise in enhancing the efficiency and precision of deduplication of large and complex pharmacovigilance databases. CONCLUSION Duplication in pharmacovigilance databases compromises risk assessment and decision-making, potentially threatening patient safety. Therefore, efficient duplicate prevention, detection and management are essential for more reliable pharmacovigilance data. To minimise duplication, consistent use of worldwide unique identifiers as the key case identifiers is recommended alongside recent advances in artificial intelligence.
Collapse
Affiliation(s)
- Ronald Kiguba
- Department of Pharmacology and Therapeutics, College of Health Sciences, Makerere University, Kampala, Uganda
| | - Gerald Isabirye
- National Pharmacovigilance Centre, National Drug Authority, Kampala, Uganda
| | - Julius Mayengo
- National Pharmacovigilance Centre, National Drug Authority, Kampala, Uganda
| | - Jonathan Owiny
- National Pharmacovigilance Centre, National Drug Authority, Kampala, Uganda
| | - Phil Tregunno
- Safety and Surveillance Group, Medicines and Healthcare Products Regulatory Agency, London, UK
| | - Kendal Harrison
- Safety and Surveillance Group, Medicines and Healthcare Products Regulatory Agency, London, UK
| | - Munir Pirmohamed
- Centre for Drug Safety Science and Wolfson Centre for Personalised Medicine, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
| | | |
Collapse
|
4
|
Bergman E, Dürlich L, Arthurson V, Sundström A, Larsson M, Bhuiyan S, Jakobsson A, Westman G. BERT based natural language processing for triage of adverse drug reaction reports shows close to human-level performance. PLOS DIGITAL HEALTH 2023; 2:e0000409. [PMID: 38055685 DOI: 10.1371/journal.pdig.0000409] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Accepted: 11/09/2023] [Indexed: 12/08/2023]
Abstract
Post-marketing reports of suspected adverse drug reactions are important for establishing the safety profile of a medicinal product. However, a high influx of reports poses a challenge for regulatory authorities as a delay in identification of previously unknown adverse drug reactions can potentially be harmful to patients. In this study, we use natural language processing (NLP) to predict whether a report is of serious nature based solely on the free-text fields and adverse event terms in the report, potentially allowing reports mislabelled at time of reporting to be detected and prioritized for assessment. We consider four different NLP models at various levels of complexity, bootstrap their train-validation data split to eliminate random effects in the performance estimates and conduct prospective testing to avoid the risk of data leakage. Using a Swedish BERT based language model, continued language pre-training and final classification training, we achieve close to human-level performance in this task. Model architectures based on less complex technical foundation such as bag-of-words approaches and LSTM neural networks trained with random initiation of weights appear to perform less well, likely due to the lack of robustness that a base of general language training provides.
Collapse
Affiliation(s)
- Erik Bergman
- Swedish Medical Products Agency, Uppsala, Sweden
| | - Luise Dürlich
- Swedish Medical Products Agency, Uppsala, Sweden
- Department of Computer Science, RISE Research Institutes of Sweden, Kista, Sweden
- Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden
| | | | | | | | | | | | - Gabriel Westman
- Swedish Medical Products Agency, Uppsala, Sweden
- Department of Medical Sciences, Uppsala University, Uppsala, Sweden
| |
Collapse
|
5
|
Kara V, Powell G, Mahaux O, Jayachandra A, Nyako N, Golds C, Bate A. Finding Needles in the Haystack: Clinical Utility Score for Prioritisation (CUSP), an Automated Approach for Identifying Spontaneous Reports with the Highest Clinical Utility. Drug Saf 2023; 46:847-855. [PMID: 37535258 PMCID: PMC10442257 DOI: 10.1007/s40264-023-01327-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/11/2023] [Indexed: 08/04/2023]
Abstract
INTRODUCTION Spontaneous reporting of adverse events has increased steadily over the past decades, and although this trend has contributed to improving post-marketing surveillance pharmacovigilance activities, the consequent amount of data generated is challenging to manually review during assessment, with each individual report requiring review by pharmacovigilance experts. This highlights a clear need for alternative or complementary methodologies to help prioritise review. OBJECTIVE Here, we aimed to develop and test an automated methodology, the Clinical Utility Score for Prioritisation (CUSP), to assist pharmacovigilance experts in prioritising clinical assessment of safety data to improve the rapidity of case series review when case volumes are large. METHODS The CUSP method was tested on a reference dataset of individual case safety reports (ICSRs) associated to five drug-event pairs that led to labelling changes. The selected drug-event pairs were of varying characteristics across the portfolio of GSK's products. RESULTS The mean CUSP score for 'key cases' and 'cases of low utility' was 19.7 (median: 21; range: 7-27) and 17.3 (median: 19; range: 4-27), respectively. CUSP distribution for 'key cases' were skewed toward the higher range of scores compared with 'all cases'. The overall performance across each individual drug-event pair varied considerably, showing higher predictive power for 'key cases' for three of the drug-event pairs (average CUSP between these three: 22.8; range: 22.5-23.0) and lesser power for the remaining two (average CUSP between these two: 17.6; range: 14.5-20.7). CONCLUSION Although several tools have been developed to assess ICSR completeness and regulatory utility, this is the first attempt to successfully develop an automated clinical utility scoring system that can support the prioritisation of ICSRs for clinical review.
Collapse
Affiliation(s)
- Vijay Kara
- GSK, 980 Great West Road, London, TW8 9GS, UK.
| | | | | | | | | | | | - Andrew Bate
- GSK, 980 Great West Road, London, TW8 9GS, UK
- Department of Non-Communicable Disease Epidemiology, LSHTM, London, UK
| |
Collapse
|
6
|
Salvo F, Micallef J, Lahouegue A, Chouchana L, Létinier L, Faillie JL, Pariente A. Will the future of pharmacovigilance be more automated? Expert Opin Drug Saf 2023; 22:541-548. [PMID: 37435796 DOI: 10.1080/14740338.2023.2227091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Accepted: 06/15/2023] [Indexed: 07/13/2023]
Abstract
INTRODUCTION Artificial intelligence (AI) based tools offer new opportunities for pharmacovigilance (PV) activities. Nevertheless, their contribution to PV needs to be tailored to preserve and strengthen medical and pharmacological expertise in drug safety. AREAS COVERED This work aims to describe PV tasks in which the contribution of AI and intelligent automation (IA) tools is required, in the context of a continuous increase of spontaneous reporting cases and regulatory tasks. A narrative review with expert selection of pertinent references was performed through Medline. Two areas were covered, management of spontaneous reporting cases and signal detection. PERSPECTIVE The use of AI and IA tools will assist a large spectrum of PV activities, both in public and private PV systems, in particular for tasks of low added value (e.g. initial quality check, verification of essential regulatory information, search for duplicates). Testing, validating, and integrating these tools in the PV routine are the actual challenges for modern PV systems, to guarantee high-quality standards in terms of case management and signal detection.
Collapse
Affiliation(s)
- Francesco Salvo
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
| | - Joelle Micallef
- Pharmacovigilance Centre, Department of Clinical Pharmacology and Pharmacovigilance, University of Aix Marseille, INSERM UMR 1106 Institut de Neurosciences des Systèmes, Marseille, France
| | - Amir Lahouegue
- Department of Pharmacovigilance and Medical Information, Astrazeneca, Courbevoie, France
| | - Laurent Chouchana
- Regional Center of Pharmacovigilance, Pharmacology Department, Cochin Port Royal University Hospital, Paris, France
| | - Louis Létinier
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
- Synapse Medicine, Bordeaux, France
| | - Jean-Luc Faillie
- Inserm, Departement de Pharmacologie Medicale Et Toxicologie, Centre Regional de PV, Institut Desbrest D'epidemiologie Et de Sante Publique, CHU de Montpellier, Universite Montpellier, Montpellier, France
| | - Antoine Pariente
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
| |
Collapse
|
7
|
Signals of Adverse Drug Reactions Communicated by Pharmacovigilance Stakeholders: A Scoping Review of the Global Literature. Drug Saf 2023; 46:109-120. [PMID: 36469249 PMCID: PMC9883307 DOI: 10.1007/s40264-022-01258-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/10/2022] [Indexed: 12/12/2022]
Abstract
INTRODUCTION AND OBJECTIVE Signals of adverse drug reactions (ADRs) can be supported by reports of ADRs and by interventional and non-interventional studies. The evidence base and features of ADR reports that are used to support signals remain to be comprehensively described. To this end, we have undertaken a scoping review. METHODS We searched the following databases: PubMed, EMBASE, PsycINFO, Web of Science, and Google Scholar, without language or time restrictions. We also hand searched the bibliographies of relevant studies. We included studies of any design if the results were described as signals. We assessed the levels of evidence using the Oxford Centre for Evidence-Based Medicine (OCEBM) criteria and coded features of reports of ADRs using the Bradford Hill guidelines. RESULTS Overall, 1974 publications reported 2421 studies of signals; 1683/2421 were clinical assessments of anecdotal reports of ADRs, but only 225 (13%) of these included explicit judgments on which features of the ADR reports were supportive of a signal. These 225 studies yielded 228 signals; these were supported by features, which were: 'experimental evidence' (i.e., positive dechallenge or rechallenge, 154 instances [68%]), 'temporality' (i.e., time to onset, 130 [57%]), 'exclusion of competing causes' (49 [21%]), and others (40 [17%]). Positive dechallenge/rechallenge often co-occurred with temporality (77/228). OCEBM 4 (i.e., case series and case-control studies) was the most frequent level of evidence (2078 studies). Between 2013 and 2019, there was a three-fold increase in clinical assessments of reports of ADRs compared with a less than two-fold increase in studies supported by higher levels of evidence (i.e., OCEBM 1-3). We identified an increased rate between 2013 and 2019 in disproportionality analyses (about 15 studies per year), mostly from academia. CONCLUSIONS Most signals were supported by temporality and dechallenge/rechallenge, but clear reporting of judgments on causality remains infrequent. The number of studies supported only by anecdotal reports of ADRs increased from year to year. The impact of a growing number of signals of disproportionate reporting communicated without an accompanying clinical assessment should be evaluated.
Collapse
|
8
|
Gosselt HR, Bazelmans EA, Lieber T, van Hunsel FPAM, Härmark L. Development of a multivariate prediction model to identify individual case safety reports which require clinical review. Pharmacoepidemiol Drug Saf 2022; 31:1300-1307. [PMID: 36251280 DOI: 10.1002/pds.5553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 10/06/2022] [Accepted: 10/10/2022] [Indexed: 12/15/2022]
Abstract
BACKGROUND The number of Individual Case Safety Reports (ICSRs) in pharmacovigilance databases are rapidly increasing world-wide. The majority of ICSRs at the Netherlands Pharmacovigilance Centre Lareb is reviewed manually to identify potential signal triggering reports (PSTR) or ICSRs which need further clinical assessment for other reasons. OBJECTIVES To develop a prediction model to identify ICSRs that require clinical review, including PSTRs. Secondly, to identify the most important features of these reports. METHODS All ICSRs (n = 30 424) received by Lareb between October 1, 2017 and February 26, 2021 were included. ICSRs originating from marketing authorisation holders and ICSRs reported on vaccines were excluded. The outcome was defined as PSTR (yes/no), where PSTR 'yes' was defined as an ICSR discussed at a signal detection meeting. Nineteen features were included, concerning structured information on: patients, adverse drug reactions (ADR) or drugs. Data were divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net logistic regression and eXtreme Gradient Boosting models were trained and tuned on a training set. Random down-sampling of negative controls was applied on the training set to adjust for the imbalanced dataset. Final models were evaluated on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re-train and test model performances with fewer features. RESULTS 1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73-0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: 'absence of ADR in the Summary of product characteristics', 'ADR reported as serious', 'ADR labelled as an important medical event', 'ADR reported by physician' and 'positive rechallenge'. Model performances were similar when using only nine of the most important features. CONCLUSIONS We developed a prediction model with moderate performances to identify PSTRs with nine commonly available features. Optimisation of the model using more ICSR information (e.g., free text fields) to increase model precision is required before implementation.
Collapse
Affiliation(s)
- Helen R Gosselt
- Netherlands Pharmacovigilance Centre Lareb, 's-Hertogenbosch, The Netherlands
| | | | - Thomas Lieber
- Netherlands Pharmacovigilance Centre Lareb, 's-Hertogenbosch, The Netherlands
| | | | - Linda Härmark
- Netherlands Pharmacovigilance Centre Lareb, 's-Hertogenbosch, The Netherlands
| |
Collapse
|
9
|
Cherkas Y, Ide J, van Stekelenborg J. Leveraging Machine Learning to Facilitate Individual Case Causality Assessment of Adverse Drug Reactions. Drug Saf 2022; 45:571-582. [PMID: 35579819 DOI: 10.1007/s40264-022-01163-6] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/13/2022] [Indexed: 11/29/2022]
Abstract
INTRODUCTION Causality assessment of individual case safety reports (ICSRs) is an important step in pharmacovigilance case-level review and aims to establish a position on whether a patient's exposure to a drug is causally related to the patient experiencing an untoward adverse event. There are many different approaches for case causality adjudication, including the use of expert opinions and algorithmic frameworks; however, a great deal of variability exists between assessment methods, products, therapeutic classes, individual physicians, change of process and conventions over time, and other factors. OBJECTIVE The objective of this study was to develop a machine learning-based model that can predict the likelihood of a causal association of an observed drug-reaction combination in an ICSR. METHODS In this study, we used a set of annotated solicited ICSRs (50K cases) from a company post-marketing database. These data were enriched with novel supplementary features from external and internal data sources that aim to capture facets such as temporal plausibility, scientific validity, and confoundedness that have been shown to contribute to causality adjudication. Using these features, we constructed a Bayesian network (BN) model to predict drug-event pair causality assessment. BN topology was driven by an internally developed ICSR causality decision support tool. Performance of the model was evaluated through examination of sensitivity, positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC) on an independent set of data from a temporally adjacent interval (20K cases). No external validation was performed because of a lack of publicly available ICSRs with causality assessments for drug-event pairs. RESULTS The model demonstrated high performance in predicting the causality assessment of drug-event pairs compared with clinical judgment using global introspection (AUC 0.924; 95% confidence interval [CI] 0.922-0.927). The sensitivity of the model was 0.900 (95% CI 0.896-0.904), and the PPV of the model was 0.778 (95% CI 0.773-0.783). CONCLUSION These results show that robust probabilistic modeling of ICSR causality is feasible, and the approach used in the development of the model can serve as a framework for such causality assessments, leading to improvements in safety decision making.
Collapse
Affiliation(s)
| | - Joshua Ide
- Johnson & Johnson Consumer, Inc, Skillman, NJ, USA
| | | |
Collapse
|
10
|
Ball R, Dal Pan G. "Artificial Intelligence" for Pharmacovigilance: Ready for Prime Time? Drug Saf 2022; 45:429-438. [PMID: 35579808 PMCID: PMC9112277 DOI: 10.1007/s40264-022-01157-4] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/10/2022] [Indexed: 01/28/2023]
Abstract
There is great interest in the application of 'artificial intelligence' (AI) to pharmacovigilance (PV). Although US FDA is broadly exploring the use of AI for PV, we focus on the application of AI to the processing and evaluation of Individual Case Safety Reports (ICSRs) submitted to the FDA Adverse Event Reporting System (FAERS). We describe a general framework for considering the readiness of AI for PV, followed by some examples of the application of AI to ICSR processing and evaluation in industry and FDA. We conclude that AI can usefully be applied to some aspects of ICSR processing and evaluation, but the performance of current AI algorithms requires a 'human-in-the-loop' to ensure good quality. We identify outstanding scientific and policy issues to be addressed before the full potential of AI can be exploited for ICSR processing and evaluation, including approaches to quality assurance of 'human-in-the-loop' AI systems, large-scale, publicly available training datasets, a well-defined and computable 'cognitive framework', a formal sociotechnical framework for applying AI to PV, and development of best practices for applying AI to PV. Practical experience with stepwise implementation of AI for ICSR processing and evaluation will likely provide important lessons that will inform the necessary policy and regulatory framework to facilitate widespread adoption and provide a foundation for further development of AI approaches to other aspects of PV.
Collapse
Affiliation(s)
- Robert Ball
- grid.483500.a0000 0001 2154 2448US Food and Drug Administration, Center for Drug Evaluation and Research, Office of Surveillance and Epidemiology, Silver Spring, MD USA
| | - Gerald Dal Pan
- grid.483500.a0000 0001 2154 2448US Food and Drug Administration, Center for Drug Evaluation and Research, Office of Surveillance and Epidemiology, Silver Spring, MD USA
| |
Collapse
|
11
|
Bate A, Stegmann JU. Safety of medicines and vaccines - building next generation capability. Trends Pharmacol Sci 2021; 42:1051-1063. [PMID: 34635346 DOI: 10.1016/j.tips.2021.09.007] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Revised: 09/10/2021] [Accepted: 09/14/2021] [Indexed: 10/20/2022]
Abstract
The systematic safety surveillance of real-world use of medicinal products and related activities (pharmacovigilance) started in earnest as a scientific field only in the 1960s. While developments have occurred over the past 50 years, adding to its complexity and sophistication, the extent to which some of these advances have positively impacted the capability for ensuring patient safety is questionable. We review how the conduct of safety surveillance has changed, highlight recent scientific advances, and argue how they need to be harnessed to enhance pharmacovigilance in the future. Specifically, we describe five changes that we believe should and will need to happen globally in the coming years: (i) better, more diverse data used for safety; (ii) the switch from manual activities to automation; (iii) removal of limited value, extraneous transactional activities and replacement with sharpened focus on scientific efforts to improve patient safety; (iv) patient-involved and focussed safety; and (v) personalised safety.
Collapse
Affiliation(s)
- Andrew Bate
- GSK, London, UK; London School of Hygiene and Tropical Medicine, University of London, London, UK; New York University, New York, NY, USA.
| | | |
Collapse
|
12
|
Abstract
Objective:
In this synopsis, we give an overview of recent research and propose a selection of best papers published in 2020 in the field of Clinical Information Systems (CIS).
Method:
As CIS section editors, we annually apply a systematic process to retrieve articles for the International Medical Informatics Association Yearbook of Medical Informatics. For seven years now, we use the same query to find relevant publications in the CIS field. Each year we retrieve more than 2,400 papers which we categorize in a multi-pass review to distill a preselection of 15 candidate papers. External reviewers and yearbook editors then assess the selected candidate papers. Based on the review results, the IMIA Yearbook editorial board chooses up to four best publications for the section at a selection meeting. To get an overview of the content of the retrieved articles, we use text mining and term co-occurrence mapping techniques.
Results:
We carried out the query in mid-January 2021 and retrieved a deduplicated result set of 2,787 articles from 1,135 different journals. We nominated 15 papers as candidates and finally selected four of them as the best papers in the CIS section. As in the previous years, the content analysis of the articles revealed the broad spectrum of topics covered by CIS research. Thus, this year we could observe a significant impact of COVID-19 on CIS research.
Conclusions:
The trends in CIS research, as seen in recent years, continue to be observable. What was very visible was the impact of the Corona Virus Disease 2019 (COVID-19) pandemic, which has affected not only our lives but also CIS.
Collapse
Affiliation(s)
- W O Hackl
- Institute of Medical Informatics, UMIT - Private University for Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
| | - A Hoerbst
- Medical Technologies Department, MCI - The Entrepreneurial School, Innsbruck, Austria
| | | |
Collapse
|
13
|
Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System. Comput Biol Med 2021; 135:104517. [PMID: 34130003 DOI: 10.1016/j.compbiomed.2021.104517] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2021] [Revised: 05/20/2021] [Accepted: 05/21/2021] [Indexed: 10/21/2022]
Abstract
BACKGROUND Our objective was to support the automated classification of Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) reports for their usefulness in assessing the possibility of a causal relationship between a drug product and an adverse event. METHOD We used a data set of 326 redacted FAERS reports that was previously annotated using a modified version of the World Health Organization-Uppsala Monitoring Centre criteria for drug causality assessment by a group of SEs at the FDA and supported a similar study on the classification of reports using supervised machine learning and text engineering methods. We explored many potential features, including the incorporation of natural language processing on report text and information from external data sources, for supervised learning and developed models for predicting the classification status of reports. We then evaluated the models on a larger data set of previously unseen reports. RESULTS The best-performing models achieved recall and F1 scores on both data sets above 0.80 for the identification of assessable reports (i.e. those containing enough information to make an informed causality assessment) and above 0.75 for the identification of reports meeting at least a Possible causality threshold. CONCLUSIONS Causal inference from FAERS reports depends on many components with complex logical relationships that are yet to be made fully computable. Efforts focused on readily addressable tasks, such as quickly eliminating unassessable reports, fit naturally in SE's thought processes to provide real enhancements for FDA workflows.
Collapse
|
14
|
An Evaluation of Postmarketing Reports with an Outcome of Death in the US FDA Adverse Event Reporting System. Drug Saf 2021; 43:457-465. [PMID: 31981082 DOI: 10.1007/s40264-020-00908-5] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
Abstract
INTRODUCTION Adverse reactions with an outcome of death are inherently important for pharmacovigilance organizations to evaluate. Prior efforts to systematically evaluate individual case safety reports (ICSRs) with an outcome of death have been limited to high-level summaries. OBJECTIVE The aim of this study was to characterize ICSRs with an outcome of death contained in the US FDA Adverse Event Reporting System (FAERS) database. METHODS All ICSRs received through 31 December 2017 reporting an outcome of death were characterized by patient demographics, suspect product(s), adverse events, and reporter type. Using the ICSR's narrative and reporter information, we classified ICSRs by source to include those from industry-sponsored programs, poison control centers, specialty pharmacies, and litigation. Additionally, a random sample of ICSRs was evaluated for completeness of structured data fields and manually reviewed for the availability of key information in the narrative (i.e. cause of death, medical history, and causality assessment). RESULTS Overall, 1,053,716 ICSRs with a death outcome were received in the study period. Ten medications treating conditions for malignancies, pain, and kidney disease accounted for nearly 20% of all fatal ICSRs. ICSRs originating from industry-sponsored programs, poison control centers, litigation, and specialty pharmacies accounted for 14%, 6.5%, 5.0%, and 3.3% of all fatal ICSRs, respectively. ICSRs in which the only adverse event coded was 'death' were more likely to be missing structured data and less likely to include key information in the narrative. CONCLUSION Understanding the origins and characteristics of ICSRs with an outcome of death supports meaningful evaluations and interpretations of FAERS data. A wide variability in ICSR quality exists, even in those reports with the most serious outcome.
Collapse
|
15
|
Spiker J, Kreimeyer K, Dang O, Boxwell D, Chan V, Cheng C, Gish P, Lardieri A, Wu E, De S, Naidoo J, Lehmann H, Rosner GL, Ball R, Botsis T. Information Visualization Platform for Postmarket Surveillance Decision Support. Drug Saf 2020; 43:905-915. [DOI: 10.1007/s40264-020-00945-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
|