Towards Automating Adverse Event Review: A Prediction Model for Case Report Utility

Trust but Verify: Lessons Learned for the Application of AI to Case-Based Clinical Decision-Making From Postmarketing Drug Safety Assessment at the US Food and Drug Administration

Adverse drug reactions are a common cause of morbidity in health care. The US Food and Drug Administration (FDA) evaluates individual case safety reports of adverse events (AEs) after submission to the FDA Adverse Event Reporting System as part of its surveillance activities. Over the past decade, the FDA has explored the application of artificial intelligence (AI) to evaluate these reports to improve the efficiency and scientific rigor of the process. However, a gap remains between AI algorithm development and deployment. This viewpoint aims to describe the lessons learned from our experience and research needed to address both general issues in case-based reasoning using AI and specific needs for individual case safety report assessment. Beginning with the recognition that the trustworthiness of the AI algorithm is the main determinant of its acceptance by human experts, we apply the Diffusion of Innovations theory to help explain why certain algorithms for evaluating AEs at the FDA were accepted by safety reviewers and others were not. This analysis reveals that the process by which clinicians decide from case reports whether a drug is likely to cause an AE is not well defined beyond general principles. This makes the development of high performing, transparent, and explainable AI algorithms challenging, leading to a lack of trust by the safety reviewers. Even accounting for the introduction of large language models, the pharmacovigilance community needs an improved understanding of causal inference and of the cognitive framework for determining the causal relationship between a drug and an AE. We describe specific future research directions that underpin facilitating implementation and trust in AI for drug safety applications, including improved methods for measuring and controlling of algorithmic uncertainty, computational reproducibility, and clear articulation of a cognitive framework for causal inference in case-based reasoning.

Exploring the impact of co-exposure timing on drug-drug interactions in signal detection through spontaneous reporting system databases: a scoping review

INTRODUCTION

Drug-drug interactions (DDIs) are defined as the pharmacological effects produced by the concomitant administration of two or more drugs. To minimize false positive signals and ensure their validity when analyzing Spontaneous Reporting System (SRS) databases, it has been suggested to incorporate key pharmacological principles, such as temporal plausibility.

AREAS COVERED

The scoping review of the literature was completed using MEDLINE from inception to March 2023. Included studies had to provide detailed methods for identifying DDIs in SRS databases. Any methodological approach and adverse event were accepted. Descriptive analyzes were excluded as we focused on automatic signal detection methods. The result is an overview of all the available methods for DDI signal detection in SRS databases, with a specific focus on the evaluation of the co-exposure time of the interacting drugs. It is worth noting that only a limited number of studies (n = 3) have attempted to address the issue of overlapping drug administration times.

EXPERT OPINION

Current guidelines for signal validation focus on factors like the number of reports and temporal association, but they lack guidance on addressing overlapping drug administration times, highlighting a need for further research and method development.

Navigating duplication in pharmacovigilance databases: a scoping review

OBJECTIVES

Pharmacovigilance databases play a critical role in monitoring drug safety. The duplication of reports in pharmacovigilance databases, however, undermines their data integrity. This scoping review sought to provide a comprehensive understanding of duplication in pharmacovigilance databases worldwide.

DESIGN

A scoping review.

DATA SOURCES

Reviewers comprehensively searched the literature in PubMed, Web of Science, Wiley Online Library, EBSCOhost, Google Scholar and other relevant websites.

ELIGIBILITY CRITERIA

Peer-reviewed publications and grey literature, without language restriction, describing duplication and/or methods relevant to duplication in pharmacovigilance databases from inception to 1 September 2023.

DATA EXTRACTION AND SYNTHESIS

We used the Joanna Briggs Institute guidelines for scoping reviews and conformed with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews. Two reviewers independently screened titles, abstracts and full texts. One reviewer extracted the data and performed descriptive analysis, which the second reviewer assessed. Disagreements were resolved by discussion and consensus or in consultation with a third reviewer.

RESULTS

We screened 22 745 unique titles and 156 were eligible for full-text review. Of the 156 titles, 58 (47 peer-reviewed; 11 grey literature) fulfilled the inclusion criteria for the scoping review. Included titles addressed the extent (5 papers), prevention strategies (15 papers), causes (32 papers), detection methods (25 papers), management strategies (24 papers) and implications (14 papers) of duplication in pharmacovigilance databases. The papers overlapped, discussing more than one field. Advances in artificial intelligence, particularly natural language processing, hold promise in enhancing the efficiency and precision of deduplication of large and complex pharmacovigilance databases.

CONCLUSION

Duplication in pharmacovigilance databases compromises risk assessment and decision-making, potentially threatening patient safety. Therefore, efficient duplicate prevention, detection and management are essential for more reliable pharmacovigilance data. To minimise duplication, consistent use of worldwide unique identifiers as the key case identifiers is recommended alongside recent advances in artificial intelligence.

BERT based natural language processing for triage of adverse drug reaction reports shows close to human-level performance

Post-marketing reports of suspected adverse drug reactions are important for establishing the safety profile of a medicinal product. However, a high influx of reports poses a challenge for regulatory authorities as a delay in identification of previously unknown adverse drug reactions can potentially be harmful to patients. In this study, we use natural language processing (NLP) to predict whether a report is of serious nature based solely on the free-text fields and adverse event terms in the report, potentially allowing reports mislabelled at time of reporting to be detected and prioritized for assessment. We consider four different NLP models at various levels of complexity, bootstrap their train-validation data split to eliminate random effects in the performance estimates and conduct prospective testing to avoid the risk of data leakage. Using a Swedish BERT based language model, continued language pre-training and final classification training, we achieve close to human-level performance in this task. Model architectures based on less complex technical foundation such as bag-of-words approaches and LSTM neural networks trained with random initiation of weights appear to perform less well, likely due to the lack of robustness that a base of general language training provides.

Finding Needles in the Haystack: Clinical Utility Score for Prioritisation (CUSP), an Automated Approach for Identifying Spontaneous Reports with the Highest Clinical Utility

INTRODUCTION

Spontaneous reporting of adverse events has increased steadily over the past decades, and although this trend has contributed to improving post-marketing surveillance pharmacovigilance activities, the consequent amount of data generated is challenging to manually review during assessment, with each individual report requiring review by pharmacovigilance experts. This highlights a clear need for alternative or complementary methodologies to help prioritise review.

OBJECTIVE

Here, we aimed to develop and test an automated methodology, the Clinical Utility Score for Prioritisation (CUSP), to assist pharmacovigilance experts in prioritising clinical assessment of safety data to improve the rapidity of case series review when case volumes are large.

METHODS

The CUSP method was tested on a reference dataset of individual case safety reports (ICSRs) associated to five drug-event pairs that led to labelling changes. The selected drug-event pairs were of varying characteristics across the portfolio of GSK's products.

RESULTS

The mean CUSP score for 'key cases' and 'cases of low utility' was 19.7 (median: 21; range: 7-27) and 17.3 (median: 19; range: 4-27), respectively. CUSP distribution for 'key cases' were skewed toward the higher range of scores compared with 'all cases'. The overall performance across each individual drug-event pair varied considerably, showing higher predictive power for 'key cases' for three of the drug-event pairs (average CUSP between these three: 22.8; range: 22.5-23.0) and lesser power for the remaining two (average CUSP between these two: 17.6; range: 14.5-20.7).

CONCLUSION

Although several tools have been developed to assess ICSR completeness and regulatory utility, this is the first attempt to successfully develop an automated clinical utility scoring system that can support the prioritisation of ICSRs for clinical review.

Will the future of pharmacovigilance be more automated?

INTRODUCTION

Artificial intelligence (AI) based tools offer new opportunities for pharmacovigilance (PV) activities. Nevertheless, their contribution to PV needs to be tailored to preserve and strengthen medical and pharmacological expertise in drug safety.

AREAS COVERED

This work aims to describe PV tasks in which the contribution of AI and intelligent automation (IA) tools is required, in the context of a continuous increase of spontaneous reporting cases and regulatory tasks. A narrative review with expert selection of pertinent references was performed through Medline. Two areas were covered, management of spontaneous reporting cases and signal detection.

PERSPECTIVE

The use of AI and IA tools will assist a large spectrum of PV activities, both in public and private PV systems, in particular for tasks of low added value (e.g. initial quality check, verification of essential regulatory information, search for duplicates). Testing, validating, and integrating these tools in the PV routine are the actual challenges for modern PV systems, to guarantee high-quality standards in terms of case management and signal detection.

Signals of Adverse Drug Reactions Communicated by Pharmacovigilance Stakeholders: A Scoping Review of the Global Literature

INTRODUCTION AND OBJECTIVE

Signals of adverse drug reactions (ADRs) can be supported by reports of ADRs and by interventional and non-interventional studies. The evidence base and features of ADR reports that are used to support signals remain to be comprehensively described. To this end, we have undertaken a scoping review.

METHODS

We searched the following databases: PubMed, EMBASE, PsycINFO, Web of Science, and Google Scholar, without language or time restrictions. We also hand searched the bibliographies of relevant studies. We included studies of any design if the results were described as signals. We assessed the levels of evidence using the Oxford Centre for Evidence-Based Medicine (OCEBM) criteria and coded features of reports of ADRs using the Bradford Hill guidelines.

RESULTS

Overall, 1974 publications reported 2421 studies of signals; 1683/2421 were clinical assessments of anecdotal reports of ADRs, but only 225 (13%) of these included explicit judgments on which features of the ADR reports were supportive of a signal. These 225 studies yielded 228 signals; these were supported by features, which were: 'experimental evidence' (i.e., positive dechallenge or rechallenge, 154 instances [68%]), 'temporality' (i.e., time to onset, 130 [57%]), 'exclusion of competing causes' (49 [21%]), and others (40 [17%]). Positive dechallenge/rechallenge often co-occurred with temporality (77/228). OCEBM 4 (i.e., case series and case-control studies) was the most frequent level of evidence (2078 studies). Between 2013 and 2019, there was a three-fold increase in clinical assessments of reports of ADRs compared with a less than two-fold increase in studies supported by higher levels of evidence (i.e., OCEBM 1-3). We identified an increased rate between 2013 and 2019 in disproportionality analyses (about 15 studies per year), mostly from academia.

CONCLUSIONS

Most signals were supported by temporality and dechallenge/rechallenge, but clear reporting of judgments on causality remains infrequent. The number of studies supported only by anecdotal reports of ADRs increased from year to year. The impact of a growing number of signals of disproportionate reporting communicated without an accompanying clinical assessment should be evaluated.

Development of a multivariate prediction model to identify individual case safety reports which require clinical review

BACKGROUND

The number of Individual Case Safety Reports (ICSRs) in pharmacovigilance databases are rapidly increasing world-wide. The majority of ICSRs at the Netherlands Pharmacovigilance Centre Lareb is reviewed manually to identify potential signal triggering reports (PSTR) or ICSRs which need further clinical assessment for other reasons.

OBJECTIVES

To develop a prediction model to identify ICSRs that require clinical review, including PSTRs. Secondly, to identify the most important features of these reports.

METHODS

All ICSRs (n = 30 424) received by Lareb between October 1, 2017 and February 26, 2021 were included. ICSRs originating from marketing authorisation holders and ICSRs reported on vaccines were excluded. The outcome was defined as PSTR (yes/no), where PSTR 'yes' was defined as an ICSR discussed at a signal detection meeting. Nineteen features were included, concerning structured information on: patients, adverse drug reactions (ADR) or drugs. Data were divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net logistic regression and eXtreme Gradient Boosting models were trained and tuned on a training set. Random down-sampling of negative controls was applied on the training set to adjust for the imbalanced dataset. Final models were evaluated on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re-train and test model performances with fewer features.

RESULTS

1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73-0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: 'absence of ADR in the Summary of product characteristics', 'ADR reported as serious', 'ADR labelled as an important medical event', 'ADR reported by physician' and 'positive rechallenge'. Model performances were similar when using only nine of the most important features.

CONCLUSIONS

We developed a prediction model with moderate performances to identify PSTRs with nine commonly available features. Optimisation of the model using more ICSR information (e.g., free text fields) to increase model precision is required before implementation.

Leveraging Machine Learning to Facilitate Individual Case Causality Assessment of Adverse Drug Reactions

INTRODUCTION

Causality assessment of individual case safety reports (ICSRs) is an important step in pharmacovigilance case-level review and aims to establish a position on whether a patient's exposure to a drug is causally related to the patient experiencing an untoward adverse event. There are many different approaches for case causality adjudication, including the use of expert opinions and algorithmic frameworks; however, a great deal of variability exists between assessment methods, products, therapeutic classes, individual physicians, change of process and conventions over time, and other factors.

OBJECTIVE

The objective of this study was to develop a machine learning-based model that can predict the likelihood of a causal association of an observed drug-reaction combination in an ICSR.

METHODS

In this study, we used a set of annotated solicited ICSRs (50K cases) from a company post-marketing database. These data were enriched with novel supplementary features from external and internal data sources that aim to capture facets such as temporal plausibility, scientific validity, and confoundedness that have been shown to contribute to causality adjudication. Using these features, we constructed a Bayesian network (BN) model to predict drug-event pair causality assessment. BN topology was driven by an internally developed ICSR causality decision support tool. Performance of the model was evaluated through examination of sensitivity, positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC) on an independent set of data from a temporally adjacent interval (20K cases). No external validation was performed because of a lack of publicly available ICSRs with causality assessments for drug-event pairs.

RESULTS

The model demonstrated high performance in predicting the causality assessment of drug-event pairs compared with clinical judgment using global introspection (AUC 0.924; 95% confidence interval [CI] 0.922-0.927). The sensitivity of the model was 0.900 (95% CI 0.896-0.904), and the PPV of the model was 0.778 (95% CI 0.773-0.783).

CONCLUSION

These results show that robust probabilistic modeling of ICSR causality is feasible, and the approach used in the development of the model can serve as a framework for such causality assessments, leading to improvements in safety decision making.

"Artificial Intelligence" for Pharmacovigilance: Ready for Prime Time?

There is great interest in the application of 'artificial intelligence' (AI) to pharmacovigilance (PV). Although US FDA is broadly exploring the use of AI for PV, we focus on the application of AI to the processing and evaluation of Individual Case Safety Reports (ICSRs) submitted to the FDA Adverse Event Reporting System (FAERS). We describe a general framework for considering the readiness of AI for PV, followed by some examples of the application of AI to ICSR processing and evaluation in industry and FDA. We conclude that AI can usefully be applied to some aspects of ICSR processing and evaluation, but the performance of current AI algorithms requires a 'human-in-the-loop' to ensure good quality. We identify outstanding scientific and policy issues to be addressed before the full potential of AI can be exploited for ICSR processing and evaluation, including approaches to quality assurance of 'human-in-the-loop' AI systems, large-scale, publicly available training datasets, a well-defined and computable 'cognitive framework', a formal sociotechnical framework for applying AI to PV, and development of best practices for applying AI to PV. Practical experience with stepwise implementation of AI for ICSR processing and evaluation will likely provide important lessons that will inform the necessary policy and regulatory framework to facilitate widespread adoption and provide a foundation for further development of AI approaches to other aspects of PV.

Safety of medicines and vaccines - building next generation capability

The systematic safety surveillance of real-world use of medicinal products and related activities (pharmacovigilance) started in earnest as a scientific field only in the 1960s. While developments have occurred over the past 50 years, adding to its complexity and sophistication, the extent to which some of these advances have positively impacted the capability for ensuring patient safety is questionable. We review how the conduct of safety surveillance has changed, highlight recent scientific advances, and argue how they need to be harnessed to enhance pharmacovigilance in the future. Specifically, we describe five changes that we believe should and will need to happen globally in the coming years: (i) better, more diverse data used for safety; (ii) the switch from manual activities to automation; (iii) removal of limited value, extraneous transactional activities and replacement with sharpened focus on scientific efforts to improve patient safety; (iv) patient-involved and focussed safety; and (v) personalised safety.

Clinical Information Systems Research in the Pandemic Year 2020

Objective: In this synopsis, we give an overview of recent research and propose a selection of best papers published in 2020 in the field of Clinical Information Systems (CIS).

Method: As CIS section editors, we annually apply a systematic process to retrieve articles for the International Medical Informatics Association Yearbook of Medical Informatics. For seven years now, we use the same query to find relevant publications in the CIS field. Each year we retrieve more than 2,400 papers which we categorize in a multi-pass review to distill a preselection of 15 candidate papers. External reviewers and yearbook editors then assess the selected candidate papers. Based on the review results, the IMIA Yearbook editorial board chooses up to four best publications for the section at a selection meeting. To get an overview of the content of the retrieved articles, we use text mining and term co-occurrence mapping techniques.

Results: We carried out the query in mid-January 2021 and retrieved a deduplicated result set of 2,787 articles from 1,135 different journals. We nominated 15 papers as candidates and finally selected four of them as the best papers in the CIS section. As in the previous years, the content analysis of the articles revealed the broad spectrum of topics covered by CIS research. Thus, this year we could observe a significant impact of COVID-19 on CIS research.

Conclusions: The trends in CIS research, as seen in recent years, continue to be observable. What was very visible was the impact of the Corona Virus Disease 2019 (COVID-19) pandemic, which has affected not only our lives but also CIS.

Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System

BACKGROUND

Our objective was to support the automated classification of Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) reports for their usefulness in assessing the possibility of a causal relationship between a drug product and an adverse event.

METHOD

We used a data set of 326 redacted FAERS reports that was previously annotated using a modified version of the World Health Organization-Uppsala Monitoring Centre criteria for drug causality assessment by a group of SEs at the FDA and supported a similar study on the classification of reports using supervised machine learning and text engineering methods. We explored many potential features, including the incorporation of natural language processing on report text and information from external data sources, for supervised learning and developed models for predicting the classification status of reports. We then evaluated the models on a larger data set of previously unseen reports.

RESULTS

The best-performing models achieved recall and F1 scores on both data sets above 0.80 for the identification of assessable reports (i.e. those containing enough information to make an informed causality assessment) and above 0.75 for the identification of reports meeting at least a Possible causality threshold.

CONCLUSIONS

Causal inference from FAERS reports depends on many components with complex logical relationships that are yet to be made fully computable. Efforts focused on readily addressable tasks, such as quickly eliminating unassessable reports, fit naturally in SE's thought processes to provide real enhancements for FDA workflows.

An Evaluation of Postmarketing Reports with an Outcome of Death in the US FDA Adverse Event Reporting System

INTRODUCTION

Adverse reactions with an outcome of death are inherently important for pharmacovigilance organizations to evaluate. Prior efforts to systematically evaluate individual case safety reports (ICSRs) with an outcome of death have been limited to high-level summaries.

OBJECTIVE

The aim of this study was to characterize ICSRs with an outcome of death contained in the US FDA Adverse Event Reporting System (FAERS) database.

METHODS

All ICSRs received through 31 December 2017 reporting an outcome of death were characterized by patient demographics, suspect product(s), adverse events, and reporter type. Using the ICSR's narrative and reporter information, we classified ICSRs by source to include those from industry-sponsored programs, poison control centers, specialty pharmacies, and litigation. Additionally, a random sample of ICSRs was evaluated for completeness of structured data fields and manually reviewed for the availability of key information in the narrative (i.e. cause of death, medical history, and causality assessment).

RESULTS

Overall, 1,053,716 ICSRs with a death outcome were received in the study period. Ten medications treating conditions for malignancies, pain, and kidney disease accounted for nearly 20% of all fatal ICSRs. ICSRs originating from industry-sponsored programs, poison control centers, litigation, and specialty pharmacies accounted for 14%, 6.5%, 5.0%, and 3.3% of all fatal ICSRs, respectively. ICSRs in which the only adverse event coded was 'death' were more likely to be missing structured data and less likely to include key information in the narrative.

CONCLUSION

Understanding the origins and characteristics of ICSRs with an outcome of death supports meaningful evaluations and interpretations of FAERS data. A wide variability in ICSR quality exists, even in those reports with the most serious outcome.