Development of an automated assessment tool for MedWatch reports in the FDA adverse event reporting system

Trust but Verify: Lessons Learned for the Application of AI to Case-Based Clinical Decision-Making From Postmarketing Drug Safety Assessment at the US Food and Drug Administration

Adverse drug reactions are a common cause of morbidity in health care. The US Food and Drug Administration (FDA) evaluates individual case safety reports of adverse events (AEs) after submission to the FDA Adverse Event Reporting System as part of its surveillance activities. Over the past decade, the FDA has explored the application of artificial intelligence (AI) to evaluate these reports to improve the efficiency and scientific rigor of the process. However, a gap remains between AI algorithm development and deployment. This viewpoint aims to describe the lessons learned from our experience and research needed to address both general issues in case-based reasoning using AI and specific needs for individual case safety report assessment. Beginning with the recognition that the trustworthiness of the AI algorithm is the main determinant of its acceptance by human experts, we apply the Diffusion of Innovations theory to help explain why certain algorithms for evaluating AEs at the FDA were accepted by safety reviewers and others were not. This analysis reveals that the process by which clinicians decide from case reports whether a drug is likely to cause an AE is not well defined beyond general principles. This makes the development of high performing, transparent, and explainable AI algorithms challenging, leading to a lack of trust by the safety reviewers. Even accounting for the introduction of large language models, the pharmacovigilance community needs an improved understanding of causal inference and of the cognitive framework for determining the causal relationship between a drug and an AE. We describe specific future research directions that underpin facilitating implementation and trust in AI for drug safety applications, including improved methods for measuring and controlling of algorithmic uncertainty, computational reproducibility, and clear articulation of a cognitive framework for causal inference in case-based reasoning.

Drug Repurposing Using FDA Adverse Event Reporting System (FAERS) Database

Drug repurposing is an emerging approach to reassigning existing pre-approved therapies for new indications. The FDA Adverse Event Reporting System (FAERS) is a large database of over 28 million adverse event reports submitted by medical providers, patients, and drug manufacturers and provides extensive drug safety signal data. In this review, four common drug repurposing strategies using FAERS are described, including inverse signal detection for a single disease, drug-drug interactions that mitigate a target ADE, identifying drug-ADE pairs with opposing gene perturbation signatures and identifying drug-drug pairs with congruent gene perturbation signatures. The purpose of this review is to provide an overview of these different approaches using existing successful applications in the literature. With the fast expansion of adverse drug event reports, FAERS-based drug repurposing represents a promising strategy for discovering new uses for existing therapies.

The association between warfarin usage and international normalized ratio increase: systematic analysis of FDA Adverse Event Reporting System (FAERS)

Introduction

Elevated international normalized ratio (INR) has been commonly reported as an adverse drug event (ADE) for patients taking warfarin for anticoagulant therapy.

Aim

The purpose of this study was to determine the association between increased INR and the usage of warfarin by using the pharmacovigilance data from the FDA Adverse Event Reporting System (FAERS).

Methods

The ADEs in patients who took warfarin (N = 77,010) were analyzed using FAERS data. Association rule mining was applied to identify warfarin-related ADEs that were most associated with elevated INR (n = 15,091) as well as possible drug-drug interactions (DDIs) associated with increased INR. Lift values were used to identify ADEs that were most commonly reported alongside elevated INR based on the correlation between both item sets. In addition, this study sought to determine if the increased INR risk was influenced by sex, age, temporal distribution, and geographic distribution and were reported as reporting odds ratios (RORs).

Results

The top 5 ADEs most associated with increased INR in patients taking warfarin were decreased hemoglobin (lift = 2.31), drug interactions (lift = 1.88), hematuria (lift = 1.58), asthenia (lift = 1.44), and fall (lift = 1.32). INR risk increased as age increased, with individuals older than 80 having a 63% greater likelihood of elevated INR compared to those younger than 50. Males were 9% more likely to report increased INR as an ADE compared to females. Individuals taking warfarin concomitantly with at least one other drug were 43% more likely to report increased INR. The top 5 most frequently identified DDIs in patients taking warfarin and presenting with elevated INR were acetaminophen (lift = 1.81), ramipril (lift = 1.71), furosemide (lift = 1.64), bisoprolol (lift = 1.58), and simvastatin (lift = 1.58).

Conclusion

The risk of elevated INR increased as patient age increased, particularly among those older than 80. Elevated INR frequently co-presented with decreased hemoglobin, drug interactions, hematuria, asthenia, and fall in patients taking warfarin. This effect may be less pronounced in women due to the procoagulatory effects of estrogen signaling. Multiple possible DDIs were identified, including acetaminophen, ramipril, and furosemide.

Improving drug safety with adverse event detection using natural language processing

INTRODUCTION

Pharmacovigilance (PV) involves monitoring and aggregating adverse event information from a variety of data sources, including health records, biomedical literature, spontaneous adverse event reports, product labels, and patient-generated content like social media posts, but the most pertinent details in these sources are typically available in narrative free-text formats. Natural language processing (NLP) techniques can be used to extract clinically relevant information from PV texts to inform decision-making.

AREAS COVERED

We conducted a non-systematic literature review by querying the PubMed database to examine the uses of NLP in drug safety and distilled the findings to present our expert opinion on the topic.

EXPERT OPINION

New NLP techniques and approaches continue to be applied for drug safety use cases; however, systems that are fully deployed and in use in a clinical environment remain vanishingly rare. To see high-performing NLP techniques implemented in the real setting will require long-term engagement with end users and other stakeholders and revised workflows in fully formulated business plans for the targeted use cases. Additionally, we found little to no evidence of extracted information placed into standardized data models, which should be a way to make implementations more portable and adaptable.

A Scoping Literature Review of Natural Language Processing Application to Safety Occurrence Reports

Safety occurrence reports can contain valuable information on how incidents occur, revealing knowledge that can assist safety practitioners. This paper presents and discusses a literature review exploring how Natural Language Processing (NLP) has been applied to occurrence reports within safety-critical industries, informing further research on the topic and highlighting common challenges. Some of the uses of NLP include the ability for occurrence reports to be automatically classified against categories, and entities such as causes and consequences to be extracted from the text as well as the semantic searching of occurrence databases. The review revealed that machine learning models form the dominant method when applying NLP, although rule-based algorithms still provide a viable option for some entity extraction tasks. Recent advances in deep learning models such as Bidirectional Transformers for Language Understanding are now achieving a high accuracy while eliminating the need to substantially pre-process text. The construction of safety-themed datasets would be of benefit for the application of NLP to occurrence reporting, as this would allow the fine-tuning of current language models to safety tasks. An interesting approach is the use of topic modelling, which represents a shift away from the prescriptive classification taxonomies, splitting data into “topics”. Where many papers focus on the computational accuracy of models, they would also benefit from real-world trials to further inform usefulness. It is anticipated that NLP will soon become a mainstream tool used by safety practitioners to efficiently process and gain knowledge from safety-related text.

Development of a multivariate prediction model to identify individual case safety reports which require clinical review

BACKGROUND

The number of Individual Case Safety Reports (ICSRs) in pharmacovigilance databases are rapidly increasing world-wide. The majority of ICSRs at the Netherlands Pharmacovigilance Centre Lareb is reviewed manually to identify potential signal triggering reports (PSTR) or ICSRs which need further clinical assessment for other reasons.

OBJECTIVES

To develop a prediction model to identify ICSRs that require clinical review, including PSTRs. Secondly, to identify the most important features of these reports.

METHODS

All ICSRs (n = 30 424) received by Lareb between October 1, 2017 and February 26, 2021 were included. ICSRs originating from marketing authorisation holders and ICSRs reported on vaccines were excluded. The outcome was defined as PSTR (yes/no), where PSTR 'yes' was defined as an ICSR discussed at a signal detection meeting. Nineteen features were included, concerning structured information on: patients, adverse drug reactions (ADR) or drugs. Data were divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net logistic regression and eXtreme Gradient Boosting models were trained and tuned on a training set. Random down-sampling of negative controls was applied on the training set to adjust for the imbalanced dataset. Final models were evaluated on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re-train and test model performances with fewer features.

RESULTS

1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73-0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: 'absence of ADR in the Summary of product characteristics', 'ADR reported as serious', 'ADR labelled as an important medical event', 'ADR reported by physician' and 'positive rechallenge'. Model performances were similar when using only nine of the most important features.

CONCLUSIONS

We developed a prediction model with moderate performances to identify PSTRs with nine commonly available features. Optimisation of the model using more ICSR information (e.g., free text fields) to increase model precision is required before implementation.

Machine Learning in Causal Inference: Application in Pharmacovigilance

Monitoring adverse drug events or pharmacovigilance has been promoted by the World Health Organization to assure the safety of medicines through a timely and reliable information exchange regarding drug safety issues. We aim to discuss the application of machine learning methods as well as causal inference paradigms in pharmacovigilance. We first reviewed data sources for pharmacovigilance. Then, we examined traditional causal inference paradigms, their applications in pharmacovigilance, and how machine learning methods and causal inference paradigms were integrated to enhance the performance of traditional causal inference paradigms. Finally, we summarized issues with currently mainstream correlation-based machine learning models and how the machine learning community has tried to address these issues by incorporating causal inference paradigms. Our literature search revealed that most existing data sources and tasks for pharmacovigilance were not designed for causal inference. Additionally, pharmacovigilance was lagging in adopting machine learning-causal inference integrated models. We highlight several currently trending directions or gaps to integrate causal inference with machine learning in pharmacovigilance research. Finally, our literature search revealed that the adoption of causal paradigms can mitigate known issues with machine learning models. We foresee that the pharmacovigilance domain can benefit from the progress in the machine learning field.

"Artificial Intelligence" for Pharmacovigilance: Ready for Prime Time?

There is great interest in the application of 'artificial intelligence' (AI) to pharmacovigilance (PV). Although US FDA is broadly exploring the use of AI for PV, we focus on the application of AI to the processing and evaluation of Individual Case Safety Reports (ICSRs) submitted to the FDA Adverse Event Reporting System (FAERS). We describe a general framework for considering the readiness of AI for PV, followed by some examples of the application of AI to ICSR processing and evaluation in industry and FDA. We conclude that AI can usefully be applied to some aspects of ICSR processing and evaluation, but the performance of current AI algorithms requires a 'human-in-the-loop' to ensure good quality. We identify outstanding scientific and policy issues to be addressed before the full potential of AI can be exploited for ICSR processing and evaluation, including approaches to quality assurance of 'human-in-the-loop' AI systems, large-scale, publicly available training datasets, a well-defined and computable 'cognitive framework', a formal sociotechnical framework for applying AI to PV, and development of best practices for applying AI to PV. Practical experience with stepwise implementation of AI for ICSR processing and evaluation will likely provide important lessons that will inform the necessary policy and regulatory framework to facilitate widespread adoption and provide a foundation for further development of AI approaches to other aspects of PV.

Text Mining of Adverse Events in Clinical Trials: Deep Learning Approach

BACKGROUND

Pharmacovigilance and safety reporting, which involve processes for monitoring the use of medicines in clinical trials, play a critical role in the identification of previously unrecognized adverse events or changes in the patterns of adverse events.

OBJECTIVE

This study aims to demonstrate the feasibility of automating the coding of adverse events described in the narrative section of the serious adverse event report forms to enable statistical analysis of the aforementioned patterns.

METHODS

We used the Unified Medical Language System (UMLS) as the coding scheme, which integrates 217 source vocabularies, thus enabling coding against other relevant terminologies such as the International Classification of Diseases-10th Revision, Medical Dictionary for Regulatory Activities, and Systematized Nomenclature of Medicine). We used MetaMap, a highly configurable dictionary lookup software, to identify the mentions of the UMLS concepts. We trained a binary classifier using Bidirectional Encoder Representations from Transformers (BERT), a transformer-based language model that captures contextual relationships, to differentiate between mentions of the UMLS concepts that represented adverse events and those that did not.

RESULTS

The model achieved a high F1 score of 0.8080, despite the class imbalance. This is 10.15 percent points lower than human-like performance but also 17.45 percent points higher than that of the baseline approach.

CONCLUSIONS

These results confirmed that automated coding of adverse events described in the narrative section of serious adverse event reports is feasible. Once coded, adverse events can be statistically analyzed so that any correlations with the trialed medicines can be estimated in a timely fashion.

Reporting of Death in US Food and Drug Administration Medical Device Adverse Event Reports in Categories Other Than Death

IMPORTANCE

In the US, most postmarket medical device safety data are obtained through adverse event reports that are submitted to the US Food and Drug Administration (FDA)'s Manufacturer and User Facility Device Experience (MAUDE) database. Adverse event reports are classified by the reporter as injury, malfunction, death, or other. If the device may have caused or contributed to a death, or if the cause of death is unknown, the FDA requires that the adverse event be reported as a death.

OBJECTIVE

To determine the percentage of medical device adverse event reports submitted to the MAUDE database that were not classified as death even though the patient died.

DESIGN, SETTING, AND PARTICIPANTS

In this study, a natural language processing algorithm was applied to the MAUDE database, followed by manual text review, to identify reports in the injury, malfunction, other or missing categories that included at least 1 term that suggested a patient death, such as patient died or patient expired, from December 31, 1991, to April 30, 2020, for any medical device.

EXPOSURES

Manual review of a random sample of 1000 adverse event reports not classified as death and of selected reports for 62 terms that are associated with deaths but were not classified as death.

MAIN OUTCOMES AND MEASURES

Percentage of adverse event reports in which the patient was said to have died in the narrative section of the report but the reporter classified the report in a category other than death.

RESULTS

The terms in the natural language processing algorithm identified 290 141 reports in which a serious injury or death was reported. Of these, 151 145 (52.1%) were classified by the reporter as death and 47.9% were classified as malfunction, injury, other, or missing. For the overall sample, the percentage of reports with deaths that were not classified as deaths was 23% (95% CI, 20%-25%), suggesting that approximately 31 552 reports in our sample had deaths that were classified in other categories. The overall percentage of missed deaths, defined as the percentage of deaths that were classified in other categories, was 17% (95% CI, 16%-19%).

CONCLUSIONS AND RELEVANCE

Many of the findings of this study suggest that many medical device adverse event reports in the FDA's MAUDE database that involved a patient death are classified in categories other than death. As the FDA only routinely reviews all adverse events that are reported as patient deaths, improving the accuracy of adverse event reporting may enhance patient safety.

Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System

BACKGROUND

Our objective was to support the automated classification of Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) reports for their usefulness in assessing the possibility of a causal relationship between a drug product and an adverse event.

METHOD

We used a data set of 326 redacted FAERS reports that was previously annotated using a modified version of the World Health Organization-Uppsala Monitoring Centre criteria for drug causality assessment by a group of SEs at the FDA and supported a similar study on the classification of reports using supervised machine learning and text engineering methods. We explored many potential features, including the incorporation of natural language processing on report text and information from external data sources, for supervised learning and developed models for predicting the classification status of reports. We then evaluated the models on a larger data set of previously unseen reports.

RESULTS

The best-performing models achieved recall and F1 scores on both data sets above 0.80 for the identification of assessable reports (i.e. those containing enough information to make an informed causality assessment) and above 0.75 for the identification of reports meeting at least a Possible causality threshold.

CONCLUSIONS

Causal inference from FAERS reports depends on many components with complex logical relationships that are yet to be made fully computable. Efforts focused on readily addressable tasks, such as quickly eliminating unassessable reports, fit naturally in SE's thought processes to provide real enhancements for FDA workflows.

Evolution of Hematology Clinical Trial Adverse Event Reporting to Improve Care Delivery

PURPOSE OF REVIEW

Reporting of adverse events on hematology clinical trials is crucial to understanding the safety of standard treatments and novel agents. However, despite the importance of understanding toxicities, challenges in capturing and reporting accurate adverse event data exist.

RECENT FINDINGS

Currently, adverse events are reported manually on most hematology clinical trials. Especially on phase III trials, the highest grade of each adverse event during a reporting period is typically reported. Despite the effort committed to AE reporting, studies have identified underreporting of adverse events on hematologic malignancy clinical trials, which raises concern about the true understanding of safety of treatment that clinicians have in order to guide patients about what to expect during therapy. In order to address these concerns, recent studies have piloted alternative methods for identification of adverse events. These methods include automated extraction of adverse event data from the electronic health record, implementation of trigger or alert tools into the medical record, and analytic tools to evaluate duration of adverse events rather than only the highest adverse event grade. Adverse event reporting is a crucial component of clinical trials. Novel tools for identifying and reporting adverse events provide opportunities for honing and refining methods of toxicity capture and improving understanding of toxicities patients experience while enrolled on clinical trials.

Towards Automating Adverse Event Review: A Prediction Model for Case Report Utility

INTRODUCTION

The rapidly expanding size of the Food and Drug Administration's (FDA) Adverse Event Reporting System database requires modernized pharmacovigilance practices. Techniques to systematically identify high utility individual case safety reports (ICSRs) will support safety signal management.

OBJECTIVES

The aim of this study was to develop and validate a model predictive of an ICSR's pharmacovigilance utility (PVU).

METHODS

PVU was operationalized as an ICSR's inclusion in an FDA-authored pharmacovigilance review's case series supporting a recommendation to modify product labeling. Multivariable logistic regression models were used to examine the association between PVU and ICSR features. The best performing model was selected for bootstrapping validation. As a sensitivity analysis, we evaluated the model's performance across subgroups of safety issues.

RESULTS

We identified 10,381 ICSRs evaluated in 69 pharmacovigilance reviews, of which 2115 ICSRs were included in a case series. The strongest predictors of ICSR inclusion were reporting of a designated medical event (odds ratio (OR) 1.93, 95% CI 1.54-2.43) and positive dechallenge (OR 1.67, 95% CI 1.50-1.87). The strongest predictors of ICSR exclusion were death reported as the only outcome (OR 2.72, 95% CI 1.76-4.35), more than three suspect products (OR 2.69, 95% CI 2.23-3.24), and > 15 preferred terms reported (OR 2.69, 95% CI 1.90-3.82). The validated model showed modest discriminative ability (C-statistic of 0.71). Our sensitivity analysis demonstrated heterogeneity in model performance by safety issue (C-statistic range 0.58-0.74).

CONCLUSIONS

Our model demonstrated the feasibility of developing a tool predictive of ICSR utility. The model's modest discriminative ability highlights opportunities for further enhancement and suggests algorithms tailored to safety issues may be beneficial.

Artificial Intelligence, Real-World Automation and the Safety of Medicines

Despite huge technological advances in the capabilities to capture, store, link and analyse data electronically, there has been some but limited impact on routine pharmacovigilance. We discuss emerging research in the use of artificial intelligence, machine learning and automation across the pharmacovigilance lifecycle including pre-licensure. Reasons are provided on why adoption is challenging and we also provide a perspective on changes needed to accelerate adoption, and thereby improve patient safety. Last, we make clear that while technologies could be superimposed on existing pharmacovigilance processes for incremental improvements, these great societal advances in data and technology also provide us with a timely opportunity to reconsider everything we do in pharmacovigilance operations to maximise the benefit of these advances.

A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

CONTEXT

Adverse events in healthcare are often collated in incident reports which contain unstructured free text. Learning from these events may improve patient safety. Natural language processing (NLP) uses computational techniques to interrogate free text, reducing the human workload associated with its analysis. There is growing interest in applying NLP to patient safety, but the evidence in the field has not been summarised and evaluated to date.

OBJECTIVE

To perform a systematic literature review and narrative synthesis to describe and evaluate NLP methods for classification of incident reports and adverse events in healthcare.

METHODS

Data sources included Medline, Embase, The Cochrane Library, CINAHL, MIDIRS, ISI Web of Science, SciELO, Google Scholar, PROSPERO, hand searching of key articles, and OpenGrey. Data items were manually abstracted to a standardised extraction form.

RESULTS

From 428 articles screened for eligibility, 35 met the inclusion criteria of using NLP to perform a classification task on incident reports, or with the aim of detecting adverse events. The majority of studies used free text from incident reporting systems or electronic health records. Models were typically designed to classify by type of incident, type of medication error, or harm severity. A broad range of NLP techniques are demonstrated to perform these classification tasks with favourable performance outcomes. There are methodological challenges in how these results can be interpreted in a broader context.

CONCLUSION

NLP can generate meaningful information from unstructured data in the specific domain of the classification of incident reports and adverse events. Understanding what or why incidents are occurring is important in adverse event analysis. If NLP enables these insights to be drawn from larger datasets it may improve the learning from adverse events in healthcare.

Pharmacovigilance in India in Comparison With the USA and European Union: Challenges and Perspectives

Pharmacovigilance (PV) is an integral part of the drug regulation system. PV plays an indispensable role in the identification, assessment, and publicizing of adverse drug reactions (ADRs) through various methods. ADRs account for serious harm to the patients and even lead to morbidity and mortality. The PV databases help in the promotion of safe drug use and protection of public health safety. This article compares the PV system in the USA, Europe, and India, highlighting the challenges and future perspectives to be adapted to widen the horizon of the existing PV structure in India. In India, PV programs are still at the dawning stage when paralleled to the other countries. The National Pharmacovigilance Program and the Pharmacovigilance Program of India are the most recent advancements in this field in the country. The USA and Europe have well-established PV systems in place thanks to technological progress and other resources. India is the largest producer of pharmaceuticals in the world and a major clinical research hub; hence, it requires a more stringent PV setup. With the increase in population and novel drugs in the market each day, there is a need for an effective PV system in India.

ADEPt, a semantically-enriched pipeline for extracting adverse drug events from free-text electronic health records

Adverse drug events (ADEs) are unintended responses to medical treatment. They can greatly affect a patient's quality of life and present a substantial burden on healthcare. Although Electronic health records (EHRs) document a wealth of information relating to ADEs, they are frequently stored in the unstructured or semi-structured free-text narrative requiring Natural Language Processing (NLP) techniques to mine the relevant information. Here we present a rule-based ADE detection and classification pipeline built and tested on a large Psychiatric corpus comprising 264k patients using the de-identified EHRs of four UK-based psychiatric hospitals. The pipeline uses characteristics specific to Psychiatric EHRs to guide the annotation process, and distinguishes: a) the temporal value associated with the ADE mention (whether it is historical or present), b) the categorical value of the ADE (whether it is assertive, hypothetical, retrospective or a general discussion) and c) the implicit contextual value where the status of the ADE is deduced from surrounding indicators, rather than explicitly stated. We manually created the rulebase in collaboration with clinicians and pharmacists by studying ADE mentions in various types of clinical notes. We evaluated the open-source Adverse Drug Event annotation Pipeline (ADEPt) using 19 ADEs specific to antipsychotics and antidepressants medication. The ADEs chosen vary in severity, regularity and persistence. The average F-measure and accuracy achieved by our tool across all tested ADEs were 0.83 and 0.83 respectively. In addition to annotation power, the ADEPT pipeline presents an improvement to the state of the art context-discerning algorithm, ConText.