A prediction model-based algorithm for computer-assisted database screening of adverse drug reactions in the Netherlands

Evaluation of four machine learning models for signal detection

Background

Logistic regression-based signal detection algorithms have benefits over disproportionality analysis due to their ability to handle potential confounders and masking factors. Feature exploration and developing alternative machine learning algorithms can further strengthen signal detection.

Objectives

Our objective was to compare the signal detection performance of logistic regression, gradient-boosted trees, random forest and support vector machine models utilizing Food and Drug Administration adverse event reporting system data.

Design

Cross-sectional study.

Methods

The quarterly data extract files from 1 October 2017 through 31 December 2020 were downloaded. Due to an imbalanced outcome, two training sets were used: one stratified on the outcome variable and another using Synthetic Minority Oversampling Technique (SMOTE). A crude model and a model with tuned hyperparameters were developed for each algorithm. Model performance was compared against a reference set using accuracy, precision, F1 score, recall, the receiver operating characteristic area under the curve (ROCAUC), and the precision-recall curve area under the curve (PRCAUC).

Results

Models trained on the balanced training set had higher accuracy, F1 score and recall compared to models trained on the SMOTE training set. When using the balanced training set, logistic regression, gradient-boosted trees, random forest and support vector machine models obtained similar performance evaluation metrics. The gradient-boosted trees hyperparameter tuned model had the highest ROCAUC (0.646) and the random forest crude model had the highest PRCAUC (0.839) when using the balanced training set.

Conclusion

All models trained on the balanced training set performed similarly. Logistic regression models had higher accuracy, precision and recall. Logistic regression, random forest and gradient-boosted trees hyperparameter tuned models had a PRCAUC ⩾ 0.8. All models had an ROCAUC ⩾ 0.5. Including both disproportionality analysis results and additional case report information in models resulted in higher performance evaluation metrics than disproportionality analysis alone.

Finding Needles in the Haystack: Clinical Utility Score for Prioritisation (CUSP), an Automated Approach for Identifying Spontaneous Reports with the Highest Clinical Utility

INTRODUCTION

Spontaneous reporting of adverse events has increased steadily over the past decades, and although this trend has contributed to improving post-marketing surveillance pharmacovigilance activities, the consequent amount of data generated is challenging to manually review during assessment, with each individual report requiring review by pharmacovigilance experts. This highlights a clear need for alternative or complementary methodologies to help prioritise review.

OBJECTIVE

Here, we aimed to develop and test an automated methodology, the Clinical Utility Score for Prioritisation (CUSP), to assist pharmacovigilance experts in prioritising clinical assessment of safety data to improve the rapidity of case series review when case volumes are large.

METHODS

The CUSP method was tested on a reference dataset of individual case safety reports (ICSRs) associated to five drug-event pairs that led to labelling changes. The selected drug-event pairs were of varying characteristics across the portfolio of GSK's products.

RESULTS

The mean CUSP score for 'key cases' and 'cases of low utility' was 19.7 (median: 21; range: 7-27) and 17.3 (median: 19; range: 4-27), respectively. CUSP distribution for 'key cases' were skewed toward the higher range of scores compared with 'all cases'. The overall performance across each individual drug-event pair varied considerably, showing higher predictive power for 'key cases' for three of the drug-event pairs (average CUSP between these three: 22.8; range: 22.5-23.0) and lesser power for the remaining two (average CUSP between these two: 17.6; range: 14.5-20.7).

CONCLUSION

Although several tools have been developed to assess ICSR completeness and regulatory utility, this is the first attempt to successfully develop an automated clinical utility scoring system that can support the prioritisation of ICSRs for clinical review.

Performance of subgrouped proportional reporting ratios in the US Food and Drug Administration (FDA) adverse event reporting system

BACKGROUND

Many signal detection algorithms give the same weight to information from all products and patients, which may result in signals being masked or false positives being flagged as potential signals. Subgrouped analysis can be used to help correct for this.

RESEARCH DESIGN AND METHODS

The publicly available US Food and Drug Administration Adverse Event Reporting System quarterly data extract files from 1 January 2015 through 30 September 2017 were utilized. A proportional reporting ratio (PRR) analysis subgrouped by either age, sex, ADE report type, seriousness of ADE, or reporter was compared to the crude PRR analysis using sensitivity, specificity, precision, and c-statistic.

RESULTS

Subgrouping by age (n = 78, 34.5% increase), sex (n = 67, 15.5% increase), and reporter (n = 64, 10.3% increase) identified more signals than the crude analysis. Subgrouping by either age or sex increased both the sensitivity and precision. Subgrouping by report type or seriousness resulted in fewer signals (n = 50, -13.8% for both). Subgrouped analyses had higher c-statistic values, with age having the highest (0.468).

CONCLUSIONS

Subgrouping by either age or sex produced more signals with higher sensitivity and precision than the crude PRR analysis. Subgrouping by these variables can unmask potentially important associations.

Observed-over-Expected analysis as additional method for pharmacovigilance signal detection in large-scaled spontaneous adverse event reporting

BACKGROUND

The large-scale COVID-19 vaccination campaigns in 2021 and 2022 led to a rapid increase in numbers of received adverse event reports in spontaneous reporting systems. As background incidences of naturally occurring medical events became increasingly relevant for causality assessment of potential associations with the vaccines, a novel approach for signal detection was warranted.

OBJECTIVES

This article illustrates the Observed-over-Expected (O/E) analysis as an additional method for signal detection and risk assessment in large-scaled spontaneous reporting systems.

METHODS

All individual case safety reports (ICSRs) concerning idiopathic peripheral facial paralysis or Bell's palsy following administration of the COVID-19 vaccines (n = 291) manufactured by Pfizer/BioNTech (Comirnaty), Moderna (Spikevax), AstraZeneca (Vaxzevria) and Janssen (JCOVDEN) received by the National Pharmacovigilance Centre Lareb until 24th March 2022 were included in the O/E analysis, within a risk window of 7 and 14 days following immunisation. Dutch background incidence rates from 2019 and exposure of the Dutch population to the vaccines were obtained from the PHARMO institute and RIVM. The data was stratified in age groups, gender and administered dose in order to differentiate between population subgroups.

RESULTS

Bell's palsy was reported more frequently than expected in several population subgroups following administration of all four COVID-19 vaccines, including children and adolescents. Among children, a high O/E ratio was found for boys aged 5-14 years after receiving the Pfizer/BioNTech vaccine. Regarding adolescents and young adults, women aged 15-24 years receiving Pfizer/BioNTech and Moderna, and men aged 15-24 years receiving Janssen developed Bell's palsy more often than expected. Furthermore, O/E ratios were high for individuals aged 25-64, regarding females receiving Pfizer, Moderna and AstraZeneca and males receiving Janssen. As facial paralysis was not labelled as an adverse event for the Janssen vaccine, this analysis contributed to signalling the association and warranting further regulatory action.

CONCLUSIONS

The O/E method is a useful approach for signal detection of potential adverse reactions when handling large numbers of ICSRs. Further research is needed to attest to the causality on a clinical basis.

Development of a multivariate prediction model to identify individual case safety reports which require clinical review

BACKGROUND

The number of Individual Case Safety Reports (ICSRs) in pharmacovigilance databases are rapidly increasing world-wide. The majority of ICSRs at the Netherlands Pharmacovigilance Centre Lareb is reviewed manually to identify potential signal triggering reports (PSTR) or ICSRs which need further clinical assessment for other reasons.

OBJECTIVES

To develop a prediction model to identify ICSRs that require clinical review, including PSTRs. Secondly, to identify the most important features of these reports.

METHODS

All ICSRs (n = 30 424) received by Lareb between October 1, 2017 and February 26, 2021 were included. ICSRs originating from marketing authorisation holders and ICSRs reported on vaccines were excluded. The outcome was defined as PSTR (yes/no), where PSTR 'yes' was defined as an ICSR discussed at a signal detection meeting. Nineteen features were included, concerning structured information on: patients, adverse drug reactions (ADR) or drugs. Data were divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net logistic regression and eXtreme Gradient Boosting models were trained and tuned on a training set. Random down-sampling of negative controls was applied on the training set to adjust for the imbalanced dataset. Final models were evaluated on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re-train and test model performances with fewer features.

RESULTS

1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73-0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: 'absence of ADR in the Summary of product characteristics', 'ADR reported as serious', 'ADR labelled as an important medical event', 'ADR reported by physician' and 'positive rechallenge'. Model performances were similar when using only nine of the most important features.

CONCLUSIONS

We developed a prediction model with moderate performances to identify PSTRs with nine commonly available features. Optimisation of the model using more ICSR information (e.g., free text fields) to increase model precision is required before implementation.

Adverse Drug Reactions of Intranasal Corticosteroids in the Netherlands: An Analysis from the Netherlands Pharmacovigilance Center

BACKGROUND

Intranasal corticosteroids are one of the cornerstone treatment options for allergic rhinitis and chronic sinusitis complaints. Safety information in the summary of product characteristics may not be representative for observations in daily clinical practice. The Netherlands Pharmacovigilance Center (Lareb) collects post-marketing safety information, using spontaneous reporting systems.

OBJECTIVE

Our objective was to analyse reports of adverse drug reactions associated with intranasal corticosteroids reported in the Dutch spontaneous reporting database of the Netherlands Pharmacovigilance Center Lareb to obtain insight into real-world safety data.

METHODS

We retrospectively examined all adverse drug reactions of intranasal corticosteroids reported to the Netherlands Pharmacovigilance Center Lareb, entered into the database from 1991 until 1 July, 2020.

RESULTS

In total, 2263 adverse drug reactions after intranasal corticosteroid use were reported in 1258 individuals. Headache (n = 143), epistaxis (n = 124) and anosmia (n = 57) were reported most frequently. Nasal septum perforation (reporting odds ratio 463.2; 95% confidence interval: 186.7-1149.7) had the highest reporting odds ratio, followed by nasal mucosal disorder (reporting odds ratio 104.5; 95% confidence interval 36.3-301.3) and hyposmia (reporting odds ratio 90.8; 95% confidence interval 45.1-182.7). Moreover, 101 (4.5%) reports were classified as serious by Lareb, including reports of Cushing's syndrome, adrenal cortical hypofunction and growth retardation.

CONCLUSIONS

Many side effects are consistent with the safety information in the summary of product characteristics of intranasal corticosteroids. Several serious (systemic) side effects are reported and it is important to realise that intranasal corticosteroids may contribute to the development. Healthcare providers and patients should be aware of the potential (individual) adverse drug reactions of intranasal corticosteroids. This information could help in discussing treatment options.

Safety of medicines and vaccines - building next generation capability

The systematic safety surveillance of real-world use of medicinal products and related activities (pharmacovigilance) started in earnest as a scientific field only in the 1960s. While developments have occurred over the past 50 years, adding to its complexity and sophistication, the extent to which some of these advances have positively impacted the capability for ensuring patient safety is questionable. We review how the conduct of safety surveillance has changed, highlight recent scientific advances, and argue how they need to be harnessed to enhance pharmacovigilance in the future. Specifically, we describe five changes that we believe should and will need to happen globally in the coming years: (i) better, more diverse data used for safety; (ii) the switch from manual activities to automation; (iii) removal of limited value, extraneous transactional activities and replacement with sharpened focus on scientific efforts to improve patient safety; (iv) patient-involved and focussed safety; and (v) personalised safety.

Evaluating Drug Risk Using GAN and SMOTE Based on CFDA's Spontaneous Reporting Data

Adverse drug reactions (ADRs) pose health threats to humans. Therefore, the risk re-evaluation of post-marketing drugs has become an important part of the pharmacovigilance work of various countries. In China, drugs are mainly divided into three categories, from high-risk to low-risk drugs, namely, prescription drugs (Rx), over-the-counter drugs A (OTC-A), and over-the-counter drugs B (OTC-B). Until now, there has been a lack of automated evaluation methods for the three status switch of drugs. Based on China Food and Drug Administration's (CFDA) spontaneous reporting database (CSRD), we proposed a classification model to predict risk level of drugs by using feature enhancement based on Generative Adversarial Networks (GAN) and Synthetic Minority Over-Sampling Technique (SMOTE). A total of 985,960 spontaneous reports from 2011 to 2018 were selected from CSRD in Jiangsu Province as experimental data. After data preprocessing, a class-imbalance data set was obtained, which contained 887 Rx (accounting for 84.72%), 113 OTC-A (10.79%), and 47 OTC-B (4.49%). Taking drugs as the samples, ADRs as the features, and signal detection results obtained by proportional reporting ratio (PRR) method as the feature values, we constructed the original data matrix, where the last column represents the category label of each drug. Our proposed model expands the ADR data from both the sample space and the feature space. In terms of feature space, we use feature selection (FS) to screen ADR symptoms with higher importance scores. Then, we use GAN to generate artificial data, which are added to the feature space to achieve feature enhancement. In terms of sample space, we use SMOTE technology to expand the minority samples to balance three categories of drugs and minimize the classification deviation caused by the gap in the sample size. Finally, we use random forest (RF) algorithm to classify the feature-enhanced and balanced data set. The experimental results show that the accuracy of the proposed classification model reaches 98%. Our proposed model can well evaluate drug risk levels and provide automated methods for status switch of post-marketing drugs.

Signals from the Dutch national spontaneous reporting system: Characteristics and regulatory actions

PURPOSE

The aim of the study is to characterise safety signals based on the Dutch spontaneous reporting system (SRS) and to investigate the association between signal characteristics and Product Information (PI) update stratified by approval type: centrally authorised products (CAPs) versus nationally and decentralised authorised products (NAPs).

METHODS

This study evaluates the full cohort of signals disseminated from the Dutch SRS in the period from 2008 to 2017. Each retrieved signal was characterised on a number of aspects. The signal management process from signal generation to a potential PI update was analysed in four steps: (1) signal characterisation; (2) proposed actions by the Dutch national competent authority (NCA) for the signals; (3) presence of PI update (yes/no) and association with signal characteristics; (4) timing from the moment the signal was issued to PI update. For step 1-3 we stratified products in CAPs and NAPs.

RESULTS

Of all signals, 88.7% led to a proposed regulatory action by the NCA. Signals from the Dutch SRS for CAPs versus NAPs more often concerned biologicals, important medical events, class effects and shorter periods since marketing authorization. We detected PI updates for 26.2% of CAP signals and 61.3% of NAP signals.

CONCLUSIONS

The Dutch SRSs remains an important source of signals. There are some notable differences in the characteristics of signals for CAPs versus NAPs. Signals for NAPs more frequently led to PI updates.

Towards Automating Adverse Event Review: A Prediction Model for Case Report Utility

INTRODUCTION

The rapidly expanding size of the Food and Drug Administration's (FDA) Adverse Event Reporting System database requires modernized pharmacovigilance practices. Techniques to systematically identify high utility individual case safety reports (ICSRs) will support safety signal management.

OBJECTIVES

The aim of this study was to develop and validate a model predictive of an ICSR's pharmacovigilance utility (PVU).

METHODS

PVU was operationalized as an ICSR's inclusion in an FDA-authored pharmacovigilance review's case series supporting a recommendation to modify product labeling. Multivariable logistic regression models were used to examine the association between PVU and ICSR features. The best performing model was selected for bootstrapping validation. As a sensitivity analysis, we evaluated the model's performance across subgroups of safety issues.

RESULTS

We identified 10,381 ICSRs evaluated in 69 pharmacovigilance reviews, of which 2115 ICSRs were included in a case series. The strongest predictors of ICSR inclusion were reporting of a designated medical event (odds ratio (OR) 1.93, 95% CI 1.54-2.43) and positive dechallenge (OR 1.67, 95% CI 1.50-1.87). The strongest predictors of ICSR exclusion were death reported as the only outcome (OR 2.72, 95% CI 1.76-4.35), more than three suspect products (OR 2.69, 95% CI 2.23-3.24), and > 15 preferred terms reported (OR 2.69, 95% CI 1.90-3.82). The validated model showed modest discriminative ability (C-statistic of 0.71). Our sensitivity analysis demonstrated heterogeneity in model performance by safety issue (C-statistic range 0.58-0.74).

CONCLUSIONS

Our model demonstrated the feasibility of developing a tool predictive of ICSR utility. The model's modest discriminative ability highlights opportunities for further enhancement and suggests algorithms tailored to safety issues may be beneficial.

Assessment of factors associated with completeness of spontaneous adverse event reporting in the United States: A comparison between consumer reports and healthcare professional reports

What is known and objective

The objectives of this study were to explore completeness of direct adverse event (AE) reports from consumers and healthcare professionals (HCPs), and to discuss the reasons completeness varied among reporters with different occupations.

Methods

We used a total of 5475 direct AE reports to the United States (US) Food and Drug Administration (FDA) from the first and second quarters of 2016 and assessed completeness of basic information (eg, patient sex, age, weight) and information relevant to AEs (eg, suspect and concomitant drugs). Logistic regression analysis was conducted to evaluate the associations between report completeness and reporting backgrounds.

Results and discussion

The completeness of AE reports from consumers was generally greater than that of reports from HCPs. Completeness of specific items varied among different occupations, which may reflect accessibility to, and/or availability of, relevant information for each type of reporter. There was a clear association between the proportion of ‘known’ ADRs in a report and completeness, suggesting that consumers and HCPs are likely to consult labelling information when reporting AEs.

What is new and conclusion

The quality of AE reports seemed to depend on information costs accrued to potential reporters. Researchers should consider the impact of database heterogeneity and possible sample selection bias when using spontaneous AE reports as a sample of events in the United States.

Future Directions of Pharmacovigilance Studies Using Electronic Medical Recording and Human Genetic Databases

Adverse drug reactions (ADRs) constitute key factors in determining successful medication therapy in clinical situations. Integrative analysis of electronic medical record (EMR) data and use of proper analytical tools are requisite to conduct retrospective surveillance of clinical decisions on medications. Thus, we suggest that electronic medical recording and human genetic databases are considered together in future directions of pharmacovigilance. We analyzed EMR-based ADR studies indexed on PubMed during the period from 2005 to 2017 and retrospectively acquired 1161 (29.6%) articles describing drug-induced adverse reactions (e.g., liver, kidney, nervous system, immune system, and inflammatory responses). Of them, only 102 (8.79%) articles contained useful information to detect or predict ADRs in the context of clinical medication alerts. Since insufficiency of EMR datasets and their improper analyses may provide false warnings on clinical decision, efforts should be made to overcome possible problems on data-mining, analysis, statistics, and standardization. Thus, we address the characteristics and limitations on retrospective EMR database studies in hospital settings. Since gene expression and genetic variations among individuals impact ADRs, pharmacokinetics, and pharmacodynamics, appropriate paths for pharmacovigilance may be optimized using suitable databases available in public domain (e.g., genome-wide association studies (GWAS), non-coding RNAs, microRNAs, proteomics, and genetic variations), novel targets, and biomarkers. These efforts with new validated biomarker analyses would be of help to repurpose clinical and translational research infrastructure and ultimately future personalized therapy considering ADRs.

Time to onset in statistical signal detection revisited: A follow-up study in long-term onset adverse drug reactions

Purpose

In a previous study, we developed a signal detection method using the time to onset (TTO) of adverse drug reactions (ADRs). The aim of the current study was to investigate this method in a subset of ADRs with a longer TTO and to compare its performance with disproportionality analysis.

Methods

Using The Netherlands's spontaneous reporting database, TTO distributions for drug—ADR associations with a median TTO of 7 days or more were compared with other drugs with the same ADR using the two‐sample Anderson–Darling (AD) test. Presence in the Summary of Product Characteristics (SPC) was used as the gold standard for identification of a true ADR. Twelve combinations with different values for the number of reports and median TTO were tested. Performance in terms of sensitivity and positive predictive value (PPV) was compared with disproportionality analysis. A sensitivity analysis was performed to compare the results with those from the previous study.

Results

A total of 38 017 case reports, containing 32 478 unique drug—ADR associations. Sensitivity was lower for the TTO method (range 0.08‐0.34) compared with disproportionality analysis (range 0.60‐0.87), whereas PPV was similar for both methods (range 0.93‐1.0). The results from the sensitivity analysis were similar to the original analysis.

Conclusions

Because of its low sensitivity, the developed TTO method cannot replace disproportionality analysis as a signal detection tool. It may be useful in combination with other methods.

NSAIDs hypersensitivity: questions not resolved

PURPOSE OF REVIEW

NSAIDs are the drugs most frequently involved in hypersensitivity reactions (HSR). These are frequently prescribed at all ages. HSR are of great concern and can affect people at any age. These drugs can induce reactions by stimulating the adaptive immune system (IgE or T cell), known as selective responders or more frequently by abnormalities in biochemical pathways related with prostaglandin metabolism. These are known as cross-intolerant. With some exceptions, skin testing and in-vitro studies are of little value in selective responders.

RECENT FINDINGS

In the last years, several classifications have been provided based on clinical symptoms, time interval between drug intake and appearance of symptoms, response to other nonchemically related NSAIDs and the underlying disease. Based on this classification, several well differentiated categories within each group of entities cross-intolerant and selective responders are now recognized. The most complex groups for evaluation are cross-intolerant in which three major groups exist: NSAIDs exacerbated respiratory disease, NSAIDs exacerbated cutaneous disease and NSAIDs-induced urticaria/angioedema in the absence of chronic spontaneous urticaria. Within the selective responders, there are two mechanisms involved: drug-specific IgE or T-cell effector responses. New entities have been added to this classification like mixed reactions within the cross-intolerant category, that must manifest as anaphylaxis and multiple immediate selective reactions.

SUMMARY

The precise evaluation of patients with NSAIDs hypersensitivity following established guidelines will improve not only our understanding but also the management of these entities. As the number of patients affected with NSAIDs is important, further studies are warranted.

A prediction model-based algorithm for computer-assisted database screening of adverse drug reactions in the Netherlands

Purpose

The statistical screening of pharmacovigilance databases containing spontaneously reported adverse drug reactions (ADRs) is mainly based on disproportionality analysis. The aim of this study was to improve the efficiency of full database screening using a prediction model‐based approach.

Methods

A logistic regression‐based prediction model containing 5 candidate predictors was developed and internally validated using the Summary of Product Characteristics as the gold standard for the outcome. All drug‐ADR associations, with the exception of those related to vaccines, with a minimum of 3 reports formed the training data for the model. Performance was based on the area under the receiver operating characteristic curve (AUC). Results were compared with the current method of database screening based on the number of previously analyzed associations.

Results

A total of 25 026 unique drug‐ADR associations formed the training data for the model. The final model contained all 5 candidate predictors (number of reports, disproportionality, reports from healthcare professionals, reports from marketing authorization holders, Naranjo score). The AUC for the full model was 0.740 (95% CI; 0.734–0.747). The internal validity was good based on the calibration curve and bootstrapping analysis (AUC after bootstrapping = 0.739). Compared with the old method, the AUC increased from 0.649 to 0.740, and the proportion of potential signals increased by approximately 50% (from 12.3% to 19.4%).

Conclusions

A prediction model‐based approach can be a useful tool to create priority‐based listings for signal detection in databases consisting of spontaneous ADRs.