Are all quantitative postmarketing signal detection methods equal? Performance characteristics of logistic regression and Multi-item Gamma Poisson Shrinker

Real-World evidence revelations: The potential of patient support programmes to provide data on medication usage

Patient Support Programmes (PSPs) are used by the pharmaceutical industry to provide education and support to consumers to overcome the challenges they face managing their condition and treatment. Whilst there is an increasing number of PSPs, limited information is available on whether these programmes contribute to safety signals. PSPs do not have a scientific hypothesis, nor are they governed by a protocol. However, by their nature, PSPs inevitably generate adverse event (AE) reports. The main goal of the research was to gather all Novartis-initiated PSPs for sacubitril/valsartan, followed by research in the company safety database to identify all AE reports emanating from these PSPs. Core data sheets (CDS) were reviewed to assess if these PSPs contributed to any new, regulatory-authority approved, validated signals. Overall, AEs entered into the safety database from PSPs confirmed no contribution to CDS updates. Detailed review of real-world data revealed tablet splitting or taking one higher dose tablet a day instead of twice daily. This research, and subsequent analyses, revealed that PSPs did not impact safety label changes for sacubitril/valsartan. It revealed an important finding concerning drug utilisation i.e. splitting of sacubitril/valsartan tablets to reduce cost. This finding suggests that PSPs may contribute important real-world data on patterns of medication usage. There remains a paucity of literature available on this topic, hence further research is required to assess if it would be worth designing PSPs for collecting data on drug utilisation and (lack of) efficacy. Such information from PSPs could be important for all stakeholders.

Adverse events in the digital age and where to find them

Exponential growth of health-related data collected by digital tools is a reality within pharmaceutical and medical device research and development. Data generated through digital tools may be categorized as relevant to efficacy and/or safety. The enormity of these data requires the adoption of new approaches for processing and evaluation. Recognition of patterns within the safety data is vital for sponsors seeking regulatory approval for their new products. Non-traditional data sources may contain relevant safety information; early evaluation of these data will help to determine the product safety profile. Advanced technologies have allowed the development of digital tools to screen these data, which in some situations are classified as software as a medical devices and subject to clinical evaluation and post-marketing surveillance. Artificial intelligence may help to reduce or even eliminate noise from within these data, allowing safety experts to focus on the most pertinent evidence. We propose a data typology and provide considerations on how to define adverse events within different types of data, even where no human reporter exists. Proposals are made for the automation of screening processes. We consider validation aspects to support solutions that are proven to produce reliable results, and to deliver trusted outputs to stakeholders.

Web-Based Signal Detection Using Medical Forums Data in France: Comparative Analysis

BACKGROUND

While traditional signal detection methods in pharmacovigilance are based on spontaneous reports, the use of social media is emerging. The potential strength of Web-based data relies on their volume and real-time availability, allowing early detection of signals of disproportionate reporting (SDRs).

OBJECTIVE

This study aimed (1) to assess the consistency of SDRs detected from patients' medical forums in France compared with those detected from the traditional reporting systems and (2) to assess the ability of SDRs in identifying earlier than the traditional reporting systems.

METHODS

Messages posted on patients' forums between 2005 and 2015 were used. We retained 8 disproportionality definitions. Comparison of SDRs from the forums with SDRs detected in VigiBase was done by describing the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, receiver operating characteristics curve, and the area under the curve (AUC). The time difference in months between the detection dates of SDRs from the forums and VigiBase was provided.

RESULTS

The comparison analysis showed that the sensitivity ranged from 29% to 50.6%, the specificity from 86.1% to 95.5%, the PPV from 51.2% to 75.4%, the NPV from 68.5% to 91.6%, and the accuracy from 68% to 87.7%. The AUC reached 0.85 when using the metric empirical Bayes geometric mean. Up to 38% (12/32) of the SDRs were detected earlier in the forums than that in VigiBase.

CONCLUSIONS

The specificity, PPV, and NPV were high. The overall performance was good, showing that data from medical forums may be a valuable source for signal detection. In total, up to 38% (12/32) of the SDRs could have been detected earlier, thus, ensuring the increased safety of patients. Further enhancements are needed to investigate the reliability and validation of patients' medical forums worldwide, the extension of this analysis to all possible drugs or at least to a wider selection of drugs, as well as to further assess performance against established signals.

Cardiovascular safety signals with dipeptidyl peptidase-4 inhibitors: A disproportionality analysis among high-risk patients

PURPOSE

In 2008, the US Food and Drug Administration (FDA) issued Draft Guidance on investigating cardiovascular risk with oral diabetic drugs, including dipeptidyl peptidase-4 inhibitors (DPP-4i). In 2014, underpowered, post hoc analyses of clinical trials suggested an increased risk of heart failure with the use of these products. As such, we assessed disproportionate reporting of major adverse cardiac events (MACE) among reports for DPP-4i submitted to the FDA Adverse Event Reporting System (FAERS) from 2006 to 2015.

METHODS

We assessed the empirical Bayes geometric mean (EBGM) and its lower bound (EB05) of the relative reporting ratio for MACE among DPP-4i reports in the full FAERS database and in a subset of reports limited to cardiovascular and diabetic drugs. We then compared the EB05 in these 2 analyses and calculated the percent positive agreement for signals of disproportional reporting (SDRs) involving MACE.

RESULTS

Of 180.3 million adverse event reports, 13.4 million were for diabetic and cardiovascular drugs. In the cardiovascular subset, there was an SDR for heart failure with linagliptin (EB05 = 2782.47) and saxagliptin (EB05 = 2.40), myocardial infarction with alogliptin (EB05 = 290.11), and cerebral infarction with sitagliptin (EB05 = 2.80). Of the 14 MACE, 8 had a percent positive agreement ≥50% for an SDR in both analyses. Overall, the cardiovascular subset elicited 11 more SDRs for DPP-4i than the full dataset.

CONCLUSIONS

Postmarketing surveillance of DPP-4i through FAERS suggest increased reporting of MACE, supporting the current FDA warning of heart failure risk. This suggests the need for additional longitudinal, observational research into the association of DPP-4i and other MACE.

Risk of Cleft Lip and/or Palate Associated With Antiepileptic Drugs: Postmarketing Safety Signal Detection and Evaluation of Information Presented to Prescribers and Patients

BACKGROUND

The aim was to analyze safety data associated with the maternal use of antiepileptic drugs in pregnancy and to assess the risk of cleft lip and/or palate (CL/P) as an outcome in the neonate. A parallel objective was to assess the completeness of the safety information concerning pregnancy exposures in the Summary of Product Characteristics (SmPCs) and the Patient Information (PI) in the USA and the UK.

METHODS

We analyzed individual case safety reports of CL/P associated with antiepileptic drugs in the FDA Adverse Event Reporting System. For the antiepileptic drugs with signals (EB05 ≥ 2), we reviewed Drug Analysis Prints for CL/P cases in the UK Medicines and Healthcare products Regulatory Agency (MHRA). We performed descriptive analyses of relevant SmPCs and PIs in the UK and the USA using a checklist of recommendations collected from the literature.

RESULTS

In total 817 CL/P reports were identified for 12 antiepileptic drugs in the FDA Adverse Event Reporting System. Ten of the 12 antiepileptic drugs were associated with 156 CL/P cases in the MHRA Sentinel. Safety information concerning pregnancy was found to be more comprehensive in UK SmPCs than in the US equivalents.

CONCLUSIONS

There is statistical disproportionality in individual case safety reports indicative of an increased risk of CL/P with 12 antiepileptic drugs studied. More studies are required to explore the association between in utero exposure to antiepileptic drugs and the risk of CL/P. There are inconsistencies between the UK and US safety labels. CL/P associated with antiepileptic drugs is an important topic and requires providing inclusive, unbiased, up-to-date information to prescribers and women of childbearing age.

Identification of Substandard Medicines via Disproportionality Analysis of Individual Case Safety Reports

INTRODUCTION

The distribution and use of substandard medicines (SSMs) is a public health concern worldwide. The detection of SSMs is currently limited to expensive large-scale assay techniques such as high-performance liquid chromatography (HPLC). Since 2013, the Pharmacovigilance Department at Novartis Pharma AG has been analyzing drug-associated adverse events related to 'product quality issues' with the aim of detecting defective medicines using spontaneous reporting. The method of identifying SSMs with spontaneous reporting was pioneered by the Monitoring Medicines project in 2011.

METHODS

This retrospective review was based on data from the World Health Organization (WHO) Global individual case safety report (ICSR) database VigiBase^® collected from January 2001 to December 2014. We conducted three different stratification analyses using the Multi-item Gamma Poisson Shrinker (MGPS) algorithm through the Oracle Empirica data-mining software. In total, 24 preferred terms (PTs) from the Medical Dictionary for Regulatory Activities (MedDRA^®) were used to identify poor-quality medicines. To identify potential SSMs for further evaluation, a cutoff of 2.0 for EB05, the lower 95% interval of the empirical Bayes geometric mean (EBGM) was applied. We carried out a literature search for advisory letters related to defective medicinal products to validate our findings. Furthermore, we aimed to assess whether we could confirm two SSMs first identified by the Uppsala Monitoring Centre (UMC) with our stratification method.

RESULTS

The analysis of ICSRs based on the specified selection criteria and threshold yielded 2506 hits including medicinal products with an excess of reports of product quality defects relative to other medicines in the database. Further investigations and a pilot study in five authorized medicinal products (proprietary and generic) licensed by a single marketing authorization holder, containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine, were performed. This resulted in an output of 23 potential SSMs. The literature search identified two communications issued to health professionals concerning a substandard rivastigmine patch, which validated our initial findings. Furthermore, we identified excess reporting of product quality issues with an ethinyl estradiol/norgestrel combination and with salbutamol. These were categorized as confirmed clusters of substandard/spurious/falsely labelled/falsified/counterfeit (SSFFC) medical products by the UMC in 2014.

CONCLUSION

This study illustrates the value of data mining of spontaneous adverse event reports and the applicability of disproportionality analysis to identify potential SSMs.

Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank

Background

Detection of unknown risks with marketed medicines is key to securing the optimal care of individual patients and to reducing the societal burden from adverse drug reactions. Large collections of individual case reports remain the primary source of information and require effective analytics to guide clinical assessors towards likely drug safety signals. Disproportionality analysis is based solely on aggregate numbers of reports and naively disregards report quality and content. However, these latter features are the very fundament of the ensuing clinical assessment.

Objective

Our objective was to develop and evaluate a data-driven screening algorithm for emerging drug safety signals that accounts for report quality and content.

Methods

vigiRank is a predictive model for emerging safety signals, here implemented with shrinkage logistic regression to identify predictive variables and estimate their respective contributions. The variables considered for inclusion capture different aspects of strength of evidence, including quality and clinical content of individual reports, as well as trends in time and geographic spread. A reference set of 264 positive controls (historical safety signals from 2003 to 2007) and 5,280 negative controls (pairs of drugs and adverse events not listed in the Summary of Product Characteristics of that drug in 2012) was used for model fitting and evaluation; the latter used fivefold cross-validation to protect against over-fitting. All analyses were performed on a reconstructed version of VigiBase^® as of 31 December 2004, at around which time most safety signals in our reference set were emerging.

Results

The following aspects of strength of evidence were selected for inclusion into vigiRank: the numbers of informative and recent reports, respectively; disproportional reporting; the number of reports with free-text descriptions of the case; and the geographic spread of reporting. vigiRank offered a statistically significant improvement in area under the receiver operating characteristics curve (AUC) over screening based on the Information Component (IC) and raw numbers of reports, respectively (0.775 vs. 0.736 and 0.707, cross-validated).

Conclusions

Accounting for multiple aspects of strength of evidence has clear conceptual and empirical advantages over disproportionality analysis. vigiRank is a first-of-its-kind predictive model to factor in report quality and content in first-pass screening to better meet tomorrow’s post-marketing drug safety surveillance needs.

Choosing thresholds for statistical signal detection with the proportional reporting ratio

BACKGROUND

Identification of potential drug safety problems using statistical screening algorithms in routinely collected databases of adverse drug reactions (ADRs) requires decision rules based on thresholds of the chosen parameters. Choosing higher or lower thresholds changes both the sensitivity of the screening and the number of false alarms produced, and thus has an impact on the effectiveness of the detection process.

OBJECTIVE

The aim of this study was to evaluate the impact on the effectiveness of signal detection activities of choosing different warning thresholds for the proportional reporting ratio (PRR) and for the count of reports of any drug-event combination.

METHODS

Signal detection methods were tested within the EudraVigilance database of suspected ADRs. Using an established set of known ADRs, the number that could be detected and the changes in time gained for earlier investigation of the signal were calculated over a range of signal detection thresholds. These figures were set against the number of false positive signals produced by the statistical signal detection algorithms.

RESULTS

Higher thresholds for the lower confidence bound of the PRR produced fewer false positives but this benefit was offset by important losses of sensitivity in the detection of ADRs. By contrast, increases in the threshold for the count of a specific drug-event combination produced fewer false positives with little loss of either sensitivity or time gained for investigation of adverse events. A threshold of five compared with the current European Medicines Agency threshold of three gave a reduction of 25 % in false positive signals in return for a loss of 12 % in true signals detected early.

CONCLUSION

Changes in the standard threshold for the count of drug-event combinations can result in a substantial improvement in efficiency of the signal detection process. Initially this change might be applied only to products with a well-established safety profile.

Toward enhanced pharmacovigilance using patient-generated data on the internet

The promise of augmenting pharmacovigilance with patient-generated data drawn from the Internet was called out by a scientific committee charged with conducting a review of the current and planned pharmacovigilance practices of the US Food and Drug Administration (FDA). To this end, we present a study on harnessing behavioral data drawn from Internet search logs to detect adverse drug reactions (ADRs). By analyzing search queries collected from 80 million consenting users and by using a widely recognized benchmark of ADRs, we found that the performance of ADR detection via search logs is comparable and complementary to detection based on the FDA’s adverse event reporting system (AERS). We show that by jointly leveraging data from the AERS and search logs, the accuracy of ADR detection can be improved by 19% relative to the use of each data source independently. The results suggest that leveraging nontraditional sources such as online search logs could supplement existing pharmacovigilance approaches.

Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system

Signal-detection algorithms (SDAs) are recognized as vital tools in pharmacovigilance. However, their performance characteristics are generally unknown. By leveraging a unique gold standard recently made public by the Observational Medical Outcomes Partnership (OMOP) and by conducting a unique systematic evaluation, we provide new insights into the diagnostic potential and characteristics of SDAs that are routinely applied to the US Food and Drug Administration (FDA) Adverse Event Reporting System (AERS). We find that SDAs can attain reasonable predictive accuracy in signaling adverse events. Two performance classes emerge, indicating that the class of approaches that address confounding and masking effects benefits safety surveillance. Our study shows that not all events are equally detectable, suggesting that specific events might be monitored more effectively using other data sources. We provide performance guidelines for several operating scenarios to inform the trade-off between sensitivity and specificity for specific use cases. We also propose an approach and demonstrate its application in identifying optimal signaling thresholds, given specific misclassification tolerances.

Optimization of a quantitative signal detection algorithm for spontaneous reports of adverse events post immunization

PURPOSE

To optimize the efficiency of signal detection by maximizing the proportion of true positive (TP) signals among signals detected by a disproportionality algorithm.

METHODS

We compared 176 different combinations of stratification factors, sex (S), age (A), region (R) and year of report (Y), and cut-off values of a Multi-Item Gamma Poisson Schrinker (MGPS) algorithm. Spontaneous adverse event reports of eight vaccines from the GlaxoSmithKline Biologicals safety database were used. Defining events listed in the Product Information as proxy of true safety signals, we compared each algorithm performance in terms of positive predictive value (PPV). For each vaccine, each algorithm was ranked according to PPV. Median rank and overall PPV were computed across vaccines.

RESULTS

For a standard cut-off of 2, the optimal stratification factors differed by vaccine and led to a set of algorithms with a median rank of 34.5 (PPV = 0.22; 34 TP). Keeping the original SARY stratification led to optimal cut-offs that differed by vaccine and a set of algorithms with a median rank of 1.75 (PPV = 0.20; 142 TP). The optimal combination of cut-off and stratification led to different algorithms by vaccine with a median rank of 1 (PPV = 0.19; 139 TP). The best unique algorithm parameterization across vaccines was 0.8-SARY (cut-off-stratification), with a median rank of 3 (PPV = 0.20; 195 TP). The original 2-SARY was one of the worst algorithms, with a median rank of 150.75 (PPV = 0.13; 8 TP).

CONCLUSION

Within the scope of this study, a unique MGPS algorithm across vaccines with the original full stratification but a lowered cut-off provided major performance improvement.