Response to letter by Levine et al

Revisiting the reported signal of acute pancreatitis with rasburicase: an object lesson in pharmacovigilance

INTRODUCTION

There is an interest in methodologies to expeditiously detect credible signals of drug-induced pancreatitis. An example is the reported signal of pancreatitis with rasburicase emerging from a study [the 'index publication' (IP)] combining quantitative signal detection findings from a spontaneous reporting system (SRS) and electronic health records (EHRs). The signal was reportedly supported by a clinical review with a case series manuscript in progress. The reported signal is noteworthy, being initially classified as a false-positive finding for the chosen reference standard, but reclassified as a 'clinically supported' signal.

OBJECTIVE

This paper has dual objectives: to revisit the signal of rasburicase and acute pancreatitis and extend the original analysis via reexamination of its findings, in light of more contemporary data; and to motivate discussions on key issues in signal detection and evaluation, including recent findings from a major international pharmacovigilance research initiative.

METHODOLOGY

We used the same methodology as the IP, including the same disproportionality analysis software/dataset for calculating observed to expected reporting frequencies (O/Es), Medical Dictionary for Regulatory Activities Preferred Term, and O/E metric/threshold combination defining a signal of disproportionate reporting. Baseline analysis results prompted supplementary analyses using alternative analytical choices. We performed a comprehensive literature search to identify additional published case reports of rasburicase and pancreatitis.

RESULTS

We could not replicate positive findings (e.g. a signal or statistic of disproportionate reporting) from the SRS data using the same algorithm, software, dataset and vendor specified in the IP. The reporting association was statistically highlighted in default and supplemental analysis when more sensitive forms of disproportionality analysis were used. Two of three reports in the FAERS database were assessed as likely duplicate reports. We did not identify any additional reports in the FAERS corresponding to the three cases identified in the IP using EHRs. We did not identify additional published reports of pancreatitis associated with rasburicase.

DISCUSSION

Our exercise stimulated interesting discussions of key points in signal detection and evaluation, including causality assessment, signal detection algorithm performance, pharmacovigilance terminology, duplicate reporting, mechanisms for communicating signals, the structure of the FAERs database, and recent results from a major international pharmacovigilance research initiative.

An experimental investigation of masking in the US FDA adverse event reporting system database

BACKGROUND

A phenomenon of 'masking' or 'cloaking' in pharmacovigilance data mining has been described, which can potentially cause signals of disproportionate reporting (SDRs) to be missed, particularly in pharmaceutical company databases. Masking has been predicted theoretically, observed anecdotally or studied to a limited extent in both pharmaceutical company and health authority databases, but no previous publication systematically assesses its occurrence in a large health authority database.

OBJECTIVE

To explore the nature, extent and possible consequences of masking in the US FDA Adverse Event Reporting System (AERS) database by applying various experimental unmasking protocols to a set of drugs and events representing realistic pharmacovigilance analysis conditions.

METHODS

This study employed AERS data from 2001 through 2005. For a set of 63 Medical Dictionary for Regulatory Activities (MedDRA®) Preferred Terms (PTs), disproportionality analysis was carried out with respect to all drugs included in the AERS database, using a previously described urn-model-based algorithm. We specifically sought masking in which drug removal induced an increase in the statistical representation of a drug-event combination (DEC) that resulted in the emergence of a new SDR. We performed a series of unmasking experiments selecting drugs for removal using rational statistical decision rules based on the requirement of a reporting ratio (RR) >1, top-ranked statistical unexpectedness (SU) and relatedness as reflected in the WHO Anatomical Therapeutic Chemical level 4 (ATC4) grouping. In order to assess the possible extent of residual masking we performed two supplemental purely empirical analyses on a limited subset of data. This entailed testing every drug and drug group to determine which was most influential in uncovering masked SDRs. We assessed the strength of external evidence for a causal association for a small number of masked SDRs involving a subset of 29 drugs for which level of evidence adjudication was available from a previous study.

RESULTS

The original disproportionality analysis identified 8719 SDRs for the 63 PTs. The SU-based unmasking protocols generated variable numbers of masked SDRs ranging from 38 to 156, representing a 0.43-1.8% increase over the number of baseline SDRs. A significant number of baseline SDRs were also lost in the course of our experiments. The trend in the number of gained SDRs per report removed was inversely related to the number of lost SDRs per protocol. Both the number and nature of the reports removed influenced the number of gained SDRs observed. The purely empirical protocols unmasked up to ten times as many SDRs. None of the masked SDRs had strong external evidence supporting a causal association. Most involved associations for which there was no external supporting evidence or were in the original product label. For two masked SDRs, there was external evidence of a possible causal association.

CONCLUSIONS

We documented masking in the FDA AERS database. Attempts at unmasking SDRs using practically implementable protocols produced only small changes in the output of SDRs in our analysis. This is undoubtedly related to the large size and diversity of the database, but the complex interdependencies between drugs and events in authentic spontaneous reporting system (SRS) databases, and the impact of measures of statistical variability that are typically used in real-world disproportionality analysis, may be additional factors that constrain the discovery of masked SDRs and which may also operate in pharmaceutical company databases. Empirical determination of the most influential drugs may uncover significantly more SDRs than protocols based on predetermined statistical selection rules but are impractical except possibly for evaluating specific events. Routine global exercises to elicit masking, especially in large health authority databases are not justified based on results available to date. Exercises to elicit unmasking should be driven by prior knowledge or obvious data imbalances.

Gold standards in pharmacovigilance: the use of definitive anecdotal reports of adverse drug reactions as pure gold and high-grade ore

Anecdotal reports of adverse drug reactions are generally regarded as being of poor evidential quality. This is especially relevant for postmarketing drug safety surveillance, which relies heavily on spontaneous anecdotal reports. The numerous limitations of spontaneous reports cannot be overemphasised, but there is another side to the story: these datasets also contain anecdotal reports that can be considered to describe definitive adverse reactions, without the need for further formal verification. We have previously defined four categories of such adverse reactions: (i) extracellular or intracellular tissue deposition of the drug or a metabolite; (ii) a specific anatomical location or pattern of injury; (iii) physiological dysfunction or direct tissue damage demonstrable by physicochemical testing; and (iv) infection, as a result of the administration of an infective agent as the therapeutic substance or because of demonstrable contamination. In this article, we discuss the implications of these definitive ('between-the-eyes') adverse effects for pharmacovigilance.

Potential Use of Data-Mining Algorithms for the Detection of ???Surprise??? Adverse Drug Reactions

BACKGROUND AND OBJECTIVE

Various data mining algorithms (DMAs) that perform disporportionality analysis on spontaneous reporting system (SRS) data are being heavily promoted to improve drug safety surveillance. The incremental value of DMAs is ultimately related to their ability to detect truly unexpected associations that would have escaped traditional surveillance and/or their ability to identify the same associations as traditional methods but with greater scientific efficiency. As to the former potential benefit, in the course of evaluating DMAs, we have observed what we call 'surprise reactions'. These adverse reactions may be discounted in manual review of adverse drug reaction (ADR) lists because they are less clinically dramatic, less characteristic of drug effects in general, less serious than the classical type B hypersensitivity reactions or may have subtle pharmacological explanations. Thus these reactions may only become recognised when post hoc explanations are sought based on more refined pharmacological knowledge of the formulation. The objective of this study was to explore notions of 'unexpectedness' as relates to signal detection and data mining by introducing the concept of 'surprise reactions' and to determine if the latter associations, often first reported in the literature, represent a type of ADR amenable to detection with the assistance of adjunctive statistical calculations on SRS data.

METHODS

Using commonly cited thresholds, the multi-item gamma Poisson shrinker (MGPS) and proportional reporting ratios (PRRs) were applied to reports in the US FDA Adverse Event Reporting System (AERS) database of well documented 'surprise reactions' compiled by the authors.

RESULTS

There were 34 relevant surprise reactions involving 29 separate drugs in 17 different drug classes. Using PRRs (PRR >2, chi(2) >4, N >2), 12 drug-event combinations were signalled before the first ADR citation appeared in MEDLINE, four occurred concurrently and 11 after. With empirical Bayes geometric mean (EBGM) analysis (EBGM >2, N >0), 12 signals occurred before, three concurrently and 11 after publication of the first literature citation. With EB(05) (EB(05)> or =2, N >0), six occurred before, two concurrently and 14 after MEDLINE citation.

DISCUSSION

Pharmacovigilance is rather unique in terms of the number and variety of events under surveillance. Some events may be more appropriate targets for statistical approaches than others. The experience of many organisations is that most statistical disproportionalities represent known associations but our findings suggest there could be events that may be discounted on manual review of adverse event lists, which may be usefully highlighted via DMAs.

CONCLUSIONS

Identification of surprise reactions may serve as an important niche for DMAs.