A Synthesis of Current Surveillance Planning Methods for the Sequential Monitoring of Drug and Vaccine Adverse Effects Using Electronic Health Care Data

Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance

INTRODUCTION

Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' The extent to which serial testing in real-world studies improves or hinders overall performance in terms of sensitivity and specificity remains unknown.

METHODS

We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared type I and II errors before and after empirical calibration for historical comparator, self-controlled case series (SCCS), and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes.

RESULTS

The historical comparator design mostly had fewer type II errors than SCCS. SCCS had fewer type I errors than the historical comparator. Before empirical calibration, the serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, type I errors returned to nominal; sensitivity was lowest when the methods were combined.

CONCLUSION

While serial combination produced fewer false-positive signals compared with the most specific method, it generated more false-negative signals compared with the most sensitive method. Using a historical comparator design followed by an SCCS analysis yielded decreased sensitivity in evaluating safety signals relative to a one-stage SCCS approach. While the current use of serial testing in vaccine surveillance may provide a practical paradigm for signal identification and triage, single epidemiological designs should be explored as valuable approaches to detecting signals.

Leveraging the entire cohort in drug safety monitoring: part 1 methods for sequential surveillance that use regression adjustment or weighting to control confounding in a multisite, rare event, distributed data setting

OBJECTIVE

Study designs involving self-controlled or exposure-matched samples are commonly used to monitor postmarket vaccine and drug safety, and they use a subset of the available larger cohort. This article overviews group sequential methods designed for observational data safety monitoring that use the whole exposed and unexposed cohorts by implementing regression adjustment or weighting to control confounding.

METHODS

We summarize what is known about the performance of "whole cohort" methods in multisite health plan data networks such as the Sentinel System of the Food and Drug Administration, where outcomes are rare, individual-level patient data cannot be pooled across sites, site heterogeneity is large, and data are dynamically updated over time.

RESULTS

Group sequential estimation and testing methods that use regression or weighting can flexibly handle electronic health care data's unpredictability, including an uncertain rate of new product uptake, variable composition of the population over time, and data changes due to dynamic administrative updates. Regression and weighting methods generally have higher power, faster signal detection, and fewer practical challenges compared with some design-based confounder adjustment methods.

CONCLUSION

Group sequential regression adjustment and weighting approaches are feasible and underused in practice. They leverage more information than designs that involved sampling and increase power to detect rare adverse effects without increasing bias.

Conditional power as an aid in making interim decisions in observational studies

Conditional power combines the findings of a partially completed study with assumptions about the future. The goal is to estimate the probability that the eventual study result will be incompatible with a criterion value, such as acceptable risk or the null hypothesis. Some history and motivation for conditional power calculations are provided, with examples illustrating the application to drug safety studies. This is an expository article suggesting that conditional power, which is well-established in clinical trials research, also has application to observational studies. The utility may be highest in regulatory settings where resources are limited and interim decisions have to be made accurately in the shortest possible time.

Assessing performance of sequential analysis methods for active drug safety surveillance using observational data

The routine use of sequential methods is well established in clinical studies. Recently, there has been increasing interest in applying these methods to prospectively monitor the safety of newly approved drugs through accrual of real-world data. However, the application to marketed drugs using real-world data has been limited and work is needed to determine which sequential approaches are most suited to such data. In this study, the conditional sequential sampling procedure (CSSP), a group sequential method, was compared with a log-linear model with Poisson distribution (LLMP) through a SAS procedure (PROC GENMOD) combined with an alpha-spending function on two large longitudinal US administrative health claims databases. Relative performance in identifying known drug-outcome associations was examined using a set of 50 well-studied drug-outcome pairs. The study finds that neither method correctly identified all pairs but that LLMP often provides better ability and shorter time for identifying the known drug-outcome associations with superior computational performance when compared with CSSP, albeit with more false positives. With the features of flexible confounding control and ease of implementation, LLMP may be a good alternative or complement to CSSP.