Near real-time adverse drug reaction surveillance within population-based health networks: methodology considerations for data accrual

Near Real-time Surveillance for Consequences of Health Policies Using Sequential Analysis

BACKGROUND

New health policies may have intended and unintended consequences. Active surveillance of population-level data may provide initial signals of policy effects for further rigorous evaluation soon after policy implementation.

OBJECTIVE

This study evaluated the utility of sequential analysis for prospectively assessing signals of health policy impacts. As a policy example, we studied the consequences of the widely publicized Food and Drug Administration's warnings cautioning that antidepressant use could increase suicidal risk in youth.

METHOD

This was a retrospective, longitudinal study, modeling prospective surveillance, using the maximized sequential probability ratio test. We used historical data (2000-2010) from 11 health systems in the US Mental Health Research Network. The study cohort included adolescents (ages 10-17 y) and young adults (ages 18-29 y), who were targeted by the warnings, and adults (ages 30-64 y) as a comparison group. Outcome measures were observed and expected events of 2 possible unintended policy outcomes: psychotropic drug poisonings (as a proxy for suicide attempts) and completed suicides.

RESULTS

We detected statistically significant (P<0.05) signals of excess risk for suicidal behavior in adolescents and young adults within 5-7 quarters of the warnings. The excess risk in psychotropic drug poisonings was consistent with results from a previous, more rigorous interrupted time series analysis but use of the maximized sequential probability ratio test method allows timely detection. While we also detected signals of increased risk of completed suicide in these younger age groups, on its own it should not be taken as conclusive evidence that the policy caused the signal. A statistical signal indicates the need for further scrutiny using rigorous quasi-experimental studies to investigate the possibility of a cause-and-effect relationship.

CONCLUSIONS

This was a proof-of-concept study. Prospective, periodic evaluation of administrative health care data using sequential analysis can provide timely population-based signals of effects of health policies. This method may be useful to use as new policies are introduced.

Safety and Effectiveness of Direct Oral Anticoagulants Versus Vitamin K Antagonists: Pilot Implementation of a Near-Real-Time Monitoring Program in Italy

Background

Real‐time monitoring is used to the ends of postmarketing observational research on newly marketed drugs. We implemented a pilot near‐real‐time monitoring program on the test case of oral anticoagulants. Specifically, we evaluated the safety and effectiveness of direct oral anticoagulants compared to vitamin K antagonists in nonvalvular atrial fibrillation secondary prevention during 2013‐2015 in the Lazio Region, Italy.

Methods and Results

A cohort study was conducted using a sequential propensity‐score–matched new user parallel‐cohort design. Sequential analyses were performed using Cox models. Overall, 10 742 patients contributed to the analyses. Compared with vitamin K antagonists, direct oral anticoagulant use was associated with a reduction of all‐cause mortality (0.81; 95% confidence interval [CI] 0.66‐0.99), cardiovascular mortality (0.71; 95% CI 0.54‐0.93), myocardial infarction (0.67; 95% CI 0.43‐1.04), ischemic stroke (0.87; 95% CI 0.52‐1.45), hemorrhagic stroke (0.25; 95% CI 0.07‐0.88), and with a nonsignificant increase of gastrointestinal bleeding (1.26; 95% CI 0.69‐2.30).

Conclusions

The present pilot study is a cornerstone to develop real‐time monitoring for new drugs in our region.

Continuous Post-Market Sequential Safety Surveillance with Minimum Events to Signal

The CDC Vaccine Safety Datalink project has pioneered the use of near real-time post-market vaccine safety surveillance for the rapid detection of adverse events. Doing weekly analyses, continuous sequential methods are used, allowing investigators to evaluate the data near-continuously while still maintaining the correct overall alpha level. With continuous sequential monitoring, the null hypothesis may be rejected after only one or two adverse events are observed. In this paper, we explore continuous sequential monitoring when we do not allow the null to be rejected until a minimum number of observed events have occurred. We also evaluate continuous sequential analysis with a delayed start until a certain sample size has been attained. Tables with exact critical values, statistical power and the average times to signal are provided. We show that, with the first option, it is possible to both increase the power and reduce the expected time to signal, while keeping the alpha level the same. The second option is only useful if the start of the surveillance is delayed for logistical reasons, when there is a group of data available at the first analysis, followed by continuous or near-continuous monitoring thereafter.

A system to build distributed multivariate models and manage disparate data sharing policies: implementation in the scalable national network for effectiveness research

Background Centralized and federated models for sharing data in research networks currently exist. To build multivariate data analysis for centralized networks, transfer of patient-level data to a central computation resource is necessary. The authors implemented distributed multivariate models for federated networks in which patient-level data is kept at each site and data exchange policies are managed in a study-centric manner.

Objective The objective was to implement infrastructure that supports the functionality of some existing research networks (e.g., cohort discovery, workflow management, and estimation of multivariate analytic models on centralized data) while adding additional important new features, such as algorithms for distributed iterative multivariate models, a graphical interface for multivariate model specification, synchronous and asynchronous response to network queries, investigator-initiated studies, and study-based control of staff, protocols, and data sharing policies.

Materials and Methods Based on the requirements gathered from statisticians, administrators, and investigators from multiple institutions, the authors developed infrastructure and tools to support multisite comparative effectiveness studies using web services for multivariate statistical estimation in the SCANNER federated network.

Results The authors implemented massively parallel (map-reduce) computation methods and a new policy management system to enable each study initiated by network participants to define the ways in which data may be processed, managed, queried, and shared. The authors illustrated the use of these systems among institutions with highly different policies and operating under different state laws.

Discussion and Conclusion Federated research networks need not limit distributed query functionality to count queries, cohort discovery, or independently estimated analytic models. Multivariate analyses can be efficiently and securely conducted without patient-level data transport, allowing institutions with strict local data storage requirements to participate in sophisticated analyses based on federated research networks.

Improving postapproval drug safety surveillance: getting better information sooner

Adverse drug events (ADEs) are an important public health concern, accounting for 5% of all hospital admissions and two-thirds of all complications occurring shortly after hospital discharge. There are often long delays between when a drug is approved and when serious ADEs are identified. Recent and ongoing advances in drug safety surveillance include the establishment of government-sponsored networks of population databases, the use of data mining approaches, and the formal integration of diverse sources of drug safety information. These advances promise to reduce delays in identifying drug-related risks and in providing reassurance about the absence of such risks.

Orphan therapies: making best use of postmarket data

BACKGROUND

Postmarket surveillance of the comparative safety and efficacy of orphan therapeutics is challenging, particularly when multiple therapeutics are licensed for the same orphan indication. To make best use of product-specific registry data collected to fulfill regulatory requirements, we propose the creation of a distributed electronic health data network among registries. Such a network could support sequential statistical analyses designed to detect early warnings of excess risks. We use a simulated example to explore the circumstances under which a distributed network may prove advantageous.

METHODS

We perform sample size calculations for sequential and non-sequential statistical studies aimed at comparing the incidence of hepatotoxicity following initiation of two newly licensed therapies for homozygous familial hypercholesterolemia. We calculate the sample size savings ratio, or the proportion of sample size saved if one conducted a sequential study as compared to a non-sequential study. Then, using models to describe the adoption and utilization of these therapies, we simulate when these sample sizes are attainable in calendar years. We then calculate the analytic calendar time savings ratio, analogous to the sample size savings ratio. We repeat these analyses for numerous scenarios.

KEY RESULTS

Sequential analyses detect effect sizes earlier or at the same time as non-sequential analyses. The most substantial potential savings occur when the market share is more imbalanced (i.e., 90% for therapy A) and the effect size is closest to the null hypothesis. However, due to low exposure prevalence, these savings are difficult to realize within the 30-year time frame of this simulation for scenarios in which the outcome of interest occurs at or more frequently than one event/100 person-years.

CONCLUSIONS

We illustrate a process to assess whether sequential statistical analyses of registry data performed via distributed networks may prove a worthwhile infrastructure investment for pharmacovigilance.

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity

PURPOSE

Outcome misclassification in retrospective epidemiologic analyses has been well-studied, but little is known about such misclassification with respect to sequential statistical analysis during surveillance of medical product-associated risks, a planned capability of the US Food and Drug Administration's Sentinel System.

METHODS

Using a vaccine example, we model and simulate sequential database surveillance in an observational data network using a variety of outcome detection algorithms. We consider how these algorithms, as characterized by sensitivity and positive predictive value, impact the length of surveillance and timeliness of safety signal detection. We show investigators/users of these networks how they can perform preparatory study design calculations that consider outcome misclassification in sequential database surveillance.

RESULTS

Non-differential outcome misclassification generates longer surveillance times and less timely safety signal detection as compared with the case of no misclassification. Inclusive algorithms characterized by high sensitivity but low positive predictive value outperform more narrow algorithms when detecting rare outcomes. This decision calculus may change considerably if medical chart validation procedures were required.

CONCLUSIONS

These findings raise important questions regarding the design of observational data networks used for pharmacovigilance. Specifically, there are tradeoffs involved when choosing to populate such networks with component databases that are large as compared with smaller integrated delivery system databases that can more easily access laboratory or clinical data and perform medical chart validation.