Near Real-time Surveillance for Consequences of Health Policies Using Sequential Analysis

Rapid evaluation of service innovations in health and social care: key considerations

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31).

Effect of the FDA Black Box Suicidality Warnings for Antidepressants on Suicide Rates in the USA

Background: Some authors claimed that the US Food and Drug Administration (FDA) black box warning on treatment-emergent suicidality with antidepressants in adolescents (issued 2004) and young adults (issued 2006) led to an increase of suicides, based on the analyses of ecological data with debatable assumptions about putative changes in suicide rates. Aims: To explore if putative changes in suicide rates in adolescents and young adults at the time of the FDA warnings is a detectable signal in the data or compatible with random fluctuations. Method: We applied different changepoint analyses for adolescent and young adult suicide rates from 1981 to 2019 in the USA. Results: Changepoint analysis did not support a detrimental effect of the FDA black box warnings. The downward trend of suicides reversed several years after the warning in adolescents (2007-2009) and many years before in young adults (1999-2001). Limitations: Our analyses cannot rule out detrimental effects of the FDA warnings. However, even if there was such an effect, it was likely small and indistinguishable from random fluctuations in the available suicide data. Conclusion: There is no detectable change of trend in adolescent or young adult suicide rates in line with a detrimental effect of the FDA black box warnings on treatment-emergent suicidality.

Design, analysis, power, and sample size calculation for three-phase interrupted time series analysis in evaluation of health policy interventions

OBJECTIVE

To discuss the study design and data analysis for three-phase interrupted time series (ITS) studies to evaluate the impact of health policy, systems, or environmental interventions. Simulation methods are used to conduct power and sample size calculation for these studies.

METHODS

We consider the design and analysis of three-phase ITS studies using a study funded by National Institutes of Health as an exemplar. The design and analysis of both one-arm and two-arm three-phase ITS studies are introduced.

RESULTS

A simulation-based approach, with ready-to-use computer programs, was developed to determine the power for two types of three-phase ITS studies. Simulations were conducted to estimate the power of segmented autoregressive (AR) error models when autocorrelation ranged from -0.9 to 0.9 with various effect sizes. The power increased as the sample size or the effect size increased. The power to detect the same effect sizes varied largely, depending on testing level change, trend changes, or both.

CONCLUSION

This article provides a convenient tool for investigators to generate sample sizes to ensure sufficient statistical power when three-phase ITS study design is implemented.

Simulation-based power and sample size calculation for designing interrupted time series analyses of count outcomes in evaluation of health policy interventions

Objective

The purpose of this study was to present the design, model, and data analysis of an interrupted time series (ITS) model applied to evaluate the impact of health policy, systems, or environmental interventions using count outcomes. Simulation methods were used to conduct power and sample size calculations for these studies.

Methods

We proposed the models and analyses of ITS designs for count outcomes using the Strengthening Translational Research in Diverse Enrollment (STRIDE) study as an example. The models we used were observation-driven models, which bundle a lagged term on the conditional mean of the outcome for a time series of count outcomes.

Results

A simulation-based approach with ready-to-use computer programs was developed to calculate the sample size and power of two types of ITS models, Poisson and negative binomial, for count outcomes. Simulations were conducted to estimate the power of segmented autoregressive (AR) error models when autocorrelation ranged from −0.9 to 0.9, with various effect sizes. The power to detect the same magnitude of parameters varied largely, depending on the testing level change, the trend change, or both. The relationships between power and sample size and the values of the parameters were different between the two models.

Conclusion

This article provides a convenient tool to allow investigators to generate sample sizes that will ensure sufficient statistical power when the ITS study design of count outcomes is implemented.

Counter-Point: Early Warning Systems Are Imperfect, but Essential

Sequential analysis can be used as an early warning system about potential unintended consequences of health policy decisions, generating follow-up investigations, but it should not be used as causal evidence.

Counter-Point: Staying Honest When Policy Changes Backfire

Despite the good intentions of the Food and Drug Administration (FDA), many drug warnings are ineffective or have unintended consequences, particularly if the media exaggerates the messages and scares the public. The controversial 2003 to 2004 FDA warnings on youth suicidality associated with antidepressant use are a case in point. In a 10-year interrupted time series (ITS) analysis in 11 health plans, we found that the warnings and hyped media coverage led to substantial reductions in antidepressant use (declines in antidepressant use and overall care corroborated in several studies), and small, visible increases in emergency room and inpatient poisonings with psychotropic drugs. In a gross misunderstanding of the method, Dr Stone calls ITS, "an intuition based upon false analogies, fallacious assumptions and analytical error." We demonstrate visually using published studies that the ITS method is one of the oldest (hundreds of years) and strongest quasi-experimental study designs, and that the alternative data analyses proposed by Dr Stone do not have rates (denominators), nor baselines, so the measures of change are invalid.