A Trial Emulation Approach for Policy Evaluations with Group-level Longitudinal Data

Staggered interventions with no control groups

The limitations of the two-way fixed effects for the impact evaluation of interventions that occur at different times for each group have meant that 'staggered interventions' have been highlighted in recent years in the econometric literature and, more recently, in epidemiology. Although many alternative strategies (such as staggered difference-in-differences) have been proposed, the focus has predominantly been on scenarios in which one or more control groups are available. However, control groups are often unavailable, due to limitations in the available data or because all units eventually receive the intervention. In this context, interrupted time series (ITS) designs can constitute an appropriate alternative. The extent to which common model specifications for ITS analyses are limited in the case of staggered interventions remains an underexplored area in the methodological literature. In this work, we aim to demonstrate that standard ITS model specifications typically yield biased results for staggered interventions and we propose alternative model specifications that were inspired by recent developments in the difference-in-differences literature to propose adapted analytical strategies.

Target Trial Emulation for Evaluating Health Policy

Target trial emulation is an approach to designing rigorous nonexperimental studies by "emulating" key features of a clinical trial. Most commonly used outside of policy contexts, this approach is also valuable for policy evaluation as policies typically are not randomly assigned. In this article, we discuss the application of the target trial emulation framework in a policy evaluation context. The policy trial emulation framework includes 7 components: the units and eligibility criteria, definitions of the exposure and comparison conditions, assignment mechanism, baseline ("time zero") and follow-up, outcomes, causal estimand, and statistical analysis and assumptions. Policy evaluations that emulate a randomized trial across these dimensions can yield estimates of the causal effects of the policy on outcomes. Using the policy trial emulation framework to conduct and report on research design and methods supports transparent assessment of threats to causal inference in nonexperimental studies intended to assess the effect of a health policy on clinical or population health outcomes.

Reporting and Description of Research Methodology in Studies Estimating Effects of Firearm Policies

BACKGROUND

Evidence about which firearm policies work, to what extent, and for whom is hotly debated, perhaps partly because variation in research methodology has produced mixed and inconclusive effect estimates. We conducted a scoping review of firearm policy research in the health sciences in the United States, focusing on methodological considerations for causal inference.

METHODS

We identified original, empirical articles indexed in PubMed from 1 January 2000 to 1 September 2021 that examined any of 18 prespecified firearm policies. We extracted key study components, including policy type(s) examined, policy operationalization, outcomes, study setting and population, study approach and design, causal language, and whether and how authors acknowledged potential sources of bias.

RESULTS

We screened 7733 articles and included 124. A plurality of studies used a legislative score as their primary exposure (n = 39; 32%) and did not examine change in policies over time (n = 47; 38%). Most examined firearm homicide (n = 51; 41%) or firearm suicide (n = 40; 32%) as outcomes. One-third adjusted for other firearm policies (n = 41; 33%). Three studies (2%) explicitly mentioned that their goal was to estimate causal effects, but over half used language implying causality (n = 72; 58%). Most acknowledged causal identification assumptions of temporality (n = 91; 73%) and exchangeability (n = 111; 90%); other assumptions were less often acknowledged. One-third of studies included bias analyses (n = 42; 34%).

CONCLUSIONS

We identified a range of methodologic approaches in firearm policy research in the health sciences. Acknowledging the imitations of data availability and quality, we identify opportunities to improve causal inferences about and reporting on the effects of firearm policies on population health.

Has the opening of Amazon fulfillment centers affected demand for disability insurance?

An estimated 17.6% of blue-collar, manufacturing jobs were lost in the United States between 1970 and 2016. These jobs, often union-represented, provided relatively generous pay and benefits, creating a path to the middle class for individuals without a four-year college degree. Evidence suggests the closure of manufacturing facilities and resulting decline in economic opportunity increased demand for disability insurance (SSDI) among blue-collar workers. In recent years, the opening of Amazon Fulfillment Centers (FCs) has accelerated around the country, driving a wave of blue-collar job creation. We estimated the extent to which the opening of FCs affected SSDI application rates, including rates of approvals and denials, using a synthetic control group approach. We found that FC openings were associated with a 1.4% reduction in the SSDI application rate over the subsequent three years, translating to 5,528 fewer applications per year across commuting zones with an FC opening. Our findings are consistent with FC openings improving economic opportunities in local labor markets, though our confidence intervals were wide and included the null.

The implementation of target trial emulation for causal inference: a scoping review

OBJECTIVES

We aim to investigate the implementation of Target Trial Emulation (TTE) for causal inference, involving research topics, frequently used strategies, and issues indicating the need for future improvements.

STUDY DESIGN AND SETTING

We performed a scoping review by following the Joanna Briggs Institute (JBI) guidance and Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist. A health research-focused librarian searched multiple medical databases, and two independent reviewers completed screening and extraction within covidence review management software.

RESULTS

Our search resulted in 1,240 papers, of which 96 papers were eligible for data extraction. Results show a significant increase in the use of TTE in 2018 and 2021. The study topics varied and focused primarily on cancer, cardiovascular and cerebrovascular diseases, and infectious diseases. However, not all papers specified well all three critical components for generating robust causal evidence: time-zero, random assignment simulation, and comparison strategy. Some common issues were observed from retrieved papers, and key limitations include residual confounding, limited generalizability, and a lack of reporting guidance that need to be improved.

CONCLUSION

Uneven adherence to the TTE framework exists, and future improvements are needed to progress applications using causal inference with observational data.

Eliminating Food Insecurity in the USA: a Target Trial Emulation Using Observational Data to Estimate Effects on Health-Related Quality of Life

BACKGROUND

Food insecurity is associated with many aspects of poor health. However, trials of food insecurity interventions typically focus on outcomes of interest to funders, such as healthcare use, cost, or clinical performance metrics, rather than quality of life outcomes that may be prioritized by individuals who experience food insecurity.

OBJECTIVE

To emulate a trial of a food insecurity elimination intervention, and quantify its estimated effects on health utility, health-related quality of life, and mental health.

DESIGN

Target trial emulation using longitudinal, nationally representative data, from the USA, 2016-2017.

PARTICIPANTS

A total of 2013 adults in the Medical Expenditure Panel Survey screened positive for food insecurity, representing 32 million individuals.

MAIN MEASURES

Food insecurity was assessed using the Adult Food Security Survey Module. The primary outcome was the SF-6D (Short-Form Six Dimension) measure of health utility. Secondary outcomes were mental component score (MCS) and physical component score (PCS) of the Veterans RAND 12-Item Health Survey (a measure of health-related quality of life), Kessler 6 (K6) psychological distress, and Patient Health Questionnaire 2-item (PHQ2) depressive symptoms.

KEY RESULTS

We estimated that food insecurity elimination would improve health utility by 80 QALYs per 100,000 person-years, or 0.008 QALYs per person per year (95% CI 0.002 to 0.014, p = 0.005), relative to the status quo. We also estimated that food insecurity elimination would improve mental health (difference in MCS [95% CI]: 0.55 [0.14 to 0.96]), physical health (difference in PCS: 0.44 [0.06 to 0.82]), psychological distress (difference in K6: -0.30 [-0.51 to -0.09]), and depressive symptoms (difference in PHQ-2: -0.13 [-0.20 to  -0.07]).

CONCLUSIONS

Food insecurity elimination may improve important, but understudied, aspects of health. Evaluations of food insecurity interventions should holistically investigate their potential to improve many different aspects of health.

Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example

BACKGROUND

Two-way fixed effects methods have been used to estimate effects of policies adopted in different places over time, but they can provide misleading results when effects are heterogeneous or dynamic, and alternate methods have been proposed.

METHODS

We compared methods for estimating the average treatment effect on the treated (ATT) under staggered adoption of policies, including two-way fixed effects, group-time ATT, cohort ATT, and target-trial approaches. We applied each method to assess the impact of Medicaid expansion on preterm birth using the National Center for Health Statistics' birth records. We compared each estimator's performance in a simulation parameterized to mimic the empirical example. We generated constant, heterogeneous, and dynamic effects and calculated bias, mean squared error, and confidence interval coverage of each estimator across 1000 iterations.

RESULTS

Two-way fixed effects estimated that Medicaid expansion increased the risk of preterm birth (risk difference [RD], 0.12; 95% CI = 0.02, 0.22), while the group-time ATT, cohort ATT, and target-trial approaches estimated protective or null effects (group-time RD, -0.16; 95% CI = -0.58, 0.26; cohort RD, -0.02; 95% CI = -0.46, 0.41; target trial RD, -0.16; 95% CI = -0.59, 0.26). In simulations, two-way fixed effects performed well when treatment effects were constant and less well under heterogeneous and dynamic effects.

CONCLUSIONS

We demonstrated why new approaches perform better than two-way fixed effects when treatment effects are heterogeneous or dynamic under a staggered policy adoption design, and created simulation and analysis code to promote understanding and wider use of these methods in the epidemiologic literature.

Target Trial Emulation Using Hospital-Based Observational Data: Demonstration and Application in COVID-19

Methodological biases are common in observational studies evaluating treatment effectiveness. The objective of this study is to emulate a target trial in a competing risks setting using hospital-based observational data. We extend established methodology accounting for immortal time bias and time-fixed confounding biases to a setting where no survival information beyond hospital discharge is available: a condition common to coronavirus disease 2019 (COVID-19) research data. This exemplary study includes a cohort of 618 hospitalized patients with COVID-19. We describe methodological opportunities and challenges that cannot be overcome applying traditional statistical methods. We demonstrate the practical implementation of this trial emulation approach via clone-censor-weight techniques. We undertake a competing risk analysis, reporting the cause-specific cumulative hazards and cumulative incidence probabilities. Our analysis demonstrates that a target trial emulation framework can be extended to account for competing risks in COVID-19 hospital studies. In our analysis, we avoid immortal time bias, time-fixed confounding bias, and competing risks bias simultaneously. Choosing the length of the grace period is justified from a clinical perspective and has an important advantage in ensuring reliable results. This extended trial emulation with the competing risk analysis enables an unbiased estimation of treatment effects, along with the ability to interpret the effectiveness of treatment on all clinically important outcomes.

Intensity and lag-time of non-pharmaceutical interventions on COVID-19 dynamics in German hospitals

Introduction

Evaluating the potential effects of non-pharmaceutical interventions on COVID-19 dynamics is challenging and controversially discussed in the literature. The reasons are manifold, and some of them are as follows. First, interventions are strongly correlated, making a specific contribution difficult to disentangle; second, time trends (including SARS-CoV-2 variants, vaccination coverage and seasonality) influence the potential effects; third, interventions influence the different populations and dynamics with a time delay.

Methods

In this article, we apply a distributed lag linear model on COVID-19 data from Germany from January 2020 to June 2022 to study intensity and lag time effects on the number of hospital patients and the number of prevalent intensive care patients diagnosed with polymerase chain reaction tests. We further discuss how the findings depend on the complexity of accounting for the seasonal trends.

Results and discussion

Our findings show that the first reducing effect of non-pharmaceutical interventions on the number of prevalent intensive care patients before vaccination can be expected not before a time lag of 5 days; the main effect is after a time lag of 10-15 days. In general, we denote that the number of hospital and prevalent intensive care patients decrease with an increase in the overall non-pharmaceutical interventions intensity with a time lag of 9 and 10 days. Finally, we emphasize a clear interpretation of the findings noting that a causal conclusion is challenging due to the lack of a suitable experimental study design.

Considering multiple governance levels in epidemiologic analysis of public policies

Epidemiology is increasingly asking questions about the use of policies to address structural inequities and intervene on health disparities and public health challenges. However, there has been limited explicit consideration of governance structures in the design of epidemiologic policy analysis. To advance empirical and theoretical inquiry in this space, we propose a model of governance analysis in which public health researchers consider at what level 1) decision-making authority for policy sits, 2) policy is implemented, 3) and accountability for policy effects appear. We follow with examples of how these considerations might improve the evaluation of the policy drivers of population health. Consideration and integration of multiple levels of governance, as well as interactions between levels, can help epidemiologists design studies including new opportunities for quasi-experimental designs and stronger counterfactuals, better quantify the policy drivers of inequities, and aid research evidence and policy development work in targeting multiple levels of governance, ultimately supporting evidence-based policy making.

The Impact of Keeping Indoor Dining Closed on COVID-19 Rates Among Large US Cities: A Quasi-Experimental Design

BACKGROUND

Indoor dining is one of the potential drivers of COVID-19 transmission. We used the heterogeneity among state government preemption of city indoor dining closures to estimate the impact of keeping indoor dining closed on COVID-19 incidence.

METHODS

We obtained case rates and city or state reopening dates from March to October 2020 in 11 US cities. We categorized cities as treatment cities that were allowed by the state to reopen but kept indoor dining closed or comparison cities that would have kept indoor dining closed but that were preempted by their state and had to reopen indoor dining. We modeled associations using a difference-in-difference approach and an event study specification. We ran negative binomial regression models, with city-day as the unit of analysis, city population as an offset, and controlling for time-varying nonpharmaceutical interventions, as well as city and time fixed effects in sensitivity analysis and the event study specification.

RESULTS

Keeping indoor dining closed was associated with a 55% (IRR = 0.45; 95% confidence intervals = 0.21, 0.99) decline in the new COVID-19 case rate over 6 weeks compared with cities that reopened indoor dining, and these results were consistent after testing alternative modeling strategies.

CONCLUSIONS

Keeping indoor dining closed may be directly or indirectly associated with reductions in COVID-19 spread. Evidence of the relationship between indoor dining and COVID-19 case rates can inform policies to restrict indoor dining as a tailored strategy to reduce COVID-19 incidence. See video abstract at, http://links.lww.com/EDE/B902.