Identifying Optimal Methods for Addressing Confounding Bias When Estimating the Effects of State-level Policies

BACKGROUND

Policy evaluation studies that assess how state-level policies affect health-related outcomes are foundational to health and social policy research. The relative ability of newer analytic methods to address confounding, a key source of bias in observational studies, has not been closely examined.

METHODS

We conducted a simulation study to examine how differing magnitudes of confounding affected the performance of 4 methods used for policy evaluations: (1) the two-way fixed effects difference-in-differences model; (2) a 1-period lagged autoregressive model; (3) augmented synthetic control method; and (4) the doubly robust difference-in-differences approach with multiple time periods from Callaway-Sant'Anna. We simulated our data to have staggered policy adoption and multiple confounding scenarios (i.e., varying the magnitude and nature of confounding relationships).

RESULTS

Bias increased for each method: (1) as confounding magnitude increases; (2) when confounding is generated with respect to prior outcome trends (rather than levels), and (3) when confounding associations are nonlinear (rather than linear). The autoregressive model and augmented synthetic control method had notably lower root mean squared error than the two-way fixed effects and Callaway-Sant'Anna approaches for all scenarios; the exception is nonlinear confounding by prior trends, where Callaway-Sant'Anna excels. Coverage rates were unreasonably high for the augmented synthetic control method (e.g., 100%), reflecting large model-based standard errors and wide confidence intervals in practice.

CONCLUSIONS

In our simulation study, no single method consistently outperformed the others, but a researcher's toolkit should include all methodologic options. Our simulations and associated R package can help researchers choose the most appropriate approach for their data.

The Effect of Medicaid Expansion on Caregiver's Quality of Life

Medicaid expansion has been shown to improve access to care, health, and finances in general populations. Until now no studies have considered how Medicaid expansion may affect informal family caregivers who are the backbone of the long term supports and services infrastructure. Family caregivers provide substantial cost savings to Medicare and Medicaid. Yet, they sustain financial, physical, and mental health strain from their caregiving role which Medicaid expansion may offset. This study evaluated the impact of Medicaid expansion on caregivers' mental health using 2015-2018 data from the Behavioral Risk Factor Surveillance System. After adjusting for demographics, socioeconomic status, and health behaviors, caregivers in Medicaid expansion states had a significantly fewer number of poor mental health days in the previous month than caregivers in non-expansion states (ß = -0.528, CI -1.019, -0.036, p < .01). Study findings indicate that Medicaid expansion state status was protective for caregiver's mental health.