The Revolution Will Be Hard to Evaluate: How Co-Occurring Policy Changes Affect Research on the Health Effects of Social Policies

Reporting and Description of Research Methodology in Studies Estimating Effects of Firearm Policies

BACKGROUND

Evidence about which firearm policies work, to what extent, and for whom is hotly debated, perhaps partly because variation in research methodology has produced mixed and inconclusive effect estimates. We conducted a scoping review of firearm policy research in the health sciences in the United States, focusing on methodological considerations for causal inference.

METHODS

We identified original, empirical articles indexed in PubMed from 1 January 2000 to 1 September 2021 that examined any of 18 prespecified firearm policies. We extracted key study components, including policy type(s) examined, policy operationalization, outcomes, study setting and population, study approach and design, causal language, and whether and how authors acknowledged potential sources of bias.

RESULTS

We screened 7733 articles and included 124. A plurality of studies used a legislative score as their primary exposure (n = 39; 32%) and did not examine change in policies over time (n = 47; 38%). Most examined firearm homicide (n = 51; 41%) or firearm suicide (n = 40; 32%) as outcomes. One-third adjusted for other firearm policies (n = 41; 33%). Three studies (2%) explicitly mentioned that their goal was to estimate causal effects, but over half used language implying causality (n = 72; 58%). Most acknowledged causal identification assumptions of temporality (n = 91; 73%) and exchangeability (n = 111; 90%); other assumptions were less often acknowledged. One-third of studies included bias analyses (n = 42; 34%).

CONCLUSIONS

We identified a range of methodologic approaches in firearm policy research in the health sciences. Acknowledging the imitations of data availability and quality, we identify opportunities to improve causal inferences about and reporting on the effects of firearm policies on population health.

Punitive legal responses to prenatal drug use in the United States: A survey of state policies and systematic review of their public health impacts

BACKGROUND

Punitive legal responses to prenatal drug use may be associated with unintended adverse health consequences. However, in a rapidly shifting policy climate, current information has not been summarized. We conducted a survey of U.S. state policies that utilize criminal or civil legal system penalties to address prenatal drug use. We then systematically identified empirical studies evaluating these policies and summarized their potential public health impacts.

METHODS

Using existing databases and original statutory research, we surveyed current U.S. state-level prenatal drug use policies authorizing explicit criminalization, involuntary commitment, civil child abuse substantiation, and parental rights termination. Next, we systematically identified quantitative associations between these policies and health outcomes, restricting to U.S.-based peer-reviewed research, published January 2000-December 2022. Results described study characteristics and synthesized the evidence on health-related harms and benefits associated with punitive policies. Validity threats were described narratively.

RESULTS

By 2022, two states had adopted policies explicitly authorizing criminal prosecution, and five states allowed pregnancy-specific and drug use-related involuntary civil commitment. Prenatal drug use was grounds for substantiating civil child abuse and terminating parental rights in 22 and five states, respectively. Of the 16 review-identified articles, most evaluated associations between punitive policies generally (k = 12), or civil child abuse policies specifically (k = 2), and multiple outcomes, including drug treatment utilization (k = 6), maltreatment reporting and foster care entry (k = 5), neonatal drug withdrawal syndrome (NDWS, k = 4) and other pregnancy and birth-related outcomes (k = 3). Most included studies reported null associations or suggested increases in adverse outcome following punitive policy adoption.

CONCLUSIONS

Nearly half of U.S. states have adopted policies that respond to prenatal drug use with legal system penalties. While additional research is needed to clarify whether such approaches engender overt health harms, current evidence indicates that punitive policies are not associated with public health benefits, and therefore constitute ineffective policy.

Structural stigma and LGBTQ+ health: a narrative review of quantitative studies

Health disparities related to sexual orientation and gender identity exist across multiple outcomes. Scholarship has begun to evaluate whether structural stigma-ie, societal-level conditions, cultural norms, and institutional policies that constrain opportunities, resources, and wellbeing-contributes to health burdens among LGBTQ+ individuals. We conducted a comprehensive review of quantitative studies examining this hypothesis. We found 98 articles that linked objective (ie, non-self-reported) measures of structural stigma to mental (n=57), behavioural (ie, substance use; n=27; HIV/AIDS or sexually transmitted infection; n=20), and physical (n=20) health outcomes. There was generally consistent evidence that structural stigma increases risk of poor health among LGBTQ+ individuals. Several methodological strengths were identified, including the use of multiple measures (eg, laws or policies [59%, 58 of 98]), designs (eg, quasi-experiments [21%, 21 of 98]), and samples (eg, probability-based [56%, 55 of 98]). However, important gaps exist. Just over half of studies included area-level covariates or non-LGBTQ+ comparison groups, which are necessary to address alternative explanations for the observed associations. Additionally, while studies (n=90) have begun to identify candidate mechanisms, only nine (10%) formally tested mediation. We offer suggestions for future research to advance this literature, which has implications not only for the identification of structural determinants of LGBTQ+ health but also for the development of public health interventions that reduce LGBTQ+ health disparities.

Building A Comprehensive, Longitudinal Dataset to Advance Research on the Efficacy of State-Level Anti-bullying Legislation: 1999 to 2017

Bullying is one of the most common forms of youth violence and is associated with myriad adverse consequences over the life course. There has been increasing interest in examining whether anti-bullying legislation is effective in preventing bullying victimization and its negative effects. However, a lack of data structures that comprehensively and longitudinally assess anti-bullying legislation and its provisions has hampered this effort. We provide 18 years of data (1999-2017) on anti-bullying legislation and amendments across 50 U.S. states and the District of Columbia, which we are making publicly available at LawAtlas.org. This article describes how the legal content analysis was conducted, provides information on the reliability of the coding, and details provisions of the legislation that were coded, such as funding provisions and enumerated groups (a total of 122 individual codes are provided). Over 90% of states had at least one amendment to their legislation during this 18-year period (range: 0-22; Mean = 6.1), highlighting both the evolving content of anti-bullying statutes and the importance of tracking these changes with longitudinal data. Additionally, we offer illustrative examples of the kinds of research questions that might be pursued with these new data. For instance, using survival analyses, we show that a variety of state characteristics (e.g., political leaning of state legislatures) predict time to adoption of key provisions of anti-bullying legislation (e.g., the comprehensiveness of legal provisions). Finally, we end with a discussion of how the dataset might be used in future research on the efficacy of anti-bullying legislation.

Identifying Optimal Methods for Addressing Confounding Bias When Estimating the Effects of State-level Policies

BACKGROUND

Policy evaluation studies that assess how state-level policies affect health-related outcomes are foundational to health and social policy research. The relative ability of newer analytic methods to address confounding, a key source of bias in observational studies, has not been closely examined.

METHODS

We conducted a simulation study to examine how differing magnitudes of confounding affected the performance of 4 methods used for policy evaluations: (1) the two-way fixed effects difference-in-differences model; (2) a 1-period lagged autoregressive model; (3) augmented synthetic control method; and (4) the doubly robust difference-in-differences approach with multiple time periods from Callaway-Sant'Anna. We simulated our data to have staggered policy adoption and multiple confounding scenarios (i.e., varying the magnitude and nature of confounding relationships).

RESULTS

Bias increased for each method: (1) as confounding magnitude increases; (2) when confounding is generated with respect to prior outcome trends (rather than levels), and (3) when confounding associations are nonlinear (rather than linear). The autoregressive model and augmented synthetic control method had notably lower root mean squared error than the two-way fixed effects and Callaway-Sant'Anna approaches for all scenarios; the exception is nonlinear confounding by prior trends, where Callaway-Sant'Anna excels. Coverage rates were unreasonably high for the augmented synthetic control method (e.g., 100%), reflecting large model-based standard errors and wide confidence intervals in practice.

CONCLUSIONS

In our simulation study, no single method consistently outperformed the others, but a researcher's toolkit should include all methodologic options. Our simulations and associated R package can help researchers choose the most appropriate approach for their data.

A statewide study of disparities in local policies and tobacco, vape, and cannabis retail environments

The current study: (1) assesses sociodemographic disparities in local policies related to tobacco and cannabis retail, and (2) examines the cross-sectional association between policy strength and retailer densities of tobacco, e-cigarette (vape), and cannabis retailers within California cities and county unincorporated areas (N = 539). We combined (a) American Community Survey data (2019 5-year estimates), (b) 2018 tobacco, vape, and cannabis retailer locations from a commercial data provider, (c) 2017 tobacco and vape retail environment policy data from American Lung Association, and (d) 2018 cannabis policy data from California Cannabis Local Laws Database. Conditional autoregressive models examined policy strength associations with sociodemographic composition and retailer density in California jurisdictions. Jurisdictions with larger percentages of Black and foreign-born residents had stronger tobacco and vape policies. For cannabis policy, only income had a small, significant positive association with policy strength. Contrary to hypothesis, tobacco/vape policies were not significantly associated with retailer density, but cannabis policy strength was associated with lower cannabis retailer density (relative rate = 0.58, 95% Uncertainty Interval 0.47-0.70)-this effect was completely driven by storefront bans. Thus, storefront cannabis bans were the only policy studied that was associated with lower cannabis retailer density. Further research is needed to understand policies and disparities in retail environments for tobacco, vape, and cannabis, including data on the prospective association between policy implementation and subsequent retailer density, and the role of enforcement.

Legislating Inequity: Structural Racism In Groups Of State Laws And Associations With Premature Mortality Rates

Most evaluations of health equity policy have focused on the effects of individual laws. However, multiple laws' combined effects better reflect the crosscutting nature of structurally racist legal regimes. To measure the combined effects of multiple laws, we used latent class analysis, a method for detecting unobserved "subgroups" in a population, to identify clusters of US states based on thirteen structural racism-related legal domains in 2013. We identified three classes of states: one with predominantly harmful laws ([Formula: see text]), another with predominantly protective laws ([Formula: see text]), and a third with a mix of both ([Formula: see text]). Premature mortality rates overall-defined as deaths before age seventy-five per 100,000 population-were highest in states with predominantly harmful laws, which included eighteen states with past Jim Crow laws. This study offers a new method for measuring structural racism on the basis of how groups of laws are associated with premature mortality rates.

A quantitative assessment of the frequency and magnitude of heterogeneous treatment effects in studies of the health effects of social policies

Substantial heterogeneity in effects of social policies on health across subgroups may be common, but has not been systematically characterized. Using a sample of 55 contemporary studies on health effects of social policies, we recorded how often heterogeneous treatment effects (HTEs) were assessed, for what subgroups (e.g., male, female), and the subgroup-specific effect estimates expressed as Standardized Mean Differences (SMDs). For each study, outcome, and dimension (e.g., gender), we fit a random-effects meta-analysis. We characterized the magnitude of heterogeneity in policy effects using the standard deviation of the subgroup-specific effect estimates (τ). Among the 44% of studies reporting subgroup-specific estimates, policy effects were generally small (<0.1 SMDs) with mixed impacts on health (67% beneficial) and disparities (50% implied narrowing of disparities). Across study-outcome-dimensions, 54% indicated any heterogeneity in effects, and 20% had τ > 0.1 SMDs. For 26% of study-outcome-dimensions, the magnitude of τ indicated that effects of opposite signs were plausible across subgroups. Heterogeneity was more common in policy effects not specified a priori. Our findings suggest social policies commonly have heterogeneous effects on health of different populations; these HTEs may substantially impact disparities. Studies of social policies and health should routinely evaluate HTEs.

US State Policy Contexts and Population Health

Policy Points This Perspective connects the dots between the polarization in US states' policy contexts and the divergence in population health across states. Key interlocking forces that fueled this polarization are the political investments of wealthy individuals and organizations and the nationalization of US political parties. Key policy priorities for the next decade include ensuring all Americans have opportunities for economic security, deterring behaviors that kill or injure hundreds of thousands of Americans each year, and protecting voting rights and democratic functioning.

Income support policies and firearm violence prevention: A scoping review

Firearm violence is a major threat to global public health and safety. Several individual, family, peer, community, and societal risk and protective factors determine or modify the risk of firearm violence. Specifically, there is a strong relationship between poverty, income inequality, and firearm violence; as such, interventions that influence upstream determinants of health by providing income support may hold much promise in affecting multiple domains of risk that are on the causal pathway to firearm violence. Guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews, we conducted a scoping review to examine the current state of evidence on the relationship between income support policies and risk of firearm violence. We searched 8 databases related to health and social sciences from inception through March 30, 2022, and placed no time, language, setting, or other publication restrictions on our search, as long as the study was quantitative or mixed-methods and addressed firearm violence specifically, rather than violence more broadly, as an outcome in relation to income support policies. We found 4 studies; of those, 3 were conducted in the United States and 1 in Brazil. All 4 found associations of policy-relevant magnitude between income support policies and reductions in risk of inter-personal firearm violence. We propose future opportunities to enhance the substantive scope and methodologic rigor of this field of research and inform policy and practice for greater impact.

When Effects Cannot be Estimated: Redefining Estimands to Understand the Effects of Naloxone Access Laws

Violations of the positivity assumption (also called the common support condition) challenge health policy research and can result in significant bias, large variance, and invalid inference. We define positivity in the single- and multiple-timepoint (i.e., longitudinal) health policy evaluation setting, and discuss real-world threats to positivity. We show empirical evidence of the practical positivity violations that can result when attempting to estimate the effects of health policies (in this case, Naloxone Access Laws). In such scenarios, an alternative is to estimate the effect of a shift in law enactment (e.g., the effect if enactment had been delayed by some number of years). Such an effect corresponds to what is called a modified treatment policy, and dramatically weakens the required positivity assumption, thereby offering a means to estimate policy effects even in scenarios with serious positivity problems. We apply the approach to define and estimate the longitudinal effects of Naloxone Access Laws on opioid overdose rates.

Changes in Racial/Ethnic Disparities in Opioid-Related Outcomes in Urban Areas during the COVID-19 Pandemic: A Rapid Review of the Literature

Opioid use disorders (OUDs) are increasingly common among minoritized populations, who have historically experienced limited access to healthcare, a situation that may have worsened during the COVID-19 pandemic. Using a structured keyword search in Pubmed, we reviewed the literature to synthesize the evidence on changes in racial/ethnic disparities in OUD-related outcomes in urban areas during the COVID-19 pandemic in the US. Nine articles were included in the final analysis. Six found increases in OUD-related outcomes during the pandemic, with four showing a widening of disparities. Results also point to the worsening of opioid outcomes among Black and Latinx individuals related to shelter-in-place or stay-at-home orders. Studies examining the use of telehealth and access to OUD treatment showed that minoritized groups have benefited from telehealth programs. The limited number of studies in a small number of jurisdictions indicate a gap in research examining the intersection between COVID-19 and OUD-related outcomes with a focus on disparities. More research is needed to understand the impact of the COVID-19 pandemic and related policies on OUD outcomes among racial/ethnic minoritized groups, including examining the impact of service disruptions on vulnerable groups with OUD.

Methodological considerations for estimating policy effects in the context of co-occurring policies

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999 to 2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (> 82%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.

Heterogeneous treatment effects in social policy studies: An assessment of contemporary articles in the health and social sciences

PURPOSE

. Social policies are important determinants of population health but may have varying effects on subgroups of people. Evaluating heterogeneous treatment effects (HTEs) of social policies is critical to determine how social policies will affect health inequities. Methods for evaluating HTEs are not standardized. Little is known about how often and by what methods HTEs are assessed in social policy and health research.

METHODS

. A sample of 55 articles from 2019 on the health effects of social policies were evaluated for frequency of reporting HTEs; for what subgroupings HTEs were reported; frequency of a priori specification of intent to assess HTEs; and methods used for assessing HTEs.

RESULTS

. 24 (44%) studies described some form of HTE assessment, including by age, gender, education, race/ethnicity, and/or geography. Among studies assessing HTEs, 63% specified HTE assessment a priori, and most (71%) used descriptive methods such as stratification; 21% used statistical tests (e.g., interaction terms in a regression); and no studies used data-driven algorithms.

CONCLUSIONS

. Although understanding HTEs could enhance policy and practice-based efforts to reduce inequities, it is not routine research practice. Increased evaluation of HTEs across relevant subgroups is needed.

Characterizing the Landscape of Safety Net Programs and Policies in California during the COVID-19 Pandemic

The COVID-19 pandemic prompted rapid and innovative policymaking around the world at the national, regional, and local levels. There has been limited work to systematically document and characterize new and expanded local U.S. pandemic-era policies, which is imperative to better understand the policy variation and resulting health impacts during this unprecedented time. California, the most populous U.S. state, provides a case example of a particularly active policy response. The aim of this Brief Report is to summarize the creation and potential areas of application of a newly created publicly available California- and US-based COVID-19 policy database. We generated an extensive list of California and US policies that were modified or created in response to the COVID-19 pandemic. From July–November 2021, we searched current and historical California and federal government websites, press releases, social media, and news sources and recorded detailed information on these policies, including coverage dates, eligibility criteria, and benefit amounts. This comprehensive dataset includes 39 public health, economic, housing, and safety net programs and policies implemented at both federal and state levels and provides details of the complex and multifaceted policy landscape in California from March 2020 to November 2021. Our database is publicly available. Future investigators can leverage the information systematically recorded in this database to rigorously assess the short- and long-term effects of these policies, which will in turn inform future preparedness response plans in California and beyond.

What to Do When Everything Happens at Once: Analytic Approaches to Estimate the Health Effects of Co-Occurring Social Policies

Social policies have great potential to improve population health and reduce health disparities. Increasingly, those doing empirical research have sought to quantify the health effects of social policies by exploiting variation in the timing of policy changes across places. Multiple social policies are often adopted simultaneously or in close succession in the same locations, creating co-occurrence that must be handled analytically for valid inferences. Although this is a substantial methodological challenge for researchers aiming to isolate social policy effects, only in a limited number of studies have researchers systematically considered analytic solutions within a causal framework or assessed whether these solutions are being adopted. We designated 7 analytic solutions to policy co-occurrence, including efforts to disentangle individual policy effects and efforts to estimate the combined effects of co-occurring policies. We used an existing systematic review of social policies and health to evaluate how often policy co-occurrence is identified as a threat to validity and how often each analytic solution is applied in practice. Of the 55 studies, only in 17 (31%) did authors report checking for any co-occurring policies, although in 36 studies (67%), at least 1 approach was used that helps address policy co-occurrence. The most common approaches were adjusting for measures of co-occurring policies; defining the outcome on subpopulations likely to be affected by the policy of interest (but not other co-occurring policies); and selecting a less-correlated measure of policy exposure. As health research increasingly focuses on policy changes, we must systematically assess policy co-occurrence and apply analytic solutions to strengthen studies on the health effects of social policies.