Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches

Food Insufficiency Increased After The Expiration Of COVID-19 Emergency Allotments For SNAP Benefits In 2023

In response to economic distress and food insecurity during the COVID-19 pandemic, Congress expanded the Supplemental Nutrition Assistance Program (SNAP) by introducing emergency allotments to increase monthly benefits, starting in March 2020. In March 2023, emergency allotments expired in the thirty-five states and territories still offering them. We provide some of the first evidence of the impacts of this loss of nutrition support-in some cases, more than $250 a month-for economically disadvantaged households. Our quasi-experimental study examined the effects of the program's expiration on food insufficiency, mental health, and financial well-being, using data from the Census Bureau's Household Pulse Survey. In difference-in-differences analyses, we compared pre-post differences among SNAP participants with pre-post differences among income-eligible nonparticipants. The emergency allotment expiration led to a substantial increase in food insufficiency (8.4 percentage points) and greater food pantry use (2.1 percentage points) and difficulty paying expenses (2.0 percentage points). Non-Hispanic Black SNAP participants experienced a greater increase in anxiety symptoms compared with non-Hispanic White SNAP participants. This study has implications for ongoing policy making with respect to nutrition and safety-net programs to support vulnerable families, especially amid inflated food prices.

Estimating the Acute Health Effects of Smoke Exposure from an Urban Factory Fire Accident: A Case Study of a Tire Factory Fire in Korea

BACKGROUND

A major industrial fire accident occurred in a tire manufacturing factory in Daejeon, Korea, on 12 March 2023 and lasted for 3 d, generating air pollutant emissions. Although evidence regarding the health effects of urban fires is limited, residents near tire factory may have experienced health hazards due to smoke exposure from fire plumes.

OBJECTIVES

Capitalizing on the timing of this fire incident as a natural experiment, we estimated the attributable excess air pollution exposure and associated disease development among residents living near the tire factory.

METHODS

We used the generalized synthetic control method to estimate air pollution exposure and health burden attributable to the accident among residents living in smoke-exposed districts. Based on satellite images and air pollution monitoring results, three administrative districts (within 1.2 km from the factory) were defined as smoke-exposed, and the other 79 districts of Daejeon were defined as controls. Among the 11 monitoring stations in Daejeon, the station located 500 m from the factory was used to estimate excess air pollution exposure (PM 10 , PM 2.5 , NO 2 , O 3 , SO 2 , and CO) for residents in the exposed districts. The number of daily district-level disease-specific incidence cases were acquired from the National Health Insurance Database and used to estimate excess health burden resulting from the fire.

RESULTS

During the first week following the factory fire, residents of exposed districts had an estimated excess exposure to 125.2 [95% confidence interval (CI): 44.9, 156.7] μ g / m 3 of PM 10 , 50.4 (95% CI: 12.7, 99.8) ppb of NO 2 , and 32.0 (95% CI: 21.0, 35.9) ppb of SO 2 . We also found an average increase in the incidence cases of other diseases of upper respiratory tract [20.6 persons (95% CI: 6.2, 37.4)], lung disease due to external agents [2.5 persons (95% CI: 2.1, 3.3)], urticaria and erythema [5.9 persons (95% CI: - 0.6 , 11.2)], and episodic and paroxysmal disorders [8.5 persons (95% CI: 3.7, 13.4)] in exposed districts.

DISCUSSION

Excessive air pollution exposure and disease incidence were identified among residents living close to the tire factory. Preventive measures, such as a warning system, to avoid health impacts to people breathing fire-related pollution may be beneficial for communities impacted by such events. https://doi.org/10.1289/EHP14115.

Quasi-experimental methods for pharmacoepidemiology: difference-in-differences and synthetic control methods with case studies for vaccine evaluation

Difference-in-differences and synthetic control methods have become common study designs for evaluating the effects of changes in policies, including health policies. They also have potential for providing real-world effectiveness and safety evidence in pharmacoepidemiology. To effectively add to the toolkit of the field, however, designs-including both their benefits and drawbacks-must be well understood. Quasi-experimental designs provide an opportunity to estimate the average treatment effect on the treated without requiring the measurement of all possible confounding factors, and to assess population-level effects. This requires, however, other key assumptions, including the parallel trends or stable weighting assumptions, a lack of other concurrent events that could alter time trends, and an absence of contamination between exposed and unexposed units. The targeted estimands are also highly specific to the settings of the study, and combining across units or time periods can be challenging. Case studies are presented for 3 vaccine evaluation studies, showcasing some of these challenges and opportunities in a specific field of pharmacoepidemiology. These methods provide feasible and valuable sources of evidence in various pharmacoepidemiologic settings and can be improved through research to identify and weigh the advantages and disadvantages in those settings. This article is part of a Special Collection on Pharmacoepidemiology.

Medicaid Expansion and Perinatal Health Outcomes: A Quasi-Experimental Study

OBJECTIVE

There has been little evidence of the impact of preventive services during pregnancy covered under the Affordable Care Act (ACA) on birthing parent and infant outcomes. To address this gap, this study examines the association between Medicaid expansion under the ACA and birthing parent and infant outcomes of low-income pregnant people.

METHODS

This study used individual-level data from the 2004-2017 annual waves of the Pregnancy Risk Assessment Monitoring System (PRAMS). PRAMS is a surveillance project of the Centers for Disease Control and Prevention and health departments that annually includes a representative sample of 1,300 to 3,400 births per state, selected from birth certificates. Birthing parents' outcomes of interest included timing of prenatal care, gestational diabetes, hypertensive disorders of pregnancy, cigarette smoking during pregnancy, and postpartum care. Infant outcomes included initiation and duration of breastfeeding, preterm birth, and birth weight. The association between ACA Medicaid expansion and the birthing parent and infant outcomes were examined using difference-in-differences estimation.

RESULTS

There was no association between Medicaid expansion and the outcomes examined after correcting for multiple testing. This finding was robust to several sensitivity analyses.

CONCLUSIONS FOR PRACTICE

Study findings suggest that expanded access to more complete insurance benefits with limited cost-sharing for pregnant people, a group that already had high rates of insurance coverage, did not impact the birthing parents' and infant health outcomes examined.

Punitive legal responses to prenatal drug use in the United States: A survey of state policies and systematic review of their public health impacts

BACKGROUND

Punitive legal responses to prenatal drug use may be associated with unintended adverse health consequences. However, in a rapidly shifting policy climate, current information has not been summarized. We conducted a survey of U.S. state policies that utilize criminal or civil legal system penalties to address prenatal drug use. We then systematically identified empirical studies evaluating these policies and summarized their potential public health impacts.

METHODS

Using existing databases and original statutory research, we surveyed current U.S. state-level prenatal drug use policies authorizing explicit criminalization, involuntary commitment, civil child abuse substantiation, and parental rights termination. Next, we systematically identified quantitative associations between these policies and health outcomes, restricting to U.S.-based peer-reviewed research, published January 2000-December 2022. Results described study characteristics and synthesized the evidence on health-related harms and benefits associated with punitive policies. Validity threats were described narratively.

RESULTS

By 2022, two states had adopted policies explicitly authorizing criminal prosecution, and five states allowed pregnancy-specific and drug use-related involuntary civil commitment. Prenatal drug use was grounds for substantiating civil child abuse and terminating parental rights in 22 and five states, respectively. Of the 16 review-identified articles, most evaluated associations between punitive policies generally (k = 12), or civil child abuse policies specifically (k = 2), and multiple outcomes, including drug treatment utilization (k = 6), maltreatment reporting and foster care entry (k = 5), neonatal drug withdrawal syndrome (NDWS, k = 4) and other pregnancy and birth-related outcomes (k = 3). Most included studies reported null associations or suggested increases in adverse outcome following punitive policy adoption.

CONCLUSIONS

Nearly half of U.S. states have adopted policies that respond to prenatal drug use with legal system penalties. While additional research is needed to clarify whether such approaches engender overt health harms, current evidence indicates that punitive policies are not associated with public health benefits, and therefore constitute ineffective policy.

A population-based, province-wide, record-linkage interrupted time series analysis of impact of the universal seasonal influenza vaccination policy on seasonal influenza vaccine uptake among 5-64-year-olds in the province of Manitoba, Canada

INTRODUCTION

Universal seasonal influenza vaccination policy (USIVP) was introduced in Manitoba, Canada in 2010. Its impact on seasonal influenza vaccine (SIV) uptake remains underexplored.

METHODS

We used population-wide data from Manitoba to assess the impact of the USIVP on SIV uptake. The study covered twenty influenza seasons (2000/01-2019/20). We summarized SIV uptake for influenza seasons before and after the USIVP. Utilizing a single-group interrupted time series analysis and appropriately accounting for autocorrelation, we estimated absolute change and annual trend in SIV uptake percentages among 5-17-, 18-44-, and 45-64-year-olds across strata of certain population socioeconomic and health-related characteristics following the USIVP.

RESULTS

Average SIV uptake percentage in all age groups was significantly higher after compared with before the USIVP. Following the USIVP, there was no significant absolute change in SIV uptake percentage among 18-44- and 45-64-year-olds overall; however, a significant decrease was observed among 18-44-year-old males in the higher income quintiles, across healthcare utilization, and in some regions of residence. A significant increase was observed among 5-17-year-olds in the lowest income quintiles, in Northern Manitoba, and among those with less healthcare utilization, and no chronic disease. Overall, there was mostly no significant annual trend in SIV uptake percentage among 18-44-year-olds, and while a significant upward and downward trend was observed among 5-17-year-olds and 45-64-year-olds, respectively, a significant downward trend was observed across all strata of population characteristics within all age groups in Northern Manitoba.

CONCLUSIONS

The USIVP in Manitoba was followed by an absolute increase in SIV uptake percentage only in some socioeconomically disadvantaged subpopulations among 5-17-year-olds. While there was mostly an upward annual trend in SIV uptake percentage among 5-17-year-olds, a downward trend was observed among 45-64-year-olds and across all age groups and subpopulations in socioeconomically disadvantaged Northern Manitoba. These findings are novel for Manitoba and require investigation and public health attention.

Medicaid Expansion and Racial-Ethnic and Sex Disparities in Cardiovascular Diseases Over 6 Years: A Generalized Synthetic Control Approach

BACKGROUND

Studies have suggested Medicaid expansion enacted in 2014 has resulted in a reduction in overall cardiovascular disease (CVD) mortality in the United States. However, it is unknown whether Medicaid expansion has a similar effect across race-ethnicity and sex. We investigated the effect of Medicaid expansion on CVD mortality across race-ethnicity and sex.

METHODS

Data come from the behavioral risk factor surveillance system and the US Centers for Disease Control's Wide-ranging Online Data for Epidemiologic Research, spanning the period 2000-2019. We used the generalized synthetic control method, a quasi-experimental approach, to estimate effects.

RESULTS

Medicaid expansion was associated with -5.36 (mean difference [MD], 95% confidence interval [CI] = -22.63, 11.91) CVD deaths per 100,000 persons per year among Blacks; -4.28 (MD, 95% CI = -30.08, 21.52) among Hispanics; -3.18 (MD, 95% CI = -8.30, 1.94) among Whites; -5.96 (MD, 95% CI = -15.42, 3.50) among men; and -3.34 (MD, 95% CI = -8.05, 1.37) among women. The difference in mean difference (DMD) between the effect of Medicaid expansion in Blacks compared with Whites was -2.18; (DMD, 95% CI = -20.20, 15.83); between that in Hispanics compared with Whites: -1.10; (DMD, 95% CI = -27.40, 25.20) and between that in women compared with men: 2.62; (DMD, 95% CI = -7.95, 13.19).

CONCLUSIONS

Medicaid expansion was associated with a reduction in CVD mortality overall and in White, Black, Hispanic, male, and female subpopulations. Also, our study did not find any difference or disparity in the effect of Medicaid on CVD across race-ethnicity and sex-gender subpopulations, likely owing to imprecise estimates.

Instrumental variables for implementation science: exploring context-dependent causal pathways between implementation strategies and evidence-based interventions

BACKGROUND

The impact of both implementation strategies (IS) and evidence-based interventions (EBI) can vary across contexts, and a better understanding of how and why this occurs presents fundamental but challenging questions that implementation science as a field will need to grapple with. We use causal epidemiologic methods to explore the mechanisms of why sharp distinctions between implementation strategies (IS) and efficacy of an evidence-based intervention (EBI) may fail to recognize that the effect of an EBI can be deeply intertwined and dependent on the context of the IS leading to its uptake.

METHODS

We explore the use of instrumental variable (IV) analyses as a critical tool for implementation science methods to isolate three relevant quantities within the same intervention context when exposure to an implementation strategy is random: (1) the effect of an IS on implementation outcomes (e.g., uptake), (2) effect of EBI uptake on patient outcomes, and (3) overall effectiveness of the IS (i.e., ~ implementation*efficacy). We discuss the mechanisms by which an implementation strategy can alter the context, and therefore effect, of an EBI using the underlying IV assumptions. We illustrate these concepts using examples of the implementation of new ART initiation guidelines in Zambia and community-based masking programs in Bangladesh.

RESULTS

Causal questions relevant to implementation science are answered at each stage of an IV analysis. The first stage assesses the effect of the IS (e.g., new guidelines) on EBI uptake (e.g., same-day treatment initiation). The second stage leverages the IS as an IV to estimate the complier average causal effect (CACE) of the EBI on patient outcomes (e.g., effect of same-day treatment initiation on viral suppression). The underlying assumptions of CACE formalize that the causal effect of EBI may differ in the context of a different IS because (1) the mechanisms by which individuals uptake an intervention may differ and (2) the subgroup of individuals who take up an EBI may differ. IV methods thus provide a conceptual framework for how IS and EBIs are linked and that the IS itself needs to be considered a critical contextual determinant. Moreover, it also provides rigorous methodologic tools to isolate the effect of an IS, EBI, and combined effect of the IS and EBI.

DISCUSSION

Leveraging IV methods when exposure to an implementation strategy is random helps to conceptualize the context-dependent nature of implementation strategies, EBIs, and patient outcomes. IV methods formalize that the causal effect of an EBI may be specific to the context of the implementation strategy used to promote uptake. This integration of implementation science concepts and theory with rigorous causal epidemiologic methods yields novel insights and provides important tools for exploring the next generation of questions related to mechanisms and context in implementation science.

The impact of team-based primary care on medication-related outcomes in older adults: A comparative analysis of two Canadian provinces

Objective

To evaluate if access to team-based primary care is related to medication management outcomes for older adults.

Methods

We completed two retrospective cohort studies using administrative health data for older adults (66+) in Ontario (n = 428,852) and Québec (n = 310,198) who were rostered with a family physician (FP) between the 2001/02 and 2017/18 fiscal years. We generated matched comparison groups of older adults rostered to an FP practicing in a team-based model, and older adults rostered to an FP in a non-team model. We compared the following outcomes between these groups: any adverse drug reactions (ADRs), any potentially inappropriate prescription (PIP), and polypharmacy. Average treatment effects of access to team-based care were estimated using a difference-in-differences estimator.

Results

The risk of an ADR was 22 % higher (RR = 1.22, 95 % CI = 1.18, 1.26) for older adults rostered to a team-based FP in Québec and 6 % lower (RR = 0.943, 95 % CI = 0.907, 0.978) in Ontario. However, absolute risk differences were less than 0.5 %. Differences in the risk of polypharmacy were small in Québec (RR = 1.005, 95 % CI = 1.001, 1.009) and Ontario (RR = 1.004, 95 % CI = 1.001, 1.007) and had absolute risk differences of less than 1 % in both provinces. Effects on PIP were not statistically or clinically significant in adjusted models.

Interpretation

We did not find evidence that access to team-based primary care in Ontario or Québec meaningfully improved medication management outcomes for older adults.

Estimating the effect of timing of earned income tax credit refunds on perinatal outcomes: a quasi-experimental study of California births

BACKGROUND

The largest poverty alleviation program in the US is the earned income tax credit (EITC), providing $60 billion to over 25 million families annually. While research has shown positive impacts of EITC receipt in pregnancy, there is little evidence on whether the timing of receipt may lead to differences in pregnancy outcomes. We used a quasi-experimental difference-in-differences design, taking advantage of EITC tax disbursement each spring to examine whether trimester of receipt was associated with perinatal outcomes.

METHODS

We conducted a difference-in-differences analysis of California linked birth certificate and hospital discharge records. The sample was drawn from the linked CA birth certificate and discharge records from 2007-2012 (N = 2,740,707). To predict eligibility, we created a probabilistic algorithm in the Panel Study of Income Dynamics and applied it to the CA data. Primary outcome measures included preterm birth, small-for-gestational age (SGA), gestational diabetes, and gestational hypertension/preeclampsia.

RESULTS

Eligibility for EITC receipt during the third trimester was associated with a lower risk of preterm birth compared with preconception. Eligibility for receipt in the preconception period resulted in improved gestational hypertension and SGA.

CONCLUSION

This analysis offers a novel method to impute EITC eligibility using a probabilistic algorithm in a data set with richer sociodemographic information relative to the clinical and administrative data sets from which outcomes are drawn. These results could be used to determine the optimal intervention time point for future income supplementation policies. Future work should examine frequent income supplementation such as the minimum wage or basic income programs.

Impact of the universal seasonal influenza vaccination policy in the province of Manitoba, Canada: A population-based, province-wide record-linkage study

INTRODUCTION

In 2010, the government of the province of Manitoba, Canada introduced universal seasonal influenza vaccination policy (USIVP), providing free-of-charge vaccination to all registered residents of the province at least six months of age. Impact of the policy on seasonal influenza vaccine (SIV) uptake (receipt of vaccine) in Manitoba remains unclear, as there is a lack of published evaluations.

METHODS

We conducted an ecological study, utilizing population-wide data from several linked de-identified Manitoba Health and Seniors Care administrative health databases. The study period was from 2000/01-2019/20 influenza seasons. The primary exposure was USIVP (five influenza seasons pre-policy [2005/06-2009/10] compared with post-policy [2010/11-2014/15]). The outcome was SIV uptake. We conducted pre/post logistic regression analysis stratified by age group (<5-, 5-17-, 18-44-, 45-64-, ≥65-year-olds) and certain population socioeconomic and health-related characteristics. Results are adjusted odds ratios with associated 95 % confidence intervals.

RESULTS

We observed significantly increased adjusted odds of SIV uptake post-policy relative to pre-policy in all age groups except ≥65-year-olds already covered from inception of the vaccination programme. The adjusted odds ratios ranged from 0.76 (0.75-0.76) among ≥65-year-olds to 2.15 (2.13-2.18) among 5-17-year-olds, and were largely homogeneous within age groups across sex, income quintiles, regions of residence, and categories of number of visits to primary care physician/hospitalization one year prior to an influenza season except among <5- and 5-17-year-olds. These findings were mostly consistent irrespective of sex and region of residence although there was variability across income quintiles in Northern Manitoba region.

CONCLUSIONS

Introduction of the USIVP in Manitoba was followed by a significant increase in SIV uptake in the five years post policy among <65-year-olds, with similar increased relative odds of vaccination observed within age groups across subpopulations. The observed variations in the relative odds of vaccination across income quintiles in Northern Manitoba region requires administrative attention.

Analysing sentiment change detection of Covid-19 tweets

The Covid-19 pandemic made a significant impact on society, including the widespread implementation of lockdowns to prevent the spread of the virus. This measure led to a decrease in face-to-face social interactions and, as an equivalent, an increase in the use of social media platforms, such as Twitter. As part of Industry 4.0, sentiment analysis can be exploited to study public attitudes toward future pandemics and sociopolitical situations in general. This work presents an analysis framework by applying a combination of natural language processing techniques and machine learning algorithms to classify the sentiment of each tweet as positive, or negative. Through extensive experimentation, we expose the ideal model for this task and, subsequently, utilize sentiment predictions to perform time series analysis over the course of the pandemic. In addition, a change point detection algorithm was applied in order to identify the turning points in public attitudes toward the pandemic, which were validated by cross-referencing the news report at that particular period of time. Finally, we study the relationship between sentiment trends on social media and, news coverage of the pandemic, providing insights into the public's perception of the pandemic and its influence on the news.

The effect of the 2009 revised U.S. guidelines for gestational weight gain on maternal and infant health: a quasi-experimental study

BACKGROUND

Excess gestational weight gain (GWG) has adverse short- and long-term effects on the health of mothers and infants. In 2009, the US Institute of Medicine revised its guidelines for GWG and reduced the recommended GWG for women who are obese. There is limited evidence on whether these revised guidelines affected GWG and downstream maternal and infant outcomes.

METHODS

We used data from the 2004-2019 waves of the Pregnancy Risk Assessment Monitoring System, a serial cross-sectional national dataset including over 20 states. We conducted a quasi-experimental difference-in-differences analysis to assess pre/post changes in maternal and infant outcomes among women who were obese, while "differencing out" the pre/post changes among a control group of women who were overweight. Maternal outcomes included GWG and gestational diabetes; infant outcomes included preterm birth (PTB), low birthweight (LBW), and very low birthweight (VLBW). Analysis began in March 2021.

RESULTS

There was no association between the revised guidelines and GWG or gestational diabetes. The revised guidelines were associated with reduced PTB (- 1.19% points, 95%CI: - 1.86, - 0.52), LBW (- 1.38% points 95%CI: - 2.07, - 0.70), and VLBW (- 1.30% points, 95%CI: - 1.68, - 0.92). Results were robust to several sensitivity analyses.

CONCLUSION

The revised 2009 GWG guidelines were not associated with changes in GWG or gestational diabetes but were associated with improvements in infant birth outcomes. These findings will help inform further programs and policies aimed at improving maternal and infant health by addressing weight gain in pregnancy.

Did Dental Insurance Expansion Improve Dental Care Needs Among Korean Adults? Difference in Difference Analysis

BACKGROUND

In 2012, the Korean government expanded dental insurance for the elderly to promote improved access to dental care. We examined the causal effect of this policy on dental care needs, focusing on low-income older adults.

METHODS

We compared data before and after policy implementation using double difference (DD) and triple difference (DDD) analyses. We used the nationally representative data from the Korea National Health and Nutrition Examination Survey from 2010 and 2016-2018. Individuals aged ≥65 years were included in the treatment group, and individuals aged <65 years were included in the control group.

RESULTS

Dental insurance expansion was associated with a paradoxical increase in perceived unmet dental needs among elderly individuals (8.8 percentage points increase, 95% CI: 4.7 to 13.0). However, there were improvements in dental prosthetics outcomes (denture wearing [4.0 percentage points, 95% CI: 0.2 to 7.9] and dental implants [5.0 percentage points, 95% CI: 2.1 to 7.9]; P < 0.01). Upon analyzing low-income elderly individuals using DDD analysis, we found that the insurance expansion led to a 21.6% smaller increase in unmet dental needs among low-income adults, compared to high-income adults (95% CI, -35.0 to -8.5; P < 0.01).

CONCLUSION

Dental insurance expansion in South Korea resulted in improvements in access to dental prosthetic services overall. It also led to a smaller increase in unmet dental needs among low-income older adults, compared to high-income adults.

Effects Of The 2021 Expanded Child Tax Credit On Adults' Mental Health: A Quasi-Experimental Study

The US Congress temporarily expanded the Child Tax Credit (CTC) during the COVID-19 pandemic to provide economic assistance for families with children. Although formerly the CTC provided $2,000 per child for mostly middle-income parents, during July-December 2021 it provided up to $3,600 per child. Eligibility criteria were also expanded to reach more economically disadvantaged families. There has been little research evaluating the effect of the policy expansion on mental health. Using data from the Census Bureau's Household Pulse Survey and a quasi-experimental study design, we examined the effects of the expanded CTC on mental health and related outcomes among low-income adults with children, and by racial and ethnic subgroup. We found fewer depressive and anxiety symptoms among low-income adults. Adults of Black, Hispanic, and other racial and ethnic backgrounds demonstrated greater reductions in anxiety symptoms compared to non-Hispanic White adults with children. There were no changes in mental health care use. These findings are important for Congress and state legislators to weigh as they consider making the expanded CTC and other similar tax credits permanent to support economically disadvantaged families.

Impact of Malakit intervention on perceptions, knowledge, attitudes, and practices related to malaria among workers in clandestine gold mines in French Guiana: results of multicentric cross-sectional surveys over time

BACKGROUND

Clandestine gold miners remain key hosts for malaria in French Guiana (FG) and contribute to imported malaria cases in Suriname and Brazil. The Malakit intervention, implemented in FG borders with Suriname and Brazil, provided gold miners with training on malaria and kits for self-diagnosis and self-treatment. Having shown a likely impact on malaria transmission, Suriname has now implemented it in routine care for cross-border moving populations. However, a decrease in malaria transmission is frequently associated with a decrease in risk perception, knowledge, and good practices regarding malaria. This study aims to describe the evolution of the perceptions, knowledge, attitudes, and practices (KAP) related to malaria among clandestine gold miners between 2015 and 2019, and to estimate the impact of Malakit on the FG/Suriname border.

METHODS

The primary outcome was the overall KAP score over time and among participants and not participants in the Malakit intervention. A propensity score matching analysis and an inverse probability of treatment weighing analysis were used to estimate the Average Treatment effect on the Treated and the Average Treatment Effect of Malakit, respectively.

RESULTS

Perception and knowledge scores were significantly lower in 2019 compared to 2015 (- 0.27 and - 0.23 points, respectively, p < 0.001) while attitude and practice scores were higher (+ 0.16 and + 0.47 points, respectively, p < 0.001). The overall KAP score was significantly higher among participants in Malakit with both propensity score matching (+ 0.72 points, 95%IC [0.29; 1.15]) and inverse probability of treatment weighting analysis (+ 0.70 points, 95%IC [0.34; 1.05]).

CONCLUSION

A decrease in perception and knowledge about malaria but an improvement of attitudes and practices as the incidence of malaria decreased are observed. The Malakit intervention seems to have a significant positive impact on the overall KAP related to malaria. The integration of this strategy into malaria control programmes could help to improve the KAP, even in areas where malaria is nearly eliminated, through optimal training and health empowerment. Trial registration ClinicalTrials.gov registration number: NCT03695770.

Improving Access to High-Value, High-Cost Medicines: The Use of Subscription Models to Treat Hepatitis C Using Direct-Acting Antivirals in the United States

State payers may face financial incentives to restrict use of high-cost medications. Yet, restrictions on access to high-value medications may have deleterious effects on population health. Direct-acting antivirals (DAAs), available since 2013, can cure chronic infection with hepatitis C virus (HCV). With prices upward of $90,000 for a treatment course, states have struggled to ensure access to DAAs for Medicaid beneficiaries and the incarcerated, populations with a disproportionate share of HCV. Advance purchase commitments (APCs), wherein a payer commits to purchase a certain quantity of medications at lower prices, offer payers incentives to increase access to high-value medications while also offering companies guaranteed revenue. This article discusses the use of subscription models, a type of APC, to support increased access to high-value DAAs for treating HCV. First, the authors provide background information about HCV, its treatment, and state financing of prescription medications. They then review the implementation of HCV subscription models in two states, Louisiana and Washington, and the early evidence of their impact. The article discusses challenges to evaluating state-sponsored subscription models, and it concludes by discussing implications of subscription models that target DAAs and other high-value, high-cost medicines.

Estimating the impact of nutrition and physical activity policies with quasi-experimental methods and simulation modelling: an integrative review of methods, challenges and synergies

BACKGROUND

The promotion of healthy lifestyles has high priority on the global public health agenda. Evidence on the real-world (cost-)effectiveness of policies addressing nutrition and physical activity is needed. To estimate short-term policy impacts, quasi-experimental methods using observational data are useful, while simulation models can estimate long-term impacts. We review the methods, challenges and potential synergies of both approaches for the evaluation of nutrition and physical activity policies.

METHODS

We performed an integrative review applying purposive literature sampling techniques to synthesize original articles, systematic reviews and lessons learned from public international workshops conducted within the European Union Policy Evaluation Network.

RESULTS

We highlight data requirements for policy evaluations, discuss the distinct assumptions of instrumental variable, difference-in-difference, and regression discontinuity designs and describe the necessary robustness and falsification analyses to test them. Further, we summarize the specific assumptions of comparative risk assessment and Markov state-transition simulation models, including their extension to microsimulation. We describe the advantages and limitations of these modelling approaches and discuss future directions, such as the adequate consideration of heterogeneous policy responses. Finally, we highlight how quasi-experimental and simulation modelling methods can be integrated into an evidence cycle for policy evaluation.

CONCLUSIONS

Assumptions of quasi-experimental and simulation modelling methods in policy evaluations should be credible, rigorously tested and transparently communicated. Both approaches can be applied synergistically within a coherent framework to compare policy implementation scenarios and improve the estimation of nutrition and physical activity policy impacts, including their distribution across population sub-groups.

A comparison of four quasi-experimental methods: an analysis of the introduction of activity-based funding in Ireland

Background

Health services research often relies on quasi-experimental study designs in the estimation of treatment effects of a policy change or an intervention. The aim of this study is to compare some of the commonly used non-experimental methods in estimating intervention effects, and to highlight their relative strengths and weaknesses. We estimate the effects of Activity-Based Funding, a hospital financing reform of Irish public hospitals, introduced in 2016.

Methods

We estimate and compare four analytical methods: Interrupted time series analysis, Difference-in-Differences, Propensity Score Matching Difference-in-Differences and the Synthetic Control method. Specifically, we focus on the comparison between the control-treatment methods and the non-control-treatment approach, interrupted time series analysis. Our empirical example evaluated the length of stay impact post hip replacement surgery, following the introduction of Activity-Based Funding in Ireland. We also contribute to the very limited research reporting the impacts of Activity-Based-Funding within the Irish context.

Results

Interrupted time-series analysis produced statistically significant results different in interpretation, while the Difference-in-Differences, Propensity Score Matching Difference-in-Differences and Synthetic Control methods incorporating control groups, suggested no statistically significant intervention effect, on patient length of stay.

Conclusion

Our analysis confirms that different analytical methods for estimating intervention effects provide different assessments of the intervention effects. It is crucial that researchers employ appropriate designs which incorporate a counterfactual framework. Such methods tend to be more robust and provide a stronger basis for evidence-based policy-making.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08657-0.

Using the Generalized Synthetic Control Method to Estimate the Impact of Extreme Weather Events on Population Health

BACKGROUND

Traditional epidemiologic approaches such as time-series or case-crossover designs are often used to estimate the effects of extreme weather events but can be limited by unmeasured confounding. Quasi-experimental methods are a family of methods that leverage natural experiments to adjust for unmeasured confounding indirectly. The recently developed generalized synthetic control method that exploits the timing of an exposure is well suited to estimate the impact of acute environmental events on health outcomes. To demonstrate how this method can be used to study extreme weather events, we examined the impact of the 20-26 October 2007 Southern California wildfire storm on respiratory hospitalizations.

METHODS

We used generalized synthetic control to compare the average number of ZIP code-level respiratory hospitalizations during the wildfire storm between ZIP codes that were classified as exposed versus unexposed to wildfire smoke. We considered wildfire exposure eligibility for each ZIP code using fire perimeters and satellite-based smoke plume data. We retrieved respiratory hospitalization discharge data from the Office of Statewide Health Planning and Development. R code to implement the generalized synthetic control method is included for reproducibility.

RESULTS

The analysis included 172 exposed and 578 unexposed ZIP codes. We estimated that the average effect of the wildfire storm among the exposed ZIP codes was an 18% (95% confidence interval: 10% to 29%) increase in respiratory hospitalizations.

CONCLUSIONS

We illustrate the use of generalized synthetic control to leverage natural experiments to quantify the health impacts of extreme weather events when traditional approaches are unavailable or limited by assumptions.

The effect of the 2009 WIC revision on maternal and child health: A quasi-experimental study

BACKGROUND

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) is the largest U.S. nutrition program for low-income pregnant women. It was revised in 2009, with the goal of improving nutritional content of food packages, enhancing nutrition education, and strengthening breast feeding support. Few studies have assessed the effects of this revision on perinatal health.

OBJECTIVES

To investigate the impact of the revised WIC program on maternal and child health in a large, multi-state data set.

METHODS

We conducted a quasi-experimental difference-in-differences analysis, comparing the pre/post changes among WIC recipients to changes among non-recipients. We adjusted for key sociodemographic covariates in multivariable linear models. We used data from the Pregnancy Risk Assessment Monitoring System (PRAMS) for 18 states from 2004 to 2017.

RESULTS

The main analysis included 331,946 mother-infant dyads. WIC recipients were more likely to be younger, Black or Hispanic/Latina, unmarried, and of greater parity. The revised WIC program was associated with reduced likelihood of more-than-recommended GWG (-1.29% points, 95% confidence interval [CI] -2.03, -0.56) and increased likelihood of ever breast fed (1.18% points, 95% CI 0.28, 2.08). We also identified heterogeneous effects on GWG, with more pronounced associations among women 35 and older. There were no associations with foetal growth.

CONCLUSIONS

The revised WIC program was associated with improvements in women's gestational weight gain and infant breast feeding.

Principles and methods of global legal epidemiology

Although the theory and methods of legal epidemiology-the scientific study and deployment of law as a factor in the cause, distribution, and prevention of disease and injury in a population-have been well developed in the context of domestic law, the challenges posed by shifting the frame of analysis to the global legal space have not yet been fully explored. While legal epidemiology rests on the foundational principles that law acts as an intervention, that law can be an object of scientific study and that law has impacts that should be evaluated, its application to the global level requires the recognition that international laws, policies and norms can cause effects independently from their legal implementation within countries. The global legal space blurs distinctions between 'hard' and 'soft' law, often operating through pathways of global agenda setting, legal language, political pressures, social mobilisation and trade pressures to have direct impacts on people, places and products. Despite these complexities, international law has been overwhelmingly studied as operating solely through national policy change, with only one global quasi-experimental evaluation of an international law's impact on health published to date. To promote greater adoption of global legal epidemiology, we expand on an existing typology of public health law studies with examples of policymaking, mapping, implementation, intervention and mechanism studies. Global legal epidemiology holds great promise as a way to produce rigorous and impactful research on the international laws, policies and norms that shape our collective health, equity and well-being.

Interventions in hypertension: systematic review and meta-analysis of natural and quasi-experiments

BACKGROUND

Hypertension is an urgent public health problem. Consistent summary from natural and quasi-experiments employed to evaluate interventions that aim at preventing or controlling hypertension is lacking in the current literature. This study aims to summarize the evidence from natural and quasi-experiments that evaluated interventions used to prevent or control hypertension.

METHODS

We searched PubMed, Embase and Web of Science for natural and quasi-experiments evaluating interventions used to prevent hypertension, improve blood pressure control or reduce blood pressure levels from January 2008 to November 2018. Descriptions of studies and interventions were systematically summarized, and a meta-analysis was conducted.

RESULTS

Thirty studies were identified, and all used quasi-experimental designs including a difference-in-difference, a pre-post with a control group or a propensity score matching design. Education and counseling on lifestyle modifications such as promoting physical activity (PA), promoting a healthy diet and smoking cessation consultations could help prevent hypertension in healthy people. The use of computerized clinical practice guidelines by general practitioners, education and management of hypertension, the screening for cardiovascular disease (CVD) goals and referral could help improve hypertension control in patients with hypertension. The educating and counseling on PA and diet, the monitoring of patients' metabolic factors and chronic diseases, the combination of education on lifestyles with management of hypertension, the screening for economic risk factors, medical needs, and CVD risk factors and referral all could help reduce blood pressure. In the meta-analysis, the largest reduction in blood pressure was seen for interventions which combined education, counseling and management strategies: weighted mean difference in systolic blood pressure was - 5.34 mmHg (95% confidence interval [CI], - 7.35 to - 3.33) and in diastolic blood pressure was - 3.23 mmHg (95% CI, - 5.51 to - 0.96).

CONCLUSIONS

Interventions that used education and counseling strategies; those that used management strategies; those that used combined education, counseling and management strategies and those that used screening and referral strategies were beneficial in preventing, controlling hypertension and reducing blood pressure levels. The combination of education, counseling and management strategies appeared to be the most beneficial intervention to reduce blood pressure levels.

Medicaid Expansion's Impact on Emergency Department Use by State and Payer

OBJECTIVES

The Affordable Care Act's Medicaid Expansion Program influences healthcare use by increasing insurance coverage. Of particular interest is how this will affect inefficient and expensive emergency department (ED) visits. We estimated the impact of the Medicaid expansion on ED use by states and payer (Medicaid, private insurance, and uninsured) 5 years after the implementation of the Medicaid expansion and illustrated the use of the generalized synthetic control method.

METHODS

In this quasi-experiment study, we implemented the generalized synthetic control method to compare states with Medicaid expansion and states without Medicaid expansion. Data were from the Healthcare Cost Utilization Project Fast Stats, which cover >95% of all ED visits. We included states with complete data from 2010 to 2018.

RESULTS

Overall, the Medicaid expansion increased Medicaid share of ED visits (average treatment effect on the treated [ATT] 11.39%; 95% confidence interval [CI] 8.76-14.02) and decreased private share of ED visits (ATT -5.80%; 95% CI -7.40 to -4.12) and uninsured share of ED visits (ATT -6.66%; 95% CI -9.78 to -3.55).

CONCLUSIONS

Medicaid Expansion Program shifted ED payer mix to Medicaid ED visits from private insurance and uninsured ED visits for adults at age of 19 to 64 years, whereas its effect on total ED volume is mixed among states. States that experienced the largest increase in Medicaid enrollment seem to experience an increase in ED visits although such results did not reach statistical significance.

Telehealth Adoption and Discontinuation by US Hospitals: Results From 2 Quasi-Natural Experiments

Background

Prior US hospital telehealth (video visit) studies have focused on describing factors that influence telehealth adoption or performance effects for specific patient segments, hospital systems, or geographic regions. To our knowledge, a larger-scale, national-level (US) study has yet to be conducted on the causal impacts of hospital telehealth adoption as well as discontinuation.

Objective

The aim of this study is to understand the causal impact of US hospital telehealth adoption or discontinuation on hospital performance from 2016 to 2018.

Methods

We analyzed impacts of telehealth adoption or discontinuation by US hospitals on emergency department visits, total ambulatory visits (minus emergency department visits), outpatient services revenue, total facility expenses, and total hospital revenue for the 2016-2018 period. We specifically focused on performance effects for hospitals that switched from not having telehealth to adopting telehealth, or vice versa, during the 2016-2018 period, thus exploiting 2 quasi-natural experiments. We applied a difference-in-differences research design to each of the 2 main analyses. We compared hospitals that have made a telehealth change to groups of hospitals with similar characteristics that did not make a telehealth change, which established a counterfactual. To appropriately match hospitals between treatment and control groups, we applied propensity score matching. Our primary data were from the American Hospital Association Annual Survey and the Healthcare Cost Report Information System data. Several control variables were obtained from additional sources, including the Area Health Resource File and the Federal Communications Commission.

Results

We found that telehealth adoption by US hospitals during the 2016-2018 period resulted in, on average, an increased number of total ambulatory visits (P=.008), increased total facility expenses (P<.001), and increased hospital revenue (P=.004) compared with the control group. We found that telehealth discontinuation during the same period resulted in, on average, decreased outpatient services revenue (P=.02) compared with the control group.

Conclusions

Our findings suggest that telehealth adoption increases use but has mixed impacts on performance, given that cost and revenue increase. However, once telehealth is offered, removing it can have a negative impact on performance, implying that returning to prior performance levels, if telehealth is removed, may be challenging.

Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for estimating effectiveness of state-level policies

BACKGROUND

Reliable evaluations of state-level policies are essential for identifying effective policies and informing policymakers' decisions. State-level policy evaluations commonly use a difference-in-differences (DID) study design; yet within this framework, statistical model specification varies notably across studies. More guidance is needed about which set of statistical models perform best when estimating how state-level policies affect outcomes.

METHODS

Motivated by applied state-level opioid policy evaluations, we implemented an extensive simulation study to compare the statistical performance of multiple variations of the two-way fixed effect models traditionally used for DID under a range of simulation conditions. We also explored the performance of autoregressive (AR) and GEE models. We simulated policy effects on annual state-level opioid mortality rates and assessed statistical performance using various metrics, including directional bias, magnitude bias, and root mean squared error. We also reported Type I error rates and the rate of correctly rejecting the null hypothesis (e.g., power), given the prevalence of frequentist null hypothesis significance testing in the applied literature.

RESULTS

Most linear models resulted in minimal bias. However, non-linear models and population-weighted versions of classic linear two-way fixed effect and linear GEE models yielded considerable bias (60 to 160%). Further, root mean square error was minimized by linear AR models when we examined crude mortality rates and by negative binomial models when we examined raw death counts. In the context of frequentist hypothesis testing, many models yielded high Type I error rates and very low rates of correctly rejecting the null hypothesis (< 10%), raising concerns of spurious conclusions about policy effectiveness in the opioid literature. When considering performance across models, the linear AR models were optimal in terms of directional bias, root mean squared error, Type I error, and correct rejection rates.

CONCLUSIONS

The findings highlight notable limitations of commonly used statistical models for DID designs, which are widely used in opioid policy studies and in state policy evaluations more broadly. In contrast, the optimal model we identified--the AR model--is rarely used in state policy evaluation. We urge applied researchers to move beyond the classic DID paradigm and adopt use of AR models.

How should we evaluate sweetened beverage tax policies? A review of worldwide experience

Over 45 jurisdictions globally have implemented sweetened beverage taxes. Researchers and policymakers need to assess whether and how these taxes change beverage demand and supply, their intended and unanticipated health, economic and equity impacts. Lessons from such evaluations can maximise the policies' success and impact on non-communicable disease prevention globally. We discuss key theoretical, design and methodological considerations to help policymakers, funders and researchers commission and conduct rigorous evaluations of these policies and related disease prevention efforts. We encourage involving the perspectives of various stakeholders on what evaluations are needed given the specific context, what data and methods are appropriate, readily available or can be collected within time and budget constraints. A logic model /conceptual system map of anticipated implications across sectors and scales should help identify optimal study design, analytical techniques and measures. These models should be updated when synthesising findings across diverse methods and integrating findings across subpopulations using similar methods.

Intended and unintended consequences of the implementation of minimum unit pricing of alcohol in Scotland: a natural experiment

Background

Scotland was the first country to implement minimum unit pricing for alcohol nationally. Minimum unit pricing aims to reduce alcohol-related harms and to narrow health inequalities. Minimum unit pricing sets a minimum retail price based on alcohol content, targeting products preferentially consumed by high-risk drinkers. This study comprised three components.

Objectives

This study comprised three components assessing alcohol consumption and alcohol-related attendances in emergency departments, investigating potential unintended effects of minimum unit pricing on alcohol source and drug use, and exploring changes in public attitudes, experiences and norms towards minimum unit pricing and alcohol use.

Design

We conducted a natural experiment study using repeated cross-sectional surveys comparing Scotland (intervention) and North England (control) areas. This involved comparing changes in Scotland following the introduction of minimum unit pricing with changes seen in the north of England over the same period. Difference-in-difference analyses compared intervention and control areas. Focus groups with young people and heavy drinkers, and interviews with professional stakeholders before and after minimum unit pricing implementation in Scotland allowed exploration of attitudes, experiences and behaviours, stakeholder perceptions and potential mechanisms of effect.

Setting

Four emergency departments in Scotland and North England (component 1), six sexual health clinics in Scotland and North England (component 2), and focus groups and interviews in Scotland (component 3).

Participants

Research nurses interviewed 23,455 adults in emergency departments, and 15,218 participants self-completed questionnaires in sexual health clinics. We interviewed 30 stakeholders and 105 individuals participated in focus groups.

Intervention

Minimum unit pricing sets a minimum retail price based on alcohol content, targeting products preferentially consumed by high-risk drinkers.

Results

The odds ratio for an alcohol-related emergency department attendance following minimum unit pricing was 1.14 (95% confidence interval 0.90 to 1.44; p = 0.272). In absolute terms, we estimated that minimum unit pricing was associated with 258 more alcohol-related emergency department visits (95% confidence interval –191 to 707) across Scotland than would have been the case had minimum unit pricing not been implemented. The odds ratio for illicit drug consumption following minimum unit pricing was 1.04 (95% confidence interval 0.88 to 1.24; p = 0.612). Concerns about harms, including crime and the use of other sources of alcohol, were generally not realised. Stakeholders and the public generally did not perceive price increases or changed consumption. A lack of understanding of the policy may have caused concerns about harms to dependent drinkers among participants from more deprived areas.

Limitations

The short interval between policy announcement and implementation left limited time for pre-intervention data collection.

Conclusions

Within the emergency departments, there was no evidence of a beneficial impact of minimum unit pricing. Implementation appeared to have been successful and there was no evidence of substitution from alcohol consumption to other drugs. Drinkers and stakeholders largely reported not noticing any change in price or consumption. The lack of effect observed in these settings in the short term, and the problem-free implementation, suggests that the price per unit set (£0.50) was acceptable, but may be too low. Our evaluation, which itself contains multiple components, is part of a wider programme co-ordinated by Public Health Scotland and the results should be understood in this wider context.

Future work

Repeated evaluation of similar policies in different contexts with varying prices would enable a fuller picture of the relationship between price and impacts.

Trial registration

Current Controlled Trials ISRCTN16039407.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 9, No. 11. See the NIHR Journals Library website for further project information.

Simulation Modeling for the Economic Evaluation of Population-Based Dietary Policies: A Systematic Scoping Review

Simulation modeling can be useful to estimate the long-term health and economic impacts of population-based dietary policies. We conducted a systematic scoping review following the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guideline to map and critically appraise economic evaluations of population-based dietary policies using simulation models. We searched Medline, Embase, and EconLit for studies published in English after 2005. Modeling studies were mapped based on model type, dietary policy, and nutritional target, and modeled risk factor-outcome pathways were analyzed. We included 56 studies comprising 136 model applications evaluating dietary policies in 21 countries. The policies most often assessed were reformulation (34/136), taxation (27/136), and labeling (20/136); the most common targets were salt/sodium (60/136), sugar-sweetened beverages (31/136), and fruit and vegetables (15/136). Model types included Markov-type (35/56), microsimulation (11/56), and comparative risk assessment (7/56) models. Overall, the key diet-related risk factors and health outcomes were modeled, but only 1 study included overall diet quality as a risk factor. Information about validation was only reported in 19 of 56 studies and few studies (14/56) analyzed the equity impacts of policies. Commonly included cost components were health sector (52/56) and public sector implementation costs (35/56), as opposed to private sector (18/56), lost productivity (11/56), and informal care costs (3/56). Most dietary policies (103/136) were evaluated as cost-saving independent of the applied costing perspective. An analysis of the main limitations reported by authors revealed that model validity, uncertainty of dietary effect estimates, and long-term intervention assumptions necessitate a careful interpretation of results. In conclusion, simulation modeling is widely applied in the economic evaluation of population-based dietary policies but rarely takes dietary complexity and the equity dimensions of policies into account. To increase relevance for policymakers and support diet-related disease prevention, economic effects beyond the health sector should be considered, and transparent conduct and reporting of model validation should be improved.

Developing logic models to inform public health policy outcome evaluation: an example from tobacco control

BACKGROUND

The evaluation of large-scale public health policy interventions often relies on observational designs where attributing causality is challenging. Logic models-visual representations of an intervention's anticipated causal pathway-facilitate the analysis of the most relevant outcomes. We aimed to develop a set of logic models that could be widely used in tobacco policy evaluation.

METHODS

We developed an overarching logic model that reflected the broad categories of outcomes that would be expected following the implementation of tobacco control policies. We subsequently reviewed policy documents to identify the outcomes expected to result from the implementation of each policy and conducted a literature review of existing evaluations to identify further outcomes. The models were revised according to feedbacks from a range of stakeholders.

RESULTS

The final models represented expected causal pathways for each policy. The models included short-term outcomes (such as policy awareness, compliance and social cognitive outcomes), intermediate outcomes (such as changes in smoking behaviour) and long-term outcomes (such as mortality, morbidity and health service usage).

CONCLUSIONS

The use of logic models enables transparent and theory-based planning of evaluation analyses and should be encouraged in the evaluation of tobacco control policy, as well as other areas of public health.

Medically underserved areas: are primary care teams efficient at attracting and retaining general practitioners?

The geographical imbalances of General Practitioners (GPs) may affect their accessibility for populations, especially in medically underserved areas. We investigate the effect of the dramatic and recent diffusion of Primary Care Teams (PCTs), especially in medically underserved areas, in order to attract and retain GPs through an improvement of their working conditions. We analyze the evolution of GPs and young GPs density between 2004 and 2017 according to a spatial taxonomy of French living areas in 6 clusters. Based on a quasi-experimental design comparing living areas, depending on the clusters, with PCTs (treated) and without PCTs (control), we used difference-in-differences models to estimate the impact of PCT new settlements on the evolution of both attraction and retention of GPs. Our results show that PCT settlements are efficient to attract young GPs and that the magnitude of the effects depends on the living area clusters. Results call for specific policies to address geographical inequalities of GPs that consider the type of place and also, in France, for new measures to attract and retain GPs in rural fringes.

Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

Factors Associated With State-Specific Medicaid Expansion and Receipt of Autologous Breast Reconstruction Among Patients Undergoing Mastectomy

IMPORTANCE

Despite demonstrated psychosocial benefits, autologous breast reconstruction remains underutilized. An analysis of the association between Medicaid expansion and autologous breast reconstruction has yet to be performed.

OBJECTIVE

To compare autologous breast reconstruction rates and determine the association between Medicaid expansion and breast reconstruction.

DESIGN, SETTING, AND PARTICIPANTS

A retrospective cross-sectional study was performed using the State Inpatient Database from January 1, 2012, through September 30, 2015, and included 51 340 patients. Patients were identified using the International Classification of Diseases, Ninth Revision, codes for breast cancer, mastectomy, and autologous breast reconstruction. Data from states that expanded Medicaid (New Jersey, New York, and Washington) were compared with states that did not expand Medicaid (Florida, North Carolina, and Wisconsin). Data were analyzed from June 1, 2020, through February 28, 2021.

EXPOSURES

The Patient Protection and Affordable Care Act's Medicaid expansion was implemented in 2014; the preexpansion period ranged from 2012 to 2013 (2 years), whereas the postexpansion period ranged from 2014 to 2015 quarter 3 (1.75 years).

MAIN OUTCOMES AND MEASURES

Primary outcomes included use of autologous breast reconstruction before and after expansion. Independent covariates included patient demographics, comorbidities, and state of residence.

RESULTS

Among 45 850 patients who underwent mastectomy and 9215 patients who received autologous breast reconstruction, 36 777 (67%) were White and 32 205 (59%) had private insurance. The use of immediate or delayed autologous reconstruction increased from 18.1% (4951 of 27 290) to 23.0% (4264 of 18 560) throughout the study period. Compared with 2012, the odds of reconstruction were 64% higher in 2015 (odds ratio [OR], 1.64; 95% CI, 1.48-1.80; P < .001). African American (OR, 1.43; 95% CI, 1.33-1.55; P < .001) and Hispanic (OR, 1.44; 95% CI, 1.31-1.60; P < .001) patients had higher odds of reconstruction compared with White patients regardless of state of residence. However, Medicaid expansion was associated with a 28% decrease in the odds of reconstruction (OR, 0.72; 95% CI, 0.61-0.87; P < .001) for African American patients, a 40% decrease (OR, 0.60; 95% CI, 0.50-0.74; P < .001) for Hispanic patients, and 20% decrease (OR, 0.80; 95% CI, 0.67-0.96; P = .01) for patients with Asian, Native American, or other minority race/ethnicity. Medicaid expansion was not associated with changes in the odds of reconstruction for White patients.

CONCLUSIONS AND RELEVANCE

In this cross-sectional study, although the odds of receiving autologous breast reconstruction increased annually, Medicaid expansion was associated with decreased odds of reconstruction for African American patients, Hispanic patients, and other patients of color.

Study Designs to Assess Real-World Interventions to Prevent COVID-19

Background: In the face of the novel virus SARS-CoV-2, scientists and the public are eager for evidence about what measures are effective at slowing its spread and preventing morbidity and mortality. Other than mathematical modeling, studies thus far evaluating public health and behavioral interventions at scale have largely been observational and ecologic, focusing on aggregate summaries. Conclusions from these studies are susceptible to bias from threats to validity such as unmeasured confounding, concurrent policy changes, and trends over time. We offer recommendations on how to strengthen frequently applied study designs which have been used to understand the impact of interventions to reduce the spread of COVID-19, and suggest implementation-focused, pragmatic designs that, moving forward, could be used to build a robust evidence base for public health practice. Methods: We conducted a literature search of studies that evaluated the effectiveness of non-pharmaceutical interventions and policies to reduce spread, morbidity, and mortality of COVID-19. Our targeted review of the literature aimed to explore strengths and weaknesses of implemented studies, provide recommendations for improvement, and explore alternative real-world study design methods to enhance evidence-based decision-making. Results:Study designs such as pre/post, interrupted time series, and difference-in-differences have been used to evaluate policy effects at the state or country level of a range of interventions, such as shelter-in-place, face mask mandates, and school closures. Key challenges with these designs include the difficulty of disentangling the effects of contemporaneous changes in policy and correctly modeling infectious disease dynamics. Pragmatic study designs such as the SMART (Sequential, Multiple-Assignment Randomized Trial), stepped wedge, and preference designs could be used to evaluate community re-openings such as schools, and other policy changes. Conclusions: As the epidemic progresses, we need to move from post-hoc analyses of available data (appropriate for the beginning of the pandemic) to proactive evaluation to ensure the most rigorous approaches possible to evaluate the impact of COVID-19 prevention interventions. Pragmatic study designs, while requiring initial planning and community buy-in, could offer more robust evidence on what is effective and for whom to combat the global pandemic we face and future policy decisions.

Application of Big Data to Support Evidence-Based Public Health Policy Decision-Making for Hearing

Ideally, public health policies are formulated from scientific data; however, policy-specific data are often unavailable. Big data can generate ecologically-valid, high-quality scientific evidence, and therefore has the potential to change how public health policies are formulated. Here, we discuss the use of big data for developing evidence-based hearing health policies, using data collected and analyzed with a research prototype of a data repository known as EVOTION (EVidence-based management of hearing impairments: public health pOlicy-making based on fusing big data analytics and simulaTION), to illustrate our points. Data in the repository consist of audiometric clinical data, prospective real-world data collected from hearing aids and an app, and responses to questionnaires collected for research purposes. To date, we have used the platform and a synthetic dataset to model the estimated risk of noise-induced hearing loss and have shown novel evidence of ways in which external factors influence hearing aid usage patterns. We contend that this research prototype data repository illustrates the value of using big data for policy-making by providing high-quality evidence that could be used to formulate and evaluate the impact of hearing health care policies.

Do educational reforms increase or decrease health inequalities: A matter of methods?

Evaluating whether social policies reduce health inequalities is complicated by the fact that these upstream determinants may also change the socioeconomic distribution. Failure to account for these compositional changes may severely bias the effect estimation procedure. In this article, we illustrate how a health inequality impact assessment of a policy that (also) changes the socioeconomic distribution may produce biased results. First, we show why analyses that do not account for compositional changes fail to estimate the correct counterfactual outcome. This problem most notably occurs when using repeated cross-sectional data, often the only available option to evaluate the health effect of large-scale policies. Second, we conducted a microsimulation study to estimate the magnitude of the bias under various conditions. The results showed that the actual impact of the policy on health inequalities is often underestimated and may even produce results that are in the opposite direction of the actual causal effect of the policy. Future studies should explore new strategies, such as simulation methods, to assess the impact of policies that (also) cause changes in the socioeconomic composition of the population, to enable researchers to accurately estimate their effect on health inequalities.

Washington's liquor license system and alcohol-related adverse health outcomes

BACKGROUND AND AIMS

In June 2012, Washington state (USA) implemented Initiative 1183, privatizing liquor sales. As a result, off-premises outlets increased from 330 to over 1400 and trading hours lengthened. Increased availability of liquor may lead to increased consumption. This study examines the impact of Initiative 1183 on alcohol-related adverse health outcomes, measured by inpatient hospitalizations for alcohol-related disorders and accidental injuries. It further assesses heterogeneity by urbanicity, because outlets increased most in metropolitan-urban areas.

DESIGN

County-by-quarter difference-in-difference linear regression models, estimated statewide and within metropolitan/rural strata.

SETTING AND PARTICIPANTS

Data are from AHRQ Healthcare Cost and Utilization State Inpatient Database 2010-2014 and HHS Area Health Resource File 2010-2014. Changes in the rates of hospitalizations in the 2.5 years following Initiative 1183 in Washington (n = 39 counties) are compared with changes in Oregon (n = 36 counties).

MEASUREMENTS

County rates of hospitalizations per 1000 residents, including all records with any-listed ICD-9 Clinical Classification Software code denoting an alcohol-related disorder, and all records with any-listed external cause of injury code denoting an accidental injury.

FINDINGS

The increase in the rate of accidental injury hospitalizations in Washington's metropolitan-urban counties was on average 0.289 hospitalizations per 1000 county residents per quarter greater than the simultaneous increase observed in Oregon (P = 0.017). This result was robust to alternative specifications using a propensity score matched sample and synthetic control methods with data from other comparison states. The evidence did not suggest that Initiative 1183 was associated with differential changes in the rate of hospitalizations for alcohol-related disorders in metropolitan-urban (P = 0.941), non-metropolitan-urban (P = 0.162), or rural counties (P = 0.876).

CONCLUSIONS

Implementing Washington's Initiative 1183 (privatizing liquor sales) appears to have been associated with a significant increase in the rate of accidental injury hospitalizations in urban counties in that state but does not appear to be significantly associated with changes in the rate of hospitalizations specifically for alcohol-related disorders within 2.5 years.

The evidence for the impact of policy on physical activity outcomes within the school setting: A systematic review

BACKGROUND

Despite the well-established health benefits of physical activity (PA) for young people (aged 4-19 years), most do not meet PA guidelines. Policies that support PA in schools may be promising, but their impact on PA behavior is poorly understood. The aim of this systematic review was to ascertain the level and type of evidence reported in the international scientific literature for policies within the school setting that contribute directly or indirectly to increasing PA.

METHODS

This systematic review is compliant with Preferred Reporting Items for Systematic Review and Meta-Analysis guidelines. Six databases were searched using key concepts of policy, school, evaluation, and PA. Following title and abstract screening of 2323 studies, 25 progressed to data synthesis. Methodological quality was assessed using standardized tools, and the strength of the evidence of policy impact was described based on pre-determined codes: positive, negative, inconclusive, or untested statistically.

RESULTS

Evidence emerged for 9 policy areas that had a direct or indirect effect on PA within the school setting. These were whole school PA policy, physical education, sport/extracurricular PA, classroom-based PA, active breaks/recess, physical environment, shared use agreements, active school transport, and surveillance. The bulk of the evidence was significantly positive (54%), 27% was inconclusive, 9% was significantly negative, and 11% was untested (due to rounding, some numbers add to 99% or 101%). Frequency of evidence was highest in the primary setting (41%), 34% in the secondary setting, and 24% in primary/secondary combined school settings. By policy area, frequency of evidence was highest for sport/extracurricular PA (35%), 17% for physical education, and 12% for whole school PA policy, with evidence for shared use agreements between schools and local communities rarely reported (2%). Comparing relative strength of evidence, the evidence for shared use agreements, though sparse, was 100% positive, while 60% of the evidence for whole school PA policy, 59% of the evidence for sport/extracurricular PA, 57% of the evidence for physical education, 50% of the evidence for PA in classroom, and 50% of the evidence for active breaks/recess were positive.

CONCLUSION

The current evidence base supports the effectiveness of PA policy actions within the school setting but cautions against a "one-size-fits-all" approach and emphasizes the need to examine policy implementation to maximize translation into practice. Greater clarity regarding terminology, measurement, and methods for evaluation of policy interventions is needed.

Methodological issues in economic evaluations of emergency transport systems in low-income and middle-income countries

A recent systematic review identified few papers on the economic evaluation of systems for emergency transport of acutely ill or injured patients. In addition, we found no articles dealing with the methodological challenges posed by such studies in low-income or middle-income countries. We therefore carried out an analysis of issues that are of particular salience to this important topic. This is an intellectual study in which we develop models, identify their limitations, suggest potential extensions to the models and discuss priorities for empirical studies to populate models. First, we develop a general model to calculate changes in survival contingent on the reduced time to treatment that an emergency transport system is designed to achieve. Second, we develop a model to estimate transfer times over an area that will be served by a proposed transfer system. Third, we discuss difficulties in obtaining parameters with which to populate the models. Fourth, we discuss costs, both direct and indirect, of an emergency transfer service. Fifth, we discuss the issue that outcomes other than survival should be considered and that the effects of a service are a weighted sum over all the conditions and severities for which the service caters. Lastly, based on the above work, we identify priorities for research. To our knowledge, this is the first study to identify and frame issues in the health economics of acute transfer systems and to develop models to calculate survival rates from basic parameters, such as time delay/survival relationships, that vary by intervention type and context.

Implementing best available evidence into practice for incontinence-associated dermatitis in Australia: A multisite multimethod study protocol

AIMS

Incontinence-associated dermatitis (IAD) is an insidious and under-reported hospital-acquired complication which substantially impacts on patients' quality of life. A published international guideline and the Ghent Global IAD Categorisation Tool (GLOBIAD) outline the best available evidence for the optimal management of IAD. This study aims to implement theguideline and the GLOBIAD tool and evaluate the effect on IAD occurrences and sacral pressure injuries as well as patient, clinician and cost-effectiveness outcomes.

MATERIALS AND METHODS

The study will employ a multi-method design across six hospitals in five health districts in Australia, and will be conducted in three phases (pre-implementation, implementation and post-implementation) over 19 months. Data collection will involve IAD and pressure injury prevalence audits for patient hospital admissions, focus groups with, and surveys of, clinicians, patient interviews, and collection of the cost of IAD hospital care and patient-related outcomes including quality of life. Eligible participants will be hospitalised adults over 18 years of age experiencing incontinence, and clinicians working in the study wards will be invited to participate in focus groups and surveys.

CONCLUSION

The implementation of health district-wide evidence-based practices for IAD using a translational research approach that engages key stakeholders will allow the standardisation of IAD care that can potentially be applicable to a range of settings. Knowledge gained will inform future practice change in patient care and health service delivery and improve the quality of care for patients with IAD. Support at the hospital, state and national levels, coupled with a refined stakeholder-inclusive strategy, will enhance this project's success, sustainability and scalability beyond this existing project.

A Difference-in-Difference Study Evaluating the Effect of Minimum Wage Policy on Body Mass Index and Related Health Behaviors

Minimum wage laws are a promising policy lever to promote health equity, but few rigorous evaluations have tested whether and how minimum wage policy affects health outcomes. This paper describes an ongoing difference-in-difference study evaluating the health effects of the 2017 Minneapolis Minimum Wage Ordinance, which incrementally increases the minimum wage to $15/hr. We present: (1) the conceptual model guiding the study including mediating mechanisms, (2) the study design, and (3) baseline findings from the study, and (4) the analytic plan for the remainder of the study. This prospective study follows a cohort of 974 low-wage workers over four years to compare outcomes among low-wage workers in Minneapolis, Minnesota, and those in a comparison city (Raleigh, North Carolina). Measures include height/weight, employment paystubs, two weeks of food purchase receipts, and a survey capturing data on participant demographics, health behaviors, and household finances. Baseline findings offer a profile of individuals likely to be affected by minimum wage laws. While the study is ongoing, the movement to increase local and state minimum wage is currently high on the policy agenda; evidence is needed to determine what role, if any, such policies play in improving the health of those affected.

Socio-economic and racial/ethnic disparities in the nutritional quality of packaged food purchases in the USA, 2008-2018

OBJECTIVE

To determine whether disparities exist in the nutritional quality of packaged foods and beverage purchases by household income, education and race/ethnicity and if they changed over time.

DESIGN

We used Nielsen Homescan, a nationally representative household panel, from 2008 to 2018 (n = 672 821 household-year observations). Multivariate, multilevel regressions were used to model the association between sociodemographic groups and a set of nutritional outcomes of public health interest, including nutrients of concern (sugar, saturated fat and Na) and calories from specific food groups (fruits, non-starchy vegetables, processed meats, sugar-sweetened beverages and junk foods).

SETTING

Household panel survey.

PARTICIPANTS

Approximately 60 000 households each year from the USA.

RESULTS

Disparities were found by income and education for most outcomes and widened for purchases of fruits, vegetables and the percentage of calories from sugar between 2008 and 2018. The magnitude of disparities was largest by education. Disparities between Black and White households include the consumption of processed meats and the percentage of calories from sugar, while no disparities were found between White and Hispanic households. Disparities have been largely persistent, as any significant changes over time have been substantively small.

CONCLUSIONS

Policies to improve the healthfulness of packaged foods must be expanded beyond SSB taxes, and future research should focus on what mediates the relationship between education and diet so as not to exacerbate disparities.

Urban vs. Rural Socioeconomic Differences in the Nutritional Quality of Household Packaged Food Purchases by Store Type

The U.S. food system is rapidly changing, including the growth of mass merchandisers and dollar stores, which may impact the quality of packaged food purchases (PFPs). Furthermore, diet-related disparities exist by socioeconomic status (SES) and rural residence. We use data from the 2010-2018 Nielsen Homescan Panel to describe the nutritional profiles of PFPs by store type and to assess whether these vary by household urbanicity and SES. Store types include grocery stores, mass merchandisers, club stores, online shopping, dollar stores, and convenience/drug stores. Food and beverage groups contributing the most calories at each store type are estimated using survey-weighted means, while the associations of urbanicity and SES with nutritional quality are estimated using multivariate regression. We find that households that are customers at particular store types purchase the same quality of food regardless of urbanicity or SES. However, we find differences in the quality of foods between store types and that the quantity of calories purchased at each store type varies according to household urbanicity and SES. Rural shoppers tend to shop more at mass merchandisers and dollar stores with less healthful PFPs. We discuss implications for the types of store interventions most relevant for improving the quality of PFPs.

The use of evaluation methods for the overall assessment of health policy: potential and limitations

Background

The implementation of public policies requires special attention from public authorities to ensure their transparency, effectiveness and efficiency. For this reason, efforts to evaluate the abovementioned politics gained attention and importance. Similar processes, by their very nature, are also noticeable in the area of health policy, however, the nature of the solutions used raises questions about the extent to which they allow to capture the assessed phenomenon in a holistic way. The current approach to the problem of evaluating health policy shows a tendency to break down this phenomenon into components including policies, programs or projects. The purpose of this publication was to assess the main methodological approaches used in evaluation studies in terms of their usefulness and limitations in conducting overall assessment of health policy.

Main body

The publication divides evaluation methods based on three main criteria identified in the literature—time, purpose and scope of evaluation. Methodological approaches to evaluation activities detailed on this basis are discussed from the point of view of their characteristics, usefulness and limitations in the creation of comprehensive health policy assessments. The growing awareness of the need for a different approach to evaluation, which was confirmed by the course of the discourse on evaluation in public health, was also pointed out.

Conclusion

Given the complexity of the matter making up the health policy, attempts aimed at its overall assessment should be based on an approach integrating many approaches, while ensuring coordination of evaluation activities that should be subject to common assumptions.

The effectiveness of plain packaging in discouraging tobacco consumption in Australia

The removal of all branding and promotions from tobacco products, known as 'plain packaging', is intended to decrease tobacco consumption. Here we examine the effectiveness of Australia's plain packaging law, which coincided with a change in graphic and text health warnings, by using nationally indicative data 5 years post implementation. We measured the effect of the law on smoking prevalence, tobacco expenditure, expenditure intensity and quantity of tobacco consumed, using New Zealand as a control country in a difference-in-differences research design. We uncover a substitution effect that is robust to different specifications and control countries. In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes. To discourage such substitution and to help the policy achieve its intended outcomes, policymakers should consider implementing auxiliary measures, such as taxes or price floors.

"Not Alone Anymore": The Experiences of Adults With Diabetes in New York's Medicaid Health Home Program

BACKGROUND

New York State Medicaid's Health Home program is an example of a natural experiment that could affect individuals with diabetes. While evaluations of interventions such as the Health Home program are generally based solely on clinical and administrative data and rarely examine patients' experience, patients may add to the understanding of the intervention's implementation and mechanisms of impact.

OBJECTIVE

The objective of this study was to qualitatively examine the health and nonmedical challenges faced by Medicaid-insured patients with diabetes and their experiences with the services provided by New York's Health Homes to address these challenges.

RESEARCH DESIGN

We performed 10 focus groups and 23 individual interviews using a guide developed in collaboration with a stakeholder board. We performed a thematic analysis to identify cross-cutting themes.

SUBJECTS

A total of 63 Medicaid-insured individuals with diabetes, 31 of whom were enrolled in New York's Health Home program.

RESULTS

While participants were not generally familiar with the term "Health Home," they described and appreciated services consistent with Health Home enrollment delivered by care managers. Services addressed challenges in access to care, especially by facilitating and reminding participants about appointments, and nonmedical needs, such as transportation, housing, and help at home. Participants valued their personal relationships with care managers and the psychosocial support they provided.

CONCLUSIONS

From the perspective of its enrollees, the Health Home program primarily addressed access to care, but also addressed material and psychosocial needs. These findings have implications for Health Home entities and for research assessing their impact.