Quantitative measures of health policy implementation determinants and outcomes: a systematic review

Design considerations for developing measures of policy implementation in quantitative evaluations of public health policy

Typical quantitative evaluations of public policies treat policies as a binary condition, without further attention to how policies are implemented. However, policy implementation plays an important role in how the policy impacts behavioral and health outcomes. The field of policy-focused implementation science is beginning to consider how policy implementation may be conceptualized in quantitative analyses (e.g., as a mediator or moderator), but less work has considered how to measure policy implementation for inclusion in quantitative work. To help address this gap, we discuss four design considerations for researchers interested in developing or identifying measures of policy implementation using three independent NIH-funded research projects studying e-cigarette, food, and mental health policies. Mini case studies of these considerations were developed via group discussions; we used the implementation research logic model to structure our discussions. Design considerations include (1) clearly specifying the implementation logic of the policy under study, (2) developing an interdisciplinary team consisting of policy practitioners and researchers with expertise in quantitative methods, public policy and law, implementation science, and subject matter knowledge, (3) using mixed methods to identify, measure, and analyze relevant policy implementation determinants and processes, and (4) building flexibility into project timelines to manage delays and challenges due to the real-world nature of policy. By applying these considerations in their own work, researchers can better identify or develop measures of policy implementation that fit their needs. The experiences of the three projects highlighted in this paper reinforce the need for high-quality and transferrable measures of policy implementation, an area where collaboration between implementation scientists and policy experts could be particularly fruitful. These measurement practices provide a foundation for the field to build on as attention to incorporating measures of policy implementation into quantitative evaluations grows and will help ensure that researchers are developing a more complete understanding of how policies impact health outcomes.

Research outcomes informing the selection of public health interventions and strategies to implement them: A cross-sectional survey of Australian policy-maker and practitioner preferences

BACKGROUND

A key role of public health policy-makers and practitioners is to ensure beneficial interventions are implemented effectively enough to yield improvements in public health. The use of evidence to guide public health decision-making to achieve this is recommended. However, few studies have examined the relative value, as reported by policy-makers and practitioners, of different broad research outcomes (that is, measures of cost, acceptability, and effectiveness). To guide the conduct of research and better inform public health policy and practice, this study aimed at describing the research outcomes that Australian policy-makers and practitioners consider important for their decision-making when selecting: (a) public health interventions; (b) strategies to support their implementation; and (c) to assess the differences in research outcome preferences between policy-makers and practitioners.

METHOD

An online value-weighting survey was conducted with Australian public health policy-makers and practitioners working in the field of non-communicable disease prevention. Participants were presented with a list of research outcomes and were asked to select up to five they considered most critical to their decision-making. They then allocated 100 points across these - allocating more points to outcomes perceived as more important. Outcome lists were derived from a review and consolidation of evaluation and outcome frameworks in the fields of public health knowledge translation and implementation. We used descriptive statistics to report relative preferences overall and for policy-makers and practitioners separately.

RESULTS

Of the 186 participants; 90 primarily identified as policy-makers and 96 as public health prevention practitioners. Overall, research outcomes of effectiveness, equity, feasibility, and sustainability were identified as the four most important outcomes when considering either interventions or strategies to implement them. Scores were similar for most outcomes between policy-makers and practitioners.

CONCLUSION

For Australian policy-makers and practitioners working in the field of non-communicable disease prevention, outcomes related to effectiveness, equity, feasibility, and sustainability appear particularly important to their decisions about the interventions they select and the strategies they employ to implement them. The findings suggest researchers should seek to meet these information needs and prioritize the inclusion of such outcomes in their research and dissemination activities. The extent to which these outcomes are critical to informing the decision of policy-makers and practitioners working in other jurisdictions or contexts warrants further investigation.

Key actors in behavioral health services availability and accessibility research: a scoping review bibliometric analysis

This bibliometric review aims to identify key actors in the behavioral health services availability/accessibility literature. Coalescing information about these actors could support subsequent research efforts to improve the availability and accessibility of behavioral health services. The authors used a scoping review method and a bibliometric approach. The articles came from Medline, Embase, Web of Science, CINAHL, and PsycINFO. Articles were included if they assessed behavioral health service availability or accessibility quantitatively and were written in English. The final sample included 265 articles. Bibliometric data were extracted, coded, and verified. The authors analyzed the data using univariate and social network analyses. Publishing in this area has become more consistent and has grown since 2002. Psychiatric Services and Graduate Theses were the most frequently used publication venues. The National Institute on Drug Abuse, National Institute of Mental Health, and the Veterans Administration funded the most research. The most frequently used keyword was "health services accessibility." The findings suggest that this literature is growing. There are a few clusters of researchers in this area. Government organizations primarily fund this research. The paper and supplementary materials list the top researchers, publication venues, funding sources, and key terms to promote further behavioral health availability/accessibility research.

A service-user digital intervention to collect real-time safety information on acute, adult mental health wards: the WardSonar mixed-methods study

Background

Acute inpatient mental health services report high levels of safety incidents. The application of patient safety theory has been sparse, particularly concerning interventions that proactively seek patient perspectives.

Objective(s)

Develop and evaluate a theoretically based, digital monitoring tool to collect real-time information from patients on acute adult mental health wards about their perceptions of ward safety.

Design

Theory-informed mixed-methods study. A prototype digital monitoring tool was developed from a co-design approach, implemented in hospital settings, and subjected to qualitative and quantitative evaluation.

Setting and methods

Phase 1: scoping review of the literature on patient involvement in safety interventions in acute mental health care; evidence scan of digital technology in mental health contexts; qualitative interviews with mental health patients and staff about perspectives on ward safety. This, alongside stakeholder engagement with advisory groups, service users and health professionals, informed the development processes. Most data collection was virtual. Phase 1 resulted in the technical development of a theoretically based digital monitoring tool that collected patient feedback for proactive safety monitoring. Phase 2: implementation of the tool in six adult acute mental health wards across two UK NHS trusts; evaluation via focused ethnography and qualitative interviews. Statistical analysis of WardSonar data and routine ward data involving construction of an hour-by-hour data set per ward, permitting detailed analysis of the use of the WardSonar tool.

Participants

A total of 8 patients and 13 mental health professionals participated in Phase 1 interviews; 33 staff and 34 patients participated in Phase 2 interviews.

Interventions

Patients could use a web application (the WardSonar tool) to record real-time perceptions of ward safety. Staff could access aggregated, anonymous data to inform timely interventions.

Results

Coronavirus disease 2019 restrictions greatly impacted the study. Stakeholder engagement permeated the project. Phase 1 delivered a theory-based, collaboratively designed digital tool for proactive patient safety monitoring. Phase 2 showed that the tool was user friendly and broadly acceptable to patients and staff. The aggregated safety data were infrequently used by staff. Feasibility depended on engaged staff and embedding use of the tool in ward routines. There is strong evidence that an incident leads to increased probability of further incidents within the next 4 hours. This puts a measure on the extent to which social/behavioural contagion persists. There is weak evidence to suggest that an incident leads to a greater use of the WardSonar tool in the following hour, but none to suggest that ward atmosphere predicts future incidents. Therefore, how often patients use the tool seems to send a stronger signal about potential incidents than patients' real-time reports about ward atmosphere.

Limitations

Implementation was limited to two NHS trusts. Coronavirus disease 2019 impacted design processes including stakeholder engagement; implementation; and evaluation of the monitoring tool in routine clinical practice. Higher uptake could enhance validity of the results.

Conclusions

WardSonar has the potential to provide a valuable route for patients to communicate safety concerns. The WardSonar monitoring tool has a strong patient perspective and uses proactive real-time safety monitoring rather than traditional retrospective data review.

Future work

The WardSonar tool can be refined and tested further in a post Coronavirus disease 2019 context.

Study registration

This study is registered as ISRCTN14470430.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR128070) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 14. See the NIHR Funding and Awards website for further award information.

A Tale of Two Taxes: Implementation of Earmarked Taxes for Behavioral Health Services in California and Washington State

OBJECTIVE

The authors sought to characterize perceptions of the impacts, attributes, and support for taxes earmarked for behavioral health services and to compare perceptions of the taxes among professionals in California and Washington, two states differing in earmarked tax designs.

METHODS

Surveys were completed by 155 public agency and community organization professionals involved in tax implementation in California (N=87) and Washington State (N=68) during 2022-2023 (29% response rate). Respondents indicated their perceptions of the taxes' impacts, attributes, and support. Responses were summed as aggregate scores and were also analyzed as individual items. Bivariate analyses were used to compare responses of professionals in California versus Washington State.

RESULTS

Earmarked taxes were generally regarded positively. Of the respondents, >80% strongly agreed that the taxes increased funding for services and were helpful, and only 10% strongly agreed that the taxes decreased behavioral health funding from other sources. Substantially more respondents in California than in Washington State strongly agreed that taxes' reporting requirements were complicated (45% vs. 5%, p<0.001) and that the taxes increased unjustified scrutiny of services or systems (33% vs. 2%, p<0.001). However, more respondents in California than in Washington State also strongly agreed that the taxes increased public awareness about behavioral health (56% vs. 15%, p<0.001) and decreased behavioral health stigma (47% vs. 14%, p<0.001).

CONCLUSIONS

Perceptions of the strengths and weaknesses of taxes earmarked for behavioral health services may vary by design features of the tax. Such features include stigma-reduction initiatives and tax spending and reporting requirements.

Acceptability and feasibility of policy implementation strategies for taxes earmarked for behavioral health services

Background

This study's aims are to: (1) Compare the acceptability and feasibility of five types of implementation strategies that could be deployed to increase the reach of evidence-based practices (EBPs) with revenue from policies that earmark taxes for behavioral health services, and (2) Illustrate how definitions of implementation strategies and measures of acceptability and feasibility can be used in policy-focused implementation science research.

Methods

Web-based surveys of public agency and community organization professionals involved with earmarked tax policy implementation were completed in 2022-2023 (N = 211, response rate = 24.9%). Respondents rated the acceptability and feasibility of five types of implementation strategies (dissemination, implementation process, integration, capacity-building, and scale-up). Aggregate acceptability and feasibility scores were calculated for each type of strategy (scoring range 4-20). Analyses of variance compared scores across strategies and between organizational actor types.

Findings

For acceptability, capacity-building strategies had the highest rating (M = 16.3, SD = 3.0), significantly higher than each of the four other strategies, p ≤ . 004), and scale-up strategies had the lowest rating (M = 15.6). For feasibility, dissemination strategies had the highest rating (M = 15.3, significantly higher than three of the other strategies, p ≤ .002) and scale-up strategies had the lowest rating (M = 14.4).

Conclusions

Capacity-building and dissemination strategies may be well-received and readily deployed by policy implementers to support EBPs implementation with revenue from taxes earmarked for behavioral health services. Adapting definitions of implementation strategies for policy-focused topics, and applying established measures of acceptability and feasibility to these strategies, demonstrates utility as an approach to advance research on policy-focused implementation strategies.

Strategies to improve implementation of cascade testing in hereditary cancer syndromes: a systematic review

Hereditary cancer syndromes constitute approximately 10% of all cancers. Cascade testing involves testing of at-risk relatives to determine if they carry the familial pathogenic variant. Despite growing efforts targeted at improving cascade testing uptake, current literature continues to reflect poor rates of uptake, typically below 30%. This study aims to systematically review current literature on intervention strategies to improve cascade testing, assess the quality of intervention descriptions and evaluate the implementation outcomes of listed interventions. We searched major databases using keywords and subject heading of "cascade testing". Interventions proposed in each study were classified according to the Effective Practice and Organization of Care (EPOC) taxonomy. Quality of intervention description was assessed using the TIDieR checklist, and evaluation of implementation outcomes was performed using Proctor's Implementation Outcomes Framework. Improvements in rates of genetic testing uptake was seen in interventions across the different EPOC taxonomy strategies. The average TIDieR score was 7.3 out of 12. Items least reported include modifications (18.5%), plans to assess fidelity/adherence (7.4%) and actual assessment of fidelity/adherence (7.4%). An average of 2.9 out of 8 aspects of implementation outcomes were examined. The most poorly reported outcomes were cost, fidelity and sustainability, with only 3.7% of studies reporting them. Most interventions have demonstrated success in improving cascade testing uptake. Uptake of cascade testing was highest with delivery arrangement (68%). However, the quality of description of interventions and assessment of implementation outcomes are often suboptimal, hindering their replication and implementation downstream. Therefore, further adoption of standardized guidelines in reporting of interventions and formal assessment of implementation outcomes may help promote translation of these interventions into routine practice.

A Semantic Approach to Describe Social and Economic Characteristics That Impact Health Outcomes (Social Determinants of Health): Ontology Development Study

BACKGROUND

Social determinants of health (SDoH) have been described by the World Health Organization as the conditions in which individuals are born, live, work, and age. These conditions can be grouped into 3 interrelated levels known as macrolevel (societal), mesolevel (community), and microlevel (individual) determinants. The scope of SDoH expands beyond the biomedical level, and there remains a need to connect other areas such as economics, public policy, and social factors.

OBJECTIVE

Providing a computable artifact that can link health data to concepts involving the different levels of determinants may improve our understanding of the impact SDoH have on human populations. Modeling SDoH may help to reduce existing gaps in the literature through explicit links between the determinants and biological factors. This in turn can allow researchers and clinicians to make better sense of data and discover new knowledge through the use of semantic links.

METHODS

An experimental ontology was developed to represent knowledge of the social and economic characteristics of SDoH. Information from 27 literature sources was analyzed to gather concepts and encoded using Web Ontology Language, version 2 (OWL2) and Protégé. Four evaluators independently reviewed the ontology axioms using natural language translation. The analyses from the evaluations and selected terminologies from the Basic Formal Ontology were used to create a revised ontology with a broad spectrum of knowledge concepts ranging from the macrolevel to the microlevel determinants.

RESULTS

The literature search identified several topics of discussion for each determinant level. Publications for the macrolevel determinants centered around health policy, income inequality, welfare, and the environment. Articles relating to the mesolevel determinants discussed work, work conditions, psychosocial factors, socioeconomic position, outcomes, food, poverty, housing, and crime. Finally, sources found for the microlevel determinants examined gender, ethnicity, race, and behavior. Concepts were gathered from the literature and used to produce an ontology consisting of 383 classes, 109 object properties, and 748 logical axioms. A reasoning test revealed no inconsistent axioms.

CONCLUSIONS

This ontology models heterogeneous social and economic concepts to represent aspects of SDoH. The scope of SDoH is expansive, and although the ontology is broad, it is still in its early stages. To our current understanding, this ontology represents the first attempt to concentrate on knowledge concepts that are currently not covered by existing ontologies. Future direction will include further expanding the ontology to link with other biomedical ontologies, including alignment for granular semantics.

Organizational readiness and implementation fidelity of an early childhood education and care-specific physical activity policy intervention: findings from the Play Active trial

BACKGROUND

Many children do not accumulate sufficient physical activity for good health and development at early childhood education and care (ECEC). This study examined the association between ECEC organizational readiness and implementation fidelity of an ECEC-specific physical activity policy intervention.

METHODS

Play Active aimed to improve the ECEC educator's physical activity practices. We investigated the implementation of Play Active using a Type 1 hybrid study (January 2021-March 2022). Associations between organizational readiness factors and service-level implementation fidelity were examined using linear regressions. Fidelity data were collected from project records, educator surveys and website analytics.

RESULTS

ECEC services with higher levels of organizational commitment and capacity at pre-implementation reported higher fidelity scores compared to services with lower organizational commitment and capacity (all Ps < 0.05). Similarly, services who perceived intervention acceptability and appropriateness at pre-implementation to be high had higher fidelity scores (P < 0.05). Perceived feasibility and organizational efficacy of Play Active were associated with higher but nonsignificant fidelity scores.

CONCLUSIONS

Results indicate that organizational readiness factors may influence the implementation of ECEC-specific physical activity policy interventions. Therefore, strategies to improve organizational readiness should be developed and tested. These findings warrant confirmation in the ECEC and other settings and with other health behavior interventions.

Policy implementation and outcome evaluation: establishing a framework and expanding capacity for advocacy organizations to assess the impact of their work in public policy

Advocacy organizations can play a crucial role in evaluating whether legislation or regulation has had its intended effect by supporting robust public policy implementation and outcome evaluation. The American Heart Association, working with expert advisors, has developed a framework for effective evaluation that can be used by advocacy organizations, in partnership with researchers, public health agencies, funders, and policy makers to assess the health and equity impact of legislation and regulation over time. Advocacy organizations can use parts of this framework to evaluate the impact of policies relevant to their own advocacy and public policy efforts and inform policy development and guide their organizational resource allocation. Ultimately, working in partnership, advocacy organizations can help bring capacity, commitment and funding to this important implementation and outcome evaluation work that informs impactful public policy for equitable population health and well-being.

Towards Tobacco-Free Generation: implementation of preventive tobacco policies in the Nordic countries

AIMS

Europe's Beating Cancer Plan set a goal of creating a Tobacco-Free Generation in Europe by 2040. Prevention is important for achieving this goal. We compare the Nordic countries' preventive tobacco policies, discuss the possible determinants for similarities and differences in policy implementation, and provide strategies for strengthening tobacco prevention.

METHODS

We used the World Health Organization Framework Convention on Tobacco Control (WHO FCTC) to identify the key policies for this narrative review. We focused on Articles 6, 8, 9, 11, 13 and 16 of the WHO FCTC, and assessed the status of the required (core) and recommended (advanced) policies and their application to novel tobacco and nicotine products. Information on the implementation of strategies, acts and regulations were searched from global and national tobacco control databases, websites and scientific articles via PubMed and MEDLINE.

RESULTS

The WHO FCTC and European regulations have ensured that the core policies are mostly in place, but also contributed to the shared deficiencies that are seen especially in the regulations on smokeless tobacco and novel products. Strong national tobacco control actors have facilitated countries to implement some advanced policies - even as the first countries in the world: point-of-sale display bans (Iceland), outdoor smoking bans (Sweden), flavour bans on electronic cigarettes (Finland), plain packaging (Norway), and plain packaging on electronic cigarettes (Denmark).

CONCLUSIONS

Collaboration and participation in reinforcing the European regulations, resources for national networking between tobacco control actors, and national regulations to provide protection from the tobacco industry's interference are needed to strengthen comprehensive implementation of tobacco policies in the Nordic countries.

Implementation costs of sugary drink policies in the United States

To support implementation of important public health policies, policymakers need information about implementation costs over time and across stakeholder groups. We assessed implementation costs of two federal sugar-sweetened beverage (SSB) policies of current policy interest and with evidence to support their effects: excise taxes and health warning labels. Our analysis encompassed the entire policy life cycle using the Exploration, Preparation, Implementation, and Sustainment framework. We identified implementation actions using key informant interviews and developed quantitative estimates of implementation costs using published literature and government documents. Results show that implementation costs vary over time and among stakeholders. Explicitly integrating implementation science theory and using mixed methods improved the comprehensiveness of our results. Although this work is specific to federal SSB policies, the process can inform how we understand the costs of many public health policies, providing crucial information for public health policy making.

Advancing the science of policy implementation: a call to action for the implementation science field

Public policies have been essential in addressing many of the most pressing public health problems in the USA and around the world. A large and convincing body of multidisciplinary research has established the impacts or effectiveness of public policies, such as smoke-free air laws and nutrition standards, on improving health outcomes and behaviors. Most of this research assumes that because an evidence-based policy is adopted or takes effect, it is implemented as intended. This assumption, however, is often incorrect. Like with clinical guidelines and other interventions, implementation science has an important role to play in promoting the uptake and implementation of evidence-based public policies that promote public health. To realize this potential, there remains a critical need to first establish a common understanding of what public policy is, the role of specific policies in the context of implementation (i.e., is it the evidence-based intervention or the implementation strategy?), and to establish an appropriate methodological foundation for the field of policy implementation science. We recommend that the field must evolve to (i) include policy experts and actors on policy implementation science study teams; (ii) identify theories, models, and frameworks that are suitable for policy implementation science; (iii) identify policy implementation strategies; (iv) adapt and/or identify study designs best suited for policy implementation science research; and (v) identify appropriate policy implementation outcome measures.

Who's "in the room where it happens"? A taxonomy and five-step methodology for identifying and characterizing policy actors

BACKGROUND

Engaging policy actors in research design and execution is critical to increasing the practical relevance and real-world impact of policy-focused dissemination and implementation science. Identifying and selecting which policy actors to engage, particularly actors involved in "Big P" public policies such as laws, is distinct from traditional engaged research methods. This current study aimed to develop a transparent, structured method for iteratively identifying policy actors involved in key policy decisions-such as adopting evidence-based interventions at systems-scale-and to guide implementation study sampling and engagement approaches. A flexible policy actor taxonomy was developed to supplement existing methods and help identify policy developers, disseminators, implementers, enforcers, and influencers.

METHODS

A five-step methodology for identifying policy actors to potentially engage in policy dissemination and implementation research was developed. Leveraging a recent federal policy as a case study-The Family First Prevention Services Act (FFPSA)-publicly available documentation (e.g., websites, reports) were searched, retrieved, and coded using content analysis to characterize the organizations and individual policy actors in the "room" during policy decisions.

RESULTS

The five steps are as follows: (1) clarify the policy implementation phase(s) of interest, (2) identify relevant proverbial or actual policymaking "rooms," (3) identify and characterize organizations in the room, (4) identify and characterize policy actors in the "room," and (5) quantify (e.g., count actors across groups), summarize, and compare "rooms" to develop or select engagement approaches aligned with the "room" and actors. The use and outcomes of each step are exemplified through the FFPSA case study.

CONCLUSIONS

The pragmatic and transparent policy actor identification steps presented here can guide researchers' methods for continuous sampling and successful policy actor engagement. Future work should explore the utility of the proposed methods for guiding selection and tailoring of engagement and implementation strategies (e.g., research-policy actor partnerships) to improve both "Big P" and "little p" (administrative guidelines, procedures) policymaking and implementation in global contexts.

Four very basic ways to think about policy in implementation science

BACKGROUND

Policy is receiving increasing attention in the field of implementation science. However, there remains a lack of clear, concise guidance about how policy can be conceptualized in implementation science research. Building on Curran's article "Implementation science made too simple"-which defines "the thing" as the intervention, practice, or innovation in need of implementation support-we offer a typology of four very basic ways to conceptualize policy in implementation science research. We provide examples of studies that have conceptualized policy in these different ways and connect aspects of the typology to established frameworks in the field. The typology simplifies and refines related typologies in the field. Four very basic ways to think about policy in implementation science research. 1) Policy as something to adopt: an evidence-supported policy proposal is conceptualized as "the thing" and the goal of research is to understand how policymaking processes can be modified to increase adoption, and thus reach, of the evidence-supported policy. Policy-focused dissemination research is well-suited to achieve this goal. 2) Policy as something to implement: a policy, evidence-supported or not, is conceptualized as "the thing" and the goal of research is to generate knowledge about how policy rollout (or policy de-implementation) can be optimized to maximize benefits for population health and health equity. Policy-focused implementation research is well-suited to achieve this goal. 3) Policy as context to understand: an evidence-supported intervention is "the thing" and policies are conceptualized as a fixed determinant of implementation outcomes. The goal of research is to understand the mechanisms through which policies affect implementation of the evidence-supported intervention. 4) Policy as strategy to use: an evidence-supported intervention is "the thing" and policy is conceptualized as a strategy to affect implementation outcomes. The goal of research is to understand, and ideally test, how policy strategies affect implementation outcomes related to the evidence-supported intervention.

CONCLUSION

Policy can be conceptualized in multiple, non-mutually exclusive ways in implementation science. Clear conceptualizations of these distinctions are important to advancing the field of policy-focused implementation science and promoting the integration of policy into the field more broadly.

Applying hybrid effectiveness-implementation studies in equity-centered policy implementation science

Policy implementation science (IS) is complex, dynamic, and fraught with unique study challenges that set it apart from biomedical or clinical research. One important consideration is the ways in which policy interacts with local contexts, such as power and social disadvantage (e.g., based on ability, race, class, sexual identity, geography). The complex nature of policy IS and the need for more intentional integration of equity principles into study approaches calls for creative adaptations to existing implementation science knowledge and guidance. Effectiveness-implementation hybrid studies were developed to enhance translation of clinical research by addressing research questions around the effectiveness of an intervention and its implementation in the same study. The original work on hybrid designs mainly focused on clinical experimental trials; however, over the last decade, researchers have applied it to a wide range of initiatives and contexts, including more widespread application in community-based studies. This perspectives article demonstrates how effectiveness-implementation hybrid studies can be adapted for and applied to equity-centered policy IS research. We draw upon principles of targeted universalism and Equity in Implementation Research frameworks to guide adaptations to hybrid study typologies, and suggest research and engagement activities to enhance equity considerations; for example, in the design and testing of implementing strategies. We also provide examples of equity-centered policy IS studies. As the field of policy IS rapidly evolves, these adapted hybrid type studies are offered to researchers as a starting guide.

County- and state-level immigration policies are associated with Supplemental Nutrition Assistance Program (SNAP) participation among Latino households

The purpose of this study was to investigate the association between county- and state-level immigrant criminalizing and integrating policies and Latino household participation in the largest safety net program against food insecurity in the U.S., the Supplemental Nutrition Assistance Program (SNAP). Our outcome, county-level proportion of SNAP-participating Latino households, and county-level covariates were obtained from the American Community Survey 1-year county files (N = 675 counties) for 13 years (2007-2019). Our exposures were county-level presence of sanctuary policies and a state-level immigrant friendliness score, created based on 19 immigrant criminalizing and integrating state-level policies obtained from the Urban Institute's State Immigration Policies Resource. We classified every county in the sample as 1) sanctuary policy + immigrant friendly state, 2) sanctuary policy + immigrant unfriendly state, 3) no sanctuary policy + immigrant friendly state, and 4) no sanctuary policy + immigrant unfriendly state. Using multivariable generalized linear models that adjusted for poverty levels and other social composition characteristics of counties, we found that county-level SNAP participation among Latino households was 1.1 percentage-point higher in counties with sanctuary policies (B = 1.12, 95%CI = 0.26-1.98), compared to counties with no sanctuary policies, and 1.6 percentage-point higher in counties with sanctuary policies in immigrant friendly states (B = 1.59, 95%CI = 0.33-2.84), compared to counties with no sanctuary policy in immigrant unfriendly states. Local and state immigration policy, even when unrelated to SNAP eligibility, may influence SNAP participation among Latino households. Jurisdictions which lack sanctuary policies or have more criminalizing and less integrating policies should consider adopting targeted outreach strategies to increase SNAP enrollment among Latino households.

Ten years of implementation outcomes research: a scoping review

BACKGROUND

Proctor and colleagues' 2011 paper proposed a taxonomy of eight implementation outcomes and challenged the field to address a research agenda focused on conceptualization, measurement, and theory building. Ten years later, this paper maps the field's progress in implementation outcomes research. This scoping review describes how each implementation outcome has been studied, research designs and methods used, and the contexts and settings represented in the current literature. We also describe the role of implementation outcomes in relation to implementation strategies and other outcomes.

METHODS

Arksey and O'Malley's framework for conducting scoping reviews guided our methods. Using forward citation tracing, we identified all literature citing the 2011 paper. We conducted our search in the Web of Science (WOS) database and added citation alerts sent to the first author from the publisher for a 6-month period coinciding with the WOS citation search. This produced 1346 titles and abstracts. Initial abstract screening yielded 480 manuscripts, and full-text review yielded 400 manuscripts that met inclusion criteria (empirical assessment of at least one implementation outcome).

RESULTS

Slightly more than half (52.1%) of included manuscripts examined acceptability. Fidelity (39.3%), feasibility (38.6%), adoption (26.5%), and appropriateness (21.8%) were also commonly examined. Penetration (16.0%), sustainability (15.8%), and cost (7.8%) were less frequently examined. Thirty-two manuscripts examined implementation outcomes not included in the original taxonomy. Most studies took place in healthcare (45.8%) or behavioral health (22.5%) organizations. Two-thirds used observational designs. We found little evidence of progress in testing the relationships between implementation strategies and implementation outcomes, leaving us ill-prepared to know how to achieve implementation success. Moreover, few studies tested the impact of implementation outcomes on other important outcome types, such as service systems and improved individual or population health.

CONCLUSIONS

Our review presents a comprehensive snapshot of the research questions being addressed by existing implementation outcomes literature and reveals the need for rigorous, analytic research and tests of strategies for attaining implementation outcomes in the next 10 years of outcomes research.

Local implementation of public health policies revealed by the COVID-19 crisis: the French case

BACKGROUND

Improving health system performance depends on the quality of health policy implementation at the local level. However, in general, the attention of researchers is mainly directed towards issues of health policy design and evaluation rather than implementation at the local level. The management of the COVID-19 crisis, especially in Europe, has particularly highlighted the complexity of implementing health policies, decided at the national or supranational level, at the local level.

METHODS

We conducted 23 semi-structured interviews with the main stakeholders in the management of the COVID-19 crisis in the second largest French region in order to identify the different actors and modes of coordination of the local implementation of health policies that this crisis management illustrates in a very visible way. Our methodology is complemented by a content analysis of the main guidelines and decisions related to this implementation.

RESULTS

The analysis of these data allows us to identify three levels of implementation of health policies at the local level (administrative, organizational and operational). Interviews also reveal the existence of different types of coordination specific to each of these levels of local implementation of health policies. These results then make it possible to identify important managerial avenues for promoting global coordination of these three levels of implementation.

CONCLUSIONS

Although research on health services emphasizes the existence of several levels of local implementation of health policies, it offers little in the way of definition or characterization of these levels. The identification in this study of the three levels of local implementation of health policies and their specific forms of coordination contribute to a more precise characterization of this implementation in order to promote, in practice, its global coordination.

Earmarked Taxes for Mental Health Services in the United States: A Local and State Legal Mapping Study

Policy Points Local governments are increasingly adopting policies that earmark taxes for mental health services, and approximately 30% of the US population lives in a jurisdiction with such a policy. Policies earmarking taxes for mental health services are heterogenous in their design, spending requirements, and oversight. In many jurisdictions, the annual per capita revenue generated by these taxes exceeds that of some major federal funding sources for mental health.

CONTEXT

State and local governments have been adopting taxes that earmark (i.e., dedicate) revenue for mental health. However, this emergent financing model has not been systematically assessed. We sought to identify all jurisdictions in the United States with policies earmarking taxes for mental health services and characterize attributes of these taxes.

METHODS

A legal mapping study was conducted. Literature reviews and 11 key informant interviews informed search strings. We then searched legal databases (HeinOnline, Cheetah tax repository) and municipal data sources. We collected information on the year the tax went into effect, passage by ballot initiative (yes/no), tax base, tax rate, and revenue generated annually (gross and per capita).

FINDINGS

We identified 207 policies earmarking taxes for mental health services (95.7% local, 4.3% state, 95.7% passed via ballot initiative). Property taxes (73.9%) and sales taxes/fees (25.1%) were most common. There was substantial heterogeneity in tax design, spending requirements, and oversight. Approximately 30% of the US population lives in a jurisdiction with a tax earmarked for mental health, and these taxes generate over $3.57 billion annually. The median per capita annual revenue generated by these taxes was $18.59 (range = $0.04-$197.09). Per capita annual revenue exceeded $25.00 in 63 jurisdictions (about five times annual per capita spending for mental health provided by the US Substance Abuse and Mental Health Services Administration).

CONCLUSIONS

Policies earmarking taxes for mental health services are diverse in design and are an increasingly common local financing strategy. The revenue generated by these taxes is substantial in many jurisdictions.

Protocol for a scoping review of health equity frameworks and models applied in empirical studies of chronic disease prevention and control

BACKGROUND

Chronic diseases, such as cancers and cardiovascular diseases, present the greatest burden of morbidity and mortality worldwide. This burden disproportionately affects historically marginalized populations. Health equity is rapidly gaining increased attention in public health, health services, and implementation research, though many health inequities persist. Health equity frameworks and models (FM) have been called upon to guide equity-focused chronic disease and implementation research. However, there is no clear synthesis of the health equity FM used in chronic disease research or how these are applied in empirical studies. This scoping review seeks to fill this gap by identifying and characterizing health equity FM applied in empirical studies along the chronic disease prevention and control continuum, describing how these FM are used, and exploring potential applications to the field of implementation science.

METHODS

We follow established guidance for conducting scoping reviews, which includes six stages: (1) identify the research question; (2) identify relevant studies; (3) select studies for inclusion; (4) data extraction; (5) collating, summarizing, and reporting the results; and (6) consultation. This protocol presents the iterative, collaborative approach taken to conceptualize this study and develop the search strategy. We describe the criteria for inclusion in this review, methods for conducting two phases of screening (title and abstract, full text), data extraction procedures, and quality assurance approaches taken throughout the project.

DISCUSSION

The findings from this review will inform health-equity focused chronic disease prevention and control research. FM identified through this review will be added to an existing website summarizing dissemination and implementation science frameworks, and we will offer case examples and recommendations for utilizing a health equity FM in empirical studies. Our search strategy and review methodology may serve as an example for scholars seeking to conduct reviews of health equity FM in other health disciplines.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework Registration https://doi.org/10.17605/OSF.IO/SFVE6.

Maternal health policy environment and the relationship with service utilization in low- and middle-income countries

Background

The extent to which a favorable policy environment influences health care utilization and outcomes for pregnant and postpartum women is largely unknown. In this study, we aimed to describe the maternal health policy environment and examines its relationship with maternal health service utilization in low- and middle-income countries (LMICs).

Methods

We used data from World Health Organization's 2018-2019 sexual, reproductive, maternal, newborn, child, and adolescent health (SRMNCAH) policy survey linked with key contextual variables from global databases, as well as UNICEF data on antenatal care (ANC), institutional delivery, and postnatal care (PNC) utilization in 113 LIMCs. We grouped maternal health policy indicators into four categories - national supportive structures and standards, service access, clinical guidelines, and reporting and review systems. For each category and overall, we calculated summative scores accounting for available policy indicators in each country. We explored variations of policy indicators by World Bank income group using χ² tests and fitted logistic regression models for ≥85% coverage for each of four or more antenatal care visits (ANC4+), institutional delivery, PNC for the mothers, and for all ANC4+, institutional delivery, and PNC for mothers, adjusting for policy scores and contextual variables.

Results

The average scores for the four policy categories were as follows: 3 for national supportive structures and standards (score range = 0-4), 5.5 for service access (score range = 0-7), 6. for clinical guidelines (score range = 0-10), and 5.7 for reporting and review systems (score range = 0-7), for an average total policy score of 21.1 (score range = 0-28) across LMICs. After adjusting for country context variables, for each unit increase in the maternal health policy score, the odds of ANC4+>85% increased by 37% (95% confidence interval (CI) = 1.13-1.64) and the odds of all ANC4+, institutional deliveries and PNC>85% by 31% (95% CI = 1.07-1.60).

Conclusions

Despite the availability of supportive structures and free maternity service access policies, there is a dire need for stronger policy support for clinical guidelines and practice regulations, as well as national reporting and review systems for maternal health. A more favorable policy environment for maternal health can improve adoption of evidence-based interventions and increase utilization of maternal health services in LMICs.

The Interaction Between General and Strategic Leadership and Climate on Their Multilevel Associations with Implementer Attitudes Toward Universal Prevention Programs for Youth Mental Health: A Cross-Sectional Study

Emerging literature has highlighted the importance of discerning general and strategic organizational context (OC) factors (e.g., leadership and climate) and their interaction effect on individual implementation behaviors (e.g., attitudes toward evidence-based practices; EBPs) in youth mental healthcare. This study aimed to examine how leadership and climate (general and strategic) are associated with implementer attitudes toward EBPs across the individual and organizational levels and their interaction effect in schools. A series of multilevel models (MLMs) were fitted on a diverse sample of schools actively implementing universal prevention programs for youth mental health (441 implementers from 52 schools). The organization-level aggregates and individual educators' perceptions of general and strategic leadership and climate, and their interaction terms, were entered as level-2 and level-1 predictors of four attitudinal dimensions (Requirement, Openness, Appeal, and Divergence) based on their level of measurement. At the organizational level, higher levels of strategic leadership and climate, but not their general counterparts, were consistently associated with more favorable attitudes in all four dimensions. At the individual/within-school level, higher levels of perceived general and strategic leadership and climate were associated with more favorable attitudes of Requirement and Openness. At the organizational/between-school level, general climate moderated the positive effect of strategic climate on implementers' perception of appeal and divergence of EBPs. Our findings indicate that leaders should make data-based decisions to allocate resources on strategic and/or general leadership and climate to foster favorable staff attitudes toward EBPs based on the level of measurement, implementation-specificity, and attitudinal dimensions.

A policy implementation study of earmarked taxes for mental health services: study protocol

BACKGROUND

Insufficient funding is frequently identified as a critical barrier to the implementation and sustainment of evidence-based practices (EBPs). Thus, increasing access to funding is recognized as an implementation strategy. Policies that create earmarked taxes-defined as taxes for which revenue can only be spent on specific activities-are an increasingly common mental health financing strategy that could improve the reach of EBPs. This project's specific aims are to (1) identify all jurisdictions in the USA that have implemented earmarked taxes for mental health and catalogue information about tax design; (2) characterize experiences implementing earmarked taxes among local (e.g., county, city) mental health agency leaders and other government and community organization officials and assess their perceptions of the acceptability and feasibility of different types of policy implementation strategies; and (3) develop a framework to guide effect earmarked tax designs, inform the selection of implementation strategies, and disseminate the framework to policy audiences.

METHODS

The project uses the Exploration, Preparation, Implementation, Sustainment (EPIS) framework to inform data collection about the determinants and processes of tax implementation and Leeman's typology of implementation strategies to examine the acceptability and feasibility strategies which could support earmarked tax policy implementation. A legal mapping will be conducted to achieve aim 1. To achieve aim 2, a survey will be conducted of 300 local mental health agency leaders and other government and community organization officials involved with the implementation of earmarked taxes for mental health. The survey will be followed by approximately 50 interviews with these officials. To achieve aim 3, quantitative and qualitative data will be integrated through a systematic framework development and dissemination process.

DISCUSSION

This exploratory policy implementation process study will build the evidence base for outer-context implementation determinants and strategies by focusing on policies that earmarked taxes for mental health services.

Tailoring dissemination strategies to increase evidence-informed policymaking for opioid use disorder treatment: study protocol

BACKGROUND

Policy is a powerful tool for systematically altering healthcare access and quality, but the research to policy gap impedes translating evidence-based practices into public policy and limits widespread improvements in service and population health outcomes. The US opioid epidemic disproportionately impacts Medicaid members who rely on publicly funded benefits to access evidence-based treatment including medications for opioid use disorder (MOUD). A myriad of misaligned policies and evidence-use behaviors by policymakers across federal agencies, state Medicaid agencies, and managed care organizations limit coverage of and access to MOUD for Medicaid members. Dissemination strategies that improve policymakers' use of current evidence are critical to improving MOUD benefits and reducing health disparities. However, no research describes key determinants of Medicaid policymakers' evidence use behaviors or preferences, and few studies have examined data-driven approaches to developing dissemination strategies to enhance evidence-informed policymaking. This study aims to identify determinants and intermediaries that influence policymakers' evidence use behaviors, then develop and test data-driven tailored dissemination strategies that promote MOUD coverage in benefit arrays.

METHODS

Guided by the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework, we will conduct a national survey of state Medicaid agency and managed care organization policymakers to identify determinants and intermediaries that influence how they seek, receive, and use research in their decision-making processes. We will use latent class methods to empirically identify subgroups of agencies with distinct evidence use behaviors. A 10-step dissemination strategy development and specification process will be used to tailor strategies to significant predictors identified for each latent class. Tailored dissemination strategies will be deployed to each class of policymakers and assessed for their acceptability, appropriateness, and feasibility for delivering evidence about MOUD benefit design.

DISCUSSION

This study will illuminate key determinants and intermediaries that influence policymakers' evidence use behaviors when designing benefits for MOUD. This study will produce a critically needed set of data-driven, tailored policy dissemination strategies. Study results will inform a subsequent multi-site trial measuring the effectiveness of tailored dissemination strategies on MOUD benefit design and implementation. Lessons from dissemination strategy development will inform future research about policymakers' evidence use preferences and offer a replicable process for tailoring dissemination strategies.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

BACKGROUND

Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings.

METHODS

Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed.

RESULTS

A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants.

CONCLUSIONS

This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field.

TRIAL REGISTRATION

This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Where is "policy" in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example

BACKGROUND

Implementation science aims to accelerate the public health impact of evidence-based interventions. However, implementation science has had too little focus on the role of health policy - and its inseparable politics, polity structures, and policymakers - in the implementation and sustainment of evidence-based healthcare. Policies can serve as determinants, implementation strategies, the evidence-based "thing" to be implemented, or another variable in the causal pathway to healthcare access, quality, and patient outcomes. Research describing the roles of policy in dissemination and implementation (D&I) efforts is needed to resolve persistent knowledge gaps about policymakers' evidence use, how evidence-based policies are implemented and sustained, and methods to de-implement policies that are ineffective or cause harm. Few D&I theories, models, or frameworks (TMF) explicitly guide researchers in conceptualizing where, how, and when policy should be empirically investigated. We conducted and reflected on the results of a scoping review to identify gaps of existing Exploration, Preparation, Implementation, and Sustainment (EPIS) framework-guided policy D&I studies. We argue that rather than creating new TMF, researchers should optimize existing TMF to examine policy's role in D&I. We describe six recommendations to help researchers optimize existing D&I TMF. Recommendations are applied to EPIS, as one example for advancing TMF for policy D&I.

RECOMMENDATIONS

(1) Specify dimensions of a policy's function (policy goals, type, contexts, capital exchanged). (2) Specify dimensions of a policy's form (origin, structure, dynamism, outcomes). (3) Identify and define the nonlinear phases of policy D&I across outer and inner contexts. (4) Describe the temporal roles that stakeholders play in policy D&I over time. (5) Consider policy-relevant outer and inner context adaptations. (6) Identify and describe bridging factors necessary for policy D&I success.

CONCLUSION

Researchers should use TMF to meaningfully conceptualize policy's role in D&I efforts to accelerate the public health impact of evidence-based policies or practices and de-implement ineffective and harmful policies. Applying these six recommendations to existing D&I TMF advances existing theoretical knowledge, especially EPIS application, rather than introducing new models. Using these recommendations will sensitize researchers to help them investigate the multifaceted roles policy can play within a causal pathway leading to D&I success.

Considering multiple governance levels in epidemiologic analysis of public policies

Epidemiology is increasingly asking questions about the use of policies to address structural inequities and intervene on health disparities and public health challenges. However, there has been limited explicit consideration of governance structures in the design of epidemiologic policy analysis. To advance empirical and theoretical inquiry in this space, we propose a model of governance analysis in which public health researchers consider at what level 1) decision-making authority for policy sits, 2) policy is implemented, 3) and accountability for policy effects appear. We follow with examples of how these considerations might improve the evaluation of the policy drivers of population health. Consideration and integration of multiple levels of governance, as well as interactions between levels, can help epidemiologists design studies including new opportunities for quasi-experimental designs and stronger counterfactuals, better quantify the policy drivers of inequities, and aid research evidence and policy development work in targeting multiple levels of governance, ultimately supporting evidence-based policy making.

Investigating implementation of school health policies through a health equity lens: A measures development study protocol

Background

School-based policies that ensure provision of nutrition, physical activity, and other health-promoting resources and opportunities are essential in mitigating health disparities among underserved populations. Measuring the implementation of such policies is imperative to bridge the gap between policy and practice. Unfortunately, limited practical, psychometrically strong measures of school policy implementation exist. Few available explicitly focus on the issues of equity and social justice as a key component of implementation, which may result in underassessment of the equity implications of policy implementation. The purpose of this study is to develop equity-focused measures in collaboration with practitioners, researchers, and other key implementation partners that will facilitate evaluation of policy implementation determinants (i.e., barriers and facilitators), processes, and outcomes.

Methods

We will actively seek engagement from practitioners, researchers, and advocacy partners (i.e., stakeholders) who have expertise in school health policy throughout each phase of this project. We propose a multi-phase, 1-year project comprising the following steps: (1) selection of relevant constructs from guiding frameworks related to health equity and implementation science; (2) initial measure development, including expert feedback on draft items; (3) pilot cognitive testing with representatives from key target populations (i.e., school administrators, teachers, food service staff, and students and parents/guardians); and (4) measure refinement based on testing and assessment of pragmatic properties. These steps will allow us to establish initial face and content validity of a set of instruments that can undergo psychometric testing in future studies to assess their reliability and validity.

Discussion

Completion of this project will result in several school policy implementation measurement tools which can be readily used by practitioners and researchers to evaluate policy implementation through a health equity lens. This will provide opportunities for better assessment and accountability of policies that aim to advance health equity among school-aged children and their families.

Trial registration

Open Science Framework Registration doi: 10.17605/OSF.IO/736ZU.

Economic evaluation of implementation science outcomes in low- and middle-income countries: a scoping review

BACKGROUND

Historically, the focus of cost-effectiveness analyses has been on the costs to operate and deliver interventions after their initial design and launch. The costs related to design and implementation of interventions have often been omitted. Ignoring these costs leads to an underestimation of the true price of interventions and biases economic analyses toward favoring new interventions. This is especially true in low- and middle-income countries (LMICs), where implementation may require substantial up-front investment. This scoping review was conducted to explore the topics, depth, and availability of scientific literature on integrating implementation science into economic evaluations of health interventions in LMICs.

METHODS

We searched Web of Science and PubMed for papers published between January 1, 2010, and December 31, 2021, that included components of both implementation science and economic evaluation. Studies from LMICs were prioritized for review, but papers from high-income countries were included if their methodology/findings were relevant to LMIC settings.

RESULTS

Six thousand nine hundred eighty-six studies were screened, of which 55 were included in full-text review and 23 selected for inclusion and data extraction. Most papers were theoretical, though some focused on a single disease or disease subset, including: mental health (n = 5), HIV (n = 3), tuberculosis (n = 3), and diabetes (n = 2). Manuscripts included a mix of methodology papers, empirical studies, and other (e.g., narrative) reviews. Authorship of the included literature was skewed toward high-income settings, with 22 of the 23 papers featuring first and senior authors from high-income countries. Of nine empirical studies included, no consistent implementation cost outcomes were measured, and only four could be mapped to an existing costing or implementation framework. There was also substantial heterogeneity across studies in how implementation costs were defined, and the methods used to collect them.

CONCLUSION

A sparse but growing literature explores the intersection of implementation science and economic evaluation. Key needs include more research in LMICs, greater consensus on the definition of implementation costs, standardized methods to collect such costs, and identifying outcomes of greatest relevance. Addressing these gaps will result in stronger links between implementation science and economic evaluation and will create more robust and accurate estimates of intervention costs.

TRIAL REGISTRATION

The protocol for this manuscript was published on the Open Science Framework. It is available at: https://osf.io/ms5fa/ (DOI: 10.17605/OSF.IO/32EPJ).

Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research

The pragmatic (i.e., practical) quality of quantitative implementation measures has received increased attention in the implementation science literature in recent years. Implementation measures that are judged to be pragmatic by implementation stakeholders are thought to be more likely to be applied in research and practice. Despite the need for pragmatic implementation measures, ambiguity and uncertainty regarding what constitutes a pragmatic measure remains. This study sought to identify and critically appraise the published literature to understand (i) how pragmatism is defined as a measurement construct/quality of implementation determinants and outcome instruments; (ii) how pragmatic qualities of instruments are evaluated; (iii) identify key gaps and limitations of the current evidence-base and (iv) identify recommendations for future research. We conducted a scoping review of the literature also employing methods of critical review. PubMed and PsycINFO databases, using the OVID interface, were searched for relevant articles published between January 2010 and September 2020. Articles that contained a definition and/or described characteristics of "pragmatism" as a measurement construct of quantitative implementation outcomes (as defined by Proctor's Implementation Outcomes taxonomy) and/or implementation determinants were eligible for inclusion. Nine articles met inclusion criteria. A degree of overlap in definitions and terms used to describe the pragmatic qualities of quantitative implementation determinant and outcome instruments were found. The most frequently cited descriptors of pragmatism were "not burdensome", "brief", "reliable", "valid" and "sensitive to change". 3 of the 9 included articles involved international implementation stakeholders in defining and conceptualizing pragmatism and employed specific methods to do so, including a systematic literature review, stakeholder interviews, concept mapping, and a Delphi process. All other articles defined pragmatism, with or without citing relevant literature. One article objectively assessed the pragmatic qualities, above and beyond the psychometric qualities, of implementation measures, using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The evidence base within the implementation instrumentation literature on what pragmatism is and how it might be assessed is limited. Some of the research identified in the review provides a strong foundation to build upon, by testing its applicability in other settings (including healthcare areas and countries) and among a more diverse group of stakeholders. We discuss directions for further development of the concept of pragmatism relating to the measurement of implementation determinants and outcomes.

Quantitative Measurements for Factors Influencing Implementation in School Settings: Protocol for A Systematic Review and A Psychometric and Pragmatic Analysis

INTRODUCTION

In order to address the effectiveness and sustainability of school-based interventions, there is a need to consider the factors affecting implementation success. The rapidly growing field of implementation-focused research is struggling to determine how to assess and measure implementation-relevant constructs. Earlier research has identified the need for strong psychometric and pragmatic measures. The aims of this review are therefore to (i) systematically review the literature to identify measurements of the factors influencing implementations which have been developed or adapted in school settings, (ii) describe each measurement's psychometric and pragmatic properties, (iii) describe the alignment between each measurement and the corresponding domain and/or construct of the Consolidated Framework for Implementation Research (CFIR).

METHODS

Six databases (Medline, ERIC, PsycInfo, Cinahl, Embase, and Web of Science) will be searched for peer-reviewed articles reporting on school settings, published from the year 2000. The identified measurements will be mapped against the CFIR, and analyzed for their psychometric and pragmatic properties.

DISCUSSION

By identifying measurements that are psychometrically and pragmatically impactful in the field, this review will contribute to the identification of feasible, effective, and sustainable implementation strategies for future research in school settings.

Research and Scholarly Methods: Implementation Science Studies

Traditional research focuses on efficacy or effectiveness of interventions but lacks evaluation of strategies needed for equitable uptake, scalable implementation, and sustainable evidence-based practice transformation. The purpose of this introductory review is to describe key implementation science (IS) concepts as they apply to medication management and pharmacy practice, and to provide guidance on literature review with an IS lens. There are five key ingredients of IS, including: (1) evidence-based intervention; (2) implementation strategies; (3) IS theory, model, or framework; (4) IS outcomes and measures; and (5) stakeholder engagement, which is key to a successful implementation. These key ingredients apply across the three stages of IS research: (1) pre-implementation; (2) implementation; and (3) sustainment. A case example using a combination of IS models, PRISM (Practical, Robust Implementation and Sustainability model) and RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance), is included to describe how an IS study is designed and conducted. This case is a cluster randomized trial comparing two clinical decision support tools to improve guideline-concordant prescribing for patients with heart failure and reduced ejection fraction. The review also includes information on the Standards for Reporting Implementation Studies (StaRI), which is used for literature review and reporting of IS studies,as well as IS-related learning resources.

Social determinants of health and cancer screening implementation and outcomes in the USA: a systematic review protocol

BACKGROUND

Improving the delivery, uptake, and implementation of cancer screening to meet evidence-based recommendations is needed to reduce persistent cancer health disparities in the USA. Current national public health targets emphasize the role of social determinants of health (SDOH) on cancer screening. However, there remains a need to explicate these linkages, toward the goal of identifying and implementing effective interventions that target and address SDOH to reduce inequities in cancer screening.

METHODS

We will conduct a systematic review of English language peer-reviewed original research articles published between 2010 and 2021 that describe observational (qualitative and quantitative) and intervention studies conducted in the USA. In alignment with Healthy People 2030, we will include studies of breast, cervical, colorectal, and/or lung cancer screening. Guided by multiple SDOH frameworks, we will broadly define SDOH by five domain areas: economic stability, education access and quality, healthcare access and quality, neighborhood and built environment, and social and community context. Following systematic literature searches in five databases (Ovid MEDLINE, Embase, CINAHL, Web of Science, Cochrane Library) and piloting of screening procedures, reviewers will independently screen titles/abstracts for potential relevance. Reviewer pairs will then screen full text articles for eligibility criteria. We will extract data items from included articles, including study characteristics, cancer screening intervention information, and coding of SDOH constructs. We will assess study quality using the Mixed Methods Appraisal Tool and synthesize our findings using narrative, descriptive statistics, tables, and figures. Our approach will adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) recommendations.

DISCUSSION

By completing this systematic review, we will summarize recent literature on SDOH and cancer screening, identify research gaps for inclusion of SDOH, and propose future opportunities for advancing equity in cancer screening by integrating SDOH as part of the implementation context to promote uptake, sustainability, and scale-up in the implementation of screening guidelines.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021276582 .

Revisiting concepts of evidence in implementation science

Background

Evidence, in multiple forms, is a foundation of implementation science. For public health and clinical practice, evidence includes the following: type 1 evidence on etiology and burden; type 2 evidence on effectiveness of interventions; and type 3: evidence on dissemination and implementation (D&I) within context. To support a vision for development and use of evidence in D&I science that is more comprehensive and equitable (particularly for type 3 evidence), this article aims to clarify concepts of evidence, summarize ongoing debates about evidence, and provide a set of recommendations and tools/resources for addressing the “how-to” in filling evidence gaps most critical to advancing implementation science.

Main text

Because current conceptualizations of evidence have been relatively narrow and insufficiently characterized in our opinion, we identify and discuss challenges and debates about the uses, usefulness, and gaps in evidence for implementation science. A set of questions is proposed to assist in determining when evidence is sufficient for dissemination and implementation. Intersecting gaps include the need to (1) reconsider how the evidence base is determined, (2) improve understanding of contextual effects on implementation, (3) sharpen the focus on health equity in how we approach and build the evidence-base, (4) conduct more policy implementation research and evaluation, and (5) learn from audience and stakeholder perspectives. We offer 15 recommendations to assist in filling these gaps and describe a set of tools for enhancing the evidence most needed in implementation science.

Conclusions

To address our recommendations, we see capacity as a necessary ingredient to shift the field’s approach to evidence. Capacity includes the “push” for implementation science where researchers are trained to develop and evaluate evidence which should be useful and feasible for implementers and reflect community or stakeholder priorities. Equally important, there has been inadequate training and too little emphasis on the “pull” for implementation science (e.g., training implementers, practice-based research). We suggest that funders and reviewers of research should adopt and support a more robust definition of evidence. By critically examining the evolving nature of evidence, implementation science can better fulfill its vision of facilitating widespread and equitable adoption, delivery, and sustainment of scientific advances.

Moving toward equity: fostering transdisciplinary research between the social and behavioral sciences and implementation science to end the HIV epidemic

PURPOSE OF REVIEW

Since the beginning of the HIV epidemic, social and behavioral scientists have developed interventions to stem the spread of the virus. The dissemination of these interventions has traditionally been a lengthy process; however, implementation science (IS) offers a route toward hastening delivery of effective interventions. A transdisciplinary approach, wherein IS informs and is informed by social and behavioral sciences (SBS) as well as community participation, offers a strategy for more efficiently moving toward health equity and ending the HIV epidemic.

RECENT FINDINGS

There has been considerable growth in HIV research utilizing IS theories, methods and frameworks. Many of these studies have been multi or interdisciplinary in nature, demonstrating the ways that IS and SBS can strengthen one another. We also find areas for continued progress toward transdisciplinarity.

SUMMARY

We review literature from 2020 to 2021, exploring the ways IS and SBS have been used in tandem to develop, evaluate and disseminate HIV interventions. We highlight the interplay between disciplines and make a case for moving toward transdisciplinarity, which would yield new, integrated frameworks that can improve prevention and treatment efforts, moving us closer to achieving health equity.

Translating Medicaid policy into practice: policy implementation strategies from three US states' experiences enhancing substance use disorder treatment

BACKGROUND

Despite the important upstream impact policy has on population health outcomes, few studies in implementation science in health have examined implementation processes and strategies used to translate state and federal policies into accessible services in the community. This study examines the policy implementation strategies and experiences of Medicaid programs in three US states that responded to a federal prompt to improve access to evidence-based practice (EBP) substance use disorder (SUD) treatment.

METHODS

Three US state Medicaid programs implementing American Society of Addiction Medicine (ASAM) Criteria-driven SUD services under Section 1115 waiver authority were used as cases. We conducted 44 semi-structured interviews with Medicaid staff, providers and health systems partners in California, Virginia, and West Virginia. Interviews were triangulated with document review of state readiness and implementation plans. The Exploration, Preparation, Implementation, Sustainment Framework (EPIS) guided qualitative theme analysis. The Expert Recommendations for Implementing Change and Specify It criteria were used to create a taxonomy of policy implementation strategies used by policymakers to promote providers' uptake of statewide EBP SUD care continuums.

RESULTS

Four themes describe states' experiences and outcomes implementing a complex EBP SUD treatment policy directive: (1) Medicaid agencies adapted their inner/outer contexts to align with EBPs and adapted EBPs to fit their local context; (2) enhanced financial reimbursement arrangements were inadequate bridging factors to achieve statewide adoption of new SUD services; (3) despite trainings, service providers and managed care organizations demonstrated poor fidelity to the ASAM Criteria; and (4) successful policy adoption at the state level did not guarantee service providers' uptake of EBPs. States used 29 implementation strategies to implement EBP SUD care continuums. Implementation strategies were used in the Exploration (n=6), Preparation (n=10), Implementation (n=19), and Sustainment (n=6) phases, and primarily focused on developing stakeholder interrelationships, evaluative and iterative approaches, and financing.

CONCLUSIONS

This study enhances our understanding of statewide policy implementation outcomes in low-resource, public healthcare settings. Themes highlight the need for additional pre-implementation and sustainment focused implementation strategies. The taxonomy of detailed policy implementation strategies employed by policymakers across states should be tested in future policy implementation research.

Applying the EPIS framework to policy-level considerations: Tobacco cessation policy implementation among California Medicaid managed care plans

Background

In 2016, the California Department of Healthcare Services (DHCS) released an "All Plan Letter" (APL 16-014) to its Medicaid managed care plans (MCPs) providing guidance on implementing tobacco-cessation coverage among Medicaid beneficiaries. However, implementation remains poor. We apply the Exploration, Preparation, Implementation, Sustainment (EPIS) framework to identify barriers and facilitators to fidelity to APL 16-014 across California Medicaid MCPs.

Methods

We assessed fidelity through semi-structured interviews with MCP health educators (N = 24). Interviews were recorded, transcribed, and reviewed to develop initial themes regarding barriers and facilitators to implementation. Initial thematic summaries were discussed and mapped onto EPIS constructs.

Results

The APL (Innovation) was described as lacking clarity and specificity in its guidelines, hindering implementation. Related to the Inner Context, MCPs described the APL as beyond the scope of their resources, pointing to their own lack of educational materials, human resources, and poor technological infrastructure as implementation barriers. In the Outer Context, MCPs identified a lack of incentives for providers and beneficiaries to offer and participate in tobacco-cessation programs, respectively. A lack of communication, educational materials, and training resources between the state and MCPs (missing Bridging Factors) were barriers to preventing MCPs from identifying smoking rates or gauging success of tobacco-cessation efforts. Facilitators included several MCPs collaborating with each other and using external resources to promote tobacco cessation. Additionally, a few MCPs used fidelity monitoring staff as Bridging Factors to facilitate provider training, track providers' identification of smokers, and follow-up with beneficiaries participating in tobacco-cessation programs.

Conclusions

The release of the evidence-based APL 16-014 by California's DHCS was an important step forward in promoting tobacco-cessation services for Medicaid MCP beneficiaries. Improved communication on implementation in different environments and improved Bridging Factors such as incentives for providers and patients are needed to fully realize policy goals.

Plan Language Summary

In 2016, the California Department of Healthcare Services (DHCS) in California released an "All Plan Letter" (APL 16-014) to its Medicaid managed care plans (MCPs) providing guidance on implementing tobacco-cessation coverage to address tobacco use among Medicaid beneficiaries. We conducted semi-structured interviews with health educators in California Medicaid MCPs to explore the barriers and facilitators to implementing the APL using the Exploration, Preparation, Implementation, Sustainment framework. According to MCPs, barriers included a lack of clarity in the APL guidelines; a lack of resources, including educational materials, infrastructure to identify smokers, and human resources; and a lack of incentives or penalties for providers to provide tobacco-cessation materials to beneficiaries. Facilitators included collaboration between MCPs and state and/or national public health programs. Overall, our findings can provide avenues for improving the implementation of tobacco-cessation services within Medicaid MCPs.

Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Background

Mental health is a critical component of wellness. Public policies present an opportunity for large-scale mental health impact, but policy implementation is complex and can vary significantly across contexts, making it crucial to evaluate implementation. The objective of this study was to (1) identify quantitative measurement tools used to evaluate the implementation of public mental health policies; (2) describe implementation determinants and outcomes assessed in the measures; and (3) assess the pragmatic and psychometric quality of identified measures.

Method

Guided by the Consolidated Framework for Implementation Research, Policy Implementation Determinants Framework, and Implementation Outcomes Framework, we conducted a systematic review of peer-reviewed journal articles published in 1995-2020. Data extracted included study characteristics, measure development and testing, implementation determinants and outcomes, and measure quality using the Psychometric and Pragmatic Evidence Rating Scale.

Results

We identified 34 tools from 25 articles, which were designed for mental health policies or used to evaluate constructs that impact implementation. Many measures lacked information regarding measurement development and testing. The most assessed implementation determinants were readiness for implementation, which encompassed training (n  =  20, 57%) and other resources (n  =  12, 34%), actor relationships/networks (n  =  15, 43%), and organizational culture and climate (n  =  11, 31%). Fidelity was the most prevalent implementation outcome (n  =  9, 26%), followed by penetration (n  =  8, 23%) and acceptability (n  =  7, 20%). Apart from internal consistency and sample norms, psychometric properties were frequently unreported. Most measures were accessible and brief, though minimal information was provided regarding interpreting scores, handling missing data, or training needed to administer tools.

Conclusions

This work contributes to the nascent field of policy-focused implementation science by providing an overview of existing measurement tools used to evaluate mental health policy implementation and recommendations for measure development and refinement. To advance this field, more valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap.

Plain Language Summary

Mental health is a critical component of wellness, and public policies present an opportunity to improve mental health on a large scale. Policy implementation is complex because it involves action by multiple entities at several levels of society. Policy implementation is also challenging because it can be impacted by many factors, such as political will, stakeholder relationships, and resources available for implementation. Because of these factors, implementation can vary between locations, such as states or countries. It is crucial to evaluate policy implementation, thus we conducted a systematic review to identify and evaluate the quality of measurement tools used in mental health policy implementation studies. Our search and screening procedures resulted in 34 measurement tools. We rated their quality to determine if these tools were practical to use and would yield consistent (i.e., reliable) and accurate (i.e., valid) data. These tools most frequently assessed whether implementing organizations complied with policy mandates and whether organizations had the training and other resources required to implement a policy. Though many were relatively brief and available at little-to-no cost, these findings highlight that more reliable, valid, and practical measurement tools are needed to assess and inform mental health policy implementation. Findings from this review can guide future efforts to select or develop policy implementation measures.

Embracing policy implementation science to ensure translation of evidence to cancer control policy

Although health-related policies are abundant, efforts to understand how to ensure that these policies serve as an effective vehicle for translating scientific evidence are relatively sparse. This paper explores how policy-focused implementation science (IS) may contribute to understanding the translation of scientific evidence to health-related policy in governmental and nongovernmental sectors. Expanding the focus of implementation science in cancer control could systematically address policy to both increase the use of scientific evidence in general and to address health equity. In this Commentary, we look to relevant work outside of IS that could be informative, most notably from the field of political science. We propose several ideas for future research that could help move the field of policy implementation science in cancer control in the USA forward. Although most efforts to increase uptake of the scientific evidence base reference translation to "practice and policy," there has been relatively little emphasis in the USA on implementation at the policy level, especially related to cancer control. If we are to achieve the full benefits of scientific discovery on population and public health, we will need to consider policy as a critical mechanism by which evidence can be translated to practice. We have a robust set of methods within implementation science that are increasing the pace of adoption and maintenance of evidence-based programs in a variety of settings. Building on these efforts, the time is right to expand our focus to include policy implementation.

A scoping review of de-implementation frameworks and models

BACKGROUND

Reduction or elimination of inappropriate, ineffective, or potentially harmful healthcare services and public health programs can help to ensure limited resources are used effectively. Frameworks and models (FM) are valuable tools in conceptualizing and guiding the study of de-implementation. This scoping review sought to identify and characterize FM that can be used to study de-implementation as a phenomenon and identify gaps in the literature to inform future model development and application for research.

METHODS

We searched nine databases and eleven journals from a broad array of disciplines (e.g., healthcare, public health, public policy) for de-implementation studies published between 1990 and June 2020. Two raters independently screened titles and abstracts, and then a pair of raters screened all full text records. We extracted information related to setting, discipline, study design, methodology, and FM characteristics from included studies.

RESULTS

The final search yielded 1860 records, from which we screened 126 full text records. We extracted data from 27 articles containing 27 unique FM. Most FM (n = 21) were applicable to two or more levels of the Socio-Ecological Framework, and most commonly assessed constructs were at the organization level (n = 18). Most FM (n = 18) depicted a linear relationship between constructs, few depicted a more complex structure, such as a nested or cyclical relationship. Thirteen studies applied FM in empirical investigations of de-implementation, while 14 articles were commentary or review papers that included FM.

CONCLUSION

De-implementation is a process studied in a broad array of disciplines, yet implementation science has thus far been limited in the integration of learnings from other fields. This review offers an overview of visual representations of FM that implementation researchers and practitioners can use to inform their work. Additional work is needed to test and refine existing FM and to determine the extent to which FM developed in one setting or for a particular topic can be applied to other contexts. Given the extensive availability of FM in implementation science, we suggest researchers build from existing FM rather than recreating novel FM.

REGISTRATION

Not registered.

Perceived determinants of implementation success of the neglected tropical diseases programme in Ghana: a qualitative study among programme officers

Background

The importance of health policy implementation cannot be overemphasized in contemporary public health. Neglected Tropical Diseases (NTDs) have negatively impacted society, affect quality of life and make the poor societies poorer. Several policies and strategies have been put in place across the world including the neglected tropical diseases programme in Ghana. Though chalked many successes, the programme continues to lag behind in the full attainment of various objectives. Several factors exist that determine how effective a programme is implemented. Identification of these factors on every programme is essential to determine where more programme resources need to be channelled. This study assessed the determinants of successful implementation of the neglected tropical diseases programme in Ghana.

Methods

A qualitative approach with the case study design was employed. Purposive and snowball sampling techniques were used to identify key programme officers at the national, regional and district levels of programme implementation. Eighteen (18) Key informant interviews were conducted at all the three levels of the Ghana Health Service NTDs programme. Data were thematically analysed and presented.

Results

Findings from the study revealed that determinants that influenced the successful implementation of the NTDs programme include donor support, education and training, partnerships, reliability of the health structure, integrative nature of the programme and management commitment. These determining factors cut across the inner settings of the implementing agency and the external environment.

Conclusion

Neglected tropical diseases continuously affect Ghanaians, especially the poor. It is important for both policy makers and implementers to identify the factors that ensure the success of the programme in the Ghanaian context. Though the factors are independently sufficient, they synergistically lead to improved programme implementation. Empowering all units involved (local to national level) and maximizing the enabling factors identified to would improve upon implementation and ensure sustainability.

Implementation of two policies to extend maternity leave and further restrict marketing of breast milk substitutes in Vietnam: a qualitative study

Policy research can reveal gaps and opportunities to enhance policy impact and implementation. In this study, we use a theoretically informed qualitative approach to investigate the implementation of two policies to promote breastfeeding in Vietnam. We conducted semi-structured interviews with national and local policy stakeholders (n = 26) in 2017. Interviews were audio-recorded, transcribed verbatim and then translated to English by certified translators. Transcript data were analysed using an integrated conceptual framework of policy implementation. Respondents identified several positive outcomes resulting from implementation of an extended maternity leave policy (Labour Code No. 10/2012/QH13) and further restrictions on marketing of breast milk substitutes (Decree No. 100/2014/ND-CP). Decree No. 100, in particular, was said to have reduced advertising of breast milk substitutes in mass media outlets and healthcare settings. Key implementation actors were national-level bureaucratic actors, local organizations and international partners. Findings reveal the importance of policy precedence and a broader set of policies to promote the rights of women and children to support implementation. Other facilitators were involvement from national-level implementing agencies and healthcare personnel and strength of government relationships and coordination with non-governmental and international organizations. Implementation challenges included insufficient funding, limited training to report violations, a cumbersome reporting process and pervasive misinformation about breast milk and breast milk substitutes. Limited reach for women employed in the informal labour sector and in rural communities was said to be a compatibility issue for the extended maternity leave policy in addition to the lack of impact on non-parental guardians and caretakers. Recommendations to improve policy implementation include designating a role for international organizations in supporting implementation, expanding maternity protections for all working women, building local-level policy knowledge to support enforcement, simplifying Decree No. 100 violation reporting processes and continuing to invest in interventions to facilitate a supportive policy environment in Vietnam.

The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation

To rigorously measure the implementation of evidence-based interventions, implementation science requires measures that have evidence of reliability and validity across different contexts and populations. Measures that can detect change over time and impact on outcomes of interest are most useful to implementers. Moreover, measures that fit the practical needs of implementers could be used to guide implementation outside of the research context. To address this need, our team developed a rating scale for implementation science measures that considers their psychometric and pragmatic properties and the evidence available. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) can be used in systematic reviews of measures, in measure development, and to select measures. PAPERS may move the field toward measures that inform robust research evaluations and practical implementation efforts.

Toward a New Strategic Public Health Science for Policy, Practice, Impact, and Health Equity

The COVID-19 pandemic and its social and health impact have underscored the need for a new strategic science agenda for public health. To optimize public health impact, high-quality strategic science addresses scientific gaps that inform policy and guide practice. At least 6 scientific gaps emerge from the US experience with COVID-19: health equity science, data science and modernization, communication science, policy analysis and translation, scientific collaboration, and climate science. Addressing these areas within a strategic public health science agenda will accelerate achievement of public health goals. Public health leadership and scientists have an unprecedented opportunity to use strategic science to guide a new era of improved and equitable public health.

A systematic review of school health policy measurement tools: implementation determinants and outcomes

BACKGROUND

Governments in some countries or states/provinces mandate school-based policies intended to improve the health and well-being of primary and secondary students and in some cases the health of school staff. Examples include mandating a minimum time spent per week in programmed physical activity, mandating provision of healthy foods and limiting fat content of school meals, and banning tobacco products or use on school campuses. Although school health researchers have studied whether schools, districts, or states/provinces are meeting requirements, it is unclear to what extent implementation processes and determinants are assessed. The purposes of the present systematic review of quantitative measures of school policy implementation were to (1) identify quantitative school health policy measurement tools developed to measure implementation at the school, district, or state/provincial levels; (2) describe the policy implementation outcomes and determinants assessed and identify the trends in measurement; and (3) assess pragmatic and psychometric properties of identified implementation measures to understand their quality and suitability for broader application.

METHODS

Peer-reviewed journal articles published 1995-2020 were included if they (1) had multiple-item quantitative measures of school policy implementation and (2) addressed overall wellness, tobacco, physical activity, nutrition, obesity prevention, or mental health/bullying/social-emotional learning. The final sample comprised 86 measurement tools from 67 peer-review articles. We extracted study characteristics, such as psychometric and pragmatic measure properties, from included articles based on three frameworks: (1) Implementation Outcomes Framework, (2) Consolidated Framework for Implementation Research, and (3) Policy Implementation Determinants Framework.

RESULTS

Most implementation tools were developed to measure overall wellness policies which combined multiple policy topics (n = 35, 40%) and were in survey form (n = 75, 87%). Fidelity was the most frequently prevalent implementation outcome (n = 70, 81%), followed by adoption (n = 32, 81%). The implementation determinants most assessed were readiness for implementation, including resources (n = 43, 50%), leadership (n = 42, 49%), and policy communication (n = 41, 48%). Overall, measures were low-cost and had easy readability. However, lengthy tools and lack of reported validity/reliability data indicate low transferability.

CONCLUSIONS

Implementation science can contribute to more complete and rigorous assessment of school health policy implementation processes, which can improve implementation strategies and ultimately the intended health benefits. Several high-quality measures of implementation determinants and implementation outcomes can be applied to school health policy implementation assessment. Dissemination and implementation science researchers can also benefit from measurement experiences of school health researchers.

"Just So You Know, It Has Been Hard": Food Retailers' Perspectives of Implementing a Food and Nutrition Policy in Public Healthcare Settings

Mandated policies to improve food environments in public settings are an important strategy for governments. Most Australian governments have mandated policies or voluntary standards for healthy food procurement in healthcare facilities, however, implementation and compliance are poor. A better understanding of the support required to successfully implement such policies is needed. This research explored food retailers' experiences in implementing a mandated food and nutrition policy (the Policy) in healthcare settings to identify barriers, enablers, and impacts of compliance. Three 90-min workshops facilitated by two public health practitioners were undertaken with 12 food retailers responsible for operating 44 outlets across four hospitals in Perth, Western Australia. Workshop discussions were transcribed non-verbatim and inductive thematic content was analyzed. Three main themes were identified: (1) food retailers had come to accept their role in implementing the Policy; (2) the Policy made it difficult for food retailers to operate successfully, and; (3) food retailers needed help and support to implement the Policy. Findings indicate the cost of implementation is borne by food retailers. Communications campaigns, centralized databases of classified products, reporting frameworks, recognition of achievements, and dedicated technical expertise would support achieving policy compliance. Feasibility assessments prior to policy implementation are recommended for policy success.