Organizational Supports for Research Evidence Use in State Public Health Agencies: A Latent Class Analysis

Understanding Misimplementation in U.S. State Health Departments: An Agent-Based Model

INTRODUCTION

The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes.

METHODS

The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation.

RESULTS

Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation.

CONCLUSIONS

The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.

Disseminating Evidence on Abortion Facilities to Health Departments: A Randomized Study of E-mail Strategies

Given the politicization of abortion, professionals working in U.S. health departments (HDs) may not be receptive to communications about abortion, despite often regulating abortion facilities. This paper reports results of a randomized, prospective, observational study to test the effects of e-mail language when disseminating evidence on abortion to HD professionals. Our sample was 302 HD employees who oversee healthcare facilities inspection/regulation in all 50 U.S. state HDs, clustered by HD and randomized into two study groups. In November-December 2019, we sent biweekly e-mails containing links to a website summarizing evidence on abortion facility regulation. E-mails/headers sent to one group emphasized public health values and did not include the word abortion; e-mails/headers to the other group used the word abortion. Primary outcome measures were e-mail open rates and click-through rates. Among 221 participants to whom e-mails were deliverable, the overall open rate was 36%. Open rate was 25% for PH values and 46% for abortion groups (p < .05). Effects were moderated by state abortion policy environment: in both supportive and restrictive environments, participants in the abortion messaging group were statistically more likely to open e-mails than those in the PH values group. There was no difference between groups in states with middle-ground abortion policy environments. Among participants opening at least one e-mail, 19% clicked through to the website, with no significant difference by group. This study demonstrates that repeated targeted e-mail campaigns can reach HD professionals with research summaries. Concerns that communications to HDs should avoid the word abortion are unsupported.

Leading the way: competencies of leadership to prevent mis-implementation of public health programs

Public health agencies are increasingly concerned with ensuring that they are maximizing limited resources by delivering effective programs to enhance population-level health outcomes. Preventing mis-implementation (ending effective activities prematurely or continuing ineffective ones) is necessary to sustain public health efforts and resources needed to improve health and well-being. The purpose of this paper is to identify the important qualities of leadership in preventing mis-implementation of public health programs. In 2019, 45 state health department chronic disease employees were interviewed via phone and audio-recorded, and the conversations were transcribed verbatim. Thematic analysis focused on items related to mis-implementation and the manners in which leadership were involved in continuing ineffective programs. Final themes were based on a Public Health Leadership Competency Framework. The following themes emerged from their interviews regarding the important leadership competencies to prevent mis-implementation: '(1) leadership and communication; (2) collaborative leadership (3) leadership to adapt programs; (4) leadership and organizational learning and development; and (5) political leadership'. This first of its kind study showed the close interrelationship between mis-implementation and leadership. Increased attention to public health leader competencies might help to reduce mis-implementation in public health practice and lead to more effective and efficient use of limited resources.

Increasing the Amount and Intensity of Stepping Training During Inpatient Stroke Rehabilitation Improves Locomotor and Non-Locomotor Outcomes

BACKGROUND

The efficacy of traditional rehabilitation interventions to improve locomotion post-stroke, including providing multiple exercises targeting impairments and activity limitations, is uncertain. Emerging evidence rather suggests attempts to prioritize stepping practice at higher cardiovascular intensities may facilitate greater locomotor outcomes.

OBJECTIVE

The present study was designed to evaluate the comparative effectiveness of high-intensity training (HIT) to usual care during inpatient rehabilitation post-stroke.

METHODS

Changes in stepping activity and functional outcomes were compared over 9 months during usual-care (n = 131 patients < 2 months post-stroke), during an 18-month transition phase with attempts to implement HIT (n = 317), and over 12 months following HIT implementation (n = 208). The transition phase began with didactic and hands-on education, and continued with meetings, mentoring, and audit and feedback. Fidelity metrics included percentage of sessions prioritizing gait interventions and documenting intensity. Demographics, training measures, and outcomes were compared across phases using linear or logistic regression analysis, Kruskal-Wallis tests, or χ² analysis.

RESULTS

Across all phases, admission scores were similar except for balance (usual-care>HIT; P < .02). Efforts to prioritize stepping and achieve targeted intensities during HIT vs transition or usual-care phases led to increased steps/day (P < .01). During HIT, gains in 10-m walk [HIT median = 0.13 m/s (interquartile range: 0-0.35) vs usual-care = 0.07 m/s (0-0.24), P = .01] and 6-min walk [50 (9.3-116) vs 2.1 (0-56) m, P < .01] were observed, with additional improvements in transfers and stair-climbing.

CONCLUSIONS

Greater efforts to prioritize walking and reach higher intensities during HIT led to increased steps/day, resulting in greater gains in locomotor and non-locomotor outcomes.

"It's good to feel like you're doing something": a qualitative study examining state health department employees' views on why ineffective programs continue to be implemented in the USA

Background

Mis-implementation, the inappropriate continuation of programs or policies that are not evidence-based or the inappropriate termination of evidence-based programs and policies, can lead to the inefficient use of scarce resources in public health agencies and decrease the ability of these agencies to deliver effective programs and improve population health. Little is known about why mis-implementation occurs, which is needed to understand how to address it. This study sought to understand the state health department practitioners’ perspectives about what makes programs ineffective and the reasons why ineffective programs continue.

Methods

Eight state health departments (SHDs) were selected to participate in telephone-administered qualitative interviews about decision-making around ending or continuing programs. States were selected based on geographic representation and on their level of mis-implementation (low and high) categorized from our previous national survey. Forty-four SHD chronic disease staff participated in interviews, which were audio-recorded and transcribed verbatim. Transcripts were consensus coded, and themes were identified and summarized. This paper presents two sets of themes, related to (1) what makes a program ineffective and (2) why ineffective programs continue to be implemented according to SHD staff.

Results

Participants considered programs ineffective if they were not evidence-based or if they did not fit well within the population; could not be implemented well due to program restraints or a lack of staff time and resources; did not reach those who could most benefit from the program; or did not show the expected program outcomes through evaluation. Practitioners described several reasons why ineffective programs continued to be implemented, including concerns about damaging the relationships with partner organizations, the presence of program champions, agency capacity, and funding restrictions.

Conclusions

The continued implementation of ineffective programs occurs due to a number of interrelated organizational, relational, human resources, and economic factors. Efforts should focus on preventing mis-implementation since it limits public health agencies’ ability to conduct evidence-based public health, implement evidence-based programs effectively, and reduce the high burden of chronic diseases. The use of evidence-based decision-making in public health agencies and supporting adaptation of programs to improve their fit may prevent mis-implementation. Future work should identify effective strategies to reduce mis-implementation, which can optimize public health practice and improve population health.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00252-4.

The Relationships Between State Health Department Practitioners' Perceptions of Organizational Supports and Evidence-Based Decision-Making Skills

OBJECTIVES

Evidence-based decision making (EBDM) allows public health practitioners to implement effective programs and policies fitting the preferences of their communities. To engage in EBDM, practitioners must have skills themselves, their agencies must engage in administrative evidence-based practices (A-EBPs), and leaders must encourage the use of EBDM. We conducted this longitudinal study to quantify perceptions of individual EBDM skills and A-EBPs, as well as the longitudinal associations between the 2.

METHODS

An online survey completed among US state health department practitioners in 2016 and 2018 assessed perceptions of respondents' skills in EBDM and A-EBPs. We used χ2 tests, t tests, and linear regressions to quantify changes over time, differences by demographic characteristics, and longitudinal associations between individual skills and A-EBPs among respondents who completed both surveys (N = 336).

RESULTS

Means of most individual EBDM skills and A-EBPs did not change significantly from 2016 to 2018. We found significant positive associations between changes in A-EBPs and changes in EBDM skill gaps: for example, a 1-point increase in the relationships and partnerships score was associated with a narrowing of the EBDM skill gap (β estimate = 0.38; 95% CI, 0.15-0.61). At both time points, perceived skills and A-EBPs related to financial practices were low.

CONCLUSIONS

Findings from this study can guide the development and dissemination of initiatives designed to simultaneously improve individual and organizational capacity for EBDM in public health settings. Future studies should focus on types of strategies most effective to build capacity in particular types of agencies and practitioners, to ultimately improve public health practice.

Determinants of using children's mental health research in policymaking: variation by type of research use and phase of policy process

Background

Research use in policymaking is multi-faceted and has been the focus of extensive study. However, virtually no quantitative studies have examined whether the determinants of research use vary according to the type of research use or phase of policy process. Understanding such variation is important for selecting the targets of implementation strategies that aim to increase the frequency of research use in policymaking.

Methods

A web-based survey of US state agency officials involved with children’s mental health policymaking was conducted between December 2019 and February 2020 (n = 224, response rate = 33.7%, 49 states responding (98%), median respondents per state = 4). The dependent variables were composite scores of the frequency of using children’s mental health research in general, specific types of research use (i.e., conceptual, instrumental, tactical, imposed), and during different phases of the policy process (i.e., agenda setting, policy development, policy implementation). The independent variables were four composite scores of determinants of research use: agency leadership for research use, agency barriers to research use, research use skills, and dissemination barriers (e.g., lack of actionable messages/recommendations in research summaries, lack of interaction/collaboration with researchers). Separate multiple linear regression models estimated associations between determinant and frequency of research use scores.

Results

Determinants of research use varied significantly by type of research use and phase of policy process. For example, agency leadership for research use was the only determinant significantly associated with imposed research use (β = 0.31, p < 0.001). Skills for research use were the only determinant associated with tactical research use (β = 0.17, p = 0.03) and were only associated with research use in the agenda-setting phase (β = 0.16, p = 0.04). Dissemination barriers were the most universal determinants of research use, as they were significantly and inversely associated with frequency of conceptual (β = −0.21, p = 0.01) and instrumental (β = −0.22, p = 0.01) research use and during all three phases of policy process.

Conclusions

Decisions about the determinants to target with policy-focused implementation strategies—and the strategies that are selected to affect these targets—should reflect the specific types of research use that these strategies aim to influence.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-021-01081-8.

Examining Training Motivations Among Public Health Workers

CONTEXT

As public health needs and priorities evolve, maintaining a trained public health workforce is critical to the success of public health efforts. Researchers have examined training needs in various contexts and subpopulations, but a nationally representative study of what motivates public health workers to seek out training has yet to be conducted. By understanding these motivations, public health agencies and policy makers can appeal to worker motivations in both training programs and organizational incentives.

OBJECTIVE

The purpose of this article was to describe overall training motivations and identify patterns of training motivations among public health workers. This study also explored whether or not training needs differ across prevalent motivational patterns.

DESIGN AND PARTICIPANTS

Using data from the 2017 Public Health Workforce Interests and Needs Survey (PH WINS), the study used latent class analysis (LCA) to identify motivational patterns and logistic regression to analyze associations with training needs.

RESULTS

The most prominent motivation to seek training was personal growth (82.7% of respondents). LCA identified 4 motivational classes of public health workers: those motivated by organizational pressure and requirements (31.8%), those motivated indiscriminately by all factors (28.4%), those motivated primarily by personal growth (21.7%), and those motivated by organizational accommodations and supports (18.2%). Motivational class was not associated with indicating training needs in any of 8 training domains, nor was it associated with indicating any training need in any domain.

CONCLUSIONS

Public health agencies should consider the different motivational classes present in the public health workforce. In particular, motivational classes that represent organizational choices suggest that public health agencies should both motivate workers with organizational requirements and pressure from managers and offer institutional support via paid travel and covered time for training.

State mental health agency officials' preferences for and sources of behavioral health research

State mental health agencies (SMHAs) are integral to public behavioral health service systems. As such, senior-level officials within SMHAs are important targets for advocacy and dissemination of behavioral health research findings. Evidence-informed decision making in SMHAs can potentially be enhanced by developing summaries of behavioral health research (e.g., policy briefs) that reflect SMHA officials' information preferences, but knowledge about these preferences is lacking. An exploratory study was conducted with the aims of characterizing senior-level SMHA officials' preferences for behavioral health research and describing where they turn for this research when making policy decisions. A cross-sectional, web-based survey of senior-level SMHA officials (1 per state) was conducted in March-May 2017 (n = 43, response rate = 84%). The features of behavioral health research that SMHA officials identified as "very important" most frequently were research being relevant to state residents (93.0%), providing data on cost-effectiveness (86.0%) and budget impact (81.4%), and being presented concisely (81.0%). The primary sources that SMHA officials turned to for behavioral research when making policy decisions were professional organizations (79.1%), SMHA agency staff (60.5%), and university researchers (55.8%). Compared with state legislators' responses to the same survey questions, results suggest that senior-level SMHA officials and legislators have similar preferences for behavioral health research but turn to different sources for this research. Advocates and researchers who seek to promote evidence-informed decision making in SMHAs should consider developing policy briefs that are concise, provide state-level prevalence data about behavioral conditions, and contain economic evaluation data, and they should disseminate these materials to multiple sources. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

SIPsmartER delivered through rural, local health districts: adoption and implementation outcomes

Background

SIPsmartER is a 6-month evidenced-based, multi-component behavioral intervention that targets sugar-sweetened beverages among adults. It consists of three in-person group classes, one teach-back call, and 11 automated phone calls. Given SIPsmartER’s previously demonstrated effectiveness, understanding its adoption, implementation, and potential for integration within a system that reaches health disparate communities is important to enhance its public health impact. During this pilot dissemination and implementation trial, SIPsmartER was delivered by trained staff from local health districts (delivery agents) in rural, Appalachian Virginia. SIPsmartER’s execution was supported by consultee-centered implementation strategies.

Methods

In this mixed-methods process evaluation, adoption and implementation indicators of the program and its implementation strategy (e.g., fidelity, feasibility, appropriateness, acceptability) were measured using tracking logs, delivery agent surveys and interviews, and fidelity checklists. Quantitative data were analyzed with descriptive statistics. Qualitative data were inductively coded.

Results

Delivery agents implemented SIPsmartER to the expected number of cohorts (n = 12), recruited 89% of cohorts, and taught 86% of expected small group classes with > 90% fidelity. The planned implementation strategies were also executed with high fidelity. Delivery agents completing the two-day training, pre-lesson meetings, fidelity checklists, and post-lesson meetings at rates of 86, 75, 100, and 100%, respectively. Additionally, delivery agents completed 5% (n = 3 of 66) and 10% (n = 6 of 59) of teach-back and missed class calls, respectively. On survey items using 6-point scales, delivery agents reported, on average, higher feasibility, appropriateness, and acceptability related to delivering the group classes (range 4.3 to 5.6) than executing missed class and teach-back calls (range 2.6 to 4.6). They also, on average, found the implementation strategy activities to be helpful (range 4.9 to 6.0). Delivery agents identified strengths and weakness related to recruitment, lesson delivery, call completion, and the implementation strategy.

Conclusions

In-person classes and the consultee-centered implementation strategies were viewed as acceptable, appropriate, and feasible and were executed with high fidelity. However, implementation outcomes for teach-back and missed class calls and recruitment were not as strong. Findings will inform the future full-scale dissemination and implementation of SIPsmartER, as well as other evidence-based interventions, into rural health districts as a means to improve population health.

Electronic supplementary material

The online version of this article (10.1186/s12889-019-7567-6) contains supplementary material, which is available to authorized users.