Rates of sustainment in the Universal Stages of Implementation Completion

A Mixed Methods Examination of Session Planning Among Public Mental Health Therapists

Session planning is a core activity for implementing evidence-based practices (EBPs), yet it is unknown whether public mental health settings provide the support for therapists to session plan. This two-part study conducted in collaboration with EBP leaders in Philadelphia's public mental health system deployed mixed methods to examine therapists' session planning practices and preferences. In Study 1, 61 public mental health therapists completed an online survey to identify session planning barriers and facilitators, current practices, and desired planning supports. In Study 2, nine therapists who ranked a session planning tool as a top choice support in Study 1 participated in two focus groups to elaborate on their survey responses and provide feedback on three session planning tool prototypes. Study 1 survey respondents cited multi-level barriers and facilitators to session planning. In both closed- and open-ended responses, analyzed descriptively and via content analysis respectively, therapists described wanting more time, lower caseloads, financial incentives for session planning, and additional clinical resources and guidance from trainings, peers, and supervisors to support session planning. Study 2 focus group participants, whose responses were analyzed using content analysis, reiterated the need for these multi-level supports and expressed the need for a "one-stop" database of session planning tools that would be free, easily searchable, and modifiable for varied clinical needs. All three session planning tool prototypes reviewed were acceptable; two were also considered feasible and appropriate. This investigation of an under-studied aspect of the EBP implementation process reveals the need for multi-level session planning supports.

Predictors of the sustainability for an evidence-based eating disorder prevention program delivered by college peer educators

BACKGROUND

Despite ongoing efforts to introduce evidence-based interventions (EBIs) into mental health care settings, little research has focused on the sustainability of EBIs in these settings. College campuses are a natural place to intervene with young adults who are at high risk for mental health disorders, including eating disorders. The current study tested the effect of three levels of implementation support on the sustainability of an evidence-based group eating disorder prevention program, the Body Project, delivered by peer educators. We also tested whether intervention, contextual, or implementation process factors predicted sustainability.

METHODS

We recruited 63 colleges with peer educator programs and randomly assigned them to (a) receive a 2-day Train-the-Trainer (TTT) training in which peer educators were trained to implement the Body Project and supervisors were taught how to train future peer educators (TTT), (b) TTT training plus a technical assistance (TA) workshop (TTT + TA), or (c) TTT plus the TA workshop and quality assurance (QA) consultations over 1-year (TTT + TA + QA). We tested whether implementation support strategies, perceived characteristics of the intervention and attitudes towards evidence-based interventions at baseline and the proportion of completed implementation activities during the implementation year predicted three school-level dichotomous sustainability outcomes (offering Body Project groups, training peer educators, training supervisors) over the subsequent two-year sustainability period using logistic regression models.

RESULTS

Implementation support strategies did not significantly predict any sustainability outcomes, although a trend suggested that colleges randomized to the TTT + TA + QA strategy were more likely to train new supervisors (OR = 5.46, 95% CI [0.89-33.38]). Colleges that completed a greater proportion of implementation activities were more likely to offer Body Project groups (OR = 1.53, 95% CI [1.19-1.98]) and train new peer educators during the sustainability phase (OR = 1.39, 95% CI [1.10-1.74]). Perceived positive characteristics of the Body Project predicted training new peer educators (OR = 18.42, 95% CI [1.48-299.66]), which may be critical for sustainability in routine settings with high provider turnover.

CONCLUSIONS

Helping schools complete more implementation activities and increasing the perceived positive characteristics of a prevention program may result in greater sustainment of prevention program implementation.

TRIAL REGISTRATION

This study was preregistered on 12/07/17 with ClinicalTrials.gov, ID NCT03409809, https://clinicaltrials.gov/ct2/show/NCT03409809 .

Understanding pathways from implementation to sustainment: a longitudinal, mixed methods analysis of promising practices implemented in the Veterans Health Administration

BACKGROUND

The Veterans Health Administration (VHA) is the United States largest learning health system. The Diffusion of Excellence (DoE) program is a large-scale model of diffusion that identifies and diffuses evidence-informed practices across VHA. During the period of 2016-2021, 57 evidence-informed practices were implemented across 82 VHA facilities. This setting provides a unique opportunity to understand sustainment determinants and pathways. Our objective was to characterize the longitudinal pathways of practices as they transition from initial implementation to long-term sustainment at each facility.

METHODS

A longitudinal, mixed-methods evaluation of 82 VHA facilities. Eighty-two facility representatives, chosen by leadership as points-of-contact for 57 DoE practices, were eligible for post-implementation interviews and annual sustainment surveys. Primary outcomes (implementation, sustainment), and secondary outcomes (institutionalization, effectiveness, anticipated sustainment) at four time-points were collected. We performed descriptive statistics and directed content analysis using Hailemariam et al.'s factors influencing sustainment.

RESULTS

After approximately five years post-implementation (e.g., 2021 sustainment outcomes), of the 82 facilities, about one-third fully sustained their practice compared to one-third that did not fully sustain their practice because it was in a "liminal" stage (neither sustained nor discontinued) or permanently discontinued. The remaining one-third of facilities had missing 2021 sustainment outcomes. A higher percentage of facilities (70%) had inconsistent primary outcomes (changing over time) compared to facilities (30%) with consistent primary outcomes (same over time). Thirty-four percent of facilities with sustained practices reported resilience since they overcame implementation and sustainment barriers. Facilities with sustained practices reported more positive secondary outcomes compared to those that did not sustain their practice. Key factors facilitating practice sustainment included: demonstrating practice effectiveness/benefit, sufficient organizational leadership, sufficient workforce, and adaptation/alignment with local context. Key factors hindering practice sustainment included: insufficient workforce, not able to maintain practice fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of practice, and no ongoing support.

CONCLUSIONS

We identified diverse pathways from implementation to sustainment, and our data underscore that initial implementation outcomes may not determine long-term sustainment outcomes. This longitudinal evaluation contributes to understanding impacts of the DoE program, including return on investment, achieving learning health system goals, and insights into achieving high-quality healthcare in VHA.

Implementing a pharmacist-integrated collaborative model of medication treatment for opioid use disorder in primary care: study design and methodological considerations

BACKGROUND

Pharmacists remain an underutilized resource in the treatment of opioid use disorder (OUD). Although studies have engaged pharmacists in dispensing medications for OUD (MOUD), few studies have evaluated collaborative care models in which pharmacists are an active, integrated part of a primary care team offering OUD care.

METHODS

This study seeks to implement a pharmacist integrated MOUD clinical model (called PrIMO) and evaluate its feasibility, acceptability, and impact across four diverse primary care sites. The Consolidated Framework for Implementation Research is used as an organizing framework for study development and interpretation of findings. Implementation Facilitation is used to support PrIMO adoption. We assess the primary outcome, the feasibility of implementing PrIMO, using the Stages of Implementation Completion (SIC). We evaluate the acceptability and impact of the PrIMO model at the sites using mixed-methods and combine survey and interview data from providers, pharmacists, pharmacy technicians, administrators, and patients receiving MOUD at the primary care sites with patient electronic health record data. We hypothesize that it is feasible to launch delivery of the PrIMO model (reach SIC Stage 6), and that it is acceptable, will positively impact patient outcomes 1 year post model launch (e.g., increased MOUD treatment retention, medication regimen adherence, service utilization for co-morbid conditions, and decreased substance use), and will increase each site's capacity to care for patients with MOUD (e.g., increased number of patients, number of prescribers, and rate of patients per prescriber).

DISCUSSION

This study will provide data on a pharmacist-integrated collaborative model of care for the treatment of OUD that may be feasible, acceptable to both site staff and patients and may favorably impact patients' access to MOUD and treatment outcomes.

TRIAL REGISTRATION

The study was registered on Clinicaltrials.gov (NCT05310786) on April 5, 2022, https://www.

CLINICALTRIALS

gov/study/NCT05310786?id=NCT05310786&rank=1.

A multi- and mixed-method adaptation study of a patient-centered perioperative mental health intervention bundle

BACKGROUND

Anxiety and depression are common among older adults and can intensify during perioperative periods, but few mental health interventions are designed for older surgical patients' unique needs. As part of the feasibility trial, we developed and adapted a perioperative mental health (PMH) bundle for older patients comprised of behavioral activation (BA) and medication optimization (MO) to ameliorate anxiety and depressive symptoms before, during, and after cardiac, orthopedic, and oncologic surgery.

METHODS

We used mixed-methods including workshop studios with patients, caregivers, clinicians, researchers, and interventionists; intervention refinement and reflection meetings; patient case review meetings; intervention session audio-recordings and documentation forms; and patient and caregiver semi-structured interviews. We used the results to refine our PMH bundle. We used multiple analytical approaches to report the nature of adaptations, including hybrid thematic analysis and content analysis informed by the Framework for Reporting Adaptations and Modifications - Expanded.

RESULTS

Adaptations were categorized by content (intervention components), context (how the intervention is delivered, based on the study, target population, intervention format, intervention delivery mode, study setting, study personnel), training, and evaluation. Of 51 adaptations, 43.1% involved content, 41.2% involved context, and 15.7% involved training and evaluation. Several key adaptations were noted: (1) Intervention content was tailored to patient preferences and needs (e.g., rewording elements to prevent stigmatization of mental health needs; adjusting BA techniques and documentation forms to improve patient buy-in and motivation). (2) Cohort-specific adaptations were recommended based on differing patient needs. (3) Compassion was identified by patients as the most important element.

CONCLUSIONS

We identified evidence-based mental health intervention components from other settings and adapted them to the perioperative setting for older adults. Informed by mixed-methods, we created an innovative and pragmatic patient-centered intervention bundle that is acceptable, feasible, and responsive to the needs of older surgical populations. This approach allowed us to identify implementation strategies to improve the reach, scalability, and sustainability of our bundle, and can guide future patient-centered intervention adaptations.

CLINICAL TRIALS REGISTRATION

NCT05110690 (11/08/2021).

Making the connection between health equity and sustainability

Sustainability and health inequities are key challenges in public health and healthcare. Research suggests that only about half of evidence-based interventions (EBIs) are sustained over time, and settings and populations experiencing systemic and structural barriers to health (e.g., poverty, racism, stigma, and discrimination) experience even greater challenges to sustainability. In this article, we argue that an enhanced focus on sustainability in the field of implementation science is critical in order to maximize the long-term health benefits and broader societal impacts of EBIs for all populations and settings. From an equity perspective, a focus on sustainability is particularly critical to prioritize among population sub-groups that have not historically received the benefits of health-related EBIs. We discuss how a health equity framing is essential to sustaining EBIs in under-resourced communities, and requires moving away from a deficit mindset that focuses on why EBIs are challenging to sustain, to one that focuses more on identifying and nurturing existing assets within individuals and communities to increase the likelihood that EBIs are sustained. We conclude with a discussion of future directions as well as recommendations and resources (e.g., frameworks, tools) to advance and make progress toward sustainability from a health equity mindset, including: (1) Actively planning early for sustainability alongside key partners; (2) Tracking progress toward enhancing sustainability and being accountable in doing so equitably for all settings and populations; and (3) Focusing on both equity and engagement early and often throughout the research process and all implementation phases.

Who's "in the room where it happens"? A taxonomy and five-step methodology for identifying and characterizing policy actors

BACKGROUND

Engaging policy actors in research design and execution is critical to increasing the practical relevance and real-world impact of policy-focused dissemination and implementation science. Identifying and selecting which policy actors to engage, particularly actors involved in "Big P" public policies such as laws, is distinct from traditional engaged research methods. This current study aimed to develop a transparent, structured method for iteratively identifying policy actors involved in key policy decisions-such as adopting evidence-based interventions at systems-scale-and to guide implementation study sampling and engagement approaches. A flexible policy actor taxonomy was developed to supplement existing methods and help identify policy developers, disseminators, implementers, enforcers, and influencers.

METHODS

A five-step methodology for identifying policy actors to potentially engage in policy dissemination and implementation research was developed. Leveraging a recent federal policy as a case study-The Family First Prevention Services Act (FFPSA)-publicly available documentation (e.g., websites, reports) were searched, retrieved, and coded using content analysis to characterize the organizations and individual policy actors in the "room" during policy decisions.

RESULTS

The five steps are as follows: (1) clarify the policy implementation phase(s) of interest, (2) identify relevant proverbial or actual policymaking "rooms," (3) identify and characterize organizations in the room, (4) identify and characterize policy actors in the "room," and (5) quantify (e.g., count actors across groups), summarize, and compare "rooms" to develop or select engagement approaches aligned with the "room" and actors. The use and outcomes of each step are exemplified through the FFPSA case study.

CONCLUSIONS

The pragmatic and transparent policy actor identification steps presented here can guide researchers' methods for continuous sampling and successful policy actor engagement. Future work should explore the utility of the proposed methods for guiding selection and tailoring of engagement and implementation strategies (e.g., research-policy actor partnerships) to improve both "Big P" and "little p" (administrative guidelines, procedures) policymaking and implementation in global contexts.

The relative value of Pre-Implementation stages for successful implementation of evidence-informed programs

BACKGROUND

Most implementations fail before the corresponding services are ever delivered. Measuring implementation process fidelity may reveal when and why these attempts fail. This knowledge is necessary to support the achievement of positive implementation milestones, such as delivering services to clients (program start-up) and competency in treatment delivery. The present study evaluates the extent to which implementation process fidelity at different implementation stages predicts achievement of those milestones.

METHODS

Implementation process fidelity data-as measured by the Stages of Implementation Completion (SIC)-from 1287 implementing sites across 27 evidence-informed programs were examined in mixed effects regression models with sites nested within programs. Implementation process fidelity, as measured by the proportion of implementation activities completed during the three stages of the SIC Pre-Implementation phase and overall Pre-Implementation (Phase 1) and Implementation (Phase 2) proportion scores, was assessed as a predictor of sites achieving program start-up (i.e., delivering services) and competency in program delivery.

RESULTS

The predicted probability of start-up across all sites was low at 35% (95% CI [33%, 38%]). When considering the evidence-informed program being implemented, that probability was nearly twice as high (64%; 95% CI [42%, 82%]), and 57% of the total variance in program start-up was attributable to the program. Implementation process fidelity was positively and significantly associated with achievement of program start-up and competency. The magnitude of this relationship varied significantly across programs for Pre-Implementation Stage 1 (i.e., Engagement) only. Compared to other stages, completing more Pre-Implementation Stage 3 (Readiness Planning) activities resulted in the most rapid gains in probability of achieving program start-up. The predicted probability of achieving competency was very low unless sites had high scores in both Pre-Implementation and Implementation phases.

CONCLUSIONS

Strong implementation process fidelity-as measured by SIC Pre-Implementation and Implementation phase proportion scores-was associated with sites' achievement of program start-up and competency in program delivery, with early implementation process fidelity being especially potent. These findings highlight the importance of a rigorous Pre-Implementation process.

A budget impact analysis of cost to implement a whole child health focused, family-based intervention in primary care for children with elevated BMI

BACKGROUND

Although the cost of implementing evidence-based interventions (EBIs) is a key determinant of adoption, lack of cost information is widespread. We previously evaluated the cost of preparing to implement Family Check-Up 4 Health (FCU4Health), an individually tailored, evidence-based parenting program that takes a whole child approach, with effects on both behavioral health and health behavior outcomes, in primary care settings. This study estimates the cost of implementation, including preparation.

METHODS

We assessed the cost of FCU4Health across the preparation and implementation phases spanning 32 months and 1 week (October 1, 2016-June 13, 2019) in a type 2 hybrid effectiveness-implementation study. This family-level randomized controlled trial took place in Arizona with n = 113 predominantly low-income, Latino families with children ages > 5.5 to < 13 years. Using electronic cost capture and time-based activity-driven methods, budget impact analysis from the perspective of a future FCU4Health adopting entity-namely, ambulatory pediatric care clinicians-was used to estimate the cost of implementation. Labor costs were based on 2021 Bureau of Labor Statistics Occupational Employment Statistics, NIH-directed salary cap levels or known salaries, plus fringe benefits at a standard rate of 30%. Non-labor costs were based on actual amounts spent from receipts and invoices.

RESULTS

The cost of FCU4Health implementation to 113 families was $268,886 ($2380 per family). Actual per family cost varied widely, as individual tailoring resulted in families receiving a range of 1-15 sessions. The estimated cost of replicating implementation for future sites ranged from $37,636-$72,372 ($333-$641 per family). Using our previously reported preparation costs (i.e., $174,489; $1544 per family), with estimated replication costs of $18,524-$21,836 ($164-$193 per family), the total cost of delivering FCU4Health was $443,375 ($3924 per family), with total estimated replication costs of $56,160-$94,208 ($497-$834 per family).

CONCLUSIONS

This study provides a baseline for costs associated with implementation of an individually tailored parenting program. Results provide critical information for decision makers and a model for future economic analysis and can be used to inform optimization thresholds for implementation and, when necessary, benchmarks for program adaptation to promote scale-up.

TRIAL REGISTRATION

This trial was prospectively registered on January 6, 2017, at ClinicalTrials.gov (NCT03013309).

Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers

Background

Economic evaluations frequently are utilized to compare the value of different interventions in medicine and health in concrete terms. Implementation science also would benefit from the incorporation of economic evaluations, but such studies are rare in the literature. The National Cancer Institute has supported a special collection of articles focusing on economic evaluations in implementation science. Even when interventions are supported by substantial evidence, they are implemented infrequently in the field. Implementation costs are important determinants for whether organizational decision-makers choose to adopt an intervention and whether the implementation process is successful. Economic evaluations, such as cost-effectiveness analyses, can help organizational decision-makers choose between implementation approaches for evidence-based interventions by accounting for costs and succinctly presenting cost/benefit tradeoffs.

Main text

This manuscript presents a discussion of important considerations for incorporating economic evaluations into implementation science. First, the distinction between intervention and implementation costs is presented, along with an explanation of why the comprehensive representation of implementation costs is elusive. Then, the manuscript describes how economic evaluations in implementation science may differ from those in medicine and health intervention studies, especially in terms of determining the perspectives and outcomes of interest. Finally, referencing a scale-up trial of an evidence-based behavioral health intervention, concrete case examples of how cost data can be collected and used in economic evaluations targeting implementation, rather than clinical outcomes, are described.

Conclusions

By gaining a greater understanding of the costs and economic impact associated with different implementation approaches, organizational decision-makers will have better transparency for future replication and scale-up. The use of economic evaluations can help to advance this understanding and provide researchers, purveyors or third-party intermediaries, and organizational decision-makers with essential information to facilitate implementation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00295-1.