Pilot evaluation of the Fiscal Mapping Process for sustainable financing of evidence-based youth mental health treatments: A comparative case study analysis

Background

Sustained delivery of evidence-based treatments (EBTs) is essential to addressing the public health impacts of youth mental health problems, but is complicated by the limited and fragmented funding available to youth mental health service agencies. Supports are needed that can guide service agencies in accessing sustainable funding for EBTs. We conducted a pilot evaluation of the Fiscal Mapping Process, an Excel-based strategic planning tool that helps service agency leaders identify and coordinate financing strategies for their EBT programs.

Method

Pilot testing of the Fiscal Mapping Process was completed with 10 youth mental health service agencies over a 12-month period, using trauma-focused cognitive-behavioral therapy or parent-child interaction therapy programs. Service agency representatives received initial training and monthly coaching in using the tool. We used case study methods to synthesize all available data (surveys, focus groups, coaching notes, document review) and contrast agency experiences to identify key findings through explanation building.

Results

Key evaluation findings related to the process and outcomes of using the Fiscal Mapping Process, as well as contextual influences. Process evaluation findings helped clarify the primary use case for the tool and identified the importance-and challenges-of engaging external collaborators. Outcome evaluation findings documented the impacts of the Fiscal Mapping Process on agency-reported sustainment capacities (strategic planning, funding stability), which fully explained reported improvements in outcomes (extent and likelihood)-although these impacts were incremental. Findings on contextual factors documented the influence of environmental and organizational capacities on engagement with the tool and concerns about equitable impacts, but also the view that the process could usefully generalize to other EBTs.

Conclusions

Our pilot evaluation of the Fiscal Mapping Process was promising. In future work, we plan to integrate the tool into EBT implementation initiatives and test its impact on long-term sustainment outcomes across various EBTs, while increasing attention to equity considerations.

Sustainment of the TeleSleep program for rural veterans

Background

In fiscal year 2021, the Veterans Health Administration (VHA) provided care for sleep disorders to 599,966 Veterans, including 189,932 rural Veterans. To further improve rural access, the VA Office of Rural Health developed the TeleSleep Enterprise-Wide Initiative (EWI). TeleSleep's telemedicine strategies include tests for sleep apnea at the Veteran's home rather than in a sleep lab; Clinical Video Telehealth applications; and other forms of virtual care. In 2017 and 2020, VHA provided 3-year start-up funding to launch new TeleSleep programs at rural-serving VA medical facilities.

Methods

In early 2022, we surveyed leaders of 24 sites that received TeleSleep funding to identify successes, failures, facilitators, and barriers relevant to sustaining TeleSleep implementations upon expiration of startup funding. We tabulated frequencies on the multiple choice questions in the survey, and, using the survey's critical incident framework, summarized the responses to open-ended questions. TeleSleep program leaders discussed the responses and synthesized recommendations for improvement.

Results

18 sites reported sustainment, while six were "on track." Sustainment involved medical centers or regional entities incorporating TeleSleep into their budgets. Facilitators included: demonstrating value; aligning with local priorities; and collaborating with spoke sites serving rural Veterans. Barriers included: misalignment with local priorities; and hiring delays. COVID was a facilitator, as it stimulated adoption of telehealth practices; and also a barrier, as it consumed attention and resources. Recommendations included: longer startup funding; dedicated funding for human resources to accelerate hiring; funders communicating with local facility leaders regarding how TeleSleep aligns with organizational priorities; hiring into job classifications aligned with market pay; and obtaining, from finance departments, projections and outcomes for the return on investment in TeleSleep.

The Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C) for serious mental illness in community mental health part 3: Study protocol to evaluate sustainment in a hybrid type 2 effectiveness-implementation cluster-randomized trial

treatments (EBPTs) has advanced rapidly, research on the sustainment of implemented EBPTs remains limited. This is concerning, given that EBPT activities and benefits regularly decline post-implementation. To advance research on sustainment, the present protocol focuses on the third and final phase - the Sustainment Phase - of a hybrid type 2 cluster-randomized controlled trial investigating the implementation and sustainment of the Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C) for patients with serious mental illness and sleep and circadian problems in community mental health centers (CMHCs). Prior to the first two phases of the trial - the Implementation Phase and Train-the-Trainer Phase - TranS-C was adapted to fit the CMHC context. Then, 10 CMHCs were cluster-randomized to implement Standard or Adapted TranS-C via facilitation and train-the-trainer. The primary goal of the Sustainment Phase is to investigate whether adapting TranS-C to fit the CMHC context predicts improved sustainment outcomes.

Methods

Data collection for the Sustainment Phase will commence at least three months after implementation efforts in partnering CMHCs have ended and may continue for up to one year. CMHC providers will be recruited to complete surveys (N = 154) and a semi-structured interview (N = 40) on sustainment outcomes and mechanisms. Aim 1 is to report the sustainment outcomes of TranS-C. Aim 2 is to evaluate whether manipulating EBPT fit to context (i.e., Standard versus Adapted TranS-C) predicts sustainment outcomes. Aim 3 is to test whether provider perceptions of fit mediate the relation between treatment condition (i.e., Standard versus Adapted TranS-C) and sustainment outcomes. Mixed methods will be used to analyze the data.

Discussion

The present study seeks to advance our understanding of sustainment predictors, mechanisms, and outcomes by investigating (a) whether the implementation strategy of adapting an EBPT (i.e., TranS-C) to the CMHC context predicts improved sustainment outcomes and (b) whether this relation is mediated by improved provider perceptions of treatment fit. Together, the findings may help inform more precise implementation efforts that contribute to lasting change.

Trial Registration

ClinicalTrials.gov identifier: NCT05956678. Registered on July 21, 2023. https://classic.clinicaltrials.gov/ct2/show/NCT05956678?term=NCT05956678&draw=2&rank=1.

Making the connection between health equity and sustainability

Sustainability and health inequities are key challenges in public health and healthcare. Research suggests that only about half of evidence-based interventions (EBIs) are sustained over time, and settings and populations experiencing systemic and structural barriers to health (e.g., poverty, racism, stigma, and discrimination) experience even greater challenges to sustainability. In this article, we argue that an enhanced focus on sustainability in the field of implementation science is critical in order to maximize the long-term health benefits and broader societal impacts of EBIs for all populations and settings. From an equity perspective, a focus on sustainability is particularly critical to prioritize among population sub-groups that have not historically received the benefits of health-related EBIs. We discuss how a health equity framing is essential to sustaining EBIs in under-resourced communities, and requires moving away from a deficit mindset that focuses on why EBIs are challenging to sustain, to one that focuses more on identifying and nurturing existing assets within individuals and communities to increase the likelihood that EBIs are sustained. We conclude with a discussion of future directions as well as recommendations and resources (e.g., frameworks, tools) to advance and make progress toward sustainability from a health equity mindset, including: (1) Actively planning early for sustainability alongside key partners; (2) Tracking progress toward enhancing sustainability and being accountable in doing so equitably for all settings and populations; and (3) Focusing on both equity and engagement early and often throughout the research process and all implementation phases.

Leadership perspectives on facilitators and barriers to sustaining evidence-based prevention interventions in the United States Military

The U.S. Department of Defense (DoD) aims to prevent suicide, harassment, sexual assault, and partner and child maltreatment by implementing evidence-based behavioral health interventions (EBIs). However, sustaining EBI implementation over time and with fidelity to result in meaningful impacts is a tremendous challenge. We interviewed 35 military leaders in positions to observe, and possibly hinder, the erosions of EBI implementations to learn what distinguishes EBIs that sustain in the military from those that fade away. Thematic analysis identified barriers and supports to EBI sustainment consistent with the Consolidated Framework for Implementation Research, reflecting the domains: outer setting, inner setting, individuals, and innovation. Participants described how factors at different levels of the social ecology interact with each other and emphasized how aspects of military culture (e.g., hierarchical structure, frequent moves, mission focus) can both support and challenge implementing and sustaining behavioral-health EBIs. EBI implementation in the military differs from most civilian settings in that service member participation in certain preventative programs is mandated. The results indicate how policy and practice can strengthen sustained EBI implementation to reduce harm and support service members.

Supporting Sustained STAR-VA Implementation: Evaluation of a Sustainment Coaching Program

OBJECTIVES

The Veterans Health Administration (VA) STAR-VA program is a person-centered, interdisciplinary intervention for managing distress behaviors in dementia in VA nursing homes, called Community Living Centers (CLCs). Teams often struggle to sustain STAR-VA after site-lead partner turnover. The STAR-VA Sustainment Coaching Program was developed to support a new local STAR-VA behavioral coordinator and/or nurse champion (ie, site-leads) and engage the team to implement sustainment strategies. The aims of this evaluation were to document sustainment needs, barriers and facilitators, and effective strategies used during a sustainment intervention.

DESIGN

We describe qualitative and quantitative findings of a program evaluation of the STAR-VA Sustainment Coaching pilot program.

SETTING AND PARTICIPANTS

Eight CLC teams participated in the pilot program, 5 training a new nurse champion, 2 a new behavioral coordinator, and 1 both new site-leads.

METHODS

CLC teams completed a needs assessment, developed sustainment goals and plans, tracked sustainment interventions implemented, and reported case outcomes. Outcome assessment and tracking reports were summarized and themes identified using a qualitative inductive approach.

RESULTS

Common sustainment needs were to train a new nurse champion, promote staff understanding of dementia and distress behaviors, and promote a culture of person-centered dementia care. Most sites selected sustainment goals of implementing training procedures and behavioral rounds. Sustainment barriers included limited staff time, staff turnover, lack of supportive routines and tools, and limited awareness of STAR-VA. Facilitators included leadership support, staff interest, and training resources. Most sites reported successful implementation of STAR-VA behavior rounds, whereas strategies to achieve training-related goals had variable success.

CONCLUSIONS AND IMPLICATIONS

The STAR-VA Sustainment Coaching Program helped CLC teams identify and implement strategies to support sustained STAR-VA implementation. The results of this project are informing ongoing efforts to integrate STAR-VA components into usual care processes for CLC residents with dementia.

The Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C) for serious mental illness in community mental health part 2: Study protocol for a hybrid type 2 effectiveness-implementation cluster- randomized trial using train-the-trainer

Background

Train-the-trainer (TTT) is a promising method for implementing evidence-based psychological treatments (EBPTs) in community mental health centers (CMHCs). In TTT, expert trainers train locally embedded individuals (i.e., Generation 1 providers) to deliver an EBPT, who then train others (i.e., Generation 2 providers). The present study will evaluate implementation and effectiveness outcomes of an EBPT for sleep and circadian dysfunction-the Transdiagnostic Intervention for Sleep and Circadian Dysfunction (TranS-C)-delivered to CMHC patients with serious mental illness by Generation 2 providers (i.e., trained and supervised within CMHCs via TTT). Specifically, we will investigate whether adapting TranS-C to fit CMHC contexts improves Generation 2 (a) patient outcomes (b) providers' perceptions of fit.

Methods

TTT will be implemented in nine CMHCs in California, United States (N= 60 providers; N= 130 patients) via facilitation. CMHCs are cluster-randomized by county to Adapted TranS-C or Standard TranS-C. Within each CMHC, patients are randomized to immediate TranS-C or usual care followed by delayed treatment with TranS-C (UC-DT). Aim 1 will assess the effectiveness of TranS-C (combined Adapted and Standard), compared to UC-DT, on improvements in sleep and circadian problems, functional impairment, and psychiatric symptoms for Generation 2 patients. Aim 2 will evaluate whether Adapted TranS-C is superior to Standard TranS-C with respect to Generation 2 providers' perceptions of fit. Aim 3 will evaluate whether Generation 2 providers' perceived fit mediates the relation between TranS-C treatment condition and patient outcomes. Exploratory analyses will: (1) evaluate whether the effectiveness of TranS-C for patient outcomes is moderated by generation, (2) compare Adapted and Standard TranS-C on patient perceptions of credibility/improvement and PhenX Toolkit outcomes (e.g., substance use, suicidality); and (3) evaluate other possible moderators.

Discussion

This trial has potential to inform the process of (a) embedding local trainers and supervisors to expand delivery of a promising transdiagnostic treatment for sleep and circadian dysfunction, (b) adding to the growing body of TTT literature by evaluating TTT outcomes with a novel treatment and population, and (c) advancing our understanding of providers' perceptions of EBPT 'fit' across TTT generations.

Trial registration

Clinicaltrials.gov identifier: NCT05805657. Registered on April 10, 2023. https://clinicaltrials.gov/ct2/show/NCT05805657.

Study Protocol: Multi-level Determinants of Implementation and Sustainment in the Education Sector

Evidence-based programs (EBPs) delivered in elementary schools show great promise in reducing risk for emotional and behavioral disorders (EBDs). However, efforts to sustain EBPs in school face barriers. Improving EBP sustainment thus represents a priority, but little research exists to inform the development of sustainment strategies. To address this gap, the Sustaining Evidenced-Based Innovations through Multi-level Implementation Constructs (SEISMIC) project will: (a) Determine if malleable individual, intervention, and organizational factors predict EBP treatment fidelity and modifications during implementation, sustainment, or both; (b) Assess the impact of EBP fidelity and modifications on child outcomes during implementation and sustainment; and (c) Explore the mechanisms through which individual, intervention, and organizational factors influence sustainment outcomes. This protocol paper describes SEISMIC, which builds upon a federally-funded RCT evaluating BEST in CLASS, a teacher-delivered program for K-3rd grade children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools. A multi-level, interrupted time series design will be used to examine the relationship between baseline factors, treatment fidelity, modifications, and child outcomes, followed by a mixed-method approach to elucidate the mechanisms that influence sustainment outcomes. Findings will be used to create a strategy to improve EBP sustainment in schools.

Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment?

Background

Sustainability science is an emerging area within implementation science. There is limited evidence regarding strategies to best support the continued delivery and sustained impact of evidence-based interventions (EBIs). To build such evidence, clear definitions, and ways to operationalize strategies specific and/or relevant to sustainment are required. Taxonomies and compilations such as the Expert Recommendations for Implementing Change (ERIC) were developed to describe and organize implementation strategies. This study aimed to adapt, refine, and extend the ERIC compilation to incorporate an explicit focus on sustainment. We also sought to classify the specific phase(s) of implementation when the ERIC strategies could be considered and applied.

Methods

We used a two-phase iterative approach to adapt the ERIC. This involved: (1) adapting through consensus (ERIC strategies were mapped against barriers to sustainment as identified via the literature to identify if existing implementation strategies were sufficient to address sustainment, needed wording changes, or if new strategies were required) and; (2) preliminary application of this sustainment-explicit ERIC glossary (strategies described in published sustainment interventions were coded against the glossary to identify if any further amendments were needed). All team members independently reviewed changes and provided feedback for subsequent iterations until consensus was reached. Following this, and utilizing the same consensus process, the Exploration, Preparation, Implementation and Sustainment (EPIS) Framework was applied to identify when each strategy may be best employed across phases.

Results

Surface level changes were made to the definitions of 41 of the 73 ERIC strategies to explicitly address sustainment. Four additional strategies received deeper changes in their definitions. One new strategy was identified: Communicate with stakeholders the continued impact of the evidence-based practice. Application of the EPIS identified that at least three-quarters of strategies should be considered during preparation and implementation phases as they are likely to impact sustainment.

Conclusion

A sustainment-explicit ERIC glossary is provided to help researchers and practitioners develop, test, or apply strategies to improve the sustainment of EBIs in real-world settings. Whilst most ERIC strategies only needed minor changes, their impact on sustainment needs to be tested empirically which may require significant refinement or additions in the future.

Review article: A primer for clinical researchers in the emergency department: Part XII. Sustainability of improvements in care: An introduction

Despite an increased focus on ways to improve implementation of evidence and de-implementation of practices with no known benefit, there is limited guidance on how to sustain these improvements. This review provides an introduction to sustainability of improvements in care and sustainability research, discussing how to support sustainability in practice and detailing a sustainability research agenda for the emergency medicine setting.

Increasing Access to Organization Theories for Implementation Science

Background

Organization theories offer numerous existing, highly relevant, yet largely untapped explanations of the organizational dynamics underlying evidence-based intervention (EBI) implementation. Rooted in ideas regarding power, autonomy, and control, organization theories can explain how and why organizations adopt, implement, and sustain EBI use. Although they have gained visibility, organization theories remain underused in implementation research, perhaps due to their inaccessibility to implementation scientists. To improve access to organization theory among implementation scientists, we summarized organization theories with relevance to implementation science.

Methods

Led by the Cancer Prevention and Control Research Network (CPCRN) Organization Theory for Implementation Science workgroup, we employed a modified Delphi process to reach a consensus among 18 experts at the intersection of organization and implementation science regarding organization theories with relevance to implementation science. From texts that described the organization theories, using standardized abstraction forms, two investigators independently abstracted information regarding constructs, propositions regarding how or why constructs might influence implementation, the potential relevance of organization theories' propositions for implementation, and overviews of each theory. The investigators then reconciled discrepancies until reaching consensus. A third investigator reviewed reconciled abstraction forms for accuracy, coherence, and completeness.

Findings

We identified nine organization theories with relevance to implementation science: contingency, complexity, institutional, network, organizational learning, resource dependence, sociotechnical, and transaction cost economics. From the theories, we abstracted 70 constructs and 65 propositions. An example proposition from institutional theory is: "Coercive, mimetic, and normative pressures contribute to organizations…within an organizational field [becoming increasingly similar]." These propositions can be operationalized as levers to facilitate EBI implementation.

Conclusions

To increase use in the field, organization theories must be made more accessible to implementation scientists. The abstraction forms developed in this study are now publicly available on the CPCRN website with the goal of increasing access to organization theories among an interdisciplinary audience of implementation scientists through the CPCRN Scholars program and other venues. Next steps include consolidating organization theory constructs into domains and translating the resulting framework for use among researchers, policymakers and practitioners, aiding them in accounting for a comprehensive set of organization theory constructs thought to influence EBI implementation.

Capacity for sustainment of recently established syringe service programs in Appalachian Kentucky: The central role of staff champions

INTRODUCTION

Ensuring adequate harm reduction infrastructure in rural areas is imperative, as drug-related epidemics expand into them. Here, we explore the capacity for sustainment of syringe service programs (SSP) in Appalachian Kentucky.

METHODS

We interviewed all staff (n = 16) of all SSPs (n = 7) in two Kentucky health districts in 2018-2019 using semi-structured one-on-one qualitative interviews; local departments of health (DOH) operated the SSPs. Interview domains encompassed: (i) SSP establishment; (ii) day-to-day operations, participation and health impacts; (iii) perceived prospects for sustainment; and (iv) perceived influences on #i-#iii. We analysed verbatim transcripts using thematic analytic methods; Schell's 'capacity for sustainment' constructs were treated as sensitising concepts during the analysis.

RESULTS

Most community members, law enforcement and DOH staff opposed SSPs before they opened, because of stigma and concerns about enabling and needlestick injuries; DOH staff also opposed SSPs because they believed they lacked the capacity to operate them. Training, technical assistance, visible evidence of the programs' public health impact and contact with SSP participants transformed DOH staff into program champions. As champions, SSP staff developed programs that had strong capacity for sustainment, as defined by Schell (e.g. visible public health impact, stable funding, political support). Staff reported that the SSPs had high prospects for sustainment.

DISCUSSION AND CONCLUSION

As in SSPs that opened in cities decades ago, staff in emerging SSPs in these rural areas appear to have become crucial champions for these controversial programs, and may serve as vital resources for expanding harm reduction programming more broadly in these underserved areas.

Stakeholder perspectives on sustainment of Housing First in a VA permanent supportive housing program

OBJECTIVE

To evaluate the sustainment of Housing First (HF) implementation in a permanent supportive housing program for homeless-experienced veterans, 5 years after practice implementation.

STUDY SETTING

From 2016 to 2017, primary data were collected from providers and veterans in the Department of Housing and Urban Development-VA Supportive Housing (HUD-VASH) program at Los Angeles.

STUDY DESIGN

Guided by the integrated sustainability framework, we performed a mixed-methods study to evaluate the sustainment of HF, an evidence-based practice implemented to improve housing outcomes. To assess sustainment, we measured fidelity to HF in six of seven HUD-VASH teams. These data were integrated with qualitative interviews with providers and veterans who described perceived sustainment to HF and contextual factors that supported or impeded sustainment.

DATA COLLECTION

Fidelity to HF at 5 years after practice implementation, as a proxy for sustainment, was quantified via surveys with HUD-VASH teams. HUD-VASH providers (n = 51) and 31 veterans participated in semi-structured interviews. Team-based template analyses were used to develop an emergent understanding of stakeholder perspectives on HF sustainment.

PRINCIPAL FINDINGS

Overall, HUD-VASH teams reported HF sustainment. The lowest fidelity scores were found in the domains of client-to-staff ratios, frequency of client-provider contact, and time to housing. Qualitative findings indicated that outer contextual factors (e.g., housing scarcity) and organizational factors (e.g., staff turnover) impacted HF sustainment. Providers identified changes in leadership and unmet resource needs as impediments to practice sustainment. All stakeholders identified positively with the HF practice and believed that the approach benefited veterans.

CONCLUSIONS

This snapshot of HF sustainment demonstrates that this practice can be sustained over time. However, strong leadership, organizational resources, and community partnerships are needed. Adaptations to HF in response to outer contextual factors and organizational capacity may result in practice sustainment while allowing for flexibility in service provision.

Implementation of the Wolverine Mental Health Program, Part 3: Sustainment Phase

Sustaining the implementation of an evidence-based practice (EBP) is the ultimate goal of often years of significant personnel and financial investment. Some conceptualize sustainment as a distinct phase following an active implementation period where the contextual factors, processes, and supports are bolstered to ensure continued EBP delivery. This study provides an overview of the sustainment strategies deployed to embed cognitive-behavioral therapy (CBT) in a Midwestern residential treatment facility serving youth with complex mental health needs. Seven key strategies and their outcomes are described: use of CBT teams, new hire orientation plans, monthly campaigns, change in job descriptions and performance evaluations, development of a behavioral reinforcement system for youth, and a pathway to CBT certification. This study provides a window into how one might sustain an EBP by addressing barriers unique to this phase of work.

Sustaining acute speech-language therapists' implementation of recommended aphasia practices: A mixed methods follow-up evaluation of a cluster RCT

BACKGROUND

While implementation studies in aphasia management have shown promising improvements to clinical practice, it is currently unknown if aphasia implementation outcomes are sustained and what factors may influence clinical sustainability.

AIMS

To evaluate the sustainment (i.e., sustained improvement of aphasia management practices and domains influencing clinicians' practice) and sustainability (i.e., factors influencing sustainability) outcomes of the Acute Aphasia IMplementation Study (AAIMS).

METHODS & PROCEDURES

A convergent interactive mixed-methods sustainability evaluation was conducted on two previously delivered implementation interventions (AAIMS). The AAIMS interventions were targeted at improving either written aphasia-friendly information provision (Intervention A) or collaborative goal-setting (Intervention B). Outcomes were collected 2 and 3 years post-implementation, addressing the research questions of sustainment (e.g., medical record audits and behavioural constructs questionnaires) and sustainability (e.g., post-study focus groups and organizational readiness surveys). Quantitative sustainability data were compared with post-implementation data, allowing for sustainment to be determined. Clinicians' perspectives on sustainability outcomes and challenges were analysed using framework analysis and integrated with the quantitative findings.

OUTCOMES & RESULTS

A total of 35 speech-language therapists (SLTs) from four hospitals participated. The medical records of 79 patients were audited in the sustainability period compared with the 107 medical records audited during AAIMS. Overall, there was variable sustainment of the target behaviours; implementation for Intervention A was not sustained at either sustainability time point (2018 = 47.8% decrease; 2019 = 22.78% decrease), but implementation for Intervention B was sustained at both time points (2018 = 7.78% increase; 2019 = 18.1% increase). There was a pattern of sustained change in the behaviour change domains targeted by the implementation interventions, where scores of the targeted domains increased over time (0.13, 95% confidence interval (CI) = -0.05 to 0.30) and scores of the non-targeted domains declined (-0.03, 95% CI = -0.11 to 0.04). Factors influencing sustainability were mainly related to 'processes', 'the inner context' and 'SLT characteristics', and these interacted dynamically to account for variation between teams.

CONCLUSIONS & IMPLICATIONS

Implementation outcomes (i.e., practice changes) were not sustained to the same level for three of the four participating SLT teams, with variable or partial sustainment most common. While the factors influencing sustainability differed depending on the context and individuals involved, the most important factor influencing outcomes seemed to be the level to which behaviour-change processes and strategies were embedded within the SLT department. Future implementation studies should incorporate sustainability measures from the onset and include follow-ups and monitoring systems to help support sustained change in the long term.

WHAT THIS PAPER ADDS

What is already known on the subject In post-stroke aphasia management, there are few examples of long-term sustainability of implementation outcomes. It is therefore unknown what factors are potentially important to sustain implementation of best-practice recommendations in aphasia services. What this paper adds to existing knowledge There is potential for implementation outcomes to be sustained long term, but sustainment is impacted by a range of factors. Ongoing facilitation or follow-up after initial implementation may to useful to promote sustainment, but is not essential if processes are sufficiently embedded. Gradual implementation into practice may lead to better sustainment than rapid change that is quickly forgotten. What are the potential or actual clinical implications of this work? Future implementation efforts should incorporate sustainability measures from the onset. Applying a sustainability framework was useful to guide evaluations and explore factors influencing the sustainment outcomes and is recommended for those interested in sustainability. Results from our evaluation can be used to guide refinement and support future development of sustainable implementation interventions.

Eleven Reasons for Adaptation of Swedish Parenting Programs

While questions about adaptation and fidelity are of great concern in many implementation projects, less attention has been paid to reasons for adaptations that remain when evidence-based interventions (EBIs) are used in clinical and community settings. This study aims to explore reasons for adaptations that can arise when using parenting programs in a community setting. Seventeen individual interviews with providers were conducted and analyzed thematically, resulting in 11 reasons for adaptations organized into four separate areas: characteristics of group leaders (supplementary skills and knowledge, preferred ways of working), characteristics of families (problem complexity, diverse or limited educational experience, non-parenting needs for support, colliding value systems), group incidents (criticism and challenges, excessive questions or discussions), and didactic challenges (lack of focus or engagement, limitations of the material, language differences). The study shows that factors triggering adaptation and fidelity decisions continuously reappear in the provision of parenting programs in community settings. Knowledge about reasons for adaptation can be used to inform decision-making during implementation planning, as well as the sustainment of implemented interventions.

Sustainment of Contingency Management within Opioid Treatment Programs: COVID-Related Barriers and Innovative Workflow Adaptations

Introduction

Contingency Management (CM) is one of the most effective interventions for persons with opioid use disorder, but one of the least available interventions in community settings, including opioid treatment programs. Project MIMIC is a NIDA-funded cluster randomized trial that is measuring CM implementation and sustainment across 30 opioid treatment programs in the New England region of the United States. The advent of the COVID-19 pandemic occurred in the midst of Project MIMIC's first cohort of eight opioid treatment programs, presenting a natural opportunity to document and analyze novel challenges to CM sustainment. Utilizing both quantitative and qualitative data collection, we aimed to identify both COVID-related barriers to CM sustainment and innovative workflow strategies to mitigate these barriers.

Methods

Quantitative analysis was conducted using data collected from a study-specific CM tracker tool on various CM implementation metrics over three distinct, successive time intervals: prior to COVID-19 social distancing orders with active support; during COVID-19 social distancing orders with active support; and during COVID-19 social distancing orders after removal of support. Semi-structured qualitative interviews were conducted with a representative from each of the eight opioid treatment programs. Using a reflexive team approach, transcripts were coded by independent raters to identify both COVID-related barriers to sustainment and innovative workflow adaptations.

Results

Quantitative data revealed a substantial decrease in the number of CM encounters following social distancing orders from 31.8 encounters weekly across eight programs to 6.9 encounters weekly across five programs. A further decline to 1.8 weekly encounters across three programs was observed after implementation support was removed. Four COVID-related barriers were identified via thematic analysis: fear of contagion; difficulty engaging patients remotely; challenges re-defining the CM attendance target due to changing regulations; and staff shortages. Potential adjustments discussed to help address one or more of these barriers included an electronic prize generator; use of technology to promote engagement; brief individual remote check-ins; and expansion of training to non-counseling staff.

Conclusion

Although CM implementation challenges emerged during the pandemic, associated workflow adaptations also emerged. The feedback solicited in this study will inform multi-level strategies to aid with CM sustainment post-pandemic.

Adaptation and Validation of the Program Sustainability Assessment Tool (PSAT) for Use in the Elementary School Setting

There is a lack of valid and reliable measures of determinants of sustainability specific to public health interventions in the elementary school setting. This study aimed to adapt and evaluate the Program Sustainability Assessment Tool (PSAT) for use in this setting. An expert reference group adapted the PSAT to ensure face validity. Elementary school teachers participating in a multi-component implementation intervention to increase their scheduling of physical activity completed the adapted PSAT. Structural validity was assessed via confirmatory factor analysis. Convergent validity was assessed using linear mixed regression evaluating the associations between scheduling of physical activity and adapted PSAT scores. Cronbach's alpha was used to evaluate internal consistency and intracluster correlation coefficients for interrater reliability. Floor and ceiling effects were also evaluated. Following adaptation and psychometric evaluation, the final measure contained 26 items. Domain Cronbach's alpha ranged from 0.77 to 0.92. Only one domain illustrated acceptable interrater reliability. Evidence for structural validity was mixed and was lacking for convergent validity. There were no floor and ceiling effects. Efforts to adapt and validate the PSAT for the elementary school setting were mixed. Future work to develop and improve measures specific to public health program sustainment that are relevant and psychometrically robust for elementary school settings are needed.

Sustainability of a biobehavioral intervention implemented by therapists and sustainment in community settings

The ultimate aim of dissemination and implementation of empirically supported treatments (ESTs) in behavioral medicine is (a) sustainability of the therapist/provider's EST usage and (b) sustainment of EST delivery in the setting. Thus far, sustainability has been understudied, and the therapist and setting variables that may be influential are unclear. The purpose of the study was to test the therapists' sustainability of a cancer-specific EST using a prospective longitudinal design and examine its predictors. Oncology mental health therapists (N = 134) from diverse settings (N = 110) completed training in the biobehavioral intervention (BBI) and were provided with 6 months of support for implementation, with no support thereafter. BBI usage (percent of patients treated) was reported at 2, 4, 6, and 12 months. Using a generalized estimating equation with a logistic link function, 12-month sustainability (a nonsignificant change in usage from 6 to 12 months) was studied along with therapist, supervisor, and setting variables as predictors. BBI usage increased through 6 months and, importantly, usage was sustained from 6 (68.4% [95% CI = 62.2%-73.9%]) to 12 months (70.9% [95% CI = 63.6%-77.3%]), with sustainment in 66 settings (60.0%). Predictors of implementation-to-sustainability usage were therapists' early intentions to use the BBI (p < .001) and from the setting, supervisors' positive attitudes toward ESTs (p = .016). Adding to the DI literature, a health psychology intervention was disseminated, implemented, and found sustainable across diverse therapists and settings. Therapists and setting predictors of usage, if modified, might facilitate future sustainability/sustainment of ESTs.

Cost analysis of a remote patient monitoring programme for post-discharge patients with type 2 diabetes

INTRODUCTION

Assessing costs of an evidence-based health promotion programme is crucial to understand the economic feasibility of adopting or sustaining the programme. This study conducted a cost analysis of a remote patient monitoring (RPM) programme to enhance the post-discharge management of type 2 diabetes.

METHODS

Using retrospective data collected during RPM implementation from September 2014 to February 2018, we estimated the costs of implementing an RPM in the primary care setting. Measures included total and average annual costs, costs per participant who was enrolled or completed the programme, and costs per person-day. We further conducted sensitivity and scenario analyses to examine variations in estimated programme costs associated with varying programme efficiencies and alternative personnel compositions of the RPM team.

RESULTS

The total RPM implementation costs were estimated at US$4,374,544 with an average annual programme costs of US$1,249,870, which translated to US$3207 per participant (n = 1364) completing the three-month programme. The per person-day cost was averaged at US$24 (182,932 person-days). Sensitivity and scenario analyses results indicate that the sustainment costs were approximately US$1.6 million annually and the per-person-day costs were between US$21 and US$29 with each nurse coach on average serving a panel of 62-93 patients.

CONCLUSION

The implementation and sustainment costs of an RPM programme, estimated under various assumptions of programme efficiency and care team compositions, as exemplified in this study, will help healthcare organizations make informed decisions in budgeting for and sustaining telehealth programmes to enhance diabetes management.

Train-to-Sustain: Predictors of Sustainment in a Large-Scale Implementation of Parent-Child Interaction Therapy

Sustainment of evidence-based practices is necessary to ensure their public health impact. The current study examined predictors of sustainment of Parent-Child Interaction Therapy (PCIT) within a large-scale system-driven implementation effort in Los Angeles County. Data were drawn from PCIT training data and county administrative claims between January 2013 and March 2018. Participants included 241 therapists from 61 programs. Two sustainment outcomes were examined at the therapist- and program-levels: 1) PCIT claim volume and 2) PCIT claim discontinuation (discontinuation of claims during study period; survival time of claiming in months). Predictors included therapist- and program-level caseload, training, and workforce characteristics. On average, therapists and programs continued claiming to PCIT for 17.7 and 32.3 months, respectively. Across the sustainment outcomes, there were both shared and unshared significant predictors. For therapists, case-mix fit (higher proportions of young child clients with externalizing disorders) and participation in additional PCIT training activities significantly predicted claims volume. Furthermore, additional training activity participation was associated with lower likelihood of therapist PCIT claim discontinuation in the follow-up period. Programs with therapists eligible to be internal trainers were significantly less likely to discontinue PCIT claiming. Findings suggest that PCIT sustainment may be facilitated by implementation strategies including targeted outreach to ensure eligible families in therapist caseloads, facilitating therapist engagement in advanced trainings, and building internal infrastructure through train-the-trainer programs.

Prospective predictors of sustainment of multiple EBPs in a system-driven implementation context: Examining sustained delivery based on administrative claims

Background

System-driven scale-up of multiple evidence-based practices (EBPs) is an increasingly common method used in public mental health to improve care. However, there are little data on the long-term sustained delivery of EBPs within these efforts, and previous studies have relied on retrospective self-report within cross-sectional studies. This study identified prospective predictors of sustained EBP delivery at the EBP-, therapist-, and organizational-levels using survey and administrative claims data within a large-scale system-driven implementation effort.

Methods

777 therapists and 162 program leaders delivering at least one of six EBPs of interest completed surveys assessing perceptions of EBPs and organizational context. These surveys were linked to administrative data to examine prospective predictors of therapists' EBP delivery over 33 months.

Results

Five of the six EBPs implemented showed sustained delivery in the system, with volume varying by EBP. Although total EBP claim volume per therapist decreased over time, the volume ratio (ratio of EBP-specific claims to total EBP and non-EBP claims) stayed relatively stable. Multilevel models revealed that EBPs that required consultation, had unstructured content, higher therapist self-efficacy with the EBP, and more positive program leader perceptions of the EBP were associated with greater sustained volume and volume ratio of the EBP. Therapists who were trained in fewer EBPs, who were unlicensed, and who worked in agencies rated by program leaders as lower on organizational staff autonomy and stress showed greater sustained EBP volume and volume ratio. Finally, more direct service hours per week provided by therapist predicted greater sustained EBP volume, but lower volume ratio.

Conclusions

The results point to the importance of EBP, therapist, and organizational factors that may be targeted in implementation strategies to promote the sustainment of EBPs.

Implementing Battlefield Acupuncture Through a Large Medical System: Overcoming Barriers

Introduction: Developing a new and innovative acupuncture technique to deal with acute and chronic pain is a considerable undertaking in itself as many barriers must be overcome. Designing the technique and gaining acceptance is even more daunting and challenging across a large system of medical facilities such as the Department of Defense and the Veterans Administration. Such was the case of Battlefield Acupuncture (BFA), a 5-needle placement in each external ear into specific auricular points to treat acute and chronic pain. Discussion: This technique geared for military deployment is rapid, safe, and easily reproducible without side effects because it uses a choice acupuncture microsystem, namely the ear. Acquiring acupuncture practical skills is the prerequisite before designing the technique and then taking into consideration, neurophysiology data, and actual clinical experience to finally focus development is demanding. Conclusions: The benefits of BFA that was promulgated in a Joint Incentive Fund for $5.4 million resulted in thousands of trained health care providers and eventually many clinical trials to help validate the technique. Process improvement data were beneficial and important for its sustainment. There are 5 steps recommended for overcoming the barriers.

An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of "maintenance" within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and "evolvability" across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

What Influences Sustainment and Nonsustainment of Facilitation Activities in Implementation? Analysis of Organizational Factors in Hospitals Implementing TeamSTEPPS

Implementation processes are often long and complex, requiring sustained facilitation efforts. Drawing on organizational and implementation literature, we examined the influence of senior management support (SMS), middle management support (MMS), facilitator team time availability (TIME) and team continuity (CONTINUITY) on sustainment of internal facilitation activities. For 2 years, we followed 10 small rural hospitals implementing TeamSTEPPS, a patient safety program, and conducted quarterly interviews with key informants. We coded, calibrated, and analyzed the data using the fuzzy-set qualitative comparative analysis. We found that five hospitals sustained facilitation activities and the combination of SMS, MMS, and CONTINUITY (i.e., presence of all three factors) was a sufficient condition for sustainment. Five other hospitals did not sustain facilitation activities and they either lacked MMS or lacked both TIME and CONTINUITY. In follow-up analyses, we found that team leadership continuity also influenced sustainment patterns. We discussed the implications for research and practice.

Using Survival Analysis to Understand Patterns of Sustainment within a System-Driven Implementation of Multiple Evidence-Based Practices for Children's Mental Health Services

Evidence-based practice (EBP) implementation requires substantial resources in workforce training; yet, failure to achieve long-term sustainment can result in poor return on investment. There is limited research on EBP sustainment in mental health services long after implementation. This study examined therapists’ continued vs. discontinued practice delivery based on administrative claims for reimbursement for six EBPs [Cognitive Behavioral Interventions for Trauma in Schools (CBITS), Child–Parent Psychotherapy, Managing and Adapting Practices (MAP), Seeking Safety (SS), Trauma-Focused Cognitive Behavior Therapy (TF-CBT), and Positive Parenting Program] adopted in a system-driven implementation effort in public mental health services for children. Our goal was to identify agency and therapist factors associated with a sustained EBP delivery. Survival analysis (i.e., Kaplan–Meier survival functions, log-rank tests, and Cox regressions) was used to analyze 19 fiscal quarters (i.e., approximately 57 months) of claims data from the Prevention and Early Intervention Transformation within the Los Angeles County Department of Mental Health. These data comprised 2,322,389 claims made by 6,873 therapists across 88 agencies. Survival time was represented by the time elapsed from therapists’ first to final claims for each practice and for any of the six EBPs. Results indicate that therapists continued to deliver at least one EBP for a mean survival time of 21.73 months (median = 18.70). When compared to a survival curve of the five other EBPs, CBITS, SS, and TP demonstrated a higher risk of delivery discontinuation, whereas MAP and TF-CBT demonstrated a lower risk of delivery discontinuation. A multivariate Cox regression model revealed that agency (centralization and service setting) and therapist (demographics, discipline, and case-mix characteristics) characteristics were significantly associated with risk of delivery discontinuation for any of the six EBPs. This study illustrates a novel application of survival analysis to administrative claims data in system-driven implementation of multiple EBPs. Findings reveal variability in the long-term continuation of therapist-level delivery of EBPs and highlight the importance of both agency and workforce characteristics in the sustained delivery of EBPs. Findings direct the field to potential targets of sustainment interventions (e.g., strategic assignment of therapists to EBP training and strategic selection of EBPs by agencies).

Implementing and Sustaining Team-Based Telecare for Bipolar Disorder: Lessons Learned from a Model-Guided, Mixed Methods Analysis

BACKGROUND

Telemental health interventions have empirical support from clinical trials and structured demonstration projects. However, their implementation and sustainability under less structured clinical conditions are not well demonstrated.

INTRODUCTION

We conducted a follow-up analysis of the implementation and sustainability of a clinical video teleconference-based collaborative care model for individuals with bipolar disorder treated in the Department of Veterans Affairs to (a) characterize the extent of implementation and sustainability of the program after its establishment and (b) identify barriers and facilitators to implementation and sustainability.

MATERIALS AND METHODS

We conducted a mixed methods program evaluation, assessing quantitative aspects of implementation according to the Reach, Efficacy, Adoption, Implementation, and Maintenance implementation framework. We conducted qualitative analysis of semistructured interviews with 16 of the providers who submitted consults, utilizing the Integrated Promoting Action on Research Implementation in the Health Services implementation framework.

RESULTS

The program demonstrated linear growth in sites (n = 35) and consults (n = 915) from late 2011 through mid-2016. Site-based analysis indicated statistically significant sustainability beyond the first year of operation. Qualitative analysis identified key facilitators, including consult content, ease of use via electronic health record, and national infrastructure. Barriers included availability of telehealth space, equipment, and staff at the sites, as well as the labor-intensive nature of scheduling.

DISCUSSION

The program achieved continuous growth over almost 5 years due to (1) successfully filling a need perceived by providers, (2) developing in a supportive context, and (3) receiving effective facilitation by national and local infrastructure.

CONCLUSION

Clinical video teleconference-based interventions, even multicomponent collaborative care interventions for individuals with complex mental health conditions, can grow vigorously under appropriate conditions.

An Overview of Research and Evaluation Designs for Dissemination and Implementation

The wide variety of dissemination and implementation designs now being used to evaluate and improve health systems and outcomes warrants review of the scope, features, and limitations of these designs. This article is one product of a design workgroup that was formed in 2013 by the National Institutes of Health to address dissemination and implementation research, and whose members represented diverse methodologic backgrounds, content focus areas, and health sectors. These experts integrated their collective knowledge on dissemination and implementation designs with searches of published evaluations strategies. This article emphasizes randomized and nonrandomized designs for the traditional translational research continuum or pipeline, which builds on existing efficacy and effectiveness trials to examine how one or more evidence-based clinical/prevention interventions are adopted, scaled up, and sustained in community or service delivery systems. We also mention other designs, including hybrid designs that combine effectiveness and implementation research, quality improvement designs for local knowledge, and designs that use simulation modeling.

De-adoption of an evidence-based trauma intervention in schools: A retrospective report from an urban school district

The de-adoption of evidence-based practices (EBPs) is a largely understudied topic. The present study examined factors related to the de-adoption of an EBP for students exposed to traumatic events in a large urban school district. Qualitative interviews conducted with school clinicians and district administrators two years after the district embarked on a large-scale roll-out of the EBP distinguished between factors that impacted partial de-adoption after one year (phase 1) and complete de-adoption by the district after two years (phase 2). Phase 1 factors included organizational consistency, workforce stability, prior success, positive student outcomes, school- and district- level supports, innovation-setting fit, and innovation-related issues. Phase 2 factors included district-level leadership changes, financial and workforce instability, and shifting priorities. Study results suggest that sustainment-enhancing strategies should be included in the early stages of program implementation to most effectively adapt to school- and system- level changes.

A systematic review: barriers to hourly rounding

AIM

The purpose of this review of the literature is to identify a comprehensive list of the barriers that impact the effective implementation and sustainment of hourly rounding on the adult inpatient medical or surgical unit.

BACKGROUND

Despite the benefits associated with hourly rounding, challenges with the implementation and sustainment of this intervention have been reported on medical-surgical units. Some hospitals have multiple campaigns to implement rounding, yet sustainment eludes them.

EVALUATION

Four databases were searched independently by two reviewers. Articles were screened according to the inclusion/exclusion criteria and 20 articles resulted.

KEY ISSUES

Clear themes of barriers emerged across the literature consisting of: workload issues, burdensome rounding logs, missing staff buy-in, patient acuity levels, lack of adequate staff education and a lack of sustainability.

CONCLUSIONS

The barriers identified are all directly related and perceived by the front line staff that is performing hourly rounding. The barriers identified may be limiting the organisation's ability to sustain the hourly rounding process.

IMPLICATIONS FOR NURSING MANAGEMENT

Leaders can utilise this information to analyse and create successful options to overcome these barriers before, during and after the implementation process. Further studies are needed to identify targeted strategies to alleviate these barriers.

Clearing Hurdles: The Challenges of Implementation of Mental Health Evidence-Based Practices in Under-resourced Schools

Schools have become the main provider of services to children with mental health needs. Although there is substantial literature on barriers to implementation of evidence-based practices (EBPs) in under-resourced school districts, less has been written on how to overcome those barriers. Providing mental health services in the school setting presents a tremendous opportunity to increase access to quality mental health care for underserved youth. This review provides a brief overview of the barriers to successful implementation and sustainment of EBPs in under-resourced public schools and provides recommendations for overcoming them. The discussion is organized around an established conceptual framework adapted for the delivery of services in under-resourced schools that focuses on interdependent factors that exist at the individual-, team, school-, and macro-levels. This manuscript explores some recommendations and strategies for effectively addressing challenges related to implementation of EBPs. Research ideas are offered to bridge the research-to-practice gap that impacts many under-resourced public school districts.

Examining the sustainment of the Adolescent-Community Reinforcement Approach in community addiction treatment settings: protocol for a longitudinal mixed method study

BACKGROUND

Although evidence-based treatments are considered the gold standard for clinical practice, it is widely recognized that evidence-based treatment implementation in real world practice settings has been limited. To address this gap, the federal government provided three years of funding, training and technical assistance to 84 community-based treatment programs to deliver an evidence-based treatment called the Adolescent-Community Reinforcement Approach (A-CRA). Little is known about whether such efforts lead to long-term A-CRA sustainment after the initial funding ends.

METHODS/DESIGN

We will use a longitudinal mixed method data analytic approach to characterize sustainment over time and to examine the factors associated with the extent to which A-CRA is sustained. We will use implementation data collected during the funding period (e.g., organizational functioning, staff certification rates and penetration) and supplement it with additional data collected during the proposed project period regarding implementation quality and the hypothesized predictors of sustainment (i.e., inner and outer contextual variables) collected over three waves from 2013 to 2015 representing program sustainment up to five years post-initial funding.

DISCUSSION

Gaining a better understanding of the factors that influence the evidence-based treatment sustainment may lead to more effective dissemination strategies and ultimately improve the quality of care being delivered in community-based addiction treatment settings.