Predicting the long-term sustainability of evidence-based practices in mental health care: an 8-year longitudinal analysis

Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study

BACKGROUND

Sustaining evidence-based practices (EBPs) is crucial to ensuring care quality and addressing health disparities. Approaches to identifying factors related to sustainability are critically needed. One such approach is Matrixed Multiple Case Study (MMCS), which identifies factors and their combinations that influence implementation. We applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM) at nine Department of Veterans Affairs (VA) outpatient mental health clinics, 3-4 years after implementation support had concluded.

METHODS

We conducted a directed content analysis of 30 provider interviews, using 6 CCM elements and 4 Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) domains as codes. Based on CCM code summaries, we designated each site as high/medium/low sustainability. We used i-PARIHS code summaries to identify relevant factors for each site, the extent of their presence, and the type of influence they had on sustainability (enabling/neutral/hindering/unclear). We organized these data into a sortable matrix and assessed sustainability-related cross-site trends.

RESULTS

CCM sustainability status was distributed among the sites, with three sites each being high, medium, and low. Twenty-five factors were identified from the i-PARIHS code summaries, of which 3 exhibited strong trends by sustainability status (relevant i-PARIHS domain in square brackets): "Collaborativeness/Teamwork [Recipients]," "Staff/Leadership turnover [Recipients]," and "Having a consistent/strong internal facilitator [Facilitation]" during and after active implementation. At most high-sustainability sites only, (i) "Having a knowledgeable/helpful external facilitator [Facilitation]" was variably present and enabled sustainability when present, while (ii) "Clarity about what CCM comprises [Innovation]," "Interdisciplinary coordination [Recipients]," and "Adequate clinic space for CCM team members [Context]" were somewhat or less present with mixed influences on sustainability.

CONCLUSIONS

MMCS revealed that CCM sustainability in VA outpatient mental health clinics may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators. These findings have informed a subsequent CCM implementation trial that prospectively examines whether enhancing the above-mentioned factors within implementation facilitation improves sustainability. MMCS is a systematic approach to multi-site examination that can be used to investigate sustainability-related factors applicable to other EBPs and across multiple contexts.

Built to last? Barriers and facilitators of healthcare program sustainability: a systematic integrative review

OBJECTIVE

To identify barriers and facilitators associated with the sustainability of implemented and evaluated improvement programs in healthcare delivery systems.

DATA SOURCES AND STUDY SETTING

Six academic databases were searched to identify relevant peer-reviewed journal articles published in English between July 2011 and June 2022. Studies were included if they reported on healthcare program sustainability and explicitly identified barriers to, and facilitators of, sustainability.

STUDY DESIGN

A systematic integrative review guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Study quality was appraised using Hawker's Quality Assessment Tool.

DATA COLLECTION/EXTRACTION METHODS

A team of reviewers screened eligible studies against the inclusion criteria and extracted the data independently using a purpose-designed Excel spreadsheet. Barriers and facilitators were extracted and mapped to the Integrated Sustainability Framework (ISF). Frequency counts of reported barriers/facilitators were performed across the included studies.

RESULTS

Of the 124 studies included in this review, almost half utilised qualitative designs (n = 52; 41.9%) and roughly one third were conducted in the USA (n = 43; 34.7%). Few studies (n = 29; 23.4%) reported on program sustainability beyond 5 years of program implementation and only 16 of them (55.2%) defined sustainability. Factors related to the ISF categories of inner setting (n = 99; 79.8%), process (n = 99; 79.8%) and intervention characteristics (n = 72; 58.1%) were most frequently reported. Leadership/support (n = 61; 49.2%), training/support/supervision (n = 54; 43.5%) and staffing/turnover (n = 50; 40.3%) were commonly identified barriers or facilitators of sustainability across included studies. Forty-six (37.1%) studies reported on the outer setting category: funding (n = 26; 56.5%), external leadership by stakeholders (n = 16; 34.8%), and socio-political context (n = 14; 30.4%). Eight studies (6.5%) reported on discontinued programs, with factors including funding and resourcing, poor fit, limited planning, and intervention complexity contributing to discontinuation.

CONCLUSIONS

This review highlights the importance of taking into consideration the inner setting, processes, intervention characteristics and outer setting factors when sustaining healthcare programs, and the need for long-term program evaluations. There is a need to apply consistent definitions and implementation frameworks across studies to strengthen evidence in this area.

TRIAL REGISTRATION

https://bmjopen.bmj.com/content/7/11/e018568 .

Sustainability of the Collaborative Chronic Care Model in Outpatient Mental Health Teams Three Years Post-Implementation: A Qualitative Analysis

Our goal was to investigate the sustainability of care practices that are consistent with the collaborative chronic care model (CCM) in nine outpatient mental health teams located within US Department of Veterans Affairs (VA) medical centers, three to four years after the completion of CCM implementation. We conducted qualitative interviews (N = 30) with outpatient mental health staff from each of the nine teams. We based our directed content analysis on the six elements of the CCM. We found variable sustainability of CCM-based care processes across sites. Some care processes, such as delivery of evidence-based psychotherapies (EBPs) and use of measurement-based care (MBC) to guide clinic decision-making, were robustly maintained or even expanded within participating teams. In contrast, other care processes-which had in some cases been developed with considerable effort-had not been sustained. For example, care manager roles were diminished in scope or eliminated completely in response to workload pressures, frontline care needs, or the COVID-19 pandemic. Similarly, processes for engaging Veterans more fully in decision-making had generally been scaled back. Leadership support in the form of adequate team staffing and time to conduct team meetings were seen as crucial for sustaining CCM-consistent care. Given the potential impact of leadership turnover on sustainability in mental health, future efforts to implement CCM-based mental health care should strive to involve multiple leaders in implementation and sustainment efforts, lest one key departure undo years of implementation work. Our results also suggest that implementing CCM processes may best be conceptualized as a partnership across multiple levels of medical center leadership.

Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention

Background

In order to achieve wide-scale impact in community settings, programs must be sustained. Theory and empirical evidence suggest that intervention characteristics, organizational context, capacity for program implementation, and processes related to implementation are associated with continued program delivery. However, few studies examine how combinations of these factors work together in different settings to influence program sustainment.

Methods

Using scales specified in the Program Sustainability Assessment Tool (PSAT), the current cross-sectional study aims to identify the necessary and sufficient conditions for the sustainment of the Strengthening Families Program for Parents and Youth 10-14 (SFP 10-14). Staff (n = 59) at SFP 10-14 implementation sites across Washington State completed an online survey reporting on their current level of SFP 10-14 sustainment. They also completed PSAT, with eight scales designed to assess conditions that consistently produce sustainment. Data were analyzed using qualitative comparative analysis.

Results

Environmental support was the only necessary condition for sustainment success. Four solutions sufficient to achieve sustainment were also identified. These included the combined presence of (1) environmental support, organizational capacity, and funding stability; (2) environmental support, organizational capacity, communication, and program evaluation, in the absence of strategic planning; (3) environmental support, organizational capacity, program evaluation, and partnerships, in the absence of strategic planning; and (4) environmental support, communication, partnerships, and funding stability, in the absence of program evaluation.

Conclusions

Environmental support in combination with organizational capacity appeared to most consistently produce sustainment of SFP 10-14 programs in Washington State. Program providers will benefit from a focusing on enhancing those conditions to increase program sustainment.

Exploring the impact of workforce turnover on the sustainability of evidence-based programs: A scoping review

Background

Evidence-based programs (EBPs) are used across disciplines to integrate research into practice and improve outcomes at the individual and/or community level. Despite widespread development and implementation of EBPs, many programs are not sustained beyond the initial implementation period due to many factors, including workforce turnover. This scoping review summarizes research on the impact of workforce turnover on the sustainability of EBPs and recommendations for mitigating these impacts.

Methods

We searched 10 databases for articles that focused on an EBP and described an association between workforce turnover and the sustainment or sustainability of the program. We created a data abstraction tool to extract relevant information from each article and applied the data abstraction tool to all included articles to create the dataset. Data were mapped and analyzed using the program sustainability framework (PSF).

Results and Discussion

A total of 30 articles were included in this scoping review and mapped to the PSF. Twenty-nine articles described impacts of workforce turnover and 18 articles proposed recommendations to address the impacts. The most frequent impacts of workforce turnover included increased need for training, loss of organizational knowledge, lack of EBP fidelity, and financial stress. Recommendations to address the impact of workforce turnover included affordable and alternative training modalities, the use of champions or volunteers, increasing program alignment with organizational goals, and generating diverse funding portfolios.

Conclusion

The sustainment of EBPs is critical to ensure and maintain the short- and long-term benefits of the EBP for all participants and communities. Understanding the impacts of workforce turnover, a determinant of sustainability, can create awareness among EBP-implementing organizations and allow for proactive planning to increase the likelihood of program sustainability.

Sustainability of a biobehavioral intervention implemented by therapists and sustainment in community settings

The ultimate aim of dissemination and implementation of empirically supported treatments (ESTs) in behavioral medicine is (a) sustainability of the therapist/provider's EST usage and (b) sustainment of EST delivery in the setting. Thus far, sustainability has been understudied, and the therapist and setting variables that may be influential are unclear. The purpose of the study was to test the therapists' sustainability of a cancer-specific EST using a prospective longitudinal design and examine its predictors. Oncology mental health therapists (N = 134) from diverse settings (N = 110) completed training in the biobehavioral intervention (BBI) and were provided with 6 months of support for implementation, with no support thereafter. BBI usage (percent of patients treated) was reported at 2, 4, 6, and 12 months. Using a generalized estimating equation with a logistic link function, 12-month sustainability (a nonsignificant change in usage from 6 to 12 months) was studied along with therapist, supervisor, and setting variables as predictors. BBI usage increased through 6 months and, importantly, usage was sustained from 6 (68.4% [95% CI = 62.2%-73.9%]) to 12 months (70.9% [95% CI = 63.6%-77.3%]), with sustainment in 66 settings (60.0%). Predictors of implementation-to-sustainability usage were therapists' early intentions to use the BBI (p < .001) and from the setting, supervisors' positive attitudes toward ESTs (p = .016). Adding to the DI literature, a health psychology intervention was disseminated, implemented, and found sustainable across diverse therapists and settings. Therapists and setting predictors of usage, if modified, might facilitate future sustainability/sustainment of ESTs.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Experiences of supporting older persons in completion of an exercise and nutrition intervention: an interview study with nursing home staff

Background

The interactions between nursing home (NH) staff and their residents are crucial not only for the atmosphere at the NH but also for achieving care goals. In order to test the potential effects of daily physical activities (sit-to-stand (STS) exercises) combined with oral nutritional supplementation (ONS), a randomized intervention trial (the Older Person’s Exercise and Nutrition (OPEN) Study) was performed in NH residents. One aspect of the study was to interview and report the NH staff’s experiences of supporting the residents in fulfilling the intervention.

Methods

In this qualitative study, individual and focus group interviews were performed in eight NH facilities with NH staff who had assisted residents in performing the 12-week ONS/STS intervention. An interview guide developed for this study was used to assess staff experiences of the intervention and its feasibility. The transcribed interviews were analyzed inductively following a constant comparative method and with input from experts in the area, described in Grounded Theory as a reliable technique for researchers to form theory and hypothesis in unexplored areas.

Results

Three main themes relating to the health-promoting intervention emerged. These included: 1) insights into attitudes towards health in general and NH care specifically; 2) intervention-related challenges, frustrations and needs, and 3) aspects of collaboration and opportunities.

The overarching hypothesis derived from the analysis reads: A health-promoting intervention such as the OPEN-concept has great potential for integration into NH life if a combined empathic and encouraging attitude, and a structure to keep it sustainable, are in place.

Conclusions

NH staff experienced the health-promoting intervention as a potentially positive concept, although it was suggested that it works best if introduced as a general routine in the unit and is integrated into the daily planning of care.

Trial registration

ClinicalTrials.govIdentifier: NCT02702037. Date of trial registration February 26, 2016. The trial was registered prospectively

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-021-02039-7.

When Do Therapists Stop Using Evidence-Based Practices? Findings from a Mixed Method Study on System-Driven Implementation of Multiple EBPs for Children

Therapist discontinuation of delivering an evidence-based practice (EBP) is a critical outcome in the community implementation of EBPs. This mixed methods study examined factors associated with therapist discontinuation within a large reimbursement-driven implementation of multiple EBPs in public children's mental health services. The study integrated quantitative survey data from 748 therapists across 65 agencies, and qualitative interviews from a subset of 79 therapists across 14 agencies. Therapists adopted, on average, 2.41 EBPs (SD = 1.05, range = 1-5), and nearly half (n = 355, 47.5%) reported discontinuing at least one EBP. Multi-level models were used to predict the binary outcome of discontinuation, and qualitative analyses were used to expand upon quantitative findings. Quantitative models revealed that therapist factors, including fewer direct service hours per week, a greater number of EBPs adopted, higher emotional exhaustion, and more negative attitudes toward EBPs in general were associated with discontinuation. In addition, EBP-specific factors including more negative perceptions of the particular EBP and lower self-efficacy for delivering the specific EBP predicted discontinuation. Themes from interview responses highlighted the importance of fit of the EBP with the agency's client base, as well as therapist perceptions of adequate EBP training supports, and the alignment of an EBP with therapists' professional goals. Together, the findings suggest the need for strategic sustainment planning interventions that target EBP fit (i.e., fit between adopted EBPs and agency target population, fit between EBP and therapist preferences and career goals) and support therapist self-efficacy in delivering EBPs.

Influence of participation in a quality improvement collaborative on staff perceptions of organizational sustainability

BACKGROUND

Sustainability capacity (SC), which is an organization's ability to implement and maintain change, is influenced by internal attributes, environmental contextual influencers, and intervention attributes. Temporal changes in staff SC perceptions, as well as the influence of quality improvement collaborative (QIC) participation, has generally not been explored. This project addresses this gap, measuring staff SC perceptions at four time points (baseline and every 9 months) for clinics participating in an intervention - the Network for the Improvement of Addiction Treatment QIC initiative (called NIATx200).

METHODS

A mixed linear model repeated measures analysis was applied to matched staff members (n = 908, representing 2329 total cases) across the evaluation timeframe. Three separate statistical models assessed potential predictors of SC perceptions: Time (Models I-III); NIATx200 intervention, staff job function, and tenure (Models II &III); and NIATx200 participation hours and four organizational variables (Model III).

RESULTS

For Model I, staff perceptions of total SC increased throughout most of the study (t1,4 = - 6.74, p < .0001; t2,4 = - 3.100, p < .036; t3,4 = - 0.23, p = ns). Model II did not change Model I's overall Time effect, but combined NIATx200 services (t = - 2.23, p = .026), staff job function (t = - 3.27, p = .001), and organizational administrators (t = - 3.50, p = .001) were also significantly associated with greater perceptions of total SC. Inclusion of additional variables in Model III demonstrated the importance of a higher participation level (t = - 3.09, p < .002) and being in a free-standing clinic (t = - 2.06, p < .04) on staff perceptions of total SC.

CONCLUSION

Although staff exposure to sustainability principals was minimal in NIATx200, staff perceptions about their organization's SC significantly differed over time. However, an organization's participation level in a QIC became the principal predictor of staff SC perceptions, regardless of other factors' influence. Given these findings, it is possible to develop and introduce specific sustainability content within the structure of a QIC to assess the impact on staff SC perceptions over time and the sustainment of organizational change.

TRIAL REGISTRATION

ClinicalTrials.gov , NCT00934141 . Registered July 6, 2009. Retrospectively registered.

Disseminating TF-CBT: A Mixed Methods Investigation of Clinician Perspectives and the Impact of Training Format and Formalized Problem-Solving Approaches on Implementation Outcomes

BACKGROUND

Trauma Focused-Cognitive Behavioural Therapy (TF-CBT) has been established as an evidence-based treatment for youth with traumatic stress symptoms. The versatility of TF-CBT in conjunction with its established effectiveness has led to its widespread dissemination. However, dissemination efforts have not always translated into sustainability, which has prompted a more thorough investigation into those factors that impact implementation and encouraged the development of strategies that promote sustainability. Toward this end, the aims of this study were to: (1) determine which components of TF-CBT clinicians found the most difficult to implement; (2) explore clinicians' perceptions as to why these components were challenging; and (3) examine whether the use of formalized problem-solving approaches (FPSAs) or training format was associated with implementation outcomes.

METHODS

A mixed methods design was utilized to analyse survey data from mental health professionals (N = 85) who received TF-CBT training. Thematic analysis explored responses to questions concerning those TF-CBT components respondents found most difficult to implement, while bivariate analyses helped determine whether training format or the use of FPSAs was associated with training engagement, confidence in delivering TF-CBT, fidelity or sustainability.

RESULTS

Clinicians perceived the same three components of TF-CBT as most difficult regardless of the type of training they received, and provider, youth, caregiver, and organizational-related themes emerged from thematic analysis. Bivariate analyses indicated more extended training formats and the use of FPSAs were associated with greater implementation success.

CONCLUSION

Findings suggest that perceived difficulty of TF-CBT components did not vary by training format, but more extended formats and the use of FPSAs were associated with more favourable implementation outcomes. Implementers should consider ways to utilize FPSAs within training programs, as well as focus on content identified by clinicians as difficult, as this may assist clinicians in developing skills and managing implementation barriers.

Importance of congruence between communicating and executing implementation programmes: a qualitative study of focus group interviews

Background

The Ostrobothnia Depression Programme (ODP) in Finland was intended to implement two evidence-based brief psychotherapy interventions, namely motivational interview and behavioural activation, in several regional psychiatric teams. A simultaneous effectiveness study was conducted. Considerable tension was encountered between these two arms, causing resistance to change. We conducted a qualitative case study to better understand this tension and to discuss how managerial and executive practices may ensure the successful running of a hybrid design programme.

Methods

We conducted focus group interviews to evaluate the phases of preparation and practical execution of the ODP from the perspectives of management and the programme executives. To gather the data, we applied the revised Socratic approach for health technology assessment and focus group interviews. We analysed the data deductively according to the Normalization Process Theory.

Results

We identified two main critical issues: (1) The ODP programme plan ignored the team leaders’ crucial role in influencing the implementation climate and mobilizing organizational strategies. The ODP had a simplistic top-down design with minimal and delayed collaboration with its target groups in the preparation phase. (2) Incongruence occurred between what the project group had explicitly communicated about being the spearhead of the ODP and what they then actually enacted. These two issues caused tension between the implementation efforts and the effectiveness study as well as resistance to change among the staff.

Conclusion

Early, open collaboration with all prospective stakeholders towards a shared understanding about the programme is the first action the programme administrators should take. Agreement on goals and the means to achieve them would lower tension between the two arms of a hybrid design programme, thereby reducing resistance to change. Congruence between the goals communicated and the actual managerial and executive actions is of paramount importance in getting the programme recipients on board.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-020-00090-w.

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention

Problematic rates of alcohol, e-cigarette, and other drug use among US adolescents highlight the need for effective implementation of evidence-based programs (EBPs), yet schools and community organizations have great difficulty implementing and sustaining EBPs. Although a growing number of studies show that implementation support interventions can improve EBP implementation, the literature on how to improve sustainability through implementation support is limited. This randomized controlled trial advances the literature by testing the effects of one such implementation intervention-Getting To Outcomes (GTO)-on sustainability of CHOICE, an after-school EBP for preventing substance use among middle-school students. CHOICE implementation was tracked for 2 years after GTO support ended across 29 Boys and Girls Club sites in the greater Los Angeles area. Predictors of sustainability were identified for a set of key tasks targeted by the GTO approach (e.g., goal setting, evaluation, collectively called "GTO performance") and for CHOICE fidelity using a series of path models. One year after GTO support ended, we found no differences between GTO and control sites on CHOICE fidelity. GTO performance was also similar between groups; however, GTO sites were superior in conducting evaluation. Better GTO performance predicted better CHOICE fidelity. Two years after GTO support ended, GTO sites were significantly more likely to sustain CHOICE implementation when compared with control sites. This study suggests that using an implementation support intervention like GTO can help low-resource settings continue to sustain their EBP implementation to help them get the most out of their investment. ClinicalTrials.gov Identifier: NCT02135991.

Community-based behavioral health administrator perspectives on sustainability of Dialectical Behavior Therapy: a qualitative evaluation

BACKGROUND

Substantial resources have been invested in evidence-based practice (EBP) implementation in community settings; however, research suggests that EBPs do not always sustain over time.

METHOD

This qualitative study explored the perspectives of 13 community behavioral health agency leaders regarding the sustainability of an EBP 25 to 28 months following the original training period. Administrators from 10 agencies were interviewed to understand the complexities of the implementation process, sustainability of Dialectical Behavior Therapy, and their recommendations to enhance implementation and sustainability.

RESULTS

A content analysis revealed five emergent themes: treatment model opinions, resource concerns, staff selection/ turnover, population characteristics, and recommendations for future implementation.

CONCLUSIONS

These themes likely would be helpful in informing the design of future implementation and sustainability initiatives sensitive to the challenges of integrating EBPs in community settings.

File audit to assess sustained fidelity to a recovery and wellbeing oriented mental health service model: an Australian case study

Background

Over the past decade there has been increasing attention to implementing recovery-oriented approaches within mental health service practice and enhancing fidelity to such approaches. However, as is often the case with evidence-based practices, less attention has been paid to the sustainability of recovery-oriented approaches over time. This study sought to investigate whether fidelity to a recovery-oriented practice framework – the Collaborative Recovery Model could be sustained over time.

Method

The study setting was an Australian community managed mental health organisation. A file audit of consumer support plans was undertaken using the Goal and Action Plan Instrument for Quality audit tool (GAP-IQ). The audit tool assessed 17 areas for quality. Consumers (n = 116) from a large community managed mental health organisation participated in the study. Sustained fidelity to the Collaborative Recovery Model (CRM) was determined by comparing results from the file audit to a similar audit conducted 3 years earlier.

Results

The file audit revealed a significant increase in fidelity to CRM practices between 2011 and 2014. Fidelity to individual audit items that comprise the GAP-IQ was also found to significantly increase across 16 of the 17 GAP-IQ audit items, with the exception of the ‘Action Plan Review’ audit item.

Conclusions

A comparison of file audit data across different time points within the same setting can provide useful feedback about whether or not a practice is being sustained over time. Although fidelity increased overtime the study design does not allow conclusions that training and coaching practices implemented by the organisation were responsible.

Comparing Agency Leader and Therapist Perspectives on Evidence-Based Practices: Associations with Individual and Organizational Factors in a Mental Health System-Driven Implementation Effort

Agency leaders and therapists are essential stakeholders in implementation of evidence-based practices (EBPs) within publicly-funded mental health services. Little is known about how these stakeholders differ in their perceptions of specific EBPs and which individual and organizational factors differentially influence these perceptions. Within the context of a system-driven implementation of multiple EBPs, survey data from 160 leaders and 720 therapists were examined to assess differences in perceptions of six EBPs. Findings indicated that leaders and therapists have unique perspectives and preferences regarding EBPs that are shaped by distinct sociodemographic and professional characteristics and aspects of organizational functioning.

Assessing long-term sustainment of clinic participation in NIATx200: Results and a new methodological approach

BACKGROUND

Sustainability frameworks differentiate between sustainability capacity and sustainment of organizational change. Multiple studies have examined sustainability capacity. Methodological approaches to assess long-term sustainment have not been explored. This study addresses this gap by describing the development of a long-term sustainment methodology and evaluating its application utilizing data from substance abuse clinics participating in a quality improvement collaborative.

METHODS

The study involved clinics (n = 121) in three states (MI, NY and WA) participating in the 2007-2009 NIATx200 quality improvement (QI) intervention. It extended the primary analysis to focus on clinics' long-term sustainment of wait time, retention and admission improvements. Long-term sustainment was defined as two years post end of the active implementation period (Calendar Years 2010 and 2011). The analysis defined case exclusion criteria and spline "knot" time intervals; allowed for C_p statistic use to address clinic data volatility; established the structure of sustainment plots and explored differences between NIATx implementation strategies.

RESULTS

Example spline and sustain plots highlight the application of the long-term sustainment methodology for NIATx200 clinics. In clinics with available longitudinal outcome data, 40.8% (n = 31 of 76 clinics) sustained improvements in wait time, 26.7% (n = 20 of 75 clinics) in retention, and 28.1% (n = 32 of 114 clinics) for admissions. Clinic assignment to a NIATx200 implementation strategy did not significantly influence a clinics' long-term sustainment except for lower wait time changes in the interest circle interventions. Thirty clinics (24.8%) sustained improvements for two outcomes and six clinics (5.0%) did so for all three outcomes. The clinics that sustained multiple outcome improvements were assigned to the interest circle (n = 12), learning session (n = 10), combination (n = 8), and coaching (n = 6) implementation strategies. Guidance for applying the long-term sustainment methodology in other quality improvement settings is described.

CONCLUSIONS

Research about sustainability capacity and sustainment of change has become increasingly important in dissemination and implementation research. Assessment of long-term sustainment in a multi-organizational quality improvement collaborative (QIC) is needed to identify when program drift and intervention decay occurs. If "cut-points" indicate when effects diminish, specific sustainability modules could be developed and introduced within the structure of a QIC to improve organizational long-term sustainment. Coaches and change teams could be trained to focus on organizational change sustainment and strengthen the likelihood of institutionalization. ClinicalTrials.gov Identifier: NCT00934141 Registered July 6, 2009. Retrospectively registered.

The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care

There is strong interest in implementation science to address the gap between research and practice in public health. Research on the sustainability of evidence-based interventions has been growing rapidly. Sustainability has been defined as the continued use of program components at sufficient intensity for the sustained achievement of desirable program goals and population outcomes. This understudied area has been identified as one of the most significant translational research problems. Adding to this challenge is uncertainty regarding the extent to which intervention adaptation and evolution are necessary to address the needs of populations that differ from those in which interventions were originally tested or implemented. This review critically examines and discusses conceptual and methodological issues in studying sustainability, summarizes the multilevel factors that have been found to influence the sustainability of interventions in a range of public health and health care settings, and highlights key areas for future research.

Factors influencing the long-term sustainment of quality improvements made in addiction treatment facilities: a qualitative study

BACKGROUND

A greater understanding of the factors that influence long-term sustainment of quality improvement (QI) initiatives is needed to promote organizational ability to sustain QI practices over time, help improve future interventions, and increase the value of QI investments.

METHODS

We approached 83 of 201 executive sponsors or change leaders at addiction treatment organizations that participated in the 2007-2009 NIATx200 QI intervention. We completed semi-structured interviews with 33 individuals between November 2015 and April 2016. NIATx200 goals were to decrease wait time, increase admissions and improve retention in treatment. Interviews sought to understand factors that either facilitated or impeded long-term sustainment of organizational QI practices made during the intervention. We used thematic analysis to organize the data and group patterns of responses. We assessed available quantitative outcome data and intervention engagement data to corroborate qualitative results.

RESULTS

We used narrative analysis to group four important themes related to long-term sustainment of QI practices: (1) finding alignment between business- and client-centered practices; (2) staff engagement early in QI process added legitimacy which facilitated sustainment; (3) commitment to integrating data into monitoring practices and the identification of a data champion; and (4) adequate organizational human resources devoted to sustainment. We found four corollary factors among agencies which did not sustain practices: (1) lack of evidence of impact on business practices led to discontinuation; (2) disengaged staff and lack of organizational capacity during implementation period led to lack of sustainment; (3) no data integration into overall business practices and no identified data champion; and (4) high staff turnover. In addition, we found that many agencies' current use of NIATx methods and tools suggested a legacy effect that might improve quality elsewhere, even absent overall sustainment of original study outcome goals. Available quantitative data on wait-time reduction demonstrated general concordance between agency perceptions of, and evidence for, sustainment 2 years following the end of the intervention. Additional quantitative data suggested that greater engagement during the intervention period showed some association with sustainment.

CONCLUSIONS

Factors identified in QI frameworks as important for short-term sustainment-organizational capacity (e.g. staffing and leadership) and intervention characteristics (e.g. flexibility and fit)-are also important to long-term sustainment.

Advancing understanding of the sustainability of lay health advisor (LHA) programs for African-American women in community settings

Lay health advisor (LHA) programs have made strong contributions towards the elimination of health disparities and are increasingly being implemented to promote health and prevent disease. Developed in collaboration with African-American survivors, the National Witness Project (NWP) is an evidence-based, community-led LHA program that improves cancer screening among African-American women. NWP has been successfully disseminated, replicated, and implemented nationally in over 40 sites in 22 states in diverse community settings, reaching over 15,000 women annually. We sought to advance understanding of barriers and facilitators to the long-term implementation and sustainability of LHA programs in community settings from the viewpoint of the LHAs, as well as the broader impact of the program on African-American communities and LHAs. In the context of a mixed-methods study, in-depth telephone interviews were conducted among 76 African-American LHAs at eight NWP sites at baseline and 12-18 months later, between 2010 and 2013. Qualitative data provides insight into inner and outer contextual factors (e.g., community partnerships, site leadership, funding), implementation processes (e.g., training), as well as characteristics of the intervention (e.g., perceived need and fit in African-American community) and LHAs (e.g., motivations, burnout) that are perceived to impact the continued implementation and sustainability of NWP. Factors at the contextual levels and related to motivations of LHAs are critical to the sustainability of LHA programs. We discuss how findings are used to inform (1) the development of the LHA Sustainability Framework and (2) strategies to support the continued implementation and sustainability of evidence-based LHA interventions in community settings.

Predicting evidence-based treatment sustainment: results from a longitudinal study of the Adolescent-Community Reinforcement Approach

BACKGROUND

Implementation support models are increasingly being used to enhance the delivery of evidence-based treatments (EBTs) in routine care settings. Little is known about the extent to which these models lead to continued EBT use after implementation support ends. Moreover, few empirical studies longitudinally examine the hypothesized factors associated with long-term psychosocial EBT use (i.e., sustainment). In an effort to address this gap, the current study examined sustainment of an EBT called the Adolescent-Community Reinforcement Approach (A-CRA) following the end of implementation support.

METHODS

Between 2006 and 2010, the Substance Abuse and Mental Health Services Administration awarded 3 years of A-CRA implementation support to 82 community-based organizations around the USA. The extent to which A-CRA was sustained following grant end and the hypothesized factors associated with EBT sustainment were collected using both retrospective and prospective data. We examined the extent to which 10 core treatment elements of A-CRA were sustained and the associations between the extent of A-CRA sustainment and hypothesized factors using a pattern-mixture longitudinal modeling approach.

RESULTS

Staff from 76 organizations participated in data collection for a 92.86% response rate. On average, about half of the 10 core treatment elements were sustained following the loss of implementation support. Factors that appeared most important to A-CRA sustainment included characteristics that were related to the outer setting (communication, funding, and partnerships), inner setting (political support, organizational capacity, and supervisor turnover rate), implementation support period (number of clinicians and supervisors certified and employed at support end and number of youth served), and staff perceptions of the intervention (implementation difficulty, relative advantage, and perceived success).

CONCLUSIONS

Even with multiple years of implementation support, community-based organizations face challenges in sustaining EBT delivery over time. Consistent with implementation theories, multiple factors appear related with EBT sustainment, including the degree of implementation during the initial support period, as well as adequate funding, infrastructure support, and staff support following the end of funding.

Sustaining complex interventions in long-term care: a qualitative study of direct care staff and managers

Background

Little is known about the sustainability of behavioral change interventions in long-term care (LTC). Following a cluster randomized trial of an intervention to improve staff communication (CONNECT), we conducted focus groups of direct care staff and managers to elicit their perceptions of factors that enhance or reduce sustainability in the LTC setting. The overall aim was to generate hypotheses about how to sustain complex interventions in LTC.

Methods

In eight facilities, we conducted 15 focus groups with 83 staff who had participated in at least one intervention session. Where possible, separate groups were conducted with direct care staff and managers. An interview guide probed for staff perceptions of intervention salience and sustainability. Framework analysis of coded transcripts was used to distill insights about sustainability related to intervention features, organizational context, and external supports.

Results

Staff described important factors for intervention sustainability that are particularly challenging in LTC. Because of the tremendous diversity in staff roles and education level, interventions should balance complexity and simplicity, use a variety of delivery methods and venues (e.g., group and individual sessions, role-play/storytelling), and be inclusive of many work positions. Intervention customizability and flexibility was particularly prized in this unpredictable and resource-strapped environment. Contextual features noted to be important include addressing the frequent lack of trust between direct care staff and managers and ensuring that direct care staff directly observe manager participation and support for the program. External supports suggested to be useful for sustainability include formalization of changes into facility routines, using “train the trainer” approaches and refresher sessions. High staff turnover is common in LTC, and providing materials for new staff orientation was reported to be important for sustainability.

Conclusions

When designing or implementing complex behavior change interventions in LTC, consideration of these particularly salient intervention features, contextual factors, and external supports identified by staff may enhance sustainability.

Trial registration

ClinicalTrial.gov, NCT00636675

The what, when, and why of implementation frameworks for evidence-based practices in child welfare and child mental health service systems

It is widely recognized that children in the child welfare system are particularly vulnerable to the adverse health and mental effects associated with exposure to abuse and neglect, making it imperative to have broad-based availability of evidence-based practices (EBPs) that can prevent child maltreatment and reduce the negative mental health outcomes for youth who are victims. A variety of EBPs exist for reducing child maltreatment risk and addressing the associated negative mental health outcomes, but the reach of these practices is limited. An emerging literature documents factors that can enhance or inhibit the success of EBP implementation in community service agencies, including how the selection of a theory-driven conceptual framework, or model, might facilitate implementation planning by providing guidance for best practices during implementation phases. However, limited research is available to guide decision makers in the selection of implementation frameworks that can boost implementation success for EBPs that focus on preventing child welfare recidivism and serving the mental health needs of maltreated youth. The aims of this conceptual paper are to (1) provide an overview of existing implementation frameworks, beginning with a discussion of definitional issues and the selection criteria for frameworks included in the review; and (2) offer recommendations for practice and policy as applicable for professionals and systems serving victims of child maltreatment and their families.

Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach

Background

Few empirical studies longitudinally examine evidence-based practice (EBP) sustainment and the hypothesized factors that influence it. In an effort to address this gap, the current study examined sustainment of an EBP for adolescent substance use called the adolescent community reinforcement approach (A-CRA).

Methods

A-CRA sustainment was assessed via information collected as part of key informant interviews and surveys with clinical staff from community-based treatment organizations that had received federal funding to implement A-CRA. Administrative data from the funding period on implementation was also used. Using discrete-time survival analysis, we regressed A-CRA sustainment on several factors theorized to influence EBP sustainment. Factors examined included outer setting, inner setting, implementation quality during the funding period, and intervention-related characteristics.

Results

Overall, data from 83 % of the targeted sample of treatment organizations was collected. A-CRA sustainment was strongly related to the time since funding loss. Strong relationships were found between sustainment status and implementation quality during the funding period, agency focus, funding stability, and political support for the treatment along with staff perceptions of the treatment’s complexity and implementation difficulty.

Conclusions

Consistent with the Consolidated Framework for Implementation Research, the current study found several factors related to the outer setting (e.g., funding stability), inner setting (e.g., agency focus), implementation quality during the funding period (e.g., staff trained, clients served), and characteristics of the intervention (e.g., implementation complexity) to be associated with EBP sustainment. Future research is warranted to examine the extent to which these relationships are stable over time. Efforts to ensure that adequate implementation occurs during the initial implementation period and that adequate funding, infrastructure, and staff support following the ending of initial support are critical to a program’s survival.

Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support

BACKGROUND

Little is known about how well or under what conditions health innovations are sustained and their gains maintained once they are put into practice. Implementation science typically focuses on uptake by early adopters of one healthcare innovation at a time. The later-stage challenges of scaling up and sustaining evidence-supported interventions receive too little attention. This project identifies the challenges associated with sustainability research and generates recommendations for accelerating and strengthening this work.

METHODS

A multi-method, multi-stage approach, was used: (1) identifying and recruiting experts in sustainability as participants, (2) conducting research on sustainability using concept mapping, (3) action planning during an intensive working conference of sustainability experts to expand the concept mapping quantitative results, and (4) consolidating results into a set of recommendations for research, methodological advances, and infrastructure building to advance understanding of sustainability. Participants comprised researchers, funders, and leaders in health, mental health, and public health with shared interest in the sustainability of evidence-based health care.

RESULTS

Prompted to identify important issues for sustainability research, participants generated 91 distinct statements, for which a concept mapping process produced 11 conceptually distinct clusters. During the conference, participants built upon the concept mapping clusters to generate recommendations for sustainability research. The recommendations fell into three domains: (1) pursue high priority research questions as a unified agenda on sustainability; (2) advance methods for sustainability research; (3) advance infrastructure to support sustainability research.

CONCLUSIONS

Implementation science needs to pursue later-stage translation research questions required for population impact. Priorities include conceptual consistency and operational clarity for measuring sustainability, developing evidence about the value of sustaining interventions over time, identifying correlates of sustainability along with strategies for sustaining evidence-supported interventions, advancing the theoretical base and research designs for sustainability research, and advancing the workforce capacity, research culture, and funding mechanisms for this important work.

Predictors of community therapists' use of therapy techniques in a large public mental health system

IMPORTANCE

Few studies have examined the effects of individual and organizational characteristics on the use of evidence-based practices in mental health care. Improved understanding of these factors could guide future implementation efforts to ensure effective adoption, implementation, and sustainment of evidence-based practices.

OBJECTIVE

To estimate the relative contribution of individual and organizational factors on therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques within the context of a large-scale effort to increase use of evidence-based practices in an urban public mental health system serving youth and families.

DESIGN, SETTING, AND PARTICIPANTS

In this observational, cross-sectional study of 23 organizations, data were collected from March 1 through July 25, 2013. We used purposive sampling to recruit the 29 largest child-serving agencies, which together serve approximately 80% of youth receiving publically funded mental health care. The final sample included 19 agencies with 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators.

MAIN OUTCOMES AND MEASURES

Therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques, as measured by the Therapist Procedures Checklist-Family Revised.

RESULTS

Individual factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 16%; family therapy techniques, 7%; and psychodynamic therapy techniques, 20%. Organizational factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 23%; family therapy techniques, 19%; and psychodynamic therapy techniques, 7%. Older therapists and therapists with more open attitudes were more likely to endorse use of cognitive-behavioral therapy techniques, as were those in organizations that had spent fewer years participating in evidence-based practice initiatives, had more resistant cultures, and had more functional climates. Women were more likely to endorse use of family therapy techniques, as were those in organizations employing more fee-for-service staff and with more stressful climates. Therapists with more divergent attitudes and less knowledge about evidence-based practices were more likely to use psychodynamic therapy techniques.

CONCLUSIONS AND RELEVANCE

This study suggests that individual and organizational factors are important in explaining therapist behavior and use of evidence-based practices, but the relative importance varies by therapeutic technique.

Examining the sustainment of the Adolescent-Community Reinforcement Approach in community addiction treatment settings: protocol for a longitudinal mixed method study

BACKGROUND

Although evidence-based treatments are considered the gold standard for clinical practice, it is widely recognized that evidence-based treatment implementation in real world practice settings has been limited. To address this gap, the federal government provided three years of funding, training and technical assistance to 84 community-based treatment programs to deliver an evidence-based treatment called the Adolescent-Community Reinforcement Approach (A-CRA). Little is known about whether such efforts lead to long-term A-CRA sustainment after the initial funding ends.

METHODS/DESIGN

We will use a longitudinal mixed method data analytic approach to characterize sustainment over time and to examine the factors associated with the extent to which A-CRA is sustained. We will use implementation data collected during the funding period (e.g., organizational functioning, staff certification rates and penetration) and supplement it with additional data collected during the proposed project period regarding implementation quality and the hypothesized predictors of sustainment (i.e., inner and outer contextual variables) collected over three waves from 2013 to 2015 representing program sustainment up to five years post-initial funding.

DISCUSSION

Gaining a better understanding of the factors that influence the evidence-based treatment sustainment may lead to more effective dissemination strategies and ultimately improve the quality of care being delivered in community-based addiction treatment settings.

Organizational factors influencing implementation of evidence-based practices for integrated treatment in behavioral health agencies

Objective. In recent years, New Mexico has prioritized integrated treatment for cooccurring mental health and substance use disorders within its public behavioral health system. This report describes factors likely to be important when implementing evidence-based practices (EBPs) in community agencies. Methods. Our mixed-method research design consisted of observations, semistructured interviews, and surveys undertaken with employees at 14 agencies at baseline and after 18 months. We developed four-agency typologies based on iterative coding and analysis of observations and interviews. We then examined survey data from employees at the four exemplar agencies to validate qualitative findings. Results. Financial resources and strong leadership impacted agency capacity to train providers and implement EBPs. Quantitative analysis of service provider survey responses from these agencies (N = 38) supported qualitative findings and demonstrated significant mean score differences in leadership, organizational climate, and attitudes toward EBPs in anticipated directions. Conclusion. The availability of strong leadership and financial resources were key components to initial implementation success in this study of community agencies in New Mexico. Reliance only on external funding poses risks for sustainment when demoralizing work climates precipitate employee turnover. Strong agency leadership does not always compensate for deficient financial resources in vulnerable communities.