Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy

Randomized trial of the leadership and organizational change for implementation (LOCI) strategy in substance use treatment clinics

INTRODUCTION

Evidence-based practice (EBP) implementation represents a strategic change that requires alignment of leadership and support throughout organizations. Leadership and Organizational Change for Implementation (LOCI) is a multifaceted implementation strategy that aims to improve implementation leadership and climate within organizations through iterative cycles of leadership and climate assessment and feedback, leadership training and coaching, and strategic planning with upper-level leaders. This study tested the effects of LOCI on transformational and implementation leadership, implementation climate, implementation citizenship behavior, and EBP reach.

METHODS

A multiple cohort, cluster randomized trial tests the effect of LOCI in 60 clinics across nine behavioral health organizations in California and Arizona, USA. The study randomized clinics within organizations to either LOCI or a leadership training webinar control condition in three consecutive cohorts. Repeated web-based surveys of direct service providers (nLOCI = 201, nControl = 179) assessed leadership, implementation climate, and implementation citizenship over time. Multilevel autoregressive modeling was the primary statistical analysis such that providers (level-1) were nested within clinics (level-2). The study predicted between-condition differences at 4-, 8-, and 12-month follow-up assessments. Provider engagement in a fidelity monitoring process assessed reach of motivational interviewing (i.e., number of sessions recorded/submitted for fidelity coding). An independent sample t-test explored between condition differences in motivational interviewing reach.

RESULTS

Results indicated between condition differences at 4 months for implementation leadership, implementation climate, and implementation citizenship behavior such that greater improvements were evidenced in the LOCI condition compared to the control condition. Reach of MI was significantly greater in the LOCI vs control condition such that LOCI providers were significantly more likely to engage in the fidelity monitoring process (chi-square (1, n = 370) = 5.59, p = .018).

CONCLUSIONS

LOCI was developed based on organizational theories of strategic leadership and climate to affect organizational change processes that communicate that innovation implementation is expected, supported, and recognized as a value of the organization. The LOCI implementation strategy resulted in more positive hypothesized outcomes compared to the control condition. Organizational change strategies have utility for implementing health innovations in complex, multilevel contexts and for greater sustainment of facilitative leader behaviors, strategic implementation climate, and improved implementation outcomes.

TRIAL REGISTRATION

This study is registered with Clinicaltrials.gov gov (NCT03042832, 2 February 2017; retrospectively registered).

Inter-organizational alignment and implementation outcomes in integrated mental healthcare for children and adolescents: a cross-sectional observational study

BACKGROUND

Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOA in general and implementation context factors are associated with implementation outcomes in integrated SBMH.

METHODS

SBMH clinicians from community-based organizations (CBOs; nclinician = 27) and their proximal student-support school staff (nschool = 99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes.

RESULTS

The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors.

CONCLUSIONS

This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

Inter-Organizational Alignment and Implementation Outcomes in Integrated Mental Healthcare for Children and Adolescents: A Cross-Sectional Observational Study

Background

Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOAs in general and implementation context factors are associated with implementation outcomes in integrated SBMH.

Methods

SBMH clinicians from community-based organizations (CBOs; nclinician=27) and their proximal student-support school staff (nschool=99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes.

Results

The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors.

Conclusions

This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

Evidence Base Update of Psychosocial Treatments for Adolescents with Disruptive Behavior

OBJECTIVE

This article expands the review of psychosocial treatments for adolescents with disruptive behavior (DB), published previously by this journal. That earlier review focused on DB treatment studies published 1966-2014; the current paper updates the evidence base by incorporating DB treatment studies published 2014-2021.

METHOD

A literature search and screening process identified 63 new studies for inclusion in this updated review. The 63 new studies were combined with 86 studies from the prior review and evaluated using Journal of Clinical Child and Adolescent Psychology level of support criteria, which classify studies as well established, probably efficacious, possibly efficacious, experimental, or of questionable efficacy based on the evidence.

RESULTS

In total, 3 well-established, 7 probably efficacious, and 10 possibly efficacious treatments for adolescents with DB were identified. Further, 52 treatments were classified as experimental and 22 treatments were determined to have questionable efficacy.

CONCLUSIONS

There continues to be a large body of literature building the evidence base for treatments of adolescent DB. With a few exceptions, treatments falling into the top three evidence levels utilized more than one theoretical approach, enhancing each treatment's ability to target DB from multiple angles. Key advances include broad representation of various demographic groups, countries of origin, treatment settings, and provider types in this body of research. Despite these advances, more research is needed to address key gaps in the field, including the need for more studies on treatments tailored to adolescents with DB who are not yet involved with the juvenile justice system.

Misalignment in Community Mental Health Leader and Therapist Ratings of Psychological Safety Climate Predicts Therapist Self-Efficacy with Evidence-Based Practices (EBPs)

Therapist self-efficacy in delivering evidence-based practices (EBPs) is associated with implementation outcomes, including adoption and sustainment in community mental health settings. Inner context organizational climate, including psychological safety, can proximally shape therapist learning experiences within EBP implementation. Psychologically safe environments are conducive to learning behaviors including taking risks, admitting mistakes, and seeking feedback. Organization leaders are instrumental in facilitating psychological safety, but may have differing perspectives of organizational climate than front-line therapists. Discrepant leader and therapist views of psychological safety may have independent associations with therapist EBP learning and implementation outcomes over and above average therapist perceptions of climate. This study examined survey data from 337 therapists and 123 leaders from 49 programs contracted to deliver multiple EBPs within a study examining determinants of sustainment within a large system-driven implementation. Both leaders and therapists completed measures of psychological safety climate and therapists reported on their self-efficacy in delivering multiple EBPs in children's mental health services. Polynomial regression and response surface analysis models were conducted to examine the associations of therapist and leader reports of psychological safety and therapist EBP self-efficacy. Greater discrepancies between leader and therapist reports of psychological safety, in either direction, were associated with lower therapist EBP self-efficacy. Alignment in leader and therapist views of psychological safety climate may impact EBP implementation outcomes. Strategies for improving alignment in perceptions and priorities among organizational members can be included in organizational implementation interventions and may represent unexamined implementation mechanisms of action.

Association of organizational culture and climate with variation in the clinical outcomes of collaborative care for maternal depression in community health centers

Background

Organizational factors may help explain variation in the effectiveness of evidence-based clinical innovations through implementation and sustainment. This study tested the relationship between organizational culture and climate and variation in clinical outcomes of the Collaborative Care Model (CoCM) for treatment of maternal depression implemented in community health centers.

Method

Organizational cultures and climates of 10 community health centers providing CoCM for depression among low-income women pregnant or parenting were assessed using the organizational social context (OSC) measure. Three-level hierarchical linear models tested whether variation in culture and climate predicted variation in improvement in depression symptoms from baseline to 6.5-month post-baseline for N = 468 women with care ±1 year of OSC assessment. Depression symptomology was measured using the Patient Health Questionnaire (PHQ-9).

Results

After controlling for patient characteristics, case mix, center size, and implementation support, patients served by centers with more proficient cultures improved significantly more from baseline to 6.5-month post-baseline than patients in centers with less proficient cultures (mean improvement = 5.08 vs. 0.14, respectively, p = .020), resulting in a large adjusted effect size of dadj = 0.78. A similar effect was observed for patients served by centers with more functional climates (mean improvement = 5.25 vs. 1.12, p < .044, dadj = 0.65). Growth models indicated that patients from all centers recovered on average after 4 months of care. However, those with more proficient cultures remained stabilized whereas patients served by centers with less proficient cultures deteriorated by 6.5-month post-baseline. A similar pattern was observed for functional climate.

Conclusions

Variation in clinical outcomes for women from historically underserved populations receiving Collaborative Care for maternal depression was associated with the organizational cultures and climates of community health centers. Implementation strategies targeting culture and climate may improve the implementation and effectiveness of integrated behavioral health care for depression.

Implementing Evidence-Based Mental-Health Treatments: Attending to Training, Fidelity, Adaptation, and Context

This article provides a very high-level overview of some key considerations in the field of implementation science as it relates to clinical psychology. The article reviews recent findings regarding treatment fidelity, adaptation of treatments, and clinical outcomes. It then details some recent findings on training therapists to provide evidence-based therapies as they were designed to be delivered, with sufficient skill and adaptations if needed to ensure better fit for clients who receive treatments in different treatment settings. Finally, the article considers implications for implementing and sustaining new treatments and supporting their fidelity and adaptation, as well as key directions for future research.

Organizational factors associated with community therapists’ self-efficacy in EBP delivery: The interplay between sustainment leadership, sustainment climate, and psychological safety

Background

Inner context organizational factors proximally shape therapist experiences with evidence-based practice (EBP) implementation and may influence therapist self-efficacy, which has been linked to sustained use of EBPs in community mental health settings. Research has primarily focused on constructs such as implementation leadership and climate. However, the effects of such factors may depend upon other inner context dimensions, such as psychological safety. Psychologically safe environments are conducive to taking risks, speaking up about problems, and requesting feedback and may promote therapist self-efficacy during implementation. This study examines whether organizational sustainment leadership and sustainment climate relate to therapist EBP self-efficacy only under conditions of psychological safety.

Methods

Data were collected from 410 clinicians in 85 programs during the sustainment phase of a system-driven implementation of multiple EBPs in children's mental health services. Therapists reported on their organization's sustainment leadership, sustainment climate, psychological safety, and their own self-efficacy in delivering specific EBPs. Multilevel regression analyses were conducted to account for nested data structure.

Results

Among program-level variables, sustainment leadership and psychological safety both significantly predicted therapist self-efficacy. However, there were no significant interactions between program-level sustainment climate and psychological safety. Exploratory post-hoc analyses revealed a significant interaction between program-level sustainment leadership and therapist-level perceptions of psychological safety such that that the conditional effect of psychological safety on EBP self-efficacy was significant at high levels of sustainment leadership, but not at low or average levels.

Conclusion

We noted independent links between sustainment leadership, organizational psychological safety and therapists feelings of confidence and mastery with EBPs. Therapists’ individual perceptions of psychological safety were linked to self-efficacy only in programs with high sustainment leadership. Thus, sustainment leadership and psychological safety may both represent implementation intervention targets, but it may not be critical to assess for perceptions of psychological safety before deploying organizational leadership strategies. Plain language abstract Therapist self-efficacy is a therapist's belief that they are capable, knowledgeable, and skilled enough to deliver evidence-based practices (EBPs), and is thought to promote improved clinical and implementation outcomes, such as therapists’ sustained use of EBPs. Conditions within community mental health organizations may influence therapists’ sense of EBP self-efficacy. Leaders’ support and expectations for EBP implementation, and collective staff perceptions about the organization's climate to support EBPs are linked to positive therapist attitudes and EBP adoption. However, less is known about how these implementation-specific organizational factors associated with therapist EBP self-efficacy in the long-term, and how this may depend on general workplace conditions. Specifically, psychologically safe environments – where therapists feel safe taking risks such as asking questions, admitting mistakes, and trying new skills – may be needed to promote self-efficacy when therapists are tasked with learning and using complex multi-component EBP innovations. The current study tested the prediction that leader-driven and program-wide focus on EBP sustainment may promote therapist EBP self-efficacy only in organizations where conditions for learning are psychologically safe. Our findings confirmed that fostering strong sustainment focused leadership and psychologically safe environments may each be important for increasing therapists’ EBP self-efficacy. The model results suggested that individual therapist perceptions of psychological safety were more strongly related to EBP self-efficacy in programs with greater implementation leadership. Findings suggest the importance of increasing EBP leadership behavior to fully potentiate other facilitating conditions for therapist learning in the sustainment phase of EBP implementation initiatives.

The impact of implementation climate on community mental health clinicians’ attitudes toward exposure: An evaluation of the effects of training and consultation

Background:

Most evidence-based treatments (EBTs) for posttraumatic stress disorder (PTSD) and anxiety disorders include exposure; however, in community settings, the implementation of exposure lags behind other EBT components. Clinician-level determinants have been consistently implicated as barriers to exposure implementation, but few organizational determinants have been studied. The current study examines an organization-level determinant, implementation climate, and clinician-level determinants, clinician demographic and background factors, as predictors of attitudes toward exposure and changes in attitudes following training.

Method:

Clinicians (n = 197) completed a 3-day training with 6 months of twice-monthly consultation. Clinicians were trained in cognitive behavioral therapy (CBT) for anxiety, depression, behavior problems, and trauma-focused CBT (TF-CBT). Demographic and background information, implementation climate, and attitudes toward exposure were assessed in a pre-training survey; attitudes were reassessed at post-consultation. Implementation climate was measured at the aggregated/group-level and clinician-level.

Results:

Attitudes toward exposure significantly improved from pre-training to post-consultation (t(193) = 9.9, p < .001; d = 0.71). Clinician-level implementation climate scores did not predict more positive attitudes at pre-training (p > .05) but did predict more positive attitudes at post-consultation (ß = −2.46; p < .05) and greater changes in those attitudes (ß = 2.28; p < .05). Group-level implementation climate scores did not predict attitudes at pre-training, post-consultation, or changes in attitudes (all ps > .05). Higher frequency of self-reported CBT use was associated with more positive attitudes at pre-training (ß = −0.81; p < .05), but no other clinician demographic or background determinants were associated with attitudes at post-consultation (all p > .05) or with changes in attitudes (all p > .05).

Conclusions:

Clinician perceptions of implementation climate predicted greater improvement of attitudes toward exposure following EBT training and consultation. Findings suggest that organizational determinants outside of training impact changes in clinicians’ attitudes. Training in four EBTs, only two of which include exposure as a component, resulted in positive changes in clinicians’ attitudes toward exposure, which suggests non-specialty trainings can be effective at changing attitudes, which may enable scale-up.

Exposure is highly effective for treating trauma symptoms and anxiety-based disorders, but it is not commonly used in community mental health settings. Clinicians who endorsed higher expectations, support, and rewards for using exposure in their agency had more positive attitudes toward exposure after training and consultation. Additionally, clinicians who endorsed that exposure is expected, supported, and rewarded in their agency showed a greater improvement in attitudes throughout the training process. Organizational culture can affect clinicians’ attitude changes in the training process, and therefore should become a focus of training efforts.

Justice community opioid innovation network (JCOIN): The TCU research hub

Recognizing the current opioid crisis among justice-involved individuals and the need to intervene at the intersection of justice and community health, the JCOIN TCU hub study explores outcomes associated with a multi-level hybrid implementation approach. The study capitalizes on facilitated collaboration, training, and cross-system data sharing to leverage improvements in criminal justice (CJ) and community behavioral health (CBH) interagency collaboration. The goal is to improve local community public health and safety outcomes for reentering justice-involved individuals who have a history of (or are at risk for) using opioids. The study compares two implementation strategies: one (vertical) in which all units in a community are trained and begin the program simultaneously and another (horizontal) in which one lead-off unit in the community is trained as a prototype of the program, the prototype is tested and refined, and then the lead-off unit helps to train other units within the community. Specific aims are to 1) increase access to and retention in CBH and medications for opioid use disorder services; 2) improve outcomes associated with public health and safety; 3) compare two implementation strategies on systems-level outcomes designed to increase service initiation and receipt of implementation and services; and 4) examine the impact of these strategies on justice-involved individuals' outcomes. The study examines both implementation and implementation-effectiveness, seeking to answer the questions of which implementation strategy is most effective for rapid and sustainable uptake of evidence-based practices and for increasing service linkage and initiation, services retention, and improved opioid-related public health safety outcomes. The study uses a hybrid type 3 study design. The study's primary aim is to compare two implementation strategies and two interventions at two levels (client and system), with a secondary aim to assess client-level outcomes associated with the trial. The study design integrates 2 robust methodologies (stepped wedge and cluster randomized trial), and plans to include 18 research performance sites (communities) located in Texas, New Mexico, and Illinois. The study will contribute to the JCOIN network's effort to establish a national consortium of investigators examining promising strategies to enhance the capabilities and capacity of the justice system to more effectively address the opioid epidemic.

Implementation of school-based services for students with autism: Barriers and facilitators across urban and rural districts and phases of implementation

LAY ABSTRACT

The law requires that schools use evidence-based practices to educate students with autism spectrum disorder. However, these practices are often not used, or are not used correctly in school programs. Understanding barriers and facilitators of use of evidence-based practices in schools will help improve the implementation process. This study uses focus groups to characterize how school-based providers representing urban or rural school districts perceive barriers and facilitators for implementing new practices for students with autism spectrum disorder. Guiding questions include the following: (1) Are contextual factors perceived as barriers or facilitators and how do these vary by district location? and (2) What are the key factors impacting implementation across the Exploration, Preparation, Implementation, and Sustainment phases? Focus group participants (n = 33) were service providers to children with autism spectrum disorder from urban- and rural-located school districts. Several personnel-related themes (attitudes and buy-in, knowledge and skills, staffing, and burnout) were shared by participants representing both urban and rural districts. However, some personnel-related themes and organizational factors were unique to rural or urban districts. For example, themes related to system and organizational factors (leadership approval, support and expectations, district structure, competing priorities, time for effective professional development, litigation and due process, and materials and resources) differed between the district locations. This project serves as an initial step in identifying implementation strategies that may improve the use of evidence-based practices in schools.

Sustainment of Trauma-Focused and Evidence-Based Practices Following Learning Collaborative Implementation

Given the need to develop and validate effective implementation models that lead to sustainable improvements, we prospectively examined changes in attitudes, behaviors, and perceived organizational support during and after statewide Community-Based Learning Collaboratives (CBLCs) promoting trauma-focused evidence-based practices (EBPs). Participants (N = 857; i.e., 492 clinicians, 218 brokers, and 139 senior leaders) from 10 CBLCs completed surveys pre- and post-CBLC; a subsample (n = 146) completed a follow-up survey approximately two years post-CBLC. Results indicated (a) medium, sustained increases in clinician-reported use of trauma-focused EBPs, (b) medium to large, sustained increases in perceived organizational support for trauma-focused EBPs, and (c) trivial to small, sustained increases in perceived organizational support for EBPs broadly. In contrast, clinician-reported overall attitudes towards EBPs decreased to a trivial degree pre- to post-CBLC, but then increased to a small, statistically significant degree from post-CBLC to follow-up. Notably, the degree of perceived improvements in organizational support for general and trauma-focused EBPs varied by professional role. Findings suggest the CBLC implementation strategies may both increase and sustain provider practices and organizational support towards EBPs, particularly those EBPs a CBLC explicitly targets.

Embedding school cultures and climates that promote evidence-based practice implementation for youth with autism: A qualitative study

LAY ABSTRACT

Schools play a major role in providing services to youth with autism; however, not all schools use evidence-based practices, defined as interventions that are proven to improve youth well-being through rigorous research. School culture and climate are strong predictors of whether or not a school uses evidence-based practices; however, little is known about how principals can create school cultures and climates that support the use of these practices. This study interviewed 32 teachers in elementary schools that implemented three closely related evidence-based practices for youth with autism to better understand how principals create school cultures and climates that support effective services. Analysis of the teachers' responses identified seven strategies principals can use to create school cultures and climates that support the implementation of effective practices for youth with autism. The strategies include the following: (a) support teachers to obtain professional development focused on autism, (b) align performance expectations and evaluations with the needs of students with autism and evidence-based practice delivery, (c) allocate resources to ensure adequate staff, materials, and training are available to implement evidence-based practices, (d) be open and flexible to allow teachers to use the building and resources as needed to meet students' needs, (e) provide direct assistance, feedback, and coaching to troubleshoot challenges or involve outside experts to do so, (f) openly value the work of special education teachers and provide recognition to those who develop expertise in evidence-based practices, and (g) look for opportunities to integrate special and general education teachers and students to foster a truly inclusive climate.

Traveling Without a Map: An Incomplete History of the Road to Implementation Science and Where We May Go from Here

This editorial provides a brief history of mental health services research over the last 30 years and how findings from large-scale studies shocked the field and led to the lines of inquiry culminating in current implementation science research. I review the manuscripts published in this special issue of Administration and Policy in Mental Health in light of that history and usethese studies as a way to assess the state of the field. Finally, I present five takeaways extracted from these articles that may be useful in considering future directions for implementation research.

The Association Between Clinician and Perceived Organizational Factors with Early Fidelity to Cognitive Processing Therapy for Posttraumatic Stress Disorder in a Randomized Controlled Implementation Trial

A common metric for determining implementation success is the measurement of clinician adherence to, and competence in, delivering a psychotherapy. The present study examined clinician and organizational factors as predictors of early adherence and competence among 78 clinicians delivering cognitive processing therapy (CPT), an evidence-based psychotherapy (EBP) for posttraumatic stress disorder, in a randomized controlled implementation trial. Results indicated that clinicians' willingness to adopt an EBP if required to do so was significantly associated with early adherence and competence in CPT delivery. Level of clinician education was significantly associated with early competence in delivering CPT. Organizational factors did not predict early adherence or competence. Implications of the findings are discussed.

Leader–member exchange and organizational climate effects on clinician turnover intentions

Purpose

Clinician turnover in mental health settings impacts service quality, including availability and delivery of evidence-based practices. Leadership is associated with organizational climate, team functioning and clinician turnover intentions (TI). This study examines leader–member exchange (LMX), reflecting the relationship between a supervisor and each supervisee, using mean team LMX, dispersion of individual clinician ratings compared to team members (i.e. relative LMX) and team level variability (i.e. LMX differentiation), in relation to organizational climate and clinician TI.

Design/methodology/approach

Survey data were collected from 363 clinicians, nested in children's mental health agency workgroups, providing county-contracted outpatient services to youth and families. A moderated mediation path analysis examined cross-level associations of leader–member exchange with organizational climate and turnover intentions.

Findings

Lower relative LMX and greater LMX differentiation were associated with higher clinician TI. Higher team-level demoralizing climate also predicted higher TI. These findings indicate that poorer LMX and more variability in LMX at the team level are related to clinician TI.

Originality/value

This study describes both team- and clinician-level factors on clinician TI. Few studies have examined LMX in mental health, and fewer still have examined relative LMX and LMX differentiation associations with organizational climate and TI. These findings highlight the importance of leader–follower relationships and organizational climate and their associations with clinician TIs. Mental health service systems and organizations can address these issues through fostering more positive supervisor–supervisee relationships.

Aligning dissemination and implementation science with health policies to improve children's mental health

The prevalence of mental health problems among children (ages 0-21) in the United States remains unacceptably high and, post-COVID-19, is expected to increase dramatically. Decades of psychological knowledge about effective treatments should inform the delivery of better services. Dissemination and implementation (D&I) science has been heralded as a solution to the persistent problem of poor quality services and has, to some extent, improved our understanding of the contexts of delivery systems that implement effective practices. However, there are few studies demonstrating clear, population-level impacts of psychological interventions on children. Momentum is growing among communities, cities, states, and some federal agencies to build "health in all policies" to address broad familial, social, and economic factors known to affect children's healthy development and mental health. These health policy initiatives offer a rare opportunity to repurpose D&I science, shifting it from a primary focus on evidence-based practice implementation, to a focus on policy development and implementation to support child and family health and well-being. This shift is critical as states develop policy responses to address the health and mental health impacts of the COVID-19 pandemic on already-vulnerable families. We provide a typology for building research on D&I and children's mental health policy. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Practitioners' Use and Evaluation of Transdiagnostic Youth Psychotherapy Years After Training and Consultation Have Ended

We examined practitioners' use of the transdiagnostic Modular Approach to Therapy for Children (MATCH) 7 years after learning MATCH for a clinical trial. The practitioners (N = 29; M_age = 52.10, SD = 12.29, 86% women, 97% white) reported using MATCH with 55% of their caseload; use of the various MATCH modules ranged from 39 to 70%. Use was positively associated with amount of MATCH experience in the trial, perceived effectiveness, and ease of implementation. Patterns of specific module use did not consistently match strength of prior evidence (e.g., exposure was least used of the anxiety modules), suggesting challenges for implementation science.

Implementation science and periodontal practice: Translation of evidence into periodontology

The advent of evidence-based practice in the 1990s led to the development of processes and resources to support the use of high-quality research in the provision of health care. As the evidence-based approach to health care continues to evolve, it has become apparent that mere creation and access to scientific knowledge is not sufficient to facilitate its routine adoption in health care. Throughout any health care system, there are inherent barriers preventing the adoption and routine use of new evidence in patient care. These barriers include provider-level factors, such as knowledge and access to new evidence, as well as each provider's attitudes and beliefs around adopting and applying the evidence with their patients. Importantly, there are also health care system-level barriers that, even among willing providers, prevent the easy adoption of new evidence and routine application in patient care. In addition to barriers, there are facilitators that help promote adoption of evidence into practice. Understanding and addressing barriers and facilitators to promote adoption of evidence into practice has led to the growth of a new field known as implementation science. Successful application of implementation science in all areas of health care, including periodontology, will help bridge the gap between what are known from clinical research to be effective treatments and what treatments should be applied routinely in clinical practice. This article reviews key concepts in implementation science and how its application in periodontology can facilitate the translation of high-quality evidence into routine periodontal practice and improved patient outcomes.

Improving Substance Use Services for Juvenile Justice-Involved Youth: Complexity of Process Improvement Plans in a Large Scale Multi-site Study

Despite the high prevalence of substance use disorders among juvenile offenders, most do not receive services. System-level process improvement plans to address unmet service needs can be optimized by combining data-driven decisions and facilitated meetings with behavioral health stakeholders. This paper operationalizes and analyzes the level of specified complexity among process improvement plans evident within 36 juvenile probation and drug courts across 7 states. To inform more effective implementation strategies, this analysis identifies and prioritizes promising courses of agency enhancement toward addressing unmet substance use needs.

The implementation of a culturally grounded, school-based, drug prevention curriculum in rural Hawai'i

This article describes the process of infusing implementation strategies in the development of a school-based drug prevention curriculum for rural Native Hawaiian youth. The curriculum (Ho'ouna Pono) is a video-enhanced, teacher-implemented curriculum developed using a culturally grounded and community-based participatory research approach. Throughout the development of the curriculum, strategies reflective of the domains of the Consolidated Framework for Implementation Research (CFIR) were integrated into the teacher training manual, to promote the implementation, adoption, and sustainability of the curriculum in rural Hawai'i. These strategies were validated through qualitative data across two interrelated studies with community stakeholders in rural Hawai'i. Implications for prevention, community, and educational practices are described in this article.

Does adoption of an evidence-based practice lead to job turnover? Results from a randomized trial

It is important to understand the impact of implementation of evidence-based practices (EBPs) on the workforce. EBP implementation can increase job demands, stress, and burnout, and may thereby exacerbate turnover. This study examined the effects of implementation of an EBP on turnover among staff at nine child welfare agencies. A total of 102 providers were randomized to either adopt an EBP, SafeCare^© , or continue providing services as usual. Participants completed a baseline survey assessing demographics, attitudes toward EBPs, and organizational functioning, and provider turnover was recorded for up to 18 months following implementation. The overall turnover rate was 35%, but did not differ by EBP assignment (odds ratio [OR] = 1.27; 95% confidence interval [0.66, 2.45]). Variables associated with turnover included age (OR = 0.92), years since degree completion (OR = 0.94), prior exposure to EBP (OR = 3.91), believing that adopting an EBP was burdensome (OR = 0.52), and motivation for change (OR = 0.89). EBP assignment moderated two aspects of negative attitudes toward EBP (divergence and monitoring) to predict turnover; those attitudes were only positively related to turnover for individuals assigned to the EBP (OR = 1.46, 1.16). Implications of the findings for implementation are discussed.

A systematic review of empirical studies examining mechanisms of implementation in health

BACKGROUND

Understanding the mechanisms of implementation strategies (i.e., the processes by which strategies produce desired effects) is important for research to understand why a strategy did or did not achieve its intended effect, and it is important for practice to ensure strategies are designed and selected to directly target determinants or barriers. This study is a systematic review to characterize how mechanisms are conceptualized and measured, how they are studied and evaluated, and how much evidence exists for specific mechanisms.

METHODS

We systematically searched PubMed and CINAHL Plus for implementation studies published between January 1990 and August 2018 that included the terms "mechanism," "mediator," or "moderator." Two authors independently reviewed title and abstracts and then full texts for fit with our inclusion criteria of empirical studies of implementation in health care contexts. Authors extracted data regarding general study information, methods, results, and study design and mechanisms-specific information. Authors used the Mixed Methods Appraisal Tool to assess study quality.

RESULTS

Search strategies produced 2277 articles, of which 183 were included for full text review. From these we included for data extraction 39 articles plus an additional seven articles were hand-entered from only other review of implementation mechanisms (total = 46 included articles). Most included studies employed quantitative methods (73.9%), while 10.9% were qualitative and 15.2% were mixed methods. Nine unique versions of models testing mechanisms emerged. Fifty-three percent of the studies met half or fewer of the quality indicators. The majority of studies (84.8%) only met three or fewer of the seven criteria stipulated for establishing mechanisms.

CONCLUSIONS

Researchers have undertaken a multitude of approaches to pursue mechanistic implementation research, but our review revealed substantive conceptual, methodological, and measurement issues that must be addressed in order to advance this critical research agenda. To move the field forward, there is need for greater precision to achieve conceptual clarity, attempts to generate testable hypotheses about how and why variables are related, and use of concrete behavioral indicators of proximal outcomes in the case of quantitative research and more directed inquiry in the case of qualitative research.

A systematic review of empirical studies examining mechanisms of implementation in health

Background

Understanding the mechanisms of implementation strategies (i.e., the processes by which strategies produce desired effects) is important for research to understand why a strategy did or did not achieve its intended effect, and it is important for practice to ensure strategies are designed and selected to directly target determinants or barriers. This study is a systematic review to characterize how mechanisms are conceptualized and measured, how they are studied and evaluated, and how much evidence exists for specific mechanisms.

Methods

We systematically searched PubMed and CINAHL Plus for implementation studies published between January 1990 and August 2018 that included the terms “mechanism,” “mediator,” or “moderator.” Two authors independently reviewed title and abstracts and then full texts for fit with our inclusion criteria of empirical studies of implementation in health care contexts. Authors extracted data regarding general study information, methods, results, and study design and mechanisms-specific information. Authors used the Mixed Methods Appraisal Tool to assess study quality.

Results

Search strategies produced 2277 articles, of which 183 were included for full text review. From these we included for data extraction 39 articles plus an additional seven articles were hand-entered from only other review of implementation mechanisms (total = 46 included articles). Most included studies employed quantitative methods (73.9%), while 10.9% were qualitative and 15.2% were mixed methods. Nine unique versions of models testing mechanisms emerged. Fifty-three percent of the studies met half or fewer of the quality indicators. The majority of studies (84.8%) only met three or fewer of the seven criteria stipulated for establishing mechanisms.

Conclusions

Researchers have undertaken a multitude of approaches to pursue mechanistic implementation research, but our review revealed substantive conceptual, methodological, and measurement issues that must be addressed in order to advance this critical research agenda. To move the field forward, there is need for greater precision to achieve conceptual clarity, attempts to generate testable hypotheses about how and why variables are related, and use of concrete behavioral indicators of proximal outcomes in the case of quantitative research and more directed inquiry in the case of qualitative research.

Burnout and Mental Health Stigma Among Juvenile Probation Officers: The Moderating Effect of Participatory Atmosphere

Despite high rates of mental health problems among juvenile justice-involved youth, mental health stigma among juvenile probation officers (JPOs) is under-studied. This cross-sectional study examined effects of job burnout and workplace participatory atmosphere on mental health stigma among JPOs across Indiana (n = 226). Participatory atmosphere moderated the relationship between JPO burnout-related cynicism and mental health stigma (interaction β = - 0.14, p = .04); burnout was related to greater mental health stigma at low levels of participatory atmosphere. Findings suggest participatory atmosphere mitigates effects of burnout on mental health stigma among JPOs. Organizational-level interventions might help to reduce mental health stigma and combat negative effects from burnout among JPOs.

Elucidating the influence of supervisors' roles on implementation climate

Background

Supervisors play an essential role in implementation by diffusing and synthesizing information, selling implementation, and translating top management’s project plans to frontline workers. Theory and emerging evidence suggest that through these roles, supervisors shape implementation climate—i.e., the degree to which innovations are expected, supported, and rewarded. However, it is unclear exactly how supervisors carry out each of these roles in ways that contribute to implementation climate—this represents a gap in the understanding of the causal mechanisms that link supervisors’ behavior with implementation climate. This study examined how supervisors’ performance of each of these roles influences three core implementation climate domains (expectations, supports, and rewards).

Materials and methods

A sequenced behavioral health screening, assessment, and referral intervention was implemented within a county-based child welfare agency. We conducted 6 focus groups with supervisors and frontline workers from implementing work units 6 months post-implementation (n = 51) and 1 year later (n = 40) (12 groups total). Participants were asked about implementation determinants, including supervision and implementation context. We audio-recorded, transcribed, and analyzed focus groups using an open coding process during which the importance of the supervisors’ roles emerged as a major theme. We further analyzed this code using concepts and definitions related to middle managers’ roles and implementation climate.

Results

In this work setting, supervisors (1) diffused information about the intervention proactively, and in response to workers’ questions, (2) synthesized information by tailoring it to workers’ individual needs, (3) translated top managements’ project plans into day-to-day tasks through close monitoring and reminders, and (4) justified implementation. All four of these roles appeared to shape the implementation climate by conveying strong expectations for implementation. Three roles (diffusing, synthesizing, and mediating) influenced climate by supporting workers during implementation. Only one role (diffusing) influenced climate by conveying rewards.

Conclusions

Supervisors shaped implementation climate by carrying out four roles (diffusing, synthesizing, mediating, and selling). Findings suggest that the interaction of these roles convey expectations and support for implementation (two implementation climate domains). Our study advances the causal theory explaining how supervisors’ behavior shapes the implementation climate, which can inform implementation practice.

Validation of the Implementation Climate Scale (ICS) in substance use disorder treatment organizations

Background

One critical factor in the implementation of evidence-based practice (EBP) in substance use disorder treatment organizations is an inner organizational context that clearly supports implementation efforts. The Implementation Climate Scale (ICS) has been developed to allow researchers and organizations to assess climate for EBP implementation in health and allied health service organizations. The ICS consists of 18 items and measures six dimensions of implementation climate: focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection for EBP, and selection for openness. The ICS was initially developed in a mental health context; thus, the goal of this study was to provide initial validation of the ICS in substance use disorder (SUD) treatment settings.

Methods

Confirmatory factor analysis (CFA) was used to assess the psychometric functioning of the ICS using survey data from 326 providers in 65 teams in SUD treatment programs. Cronbach’s alpha was examined to assess internal consistency of the ICS, and individual and team level construct-based validity was examined by comparing its correlations with service climate, molar climate, and organizational change.

Results

We found evidence for the reliability, factor structure, and validity of the ICS in SUD services. The psychometric functioning of the ICS in SUD treatment settings was comparable to that found in mental health contexts.

Conclusions

The ICS is a brief and pragmatic tool for researchers to better understand a critical antecedent for implementation effectiveness in SUD treatment and for organizational leaders in SUD treatment organizations to evaluate the extent to which providers perceive that their organization supports EBP implementation.

After the Study Ends: A Qualitative Study of Factors Influencing Intervention Sustainability

Sustaining evidence-based practices after initial training and support has ended is necessary to ensure lasting improvements in youth mental health services. This study examined factors impacting community clinicians' decisions to sustain a transdiagnostic youth intervention following participation in a study. The aim of the study was to identify potentially mutable factors impacting sustainability to inform future implementation efforts. Thirteen clinicians (85% women, 92% Caucasian, M age = 35.6) completed interviews after participating in an open trial of an evidence-based intervention for depression, anxiety, and conduct disorders. Interviews were analyzed using thematic analysis methods. All (100%) clinicians reported current use of the intervention. Four themes emerged related to sustainability. Clinicians (100%) reported that making modifications, alignment with prior training, and relative advantage influenced their current intervention use. Clinicians (100%) reported that knowledge transfer from treatment developers was vital to sustainability. They (92%) noted a number of logistical, inner-organizational, and client-level barriers to sustainability. Lastly, clinicians (92%) identified factors related to scaling up the intervention. A variety of personal, organizational, logistical, and client variables influence the sustainment of new interventions, and could be leveraged in future implementation efforts.

Modeling the Decision of Mental Health Providers to Implement Evidence-Based Children's Mental Health Services: A Discrete Choice Conjoint Experiment

Using an online, cross sectional discrete choice experiment, we modeled the influence of 14 implementation attributes on the intention of 563 providers to adopt hypothetical evidence-based children’s mental health practices (EBPs). Latent class analysis identified two segments. Segment 1 (12%) would complete 100% of initial training online, devote more time to training, make greater changes to their practices, and introduce only minor modifications to EBPs. Segment 2 (88%) preferred fewer changes, more modifications, less training, but more follow-up. Simulations suggest that enhanced supervisor support would increase the percentage of participants choosing the intensive training required to implement EBPs. The dissemination of EBPs needs to consider the views of segments of service providers with differing preferences regarding EBPs and implementation process design.

Comparing a Practice-Based Model with a Research-Based Model of social skills interventions for children with autism in schools

BACKGROUND

Social impairment is the most challenging core deficit for children with autism spectrum disorder (ASD). Several evidence-based interventions address social impairment in children with ASD; however, adoption, use, and implementation of these interventions in schools is challenging.

METHOD

Ninety-two children with ASD who received one of three intervention models: a) School personnel adapted and implemented evidence-based social skills intervention (Practice-Based Model; n = 14); b) University researcher developed and implemented evidence-based social skills intervention (Research-Based Model; n = 45); or c) standard educational practices model (Inclusion Only Model; n = 33) participated. The average age was 8.4 (SD = 1.6) years; majority was male (88%) and white (52.2%). Typically developing classmates completed sociometric ratings to determine children's social network inclusion, and independent raters observed children on the playground using a time-interval behavior coding system to record solitary engagement and frequency of initiations.

RESULTS

Separate linear regression models were conducted. Children in the Research-Based Model had significantly higher social network inclusion than children in the other two settings (p = .05). Children in the Practice-Based Model had significantly lower solitary engagement (p = .04) and more initiations on the playground than children in the University Developed Model (p = .04).

CONCLUSIONS

The results suggest that researchers: 1) may learn from public school stakeholders who have lived experiences to better understand the context in which implementation occurs; and 2) should partner with schools to learn about their processes of adaptation and adoption in order to facilitate successful implementation of evidence-based practices for children with ASD. Interventions designed with implementation in mind may be more feasible and increase the chances of use in schools.

Concordance Between Administrator and Clinician Ratings of Organizational Culture and Climate

Organizational culture and climate are important determinants of behavioral health service delivery for youth. The Organizational Social Context measure is a well validated assessment of organizational culture and climate that has been developed and extensively used in public sector behavioral health service settings. The degree of concordance between administrators and clinicians in their reports of organizational culture and climate may have implications for research design, inferences, and organizational intervention. However, the extent to which administrators' and clinicians' reports demonstrate concordance is just beginning to garner attention in public behavioral health settings in the United States. We investigated the concordance between 73 administrators (i.e., supervisors, clinical directors, and executive directors) and 247 clinicians in 28 child-serving programs in a public behavioral health system. Findings suggest that administrators, compared to clinicians, reported more positive cultures and climates. Organizational size moderated this relationship such that administrators in small programs (<466 youth clients served annually) provided more congruent reports of culture and climate in contrast to administrators in large programs (≥466 youth clients served annually) who reported more positive cultures and climates than clinicians. We propose a research agenda that examines the effect of concordance between administrators and clinicians on organizational outcomes in public behavioral health service settings.

Technology-Based Alcohol Interventions in Primary Care: Systematic Review

Background

Primary care settings are uniquely positioned to reach individuals at risk of alcohol use disorder through technology-delivered behavioral health interventions. Despite emerging effectiveness data, few efforts have been made to summarize the collective findings from these delivery approaches.

Objective

The aim of this study was to review recent literature on the use of technology to deliver, enhance, or support the implementation of alcohol-related interventions in primary care. We focused on addressing questions related to (1) categorization or target of the intervention, (2) descriptive characteristics and context of delivery, (3) reported efficacy, and (4) factors influencing efficacy.

Methods

We conducted a comprehensive search and systematic review of completed studies at the intersection of primary care, technology, and alcohol-related problems published from January 2000 to December 2018 within EBSCO databases, ProQuest Dissertations, and Cochrane Reviews. Of 2307 initial records, 42 were included and coded independently by 2 investigators.

Results

Compared with the years of 2000 to 2009, published studies on technology-based alcohol interventions in primary care nearly tripled during the years of 2010 to 2018. Of the 42 included studies, 28 (64%) were randomized controlled trials. Furthermore, studies were rated on risk of bias and found to be predominantly low risk (n=18), followed by moderate risk (n=16), and high risk (n=8). Of the 24 studies with primary or secondary efficacy outcomes related to drinking and drinking-related harms, 17 (71%) reported reduced drinking or harm in all primary and secondary efficacy outcomes. Furthermore, of the 31 studies with direct comparisons with treatment as usual (TAU), 13 (42%) reported that at least half of the primary and secondary efficacy outcomes of the technology-based interventions were superior to TAU. High efficacy was associated with provider involvement and the reported use of an implementation strategy to deliver the technology-based intervention.

Conclusions

Our systematic review has highlighted a pattern of growth in the number of studies evaluating technology-based alcohol interventions in primary care. Although these interventions appear to be largely beneficial in primary care, outcomes may be enhanced by provider involvement and implementation strategy use. This review enables better understanding of the typologies and efficacy of these interventions and informs recommendations for those developing and implementing technology-based alcohol interventions in primary care settings.

Transforming Mental Health Delivery Through Behavioral Economics and Implementation Science: Protocol for Three Exploratory Projects

BACKGROUND

Efficacious psychiatric treatments are not consistently deployed in community practice, and clinical outcomes are attenuated compared with those achieved in clinical trials. A major focus for mental health services research is to develop effective and cost-effective strategies that increase the use of evidence-based assessment, prevention, and treatment approaches in community settings.

OBJECTIVE

The goal of this program of research is to apply insights from behavioral economics and participatory design to advance the science and practice of implementing evidence-based practice (EBP) for individuals with psychiatric disorders across the life span.

METHODS

Project 1 (Assisting Depressed Adults in Primary care Treatment [ADAPT]) is patient-focused and leverages decision-making heuristics to compare ways to incentivize adherence to antidepressant medications in the first 6 weeks of treatment among adults newly diagnosed with depression. Project 2 (App for Strengthening Services In Specialized Therapeutic Support [ASSISTS]) is provider-focused and utilizes normative pressure and social status to increase data collection among community mental health workers treating children with autism. Project 3 (Motivating Outpatient Therapists to Implement: Valuing a Team Effort [MOTIVATE]) explores how participatory design can be used to design organizational-level implementation strategies to increase clinician use of EBPs. The projects are supported by a Methods Core that provides expertise in implementation science, behavioral economics, participatory design, measurement, and associated statistical approaches.

RESULTS

Enrollment for project ADAPT started in 2018; results are expected in 2020. Enrollment for project ASSISTS will begin in 2019; results are expected in 2021. Enrollment for project MOTIVATE started in 2018; results are expected in 2019. Data collection had begun for ADAPT and MOTIVATE when this protocol was submitted.

CONCLUSIONS

This research will advance the science of implementation through efforts to improve implementation strategy design, measurement, and statistical methods. First, we will test and refine approaches to collaboratively design implementation strategies with stakeholders (eg, discrete choice experiments and innovation tournaments). Second, we will refine the measurement of mechanisms related to heuristics used in decision making. Third, we will develop new ways to test mechanisms in multilevel implementation trials. This trifecta, coupled with findings from our 3 exploratory projects, will lead to improvements in our knowledge of what causes successful implementation, what variables moderate and mediate the effects of those causal factors, and how best to leverage this knowledge to increase the quality of care for people with psychiatric disorders.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03441399; https://www.clinicaltrials.gov/ct2/show/NCT03441399 (Archived by WebCite at http://www.webcitation.org/74dRbonBD).

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/12121.

Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda

The field of implementation science was developed to better understand the factors that facilitate or impede implementation and generate evidence for implementation strategies. In this article, we briefly review progress in implementation science, and suggest five priorities for enhancing the impact of implementation strategies. Specifically, we suggest the need to: (1) enhance methods for designing and tailoring implementation strategies; (2) specify and test mechanisms of change; (3) conduct more effectiveness research on discrete, multi-faceted, and tailored implementation strategies; (4) increase economic evaluations of implementation strategies; and (5) improve the tracking and reporting of implementation strategies. We believe that pursuing these priorities will advance implementation science by helping us to understand when, where, why, and how implementation strategies improve implementation effectiveness and subsequent health outcomes.

Translational Research: Bringing Science to the Provider Through Guideline Implementation

Significant variation exists in health care practice patterns that creates concerns regarding the quality of care delivered. Clinical practice based on high-quality evidence provides a rationale for clinical decision making. Resources, such as evidence-based guidelines, provide that evidence to clinicians and improve patient outcomes by decreasing unwanted variation in clinical practice. Because knowledge dissemination alone is ineffective to translate scientific evidence into clinical practice, the field of implementation science has emerged to facilitate this translation of research into routine clinical practice. This article provides an introduction to implementation science, and its application in dentistry to promote adoption of evidence-based guidelines.

Stakeholder Perceptions of the Barriers to Receiving and Delivering Exposure-Based Cognitive Behavioral Therapy for Anxiety Disorders in Adult Community Mental Health Settings

CBT is considered the first-line treatment for anxiety disorders, particularly when it involves gradual confrontation with feared stimuli (i.e., exposure); however, delivery of CBT for anxiety disorders in real-world community clinics is lacking. This study utilized surveys we developed with key stakeholder feedback (patient, provider, and administrator) to assess patient and provider/administrator perceptions of the barriers to delivering (or receiving) CBT for anxiety disorders. Providers/administrators from two counties in California (N = 106) indicated lack of training/competency as primary barriers. Patients in one large county (N = 42) reported their own symptoms most often impacted treatment receipt. Both groups endorsed acceptability of exposure but indicated that its use in treatment provided/received had been limited. Implications and recommendations are discussed.

Navigating the storm: How proficient organizational culture promotes clinician retention in the shift to evidence-based practice

Objective

Clinician turnover is a major concern as mental health systems and organizations invest substantial resources in the implementation of evidence-based practice (EBP). In this study, we identify malleable factors associated with reduced clinician turnover during a system-wide EBP implementation initiative. Specifically, we examine how proficient organizational culture (i.e., norms and behavioral expectations that clinicians prioritize improvement in client well-being and exhibit competence in up-to-date treatment practices), EBP implementation climate (i.e., perceptions that the organization’s policies, procedures, and practices support EBP use), and change in these organizational characteristics relate to clinician turnover during a system-wide EBP transformation.

Method

Data were collected from 236 clinicians in 19 mental health clinics across 3 years of a system-wide EBP implementation initiative in the City of Philadelphia. Clinicians reported on proficient organizational culture and EBP implementation climate at baseline (T₁) and two-year follow-up (T₂). Administrators reported on clinician turnover at three-year follow-up (T₃). Hypotheses were tested via multilevel mediation analyses incorporating mixed effects logistic regression models.

Results

Controlling for organization size, clinician job satisfaction, attitudes towards EBP, job tenure, and age, higher levels of proficient organizational culture and improvement in proficient culture from baseline to two-year follow-up predicted reduced clinician turnover in the year following; these effects were mediated by EBP implementation climate and by improvement in EBP implementation climate, respectively.

Conclusions

Organizations with more proficient cultures have more supportive EBP implementation climates that predict reduced clinician turnover during system-wide EBP implementation initiatives. Strategies that target these antecedents in mental health service organizations may contribute to reduced clinician turnover.

The effect of implementation climate on program fidelity and student outcomes in autism support classrooms

OBJECTIVE

An organization's implementation climate, or the extent to which use of an intervention is expected, supported, and rewarded by colleagues and supervisors, has been identified as critical to successful intervention implementation and outcomes. The effect of implementation climate has not been well studied in special education settings. The present study examines the association between teachers' perceptions of implementation climate, teacher fidelity to a school-based program for students with autism, and student outcomes (measured as changes in IQ) over time.

METHOD

Participants included 158 students from 45 classrooms and their teachers. Teachers provided a measure of implementation climate at the beginning of the academic year; program fidelity was measured monthly throughout the year. The main and interaction effects of perceived implementation climate and fidelity on student outcomes were examined using longitudinal nested linear models with random effects for classroom and student, controlling for important covariates.

RESULTS

On average, IQ scores improved 2.2 points (SD = 8.7). There were no main effects of perceived implementation climate or fidelity on student outcomes; however, the interaction between perceived implementation climate and fidelity was associated with student outcomes (p < .05, d = 0.54). Among classrooms with a strong perceived implementation climate, higher fidelity was associated with better student outcomes.

CONCLUSIONS

While preliminary and requiring replication, these findings suggest that perceived implementation climate and program fidelity each may be important but not sufficient for optimizing outcomes for students with autism. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Operationalizing the 'pragmatic' measures construct using a stakeholder feedback and a multi-method approach

Context

Implementation science measures are rarely used by stakeholders to inform and enhance clinical program change. Little is known about what makes implementation measures pragmatic (i.e., practical) for use in community settings; thus, the present study’s objective was to generate a clinical stakeholder-driven operationalization of a pragmatic measures construct.

Evidence acquisition

The pragmatic measures construct was defined using: 1) a systematic literature review to identify dimensions of the construct using PsycINFO and PubMed databases, and 2) interviews with an international stakeholder panel (N = 7) who were asked about their perspectives of pragmatic measures.

Evidence synthesis

Combined results from the systematic literature review and stakeholder interviews revealed a final list of 47 short statements (e.g., feasible, low cost, brief) describing pragmatic measures, which will allow for the development of a rigorous, stakeholder-driven conceptualization of the pragmatic measures construct.

Conclusions

Results revealed significant overlap between terms related to the pragmatic construct in the existing literature and stakeholder interviews. However, a number of terms were unique to each methodology. This underscores the importance of understanding stakeholder perspectives of criteria measuring the pragmatic construct. These results will be used to inform future phases of the project where stakeholders will determine the relative importance and clarity of each dimension of the pragmatic construct, as well as their priorities for the pragmatic dimensions. Taken together, these results will be incorporated into a pragmatic rating system for existing implementation science measures to support implementation science and practice.

Comprehensive Community-Based Intervention and Asthma Outcomes in African American Adolescents

: media-1vid110.1542/5804911922001PEDS-VA_2017-3737Video Abstract BACKGROUND: African American adolescents appear to be the most at risk for asthma morbidity and mortality even compared with other minority groups, yet there are few successful interventions for this population that are used to target poorly controlled asthma.

METHODS

African American adolescents (age 12-16 years) with moderate-to-severe persistent asthma and ≥1 inpatient hospitalization or ≥2 emergency department visits in 12 months were randomly assigned to Multisystemic Therapy-Health Care or an attention control group (N = 167). Multisystemic Therapy-Health Care is a 6-month home- and community-based treatment that has been shown to improve illness management and health outcomes in high-risk adolescents by addressing the unique barriers for each individual family with cognitive behavioral interventions. The attention control condition was weekly family supportive counseling, which was also provided for 6 months in the home. The primary outcome was lung function (forced expiratory volume in 1 second [FEV₁]) measured over 12 months of follow-up.

RESULTS

Linear mixed-effects models revealed that compared with adolescents in the comparison group, adolescents in the treatment group had significantly greater improvements in FEV₁ secondary outcomes of adherence to controller medication, and the frequency of asthma symptoms. Adolescents in the treatment group had greater reductions in hospitalizations, but there were no differences in reductions in emergency department visits.

CONCLUSIONS

A comprehensive family- and community-based treatment significantly improved FEV₁, medication adherence, asthma symptom frequency, and inpatient hospitalizations in African American adolescents with poorly controlled asthma. Further evaluation in effectiveness and implementation trials is warranted.

Factors associated with program effectiveness in the implementation of a sexual risk reduction intervention for female sex workers across Mexico: Results from a randomized trial

OBJECTIVE

The overall aim of this paper is to examine effectiveness of an evidence-based intervention in community settings, and the factors associated with effectiveness. Limited research in the area of HIV prevention has focused on evaluating intervention program effectiveness in real-world settings.

METHODS

We implemented an efficacious theory-based sexual risk reduction intervention for female sex workers (FSW) called Mujer Segura across 13 different clinics in 13 sites across Mexico. The overall design was a cluster randomized Type I design simultaneously testing intervention program effectiveness with an observational study of implementation factors. We aimed to examine the effectiveness of Mujer Segura at reducing HIV/STI incidence among FSW participants at each site, and to examine the client-, provider-, organization-, and structure-related factors associated with program effectiveness.

RESULTS

We found lower HIV/STI incidence density in the intervention relative to the control group in 5 sites we labeled as "program effective sites," but not in 8 sites we labeled as "program ineffective sites." Using generalized estimating equations controlling for site and computed mean difference effect sizes, we examined statistically and practically significant differences, respectively, between the two groups of sites along various client-, provider-, organization-, and structure-related characteristics. Results indicated that client-level HIV/AIDS related knowledge, and proficiency and engagement in the organizational social context were associated with program effectiveness.

CONCLUSIONS

Enormous resources are required to systematically and adequately test the role of multilevel factors on program effectiveness. We successfully implemented Mujer Segura in 13 sites in Mexico. Results suggest that other measures may need to be included in future implementation studies than the ones included here. We were able to point to a few specific factors that should be targeted to increase effectiveness of similar evidence-based programs in low- and other middle-income countries like Mexico.

TRIAL REGISTRATION

ClinicalTrials.gov NCT01465607.

The impact of implementation support on the use of a social engagement intervention for children with autism in public schools

Several interventions have demonstrated efficacy in improving social outcomes for children with autism; however, few have been successfully implemented in schools. This study compared two implementation strategies to improve the use of a social engagement intervention for children with autism in public schools. In total, 31 children with autism in grades K-5 and 28 school personnel participated in a randomized controlled trial. Schools were randomized to (1) training in Remaking Recess, a social engagement intervention, or (2) training in Remaking Recess with implementation support. Linear regression with random effects was used to test the intervention effects on implementation fidelity and social outcomes (peer engagement, social network inclusion, and friendship nominations). In both groups, implementation fidelity improved after training but remained low. Children in the Remaking Recess with implementation support condition had significantly higher social network inclusion and received more friendship nominations than children in the Remaking Recess-only condition (p = 0.03). Children in both groups experienced reduced solitary engagement (p < 0.001) and increased joint engagement (p < 0.001). The results suggest that implementation supports may have an effect on outcomes above and beyond the intervention, and that further research is needed into the active intervention mechanisms.

The Utility of Measuring Intentions to Use Best Practices: A Longitudinal Study Among Teachers Supporting Students With Autism

BACKGROUND

School-based interventions can result in profound social, behavioral, and academic improvement for students with autism, but teachers rarely implement them. It is important to understand why this occurs and use this information to increase the use of evidence-based practices. Toward this goal, 2 proof-of-construct studies demonstrate the theoretical and methodological advantages of measuring behavioral intentions to use specific practices.

METHODS

Two observational studies enrolled public school teachers who work with students with autism. The studies measure the strength of teachers' intentions to use each of 4 different evidence-based practices, assess variability in intentions, and test whether intentions predict future teacher behavior.

RESULTS

In adjusted analyses, intentions to use a specific evidence-based practice were strongly associated with subsequent use (adjusted odds ratio = 5.2). The proportion of teachers who reported strong intentions varied from a low of 33% to a high of 66%, depending on the practice.

CONCLUSIONS

The findings illustrate that the strength of intentions, which varies depending on the specific practice, can predict implementation. More generally, the studies demonstrate how measures of intention can aid efforts to identify implementation barriers. The approach taken can be applied to study implementation of any practices designed to improve student health.

Defining the external implementation context: an integrative systematic literature review

Background

Proper implementation of evidence-based interventions is necessary for their full impact to be realized. However, the majority of research to date has overlooked facilitators and barriers existing outside the boundaries of the implementing organization(s). Better understanding and measurement of the external implementation context would be particularly beneficial in light of complex health interventions that extend into and interact with the larger environment they are embedded within. We conducted a integrative systematic literature review to identify external context constructs likely to impact implementation of complex evidence-based interventions.

Methods

The review process was iterative due to our goal to inductively develop the identified constructs. Data collection occurred in four primary stages: (1) an initial set of key literature across disciplines was identified and used to inform (2) journal and (3) author searches that, in turn, informed the design of the final (4) database search. Additionally, (5) we conducted citation searches of relevant literature reviews identified in each stage. We carried out an inductive thematic content analysis with the goal of developing homogenous, well-defined, and mutually exclusive categories.

Results

We identified eight external context constructs: (1) professional influences, (2) political support, (3) social climate, (4) local infrastructure, (5) policy and legal climate, (6) relational climate, (7) target population, and (8) funding and economic climate.

Conclusions

This is the first study to our knowledge to use a systematic review process to identify empirically observed external context factors documented to impact implementation. Comparison with four widely-utilized implementation frameworks supports the exhaustiveness of our review process. Future work should focus on the development of more stringent operationalization and measurement of these external constructs.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3046-5) contains supplementary material, which is available to authorized users.

Mechanisms of Change in the ARC Organizational Strategy: Increasing Mental Health Clinicians' EBP Adoption Through Improved Organizational Culture and Capacity

The development of efficient and scalable implementation strategies in mental health is restricted by poor understanding of the change mechanisms that increase clinicians' evidence-based practice (EBP) adoption. This study tests the cross-level change mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians' EBP adoption and use. Four hundred seventy-five mental health clinicians in 14 children's mental health agencies were randomly assigned to the ARC intervention or a control condition. Measures of organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers were collected before, during, and upon completion of the three-year ARC intervention. EBP adoption and use were assessed at 12-month follow-up. Multilevel mediation analyses tested changes in organizational culture, clinicians' intentions to adopt EBPs, and job-related EBP barriers as linking mechanisms explaining the effects of ARC on clinicians' EBP adoption and use. ARC increased clinicians' EBP adoption (OR = 3.19, p = .003) and use (81 vs. 56 %, d = .79, p = .003) at 12-month follow-up. These effects were mediated by improvement in organizational proficiency culture leading to increased clinician intentions to adopt EBPs and by reduced job-related EBP barriers. A combined mediation analysis indicated the organizational culture-EBP intentions mechanism was the primary carrier of ARC's effects on clinicians' EBP adoption and use. ARC increases clinicians' EBP adoption and use by creating proficient organizational cultures that increase clinicians' intentions to adopt EBPs.

The impact of inter-organizational alignment (IOA) on implementation outcomes: evaluating unique and shared organizational influences in education sector mental health

BACKGROUND

Integrated healthcare delivered by work groups in nontraditional service settings is increasingly common, yet contemporary implementation frameworks typically assume a single organization-or organizational unit-within which system-level processes influence service quality and implementation success. Recent implementation frameworks predict that inter-organizational alignment (i.e., similarity in values, characteristics, activities related to implementation across organizations) may facilitate the implementation of evidence-based practices (EBP), but few studies have evaluated this premise. This study's aims examine the impact of overlapping organizational contexts by evaluating the implementation contexts of externally employed mental health clinicians working in schools-the most common integrated service delivery setting for children and adolescents. Aim 1 is to estimate the effects of unique intra-organizational implementation contexts and combined inter-organizational alignment on implementation outcomes. Aim 2 is to examine the underlying mechanisms through which inter-organizational alignment facilitates or hinders EBP implementation.

METHODS/DESIGN

This study will conduct sequential, exploratory mixed-methods research to evaluate the intra- and inter-organizational implementation contexts of schools and the external community-based organizations that most often employ school-based mental health clinicians, as they relate to mental health EBP implementation. Aim 1 will involve quantitative surveys with school-based, externally-employed mental health clinicians, their supervisors, and proximal school-employed staff (total n = 120 participants) to estimate the effects of each organization's general and implementation-specific organizational factors (e.g., climate, leadership) on implementation outcomes (fidelity, acceptability, appropriateness) and assess the moderating role of the degree of clinician embeddedness in the school setting. Aim 2 will explore the mechanisms through which inter-organizational alignment influences implementation outcomes by presenting the results of Aim 1 surveys to school-based clinicians (n = 30) and conducting semi-structured qualitative interviews. Qualitative data will be evaluated using an integrative inductive and deductive approach.

DISCUSSION

The study aims are expected to identify intra- and inter-organizational constructs that are most instrumental to EBP implementation success in school-based integrated care settings and illuminate mechanisms that may account for the influence of inter-organizational alignment. In addition to improving school-based mental health, these findings will spur future implementation science that considers the relationships across organizations and optimize the capacity of implementation science to guide practice in increasingly complex systems of care.

Applying the Policy Ecology Framework to Philadelphia's Behavioral Health Transformation Efforts

Raghavan et al. (Implement Sci 3(26):1-9, 2008) proposed that effective implementation of evidence-based practices requires implementation strategies deployed at multiple levels of the "policy ecology," including the organizational, regulatory or purchaser agency, political, and social levels. However, much of implementation research and practice targets providers without accounting for contextual factors that may influence provider behavior. This paper examines Philadelphia's efforts to work toward an evidence-based and recovery-oriented behavioral health system, and uses the policy ecology framework to illustrate how multifaceted, multilevel implementation strategies can facilitate the widespread implementation of evidence-based practices. Ongoing challenges and implications for research and practice are discussed.

Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice

A step toward the development of optimally effective, efficient, and feasible implementation strategies that increase evidence-based treatment integration in mental health services involves identification of the multilevel mechanisms through which these strategies influence implementation outcomes. This article (a) provides an orientation to, and rationale for, consideration of multilevel mediating mechanisms in implementation trials, and (b) systematically reviews randomized controlled trials that examined mediators of implementation strategies in mental health. Nine trials were located. Mediation-related methodological deficiencies were prevalent and no trials supported a hypothesized mediator. The most common reason was failure to engage the mediation target. Discussion focuses on directions to accelerate implementation strategy development in mental health.

Leading for the long haul: a mixed-method evaluation of the Sustainment Leadership Scale (SLS)

Background

Despite our progress in understanding the organizational context for implementation and specifically the role of leadership in implementation, its role in sustainment has received little attention. This paper took a mixed-method approach to examine leadership during the sustainment phase of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Utilizing the Implementation Leadership Scale as a foundation, we sought to develop a short, practical measure of sustainment leadership that can be used for both applied and research purposes.

Methods

Data for this study were collected as a part of a larger mixed-method study of evidence-based intervention, SafeCare®, sustainment. Quantitative data were collected from 157 providers using web-based surveys. Confirmatory factor analysis was used to examine the factor structure of the Sustainment Leadership Scale (SLS). Qualitative data were collected from 95 providers who participated in one of 15 focus groups. A framework approach guided qualitative data analysis. Mixed-method integration was also utilized to examine convergence of quantitative and qualitative findings.

Results

Confirmatory factor analysis supported the a priori higher order factor structure of the SLS with subscales indicating a single higher order sustainment leadership factor. The SLS demonstrated excellent internal consistency reliability. Qualitative analyses offered support for the dimensions of sustainment leadership captured by the quantitative measure, in addition to uncovering a fifth possible factor, available leadership.

Conclusions

This study found qualitative and quantitative support for the pragmatic SLS measure. The SLS can be used for assessing leadership of first-level leaders to understand how staff perceive leadership during sustainment and to suggest areas where leaders could direct more attention in order to increase the likelihood that EBIs are institutionalized into the normal functioning of the organization.