The impact of inter-organizational alignment (IOA) on implementation outcomes: evaluating unique and shared organizational influences in education sector mental health

Randomized trial of the leadership and organizational change for implementation (LOCI) strategy in substance use treatment clinics

INTRODUCTION

Evidence-based practice (EBP) implementation represents a strategic change that requires alignment of leadership and support throughout organizations. Leadership and Organizational Change for Implementation (LOCI) is a multifaceted implementation strategy that aims to improve implementation leadership and climate within organizations through iterative cycles of leadership and climate assessment and feedback, leadership training and coaching, and strategic planning with upper-level leaders. This study tested the effects of LOCI on transformational and implementation leadership, implementation climate, implementation citizenship behavior, and EBP reach.

METHODS

A multiple cohort, cluster randomized trial tests the effect of LOCI in 60 clinics across nine behavioral health organizations in California and Arizona, USA. The study randomized clinics within organizations to either LOCI or a leadership training webinar control condition in three consecutive cohorts. Repeated web-based surveys of direct service providers (nLOCI = 201, nControl = 179) assessed leadership, implementation climate, and implementation citizenship over time. Multilevel autoregressive modeling was the primary statistical analysis such that providers (level-1) were nested within clinics (level-2). The study predicted between-condition differences at 4-, 8-, and 12-month follow-up assessments. Provider engagement in a fidelity monitoring process assessed reach of motivational interviewing (i.e., number of sessions recorded/submitted for fidelity coding). An independent sample t-test explored between condition differences in motivational interviewing reach.

RESULTS

Results indicated between condition differences at 4 months for implementation leadership, implementation climate, and implementation citizenship behavior such that greater improvements were evidenced in the LOCI condition compared to the control condition. Reach of MI was significantly greater in the LOCI vs control condition such that LOCI providers were significantly more likely to engage in the fidelity monitoring process (chi-square (1, n = 370) = 5.59, p = .018).

CONCLUSIONS

LOCI was developed based on organizational theories of strategic leadership and climate to affect organizational change processes that communicate that innovation implementation is expected, supported, and recognized as a value of the organization. The LOCI implementation strategy resulted in more positive hypothesized outcomes compared to the control condition. Organizational change strategies have utility for implementing health innovations in complex, multilevel contexts and for greater sustainment of facilitative leader behaviors, strategic implementation climate, and improved implementation outcomes.

TRIAL REGISTRATION

This study is registered with Clinicaltrials.gov gov (NCT03042832, 2 February 2017; retrospectively registered).

Inter-organizational alignment and implementation outcomes in integrated mental healthcare for children and adolescents: a cross-sectional observational study

BACKGROUND

Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOA in general and implementation context factors are associated with implementation outcomes in integrated SBMH.

METHODS

SBMH clinicians from community-based organizations (CBOs; nclinician = 27) and their proximal student-support school staff (nschool = 99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes.

RESULTS

The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors.

CONCLUSIONS

This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

Building a communication and support network among quality improvement teams in nursing homes: a longitudinal study of the SCOPE trial

BACKGROUND

We applied a longitudinal network analysis approach to assess the formation of knowledge sharing and collaboration networks among care aide-led quality improvement (QI) teams in Canadian nursing homes participating in the Safer Care for Older Persons (in residential) Environments (SCOPE) trial which aimed to support unregulated front-line staff to lead unit-based quality improvement (QI) teams in nursing homes. We hypothesized that SCOPE's communicative and participatory nature would provide opportunities for peer support, knowledge sharing, and collaboration building among teams.

METHODS

Fourteen QI teams in Alberta (AB) and seventeen QI teams in British Columbia (BC) participated in the study. Communications across nursing homes occurred through a series of 4 collaborative Learning Congresses (training sessions) over a 1-year period. The senior leaders of QI teams participated in two online network surveys about the communication/collaboration between teams in their province, 1 month after the first, and 6 months later, after the fourth Learning Congress. We developed communication and collaboration network maps pertaining to three time points: before SCOPE, at 2 months, and at 9 months.

RESULTS

Over time, teams made significantly more new connections and strengthened existing ones, within and across regions. Geographic proximity and co-membership in organizational chains were important predictors of connectivity before and during SCOPE. Teams whose members were well connected at baseline disproportionately improved connectivity over time. On the other hand, teams that did not have prior opportunities to connect appeared to use SCOPE to build new ties.

CONCLUSIONS

Our findings suggest the importance of network-altering activities to the formation of collaboration networks among QI teams across nursing homes. Active strategies could be used to better connect less connected teams and facilitate collaboration among geographically proximate teams. These findings may inform the development of interventions to leverage existing networks and provide new networking opportunities to develop and sustain organizational improvements.

Different factors identified by stakeholder group for barriers and facilitators to measurement-based care implementation in behavioral health clinics

INTRODUCTION

Despite the benefits of measurement-based care (MBC) in the behavioral health setting, there have been difficulties in implementation and low saturation. Although barriers and facilitators to MBC implementation have been identified, research has generally only included the perspective of one stakeholder group. The current study aims to examine the similarities and differences-by stakeholder group-in the identified barriers to and facilitators of implementing MBC in the behavioral health setting.

METHOD

A purposeful sampling approach was used to recruit and conduct interviews and focus groups with stakeholders (clinicians, clinic leaders, and administrative staff) from four behavioral health clinics at an academic medical center that is part of a larger healthcare system. The data coding process included a directed content analytic approach whereby the coding team used an iterative process to analyze deidentified transcripts starting with a codebook based on the Consolidated Framework for Implementation Research (CFIR) constructs.

RESULTS

A total of 31 clinicians, 11 clinic leaders, and 8 administrative staff participated in the interviews and focus groups. There was convergence among all stakeholder regarding which CFIR constructs were identified as barriers and facilitators, but there were differences in the specific thematic factors identified by stakeholders as barriers and facilitators within each of these implementation constructs. The barriers and facilitators that stakeholders identified within each CFIR construct were often connected to their specific role in implementing MBC.

CONCLUSION

Collecting information on barriers and facilitators to MBC implementation from the multiple stakeholders involved in the process may enhance successful implementation of MBC given the variation between groups in identified thematic factors. Administrative staff perspectives, which have not been reported in the literature, may be of particular importance in planning for successful MBC implementation.

Protocol for a hybrid type 3 effectiveness-implementation trial of a pragmatic individual-level implementation strategy for supporting school-based prevention programming

BACKGROUND

For approximately one in five children who have social, emotional, and behavioral (SEB) challenges, accessible evidence-based prevention practices (EBPPs) are critical. In the USA, schools are the primary setting for children's SEB service delivery. Still, EBPPs are rarely adopted and implemented by front-line educators (e.g., teachers) with sufficient fidelity to see effects. Given that individual behavior change is ultimately required for successful implementation, focusing on individual-level processes holds promise as a parsimonious approach to enhance impact. Beliefs and Attitudes for Successful Implementation in Schools for Teachers (BASIS-T) is a pragmatic, multifaceted pre-implementation strategy targeting volitional and motivational mechanisms of educators' behavior change to enhance implementation and student SEB outcomes. This study protocol describes a hybrid type 3 effectiveness-implementation trial designed to evaluate the main effects, mediators, and moderators of the BASIS-T implementation strategy as applied to Positive Greetings at the Door, a universal school-based EBPP previously demonstrated to reduce student disruptive behavior and increase academic engagement.

METHODS

This project uses a blocked randomized cohort design with an active comparison control (ACC) condition. We will recruit and include approximately 276 teachers from 46 schools randomly assigned to BASIS-T or ACC conditions. Aim 1 will evaluate the main effects of BASIS-T on proximal implementation mechanisms (attitudes, subjective norms, self-efficacy, intentions to implement, and maintenance self-efficacy), implementation outcomes (adoption, reach, fidelity, and sustainment), and child outcomes (SEB, attendance, discipline, achievement). Aim 2 will examine how, for whom, under what conditions, and how efficiently BASIS-T works, specifically by testing whether the effects of BASIS-T on child outcomes are (a) mediated via its putative mechanisms of behavior change, (b) moderated by teacher factors or school contextual factors, and (c) cost-effective.

DISCUSSION

This study will provide a rigorous test of BASIS-T-a pragmatic, theory-driven, and generalizable implementation strategy designed to target theoretically-derived motivational mechanisms-to increase the yield of standard EBPP training and support strategies.

TRIAL REGISTRATION

ClinicalTrials.gov ID: NCT05989568. Registered on May 30, 2023.

Generating Actionable Evidence for School-Based Mental Health Service Delivery: Public-Academic Partnership Based Evaluations

Public-academic partnership-based program evaluations can generate actionable evidence for policymaking, program design and implementation in improving school-based mental health service delivery. The University of Pennsylvania Center for Mental Health and public behavioral health care agencies in Philadelphia in the United States have evaluated Philadelphia's school mental health programs reimbursable through Medicaid billing since 2008. The variety of evaluations include (1) examining acute mental health service use of children receiving school-based mental health care and Medicaid expenditure, (2) examining children's externalizing and internalizing behaviors to measure school mental health providers' performance, and (3) examining effects of different types of school mental health programs on children's behavioral health functioning, school outcomes, and other out-of-school service use. This paper reports key findings of these evaluations, discusses how programs have been refined based on evaluation results, and shares lessons learned for successful public-academic partnership-based evaluations to promote use of actionable evidence.

Readiness to implement screening, brief intervention, and referral to treatment in school-based health centers

The purpose of this study was to evaluate the readiness of a university-based school-based health center (SBHC) program to implement the screening, brief intervention, and referral to treatment (SBIRT) model. We completed preimplementation surveys and interviews with providers, staff, and administrators at participating SBHCs (N = 19) to measure current protocols for and barriers to addressing adolescent substance use and barriers and facilitators to implementing SBIRT. We used the R = MC² heuristic (readiness equals motivation, general organizational capacity, and innovation-specific capacity) to interpret findings from the data. Using the R = MC² heuristic, we identified factors that may aid implementation, including the prioritization of substance use and a supportive organizational culture, as well as potential barriers, including competing high-priority health issues and a lack of resources for innovation. This study contributes to the emerging literature on the implementation of the SBIRT model with adolescents and in SBHCs and demonstrates the practical applicability of the R = MC² heuristic for assessing implementation readiness. SBHCs are ideal locations for addressing substance use but SBHC providers also face unique challenges when implementing the SBIRT model.

Implementing School-Based Mental Health Services: A Scoping Review of the Literature Summarizing the Factors That Affect Implementation

Background: Mental illness in children and youths has become an increasing problem. School-based mental health services (SBMHS) are an attempt to increase accessibility to mental health services. The effects of these services seem positive, with some mixed results. To date, little is known about the implementation process of SBMHS. Therefore, this scoping review synthesizes the literature on factors that affect the implementation of SBMHS. Methods: A scoping review based on four stages: (a) identifying relevant studies; (b) study selection; (c) charting the data; and (d) collating, summarizing, and reporting the results was performed. From the searches (4414 citations), 360 were include in the full-text screen and 38 in the review. Results: Implementation-related factors were found in all five domains of the Consolidated Framework for Implementation Research. However, certain subfactors were mentioned more often (e.g., the adaptability of the programs, communication, or engagement of key stakeholders). Conclusions: Even though SBMHS differed in their goals and way they were conducted, certain common implementation factors were highlighted more frequently. To minimize the challenges associated with these types of interventions, learning about the implementation of SBMHS and using this knowledge in practice when introducing SBMHS is essential to achieving the best possible effects with SMBHSs.

Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum

BACKGROUND

The challenges of implementing evidence-based innovations (EBIs) are widely recognized among practitioners and researchers. Context, broadly defined as everything outside the EBI, includes the dynamic and diverse array of forces working for or against implementation efforts. The Consolidated Framework for Implementation Research (CFIR) is one of the most widely used frameworks to guide assessment of contextual determinants of implementation. The original 2009 article invited critique in recognition for the need for the framework to evolve. As implementation science has matured, gaps in the CFIR have been identified and updates are needed. Our team is developing the CFIR 2.0 based on a literature review and follow-up survey with authors. We propose an Outcomes Addendum to the CFIR to address recommendations from these sources to include outcomes in the framework.

MAIN TEXT

We conducted a literature review and surveyed corresponding authors of included articles to identify recommendations for the CFIR. There were recommendations to add both implementation and innovation outcomes from these sources. Based on these recommendations, we make conceptual distinctions between (1) anticipated implementation outcomes and actual implementation outcomes, (2) implementation outcomes and innovation outcomes, and (3) CFIR-based implementation determinants and innovation determinants.

CONCLUSION

An Outcomes Addendum to the CFIR is proposed. Our goal is to offer clear conceptual distinctions between types of outcomes for use with the CFIR, and perhaps other determinant implementation frameworks as well. These distinctions can help bring clarity as researchers consider which outcomes are most appropriate to evaluate in their research. We hope that sharing this in advance will generate feedback and debate about the merits of our proposed addendum.

Alignment in implementation of evidence-based interventions: a scoping review

Background

Alignment (i.e., the process of creating fit between elements of the inner and outer context of an organization or system) in conjunction with implementation of an evidence-based intervention (EBI) has been identified as important for implementation outcomes. However, research evidence has so far not been systematically summarized. The aim of this scoping review is therefore to create an overview of how the concept of alignment has been applied in the EBI implementation literature to provide a starting point for future implementation efforts in health care.

Methods

We searched for peer-reviewed English language articles in four databases (MEDLINE, Cinahl, Embase, and Web of Science) published between 2003 and 2019. Extracted data were analyzed to address the study aims. A qualitative content analysis was carried out for items with more extensive information. The review was reported according to the preferred reporting items for systematic reviews and meta-analyses extension for scoping review (PRISMA-ScR) guidelines.

Results

The database searches yielded 3629 publications, of which 235 were considered potentially relevant based on the predetermined eligibility criteria, and retrieved in full text. In this review, the results of 53 studies are presented. Different definitions and conceptualizations of alignment were found, which in general could be categorized as structural, as well as social, types of alignments. Whereas the majority of studies viewed alignment as important to understand the implementation process, only a few studies actually assessed alignment. Outcomes of alignment were focused on either EBI implementation, EBI sustainment, or healthcare procedures. Different actors were identified as important for creating alignment and five overall strategies were found for achieving alignment.

Conclusions

Although investigating alignment has not been the primary focus of studies focusing on EBI implementation, it has still been identified as an important factor for the implementation success. Based on the findings from this review, future research should incorporate alignment and put a stronger emphasize on testing the effectiveness of alignment related to implementation outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-021-01160-w.

Dissemination Science in School Mental Health: A Framework for Future Research

There has been an increase in school mental health research aimed at producing generalizable knowledge to address longstanding science-to-practice gaps to increase children's access to evidence-based mental health services. Successful dissemination and implementation are both important pieces to address science-to-practice gaps, but there is conceptual and semantic imprecision that creates confusion regarding where dissemination ends and implementation begins, as well as an imbalanced focus in research on implementation relative to dissemination. In this paper, we provide an enhanced operational definition of dissemination; offer a conceptual model that outlines elements of effective dissemination that can produce changes in awareness, knowledge, perceptions, and motivation across different stakeholder groups; and delineate guiding principles that can inform dissemination science and practice. The overarching goal of this paper is to stimulate future research that aims to advance dissemination science and practice in school mental health.

Bringing Evidence-Based Interventions into the Schools: An Examination of Organizational Factors and Implementation Outcomes

Although schools are one of the largest providers of behavioral health services for youth, many barriers exist to the implementation of evidence-based interventions in schools. This study used the Stages of Implementation Completion (SIC) to examine school-based implementation outcomes for a computer-assisted cognitive behavioral therapy intervention for anxious youth. Organizational factors and predictors of program startup also were examined. Results indicated that the SIC detected implementation variability in schools and suggested that spending more time completing pre-implementation activities may better prepare schools for active implementation of program delivery. Furthermore, proficiency emerged as a potentially important organizational factor to examine in future school-based implementation research.

Implementing and Sustaining SHARE: An Exposure-Based Psychotherapy Group for Incarcerated Women Survivors of Sexual Violence

Although incarcerated women are a highly victimized population, therapy for sexual violence victimization (SVV) sequela is not routinely offered in prison. SHARE is a group therapy for SVV survivors that was successfully implemented and sustained in a women's correction center. Here, we aimed to identify implementation factors and strategies that led to SHARE's success and describe incarcerated women's perspectives on the program. We conducted a retrospective process evaluation using interviews structured according to EPIS, a well-established implementation science framework. Participants (N = 22) were incarcerated women, members of the SHARE treatment team, and members of the correction center's leadership, therapeutic team, and volunteer program. Factors that facilitated SHARE implementation varied by EPIS phase and organization. Positive inter-organizational and interpersonal relationships were key across phases, as were the synergies between both the strengths and needs of each organization involved in implementation. Incarcerated women reported a strong need for SHARE and did not report any concerns about receiving trauma therapy in a carceral setting. Therapy for SVV sequelae, including exposure-based therapy, is possible to implement and sustain in carceral settings. Community-academic partnerships may be a particularly feasible way to expand access to SVV therapy for incarcerated women.

Protocol for a hybrid type 2 cluster randomized trial of trauma-focused cognitive behavioral therapy and a pragmatic individual-level implementation strategy

BACKGROUND

More than two-thirds of youth experience trauma during childhood, and up to 1 in 5 of these youth develops posttraumatic stress symptoms that significantly impair their functioning. Although trauma-focused cognitive behavior therapy (TF-CBT) has a strong evidence base, it is rarely adopted, delivered with adequate fidelity, or evaluated in the most common setting where youth access mental health services-schools. Given that individual behavior change is ultimately required for successful implementation, even when organizational factors are firmly in place, focusing on individual-level processes represents a potentially parsimonious approach. Beliefs and Attitudes for Successful Implementation in Schools (BASIS) is a pragmatic, motivationally focused multifaceted strategy that augments training and consultation and is designed to target precise mechanisms of behavior change to produce enhanced implementation and youth clinical outcomes. This study protocol describes a hybrid type 2 effectiveness-implementation trial designed to concurrently evaluate the main effects, mediators, and moderators of both the BASIS implementation strategy on implementation outcomes and TF-CBT on youth mental health outcomes.

METHODS

Using a cluster randomized controlled design, this trial will assign school-based mental health (SMH) clinicians and schools to one of three study arms: (a) enhanced treatment-as-usual (TAU), (b) attention control plus TF-CBT, or (c) BASIS+TF-CBT. With a proposed sample of 120 SMH clinicians who will each recruit 4-6 youth with a history of trauma (480 children), this project will gather data across 12 different time points to address two project aims. Aim 1 will evaluate, relative to an enhanced TAU condition, the effects of TF-CBT on identified mechanisms of change, youth mental health outcomes, and intervention costs and cost-effectiveness. Aim 2 will compare the effects of BASIS against an attention control plus TF-CBT condition on theoretical mechanisms of clinician behavior change and implementation outcomes, as well as examine costs and cost-effectiveness.

DISCUSSION

This study will generate critical knowledge about the effectiveness and cost-effectiveness of BASIS-a pragmatic, theory-driven, and generalizable implementation strategy designed to enhance motivation-to increase the yield of evidence-based practice training and consultation, as well as the effectiveness of TF-CBT in a novel service setting.

TRIAL REGISTRATION

ClinicalTrials.gov registration number NCT04451161 . Registered on June 30, 2020.

Sustaining Treatment for Youth with Problematic Sexual Behavior: Administrator and Stakeholder Perspectives Following Implementation

This study examines administrator and stakeholder perspectives on factors influencing the sustainability of Problematic Sexual Behavior Cognitive Behavioral Therapy (PSB-CBT). After initial implementation of PSB-CBT, qualitative interviews (N = 42) on the sustainability of services were conducted with treatment program administrators (n = 10) and community stakeholders (n = 32) from six sites across the USA. Interviewees discussed key facilitators and barriers for sustainability and identified the following factors as impacting the sustainability of PSB-CBT: public relations, policy, inter-agency collaboration, funding, and workforce issues. Results were overall consistent with the EPIS (Exploration, Preparation, Implementation, Sustainment) model for implementation in public service sectors, but proposed modifications are also noted. Findings underscore the value of qualitative research in identifying best practices for sustaining valuable therapeutic interventions. The authors argue that future efforts to sustain evidence-based interventions, especially those dealing with sensitive topics around which misinformation and stigma are associated, should prioritize public relations (i.e., outreach, education) in addition to the intervention.

Implementation Determinants and Outcomes of a Technology-Enabled Service Targeting Suicide Risk in High Schools: Mixed Methods Study

BACKGROUND

Technology-enabled services (TESs), which integrate human service and digital components, are popular strategies to increase the reach and impact of mental health interventions, but large-scale implementation of TESs has lagged behind their potential.

OBJECTIVE

This study applied a mixed qualitative and quantitative approach to gather input from multiple key user groups (students and educators) and to understand the factors that support successful implementation (implementation determinants) and implementation outcomes of a TES for universal screening, ongoing monitoring, and support for suicide risk management in the school setting.

METHODS

A total of 111 students in the 9th to 12th grade completed measures regarding implementation outcomes (acceptability, feasibility, and appropriateness) via an open-ended survey. A total of 9 school personnel (school-based mental health clinicians, nurses, and administrators) completed laboratory-based usability testing of a dashboard tracking the suicide risk of students, quantitative measures, and qualitative interviews to understand key implementation outcomes and determinants. School personnel were presented with a series of scenarios and common tasks focused on the basic features and functions of the dashboard. Directed content analysis based on the Consolidated Framework for Implementation Research was used to extract multilevel determinants (ie, the barriers or facilitators at the levels of the outer setting, inner setting, individuals, intervention, and implementation process) related to positive implementation outcomes of the TES.

RESULTS

Overarching themes related to implementation determinants and outcomes suggest that both student and school personnel users view TESs for suicide prevention as moderately feasible and acceptable based on the Acceptability of Intervention Measure and Feasibility of Intervention Measure and as needing improvements in usability based on the System Usability Scale. Qualitative results suggest that students and school personnel view passive data collection based on social media data as a relative advantage to the current system; however, the findings indicate that the TES and the school setting need to address issues of privacy, integration into existing workflows and communication patterns, and options for individualization for student-centered care.

CONCLUSIONS

Innovative suicide prevention strategies that rely on passive data collection in the school context are a promising and appealing idea. Usability testing identified key issues for revision to facilitate widespread implementation.

From evidence to impact: Joining our best school mental health practices with our best implementation strategies

There is substantial research evidence for the effectiveness of school mental health strategies across problem areas, developmental levels, and the prevention-intervention spectrum. At the same time, it is clear that the education and mental health fields continue to struggle to apply this evidence at any level of scale. This commentary reflects on ways in which education-specific applications of implementation science principles - and explicit consideration of determinants of implementation success - may guide more consistent use of evidence in school mental health. After reviewing implementation determinants and strategies across multiple levels of effect (i.e., the outer setting, inner setting, individual, and intervention levels), the commentary goes on to recommend specific areas of needed attention in school mental health implementation efforts and research. These include a need to adapt interventions to better fit the context of schools, streamlining school mental health programs and practices to make them more implementable, and recognizing the critical role of assessment and selection of evidence-based interventions by school leaders. The commentary concludes by reflecting on the substantial opportunity provided by the education sector to both apply and advance implementation science.