A Longitudinal Mixed Method Approach for Assessing Implementation Context and Process Factors: Comparison of Three Sites from a Housing First Implementation Strategy Pilot

Using capacity assessments and tailored technical assistance to advance trauma informed care integration at the organizational level

BACKGROUND

The prevalence of trauma among individuals with HIV has prompted efforts to integrate trauma-informed care (TIC) into HIV care and treatment to improve health outcomes. A TIC Implementation Model, developed by a US capacity-building organization focuses on organizational changes, aligning cultural and physical environments, emphasizing values like safety and trustworthiness, engaging leadership, and training staff in skills-based TIC services. Despite growing research, gaps remain in understanding the relationship between organizational capacity, provider knowledge, and the dosage of technical assistance (TA) required to sustain TIC integration. Researchers investigated how the project team adapted the type and amount of TA based on initial Cultural Assessment scores (measuring core TIC values) and its impact on Implementation Status scores.

METHODS

This study focuses on eight of 20 HIV care agencies in New Jersey that had largely met their TIC implementation goals by Spring 2022. As part of the TIC Implementation Model to measure agency capacity and implementation progress over time, agency staff and clients completed a Cultural Assessment (n = 72) and Physical Assessment (n = 43); staff completed a Pre/Post Training Survey (n = 296); and implementation teams at 8 agencies completed an Implementation Status Assessment Tool. Additionally, TA Logs capturing the details of TA meetings with the eight agencies were recorded by project staff. Data from these tools were analyzed in aggregate by agency using descriptive and correlational analyses.

RESULTS

Results demonstrated responsive TA correlated with agencies' baseline capacity. Agencies with lower capacity received significantly more frequent and extended TA encounters, which were associated with higher implementation scores and improvements in cultural environments for staff and clients (e.g., new protocols for staff response plans).

CONCLUSIONS

These findings underscore the importance of tailored TA in fostering diverse organizational cultures conducive to TIC implementation. For HIV care agencies, successful TIC implementation can impact health behaviors and outcomes for clients impacted by trauma. The TIC Implementation Model significantly advanced organizations' ability to transform their culture and systems, increasing their capacity to implement and sustain TIC integration. These results align with existing research that emphasizes when time is invested to shift organizational culture and develop leadership, new practices can effectively be implemented and scaled-up.

What happens after the funding ends?: A qualitative sustainability investigation of emergency department-based peer support programs implemented as part of Indiana's opioid state targeted response initiative

INTRODUCTION

In 2017, funding disseminated through the US Substance Abuse and Mental Health Services Administration's Opioid State Targeted Response (STR) program accelerated the expansion of peer recovery support services across several states to engage emergency department patients presenting with opioid use disorder. While there is some literature on the initial implementation of these programs, little is known about their sustainability after the STR funding's end. Identifying what happened to these programs is a key component of understanding their ultimate impact and can inform future activities to develop, fund, or sustain similar efforts.

METHODS

We collected qualitative data from six organizations that participated in Indiana's STR-funded Recovery Coaching and Peer Support Initiative (RCPSI). The semi-structured interview guide was designed to gather data related to eight domains of sustainability (i.e., environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, program evaluation, communications, and strategic planning). The analysts followed a deductive-inductive analysis approach, using the eight domains as an a priori coding structure and developing higher-level inductive themes.

RESULTS

A total of ten individuals (roles included 4 Program Supervisors, 2 Nurse Administrators, a Psychiatric Social Worker, a Mobile Treatment Manager, a Grant Coordinator, and a Vice President of Planning) participated in six interviews. Two programs did not sustain services, primarily because they lacked a sufficient volume of eligible patients to justify services. Factors identified as supporting sustainability in the other four programs included (1) identification of alternate funding sources, (2) evolving internal support for ED-based opioid use disorder treatment, and (3) investment in internal and external relationships. Furthermore, these themes operated across multiple sustainability domains.

CONCLUSIONS

The findings illustrate a dynamic interplay between program context and multiple theorized sustainability domains that impacted the viability of RCPSI programs after the end of STR funding. Results indicate a need for a better understanding of the factors influencing the sustainability of programs supported by federal funding to mitigate the opioid crisis, and such findings will likely apply to a broader range of grant-supported programs.

Implementation cost analysis of collaborative care for perinatal mental health in community health centers

Background

Although costs are of key importance to clinic leadership when considering adoption of new programs, few studies examine real-world resource needs associated with implementing complex interventions for chronic conditions in primary care. This analysis sought to identify the costs necessary to implement the evidence-based collaborative care model (CoCM), an integrated behavioral health program for common mental disorders in primary care.

Methods

Ten federally qualified health centers (FQHCs) adopted CoCM as part of a larger national randomized trial evaluating implementation strategies for CoCM when adapted for perinatal mental health. The Cost of Implementing New Strategies (COINS) tool was used to assess implementation costs associated with activities completed by sites as they progressed through the implementation process. National wage norms were used to calculate cost estimates for staff time.

Results

On average, clinics spent $40,778 (SD=$30,611) on implementation, with clinics ranging widely from $4,502 to $103,156. Three out of 10 participating clinics achieved competency in the intervention during the 2-year implementation period. Costs among competent clinics ranged from $20,944 to $65,415 (mean=$41,788). Clinics that did not achieve competency were more varied, with both the lowest and highest resource use. Significant staff effort was required to complete all implementation stages; clinical staff and program champions showed greatest effort.

Conclusions

Site implementation costs for this complex behavioral health intervention were substantial and varied dramatically, particularly among sites who did not achieve competence. Additional work is needed to identify optimal site resource investment related to implementation success for CoCM.

Trial registration

ClinicalTrials.gov.NCT02976025. Registered on November 23, 2016.

Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Background

Dissemination and implementation frameworks provide the scaffolding to explore the effectiveness of evidence-based practices (EBPs) targeting process of care and organizational outcomes. Few instruments, like the stages of implementation completion (SIC) examine implementation fidelity to EBP adoption and how organizations differ in their approach to implementation. Instruments to measure organizational competency in the utilization of implementation strategies are lacking.

Method

An iterative process was utilized to adapt the SIC to the NIATx implementation strategies. The new instrument, NIATx-SIC, was applied in a randomized controlled trial involving 53 addiction treatment agencies in Washington state to improve agency co-occurring capacity. NIATx-SIC data were reported by state staff and external facilitators and through participating agency documentation. Proportion and duration scores for each stage and phase of the NIATx-SIC were calculated for each agency. Competency was assessed using the NIATx fidelity tool. Comparisons of proportion, duration, and NIATx activities completed were determined using independent sample t-tests by agency competency level.

Results

The NIATx-SIC distinguished between agencies achieving competency (n  =  23) and those not achieving competency (n  =  26). Agencies achieving competency completed a greater proportion of implementation phase activities and had a significantly longer Stage 7 duration. These agencies participated in significantly more individual and group coaching calls, attended more in-person meetings, implemented more change projects, and spent approximately 64 more days, on average, engaging in all NIATx activities.

Conclusions

Organizational participation in dissemination and implementation research requires a significant investment of staff resources. The inability of an organization to achieve competency when utilizing a set of implementation strategies waste an opportunity to institutionalize knowledge of how to apply implementation strategies to future change efforts. The NIATx-SIC provides evidence that competency is not an attribute of the organization but rather a result of the application of the NIATx implementation strategies to improve agency co-occurring capacity.

Trial Registration

ClinicalTrials.gov, NCT03007940. Registered January 2, 2017, https://clinicaltrials.gov/ct2/show/NCT03007940.

Stepped implementation-to-target: a study protocol of an adaptive trial to expand access to addiction medications

BACKGROUND

In response to the US opioid epidemic, significant national campaigns have been launched to expand access to `opioid use disorder (MOUD). While adoption has increased in general medical care settings, specialty addiction programs have lagged in both reach and adoption. Elevating the quality of implementation strategy, research requires more precise methods in tailoring strategies rather than a one-size-fits-all-approach, documenting participant engagement and fidelity to the delivery of the strategy, and conducting an economic analysis to inform decision making and policy. Research has yet to incorporate all three of these recommendations to address the challenges of implementing and sustaining MOUD in specialty addiction programs.

METHODS

This project seeks to recruit 72 specialty addiction programs in partnership with the Washington State Health Care Authority and employs a measurement-based stepped implementation-to-target approach within an adaptive trial design. Programs will be exposed to a sequence of implementation strategies of increasing intensity and cost: (1) enhanced monitoring and feedback (EMF), (2) 2-day workshop, and then, if outcome targets are not achieved, randomization to either internal facilitation or external facilitation. The study has three aims: (1) evaluate the sequential impact of implementation strategies on target outcomes, (2) examine contextual moderators and mediators of outcomes in response to the strategies, and (3) document and model costs per implementation strategy. Target outcomes are organized by the RE-AIM framework and the Addiction Care Cascade.

DISCUSSION

This implementation project includes elements of a sequential multiple assignment randomized trial (SMART) design and a criterion-based design. An innovative and efficient approach, participating programs only receive the implementation strategies they need to achieve target outcomes. Findings have the potential to inform implementation research and provide key decision-makers with evidence on how to address the opioid epidemic at a systems level.

TRIAL REGISTRATION

This trial was registered at ClinicalTrials.gov (NCT05343793) on April 25, 2022.

Rates of sustainment in the Universal Stages of Implementation Completion

BACKGROUND

Sustainment is a desirable outcome of implementation, but its precise definition remains unclear, contributing to the difficulty of identifying a generalized rate of sustainment. Several studies and reviews on the topic differ on both definition and levels of analysis. Furthermore, methodological limitations might have influenced the results, including the unknown quality with which some interventions were delivered. The Universal Stages of Implementation Completion (UniSIC) is a standardized measurement tool that tracks the implementation process and milestone completion across a wide range of real-world implementations-this provides a unique opportunity to identify a generalized rate of sustainment.

METHODS

UniSIC data was captured from the SIC website on 27 September 2020 and included data from all sites (n = 1778) that had been tracked to date. Data were restricted to sites that achieved competency in program delivery, and thus had a newly adopted program worthy of sustainment. Dates and indicator variables of implementation activities were combined to form two alternate definitions of sustainment: sustained (start-up) was achieved if sites continued to deliver services 2 years past their program start-up date; sustained (competent) was achieved if sites continued to deliver services 2 years past their competence and/or certification date. Of sites eligible for inclusion based on these definitions (N = 208), descriptive analyses were conducted to determine a rate of sustainment for all programs that successfully started a program. These definitions were also applied to a combined sample for a general rate of sustainment among all sites. Rates of competency among both a sample of sites that started up and a combined sample were also identified.

RESULTS

The rate of competence was 58.5% and the rate of sustained (start-up) was 37.1%, while the rate of sustained (competent) was 25.1%. The rates of competence and sustainment among the combined samples were far lower: 15.6% for competence, 6.8% for sustained (start-up), and 4.4% for sustained (competent).

CONCLUSIONS

These identified rates of sustainment are accurate initial estimates of sustainment of community-based practices, or in general. Future research on rates of sustainment should carefully define measures of sustainment and be transparent about the real-world conditions on which analyses are centered.

Barriers impacting the POINT pragmatic trial: the unavoidable overlap between research and intervention procedures in "real-world" research

BACKGROUND

This manuscript provides a research update to the ongoing pragmatic trial of Project POINT (Planned Outreach, Intervention, Naloxone, and Treatment), an emergency department-based peer recovery coaching intervention for linking patients with opioid use disorder to evidence-based treatment. The research team has encountered a number of challenges related to the "real-world" study setting since the trial began. Using an implementation science lens, we sought to identify and describe barriers impacting both the intervention and research protocols of the POINT study, which are often intertwined in pragmatic trials due to the focus on external validity.

METHOD

Qualitative data were collected from 3 peer recovery coaches, 2 peer recovery coach supervisors, and 3 members of the research team. Questions and deductive qualitative analysis were guided by the Consolidated Framework for Implementation Research (CFIR).

RESULTS

Nine unique barriers were noted, with 5 of these barriers impacting intervention and research protocol implementation simultaneously. These simultaneous barriers were timing of intervention delivery, ineffective communication with emergency department staff, lack of privacy in the emergency department, the fast-paced emergency department setting, and patient's limited resources. Together, these barriers represent the intervention characteristics, inner setting, and outer setting domains of the CFIR.

CONCLUSION

Results highlight the utility of employing an implementation science framework to assess implementation issues in pragmatic trials and how this approach might be used as a quality assurance mechanism given the considerable overlap that exists between research and intervention protocols in real-world trial settings. Previously undocumented changes to the trial design that have been made as a result of the identified barriers are discussed.