Adaptive School-based Implementation of CBT (ASIC): clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools

Optimization of implementation strategies using the Multiphase Optimization STratgey (MOST) framework: Practical guidance using the factorial design

The Multiphase Optimization STrategy (MOST) is a framework that uses three phases-preparation, optimization, and evaluation-to develop multicomponent interventions that achieve intervention EASE by strategically balancing Effectiveness, Affordability, Scalability, and Efficiency. In implementation science, optimization of the intervention requires focus on the implementation strategies-things that we do to deliver the intervention-and implementation outcomes. MOST has been primarily used to optimize the components of the intervention related to behavioral or health outcomes. However, innovative opportunities to optimize discrete (i.e. single strategy) and multifaceted (i.e. multiple strategies) implementation strategies exist and can be done independently, or in conjunction with, intervention optimization. This article details four scenarios where the MOST framework and the factorial design can be used in the optimization of implementation strategies: (i) the development of new multifaceted implementation strategies; (ii) evaluating interactions between program components and a discrete or multifaceted implementation strategies; (iii) evaluating the independent effects of several discrete strategies that have been previously evaluated as a multifaceted implementation strategy; and (iv) modification of a discrete or multifaceted implementation strategy for the local context. We supply hypothetical school-based physical activity examples to illustrate these four scenarios, and we provide hypothetical data that can help readers make informed decisions derived from their trial data. This manuscript offers a blueprint for implementation scientists such that not only is the field using MOST to optimize the effectiveness of an intervention on a behavioral or health outcome, but also that the implementation of that intervention is optimized.

Student mental health outcomes of a clustered SMART for developing an adaptive implementation strategy to support school-based CBT delivery

PURPOSE

Most youth experiencing anxiety/depression lack access to evidence-based mental health practices (EBPs). School-delivered care improves access, and various support can help school professionals (SPs; school social workers, counselors) deliver EBPs, like Cognitive Behavioral Therapy (CBT). Understanding implementation strategies' impact on downstream mental health outcomes is crucial to scaling up EBPs to address the treatment gap, but it has rarely been assessed.

METHODS

This paper compares implementation strategies' impact on change in student outcomes, collected as exploratory outcomes from a type III hybrid implementation-effectiveness trial. A clustered, sequential, multiple-assignment randomized trial design was used, which embedded four implementation supports that differentially sequence three implementation strategies, Replicating Effective Programs (REP), Coaching, and Facilitation. Prior to the first randomization, N = 169 SPs from 94 Michigan high schools each identified up to 10 students whom they believed could benefit from CBT and facilitated student survey completion. Changes in students' depression (Patient Health Questionnaire-9, modified for teens) and anxiety symptoms (Generalized Anxiety Disorder-7) over 10 months were compared across the four sequences of implementation support using a generalization of a marginal, weighted least squares approach developed for a clustered SMARTs.

RESULTS

Small, non-clinically significant reductions in symptoms over the study period were found. Pairwise comparisons found no significant differences in symptom change across the four implementation strategies. The difference in the estimated mean PHQ-9T/GAD-7 scores between the least and the most intensive strategies (REP vs. REP+Coaching+Facilitation) was 1.04 (95%CI = -0.95, 3.04) for depression and 0.82 (95%CI = -0.89, 2.52) for anxiety.

DISCUSSION

No difference in symptom change was found across the four implementation strategies. Multiple forms of implementation support may be useful for improving student mental health outcomes.

TRIAL REGISTRATION

NCT03541317-Registered on 29 May 2018 on ClinicalTrials.gov PRS.

Adaptive Designs in Implementation Science and Practice: Their Promise and the Need for Greater Understanding and Improved Communication

The promise of adaptation and adaptive designs in implementation science has been hindered by the lack of clarity and precision in defining what it means to adapt, especially regarding the distinction between adaptive study designs and adaptive implementation strategies. To ensure a common language for science and practice, authors reviewed the implementation science literature and found that the term adaptive was used to describe interventions, implementation strategies, and trial designs. To provide clarity and offer recommendations for reporting and strengthening study design, we propose a taxonomy that describes fixed versus adaptive implementation strategies and implementation trial designs. To improve impact, (a) futureimplementation studies should prespecify implementation strategy core functions that in turn can be taught to and replicated by health system/community partners, (b) funders should support exploratory studies that refine and specify implementation strategies, and (c) investigators should systematically address design requirements and ethical considerations (e.g., randomization, blinding/masking) with health system/community partners.

SMART Binary: New Sample Size Planning Resources for SMART Studies with Binary Outcome Measurements

Sequential Multiple-Assignment Randomized Trials (SMARTs) play an increasingly important role in psychological and behavioral health research. This experimental approach enables researchers to answer scientific questions about how to sequence and match interventions to the unique, changing needs of individuals. A variety of sample size planning resources for SMART studies have been developed, enabling researchers to plan SMARTs for addressing different types of scientific questions. However, relatively limited attention has been given to planning SMARTs with binary (dichotomous) outcomes, which often require higher sample sizes relative to continuous outcomes. Existing resources for estimating sample size requirements for SMARTs with binary outcomes do not consider the potential to improve power by including a baseline measurement and/or multiple repeated outcome measurements. The current paper addresses this issue by providing sample size planning simulation procedures and approximate formulas for two-wave repeated measures binary outcomes (i.e., two measurement times for the outcome variable, before and after intervention delivery). The simulation results agree well with the formulas. We also discuss how to use simulations to calculate power for studies with more than two outcome measurement occasions. Results show that having at least one repeated measurement of the outcome can substantially improve power under certain conditions.

A mixed methods partner-focused cost and budget impact analysis to deploy implementation strategies for school-based prevention

BACKGROUND

Obtaining information on implementation strategy costs and local budget impacts from multiple perspectives is essential to data-driven decision-making about resource allocation for successful evidence-based intervention delivery. This mixed methods study determines the costs and priorities of deploying Enhanced Replicating Effective Programs (REP) to implement the Michigan Model for Health™, a universal school-based prevention intervention, from key shareholder perspectives.

METHODS

Our study included teachers in 8 high schools across 3 Michigan counties as part of a pilot cluster randomized trial. We used activity-based costing, mapping key Enhanced REP activities across implementation phases. We included multiple perspectives, including state agencies, regional education service agencies, lead organization, and implementers. We also conducted a budget impact analysis (BIA, assessing the potential financial impact of adopting Enhanced REP) and a scenario analysis to estimate replication and account for cost variability. We used an experimental embedded mixed methods approach, conducting semi-structured interviews and collecting field notes during the trial to expand and explain the cost data and the implications of costs across relevant perspectives.

RESULTS

Based on trial results, we estimate costs for deploying Enhanced REP are $11,903/school, with an estimated range between $8263/school and $15,201/school. We estimate that adding four additional schools, consistent with the pilot, would cost $8659/school. Qualitative results indicated misalignment in school and teacher priorities in some cases. Implementation activities, including training and implementation facilitation with the health coordinator, were sometimes in addition to regular teaching responsibilities. The extent to which this occurred was partly due to leadership priorities (e.g., sticking to the district PD schedule) and organizational priorities (e.g., budget).

CONCLUSIONS

Previous research findings indicate that, from a societal perspective, universal prevention is an excellent return on investment. However, notable misalignment in cost burden and priorities exists across shareholder groups. Our results indicate significant personal time costs by teachers when engaging in implementation strategy activities that impose an opportunity cost. Additional strategies are needed to improve the alignment of costs and benefits to enhance the success and sustainability of implementation. We focus on those perspectives informed by the analysis and discuss opportunities to expand a multi-level focus and create greater alignment across perspectives.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04752189. Registered on 12 February 2021.

The engagement problem: A review of engagement with digital mental health interventions and recommendations for a path forward

Purpose of the review

Digital mental health interventions (DMHIs) are an effective and accessible means of addressing the unprecedented levels of mental illness worldwide. Currently, however, patient engagement with DMHIs in real-world settings is often insufficient to see clinical benefit. In order to realize the potential of DMHIs, there is a need to better understand what drives patient engagement.

Recent findings

We discuss takeaways from the existing literature related to patient engagement with DMHIs and highlight gaps to be addressed through further research. Findings suggest that engagement is influenced by patient-, intervention- and systems-level factors. At the patient-level, variables such as sex, education, personality traits, race, ethnicity, age and symptom severity appear to be associated with engagement. At the intervention-level, integrating human support, gamification, financial incentives and persuasive technology features may improve engagement. Finally, although systems-level factors have not been widely explored, the existing evidence suggests that achieving engagement will require addressing organizational and social barriers and drawing on the field of implementation science.

Summary

Future research clarifying the patient-, intervention- and systems-level factors that drive engagement will be essential. Additionally, to facilitate improved understanding of DMHI engagement, we propose the following: (a) widespread adoption of a minimum necessary 5-element engagement reporting framework; (b) broader application of alternative clinical trial designs; and (c) directed efforts to build upon an initial parsimonious conceptual model of DMHI engagement.

The integration of behavioral health consultants within a state-wide psychiatry consultation program: a community case study

This paper describes the integration and role of masters-prepared behavioral health consultants (BHCs) within a state-wide psychiatry consultation program for children, adolescents, and perinatal women. Data from the Michigan Clinical Consultation and Care (MC3) program are reviewed, with attention to the role that BHCs play in the consultation process, integrated care, screening, and their dual roles with Community Mental Health Services Programs (CMHSPs) in Michigan. Approximately 82% of MC3 services are provided by the BHCs and involve resources or integrated care. BHCs play a role in managing provider consultations and connecting patients and providers to resources in the MC3 program.

Scaling and sustaining COVID-19 vaccination through meaningful community engagement and care coordination for underserved communities: hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial

BACKGROUND

COVID-19 inequities are abundant in low-income communities of color. Addressing COVID-19 vaccine hesitancy to promote equitable and sustained vaccination for underserved communities requires a multi-level, scalable, and sustainable approach. It is also essential that efforts acknowledge the broader healthcare needs of these communities including engagement in preventive services.

METHODS

This is a hybrid type 3 effectiveness-implementation study that will include a multi-level, longitudinal, mixed-methods data collection approach designed to assess the sustained impact of a co-created multicomponent strategy relying on bidirectional learning, shared decision-making, and expertise by all team members. The study capitalizes on a combination of implementation strategies including mHealth outreach with culturally appropriate messaging, care coordination to increase engagement in high priority preventive services, and the co-design of these strategies using community advisory boards led by Community Weavers. Community Weavers are individuals with lived experience as members of an underserved community serving as cultural brokers between communities, public health systems, and researchers to co-create community-driven, culturally sensitive public health solutions. The study will use an adaptive implementation approach operationalized in a sequential multiple assignment randomized trial design of 300 participants from three sites in a Federally Qualified Health Center in Southern California. This design will allow examining the impact of various implementation strategy components and deliver more intensive support to those who benefit from it most. The primary effectiveness outcomes are COVID-19 vaccine completion, engagement in preventive services, and vaccine confidence. The primary implementation outcomes are reach, adoption, implementation, and maintenance of the multicomponent strategy over a 12-month follow-up period. Mixed-effects logistic regression models will be used to examine program impacts and will be triangulated with qualitative data from participants and implementers.

DISCUSSION

This study capitalizes on community engagement, implementation science, health equity and communication, infectious disease, and public health perspectives to co-create a multicomponent strategy to promote the uptake of COVID-19 vaccination and preventive services for underserved communities in San Diego. The study design emphasizes broad engagement of our community and clinic partners leading to culturally sensitive and acceptable strategies to produce lasting and sustainable increases in vaccine equity and preventive services engagement.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT05841810 May 3, 2023.

Adapting adaptive design methods to accelerate adoption of a digital asthma management intervention

Investigators conducting translational research in real-world settings may experience changes that create challenges to the successful completion of the trial as well as post-trial adoption and implementation. Adaptive designs support translational research by systematically adapting content and methods to meet the needs of target populations, settings and contexts. This manuscript describes an adaptive implementation research model that provides strategies for changing content, delivery processes, and research methods to correct course when anticipated and unanticipated circumstances occur during a pragmatic trial. The Breathewell Program included two large pragmatic trials of the effectiveness of a digital communication technology intervention to improve symptom management and medication adherence in asthma care. The first trial targeted parents of children with asthma; the second targeted adults with asthma. Adaptations were made iteratively to adjust to dynamic conditions within the healthcare setting, informed by prospectively collected stakeholder input, and were categorized retrospectively by the authors as proactive or reactive. Study outcomes demonstrated improved treatment adherence and clinical efficiency. Kaiser Permanente Colorado, the setting for both studies, adopted the speech recognition intervention into routine care, however, both interventions required numerous adaptations, including changes to target population, intervention content, and internal workflows. Proactive and reactive adaptations assured that both trials were successfully completed. Adaptive research designs will continue to provide an important pathway to move healthcare delivery research into practice while conducting ongoing effectiveness evaluation.

Individual and organizational factors as predictors of early evidence-based practice adoption in Michigan high schools: Baseline data from an implementation trial

Background

Adolescents increasingly access mental health services in schools. School mental health professionals (SPs; school counselors, social workers, etc.) can offer evidence-based mental health practices (EBPs) in schools, which may address access gaps and improve clinical outcomes. Although some studies have assessed factors associated with EBP adoption in schools, additional research focusing on SP- and school-level factors is warranted to support EBP implementation as SPs' mental health delivery grows.

Methods

Baseline data were collected from SPs at Michigan high schools participating in a statewide trial to implement SP-delivered cognitive behavioral therapy (CBT) to students. Models examined factors associated with attitudes about EBPs, implementation climate, and implementation leadership, and their associations with CBT knowledge, training attendance, and pre-training CBT delivery.

Results

One hundred ninety-eight SPs at 107 schools (87%) completed a baseline survey. The mean Evidence-Based Practice Attitude Scale (EBPAS) total score was 2.9, and school-aggregated mean scores of the Implementation Climate Scale (ICS) and Implementation Leadership Scale (ILS) were 1.83 and 1.77, respectively, all on a scale ranging from 0 (low) to 4 (high). ICS and ILS scores were lower than typically reported in clinical settings, while EBPAS scores were higher. School characteristics were not significantly associated with EBPAS, ICS, or ILS scores, but scores did differ by SP role. Higher EBPAS scores were associated with more CBT knowledge (average marginal effect for 1 SD change [AME] = 0.15 points) and a higher probability of training completion (AME = 8 percentage points). Higher ICS scores were associated with a higher probability of pre-training CBT delivery (AME = 6 percentage points), and higher ILS scores were associated with higher probability of training completion (AME = 10 percentage points).

Conclusions

Our findings suggest that SPs' attitudes toward EBPs and organizational support were positively associated with early signs of implementation success. As schools increasingly fill the adolescent mental healthcare access gap, efforts to strengthen both provider attitudes toward EBP and strategic organizational factors supporting EBP delivery will be key to encouraging EBP uptake in schools.

Plain Language Summary

Schools are an important setting in which adolescents receive mental healthcare. We need to better understand how to implement evidence-based practices (EBPs) in this setting to improve student mental health. This study examined the attitudes and perceptions of school professionals (SPs) as key contributors to the implementation of a particular EBP, the delivery of cognitive behavioral therapy (CBT) in schools. The study found that implementation climate and leadership scores in participating schools were lower than scores typically reported in clinical settings, while scores for SP attitudes about EBP adoption were higher than typical scores in clinical settings. Results further suggest that SPs with more positive attitudes toward EBPs are more knowledgeable of CBT and more likely to complete a 1-day CBT training. We also found that higher implementation climate scores were associated with SPs reporting pre-training CBT delivery (although this association was not statistically significant), and more implementation leadership was associated with SPs completing the CBT training. These findings suggest that SP attitudes toward EBPs and organizational support in schools are positively associated with early signs of implementation success. Early, low-intensity efforts to (1) improve SP attitudes about mental health EBPs, and (2) increase schools' support for implementation may scaffold more intensive implementation efforts in schools down the road.

Stepped implementation-to-target: a study protocol of an adaptive trial to expand access to addiction medications

BACKGROUND

In response to the US opioid epidemic, significant national campaigns have been launched to expand access to `opioid use disorder (MOUD). While adoption has increased in general medical care settings, specialty addiction programs have lagged in both reach and adoption. Elevating the quality of implementation strategy, research requires more precise methods in tailoring strategies rather than a one-size-fits-all-approach, documenting participant engagement and fidelity to the delivery of the strategy, and conducting an economic analysis to inform decision making and policy. Research has yet to incorporate all three of these recommendations to address the challenges of implementing and sustaining MOUD in specialty addiction programs.

METHODS

This project seeks to recruit 72 specialty addiction programs in partnership with the Washington State Health Care Authority and employs a measurement-based stepped implementation-to-target approach within an adaptive trial design. Programs will be exposed to a sequence of implementation strategies of increasing intensity and cost: (1) enhanced monitoring and feedback (EMF), (2) 2-day workshop, and then, if outcome targets are not achieved, randomization to either internal facilitation or external facilitation. The study has three aims: (1) evaluate the sequential impact of implementation strategies on target outcomes, (2) examine contextual moderators and mediators of outcomes in response to the strategies, and (3) document and model costs per implementation strategy. Target outcomes are organized by the RE-AIM framework and the Addiction Care Cascade.

DISCUSSION

This implementation project includes elements of a sequential multiple assignment randomized trial (SMART) design and a criterion-based design. An innovative and efficient approach, participating programs only receive the implementation strategies they need to achieve target outcomes. Findings have the potential to inform implementation research and provide key decision-makers with evidence on how to address the opioid epidemic at a systems level.

TRIAL REGISTRATION

This trial was registered at ClinicalTrials.gov (NCT05343793) on April 25, 2022.

Michigan Model for HealthTM Learning to Enhance and Adapt for Prevention (Mi-LEAP): protocol of a pilot randomized trial comparing Enhanced Replicating Effective Programs versus standard implementation to deliver an evidence-based drug use prevention curriculum

BACKGROUND

School-based drug use prevention programs have demonstrated notable potential to reduce the onset and escalation of drug use, including among youth at risk of poor outcomes such as those exposed to trauma. Researchers have found a robust relationship between intervention fidelity and participant (i.e., student) outcomes. Effective implementation of evidence-based interventions, such as the Michigan Model for HealthTM (MMH), is critical to achieving desired public health objectives. Yet, a persistent gap remains in what we know works and how to effectively translate these findings into routine practice. The objective of this study is to design and test a multi-component implementation strategy to tailor MMH to meet population needs (i.e., students exposed to trauma), and improve the population-context fit to enhance fidelity and effectiveness.

METHODS

Using a 2-group, mixed-method randomized controlled trial design, this study will compare standard implementation versus Enhanced Replicating Effective Programs (REP) to deliver MMH. REP is a theoretically based implementation strategy that promotes evidence-based intervention (EBI) fidelity through a combination of EBI curriculum packaging, training, and as-needed technical assistance and is consistent with standard MMH implementation. Enhanced REP will tailor the intervention and training to integrate trauma-informed approaches and deploy customized implementation support (i.e., facilitation). The research will address the following specific aims: (1) design and test an implementation strategy (Enhanced REP) to deliver the MMH versus standard implementation and evaluate feasibility, acceptability, and appropriateness using mixed methods, (2) estimate the costs and cost-effectiveness of Enhanced REP to deliver MMH versus standard implementation.

DISCUSSION

This research will design and test a multi-component implementation strategy focused on enhancing the fit between the intervention and population needs while maintaining fidelity to MMH core functions. We focus on the feasibility of deploying the implementation strategy bundle and costing methods and preliminary information on cost input distributions. The substantive focus on youth at heightened risk of drug use and its consequences due to trauma exposure is significant because of the public health impact of prevention. Pilot studies of implementation strategies are underutilized and can provide vital information on designing and testing effective strategies by addressing potential design and methods uncertainties and the effects of the implementation strategy on implementation and student outcomes.

TRIAL REGISTRATION

NCT04752189-registered on 8 February 2021 on ClinicalTrials.gov PRS.

Advancing rapid adaptation for urgent public health crises: Using implementation science to facilitate effective and efficient responses

Responding rapidly to emerging public health crises is vital to reducing their escalation, spread, and impact on population health. These responses, however, are challenging and disparate processes for researchers and practitioners. Researchers often develop new interventions that take significant time and resources, with little exportability. In contrast, community-serving systems are often poorly equipped to properly adopt new interventions or adapt existing ones in a data-driven way during crises' onset and escalation. This results in significant delays in deploying evidence-based interventions (EBIs) with notable public health consequences. This prolonged timeline for EBI development and implementation results in significant morbidity and mortality that is costly and preventable. As public health emergencies have demonstrated (e.g., COVID-19 pandemic), the negative consequences often exacerbate existing health disparities. Implementation science has the potential to bridge the extant gap between research and practice, and enhance equity in rapid public health responses, but is underutilized. For the field to have a greater "real-world" impact, it needs to be more rapid, iterative, participatory, and work within the timeframes of community-serving systems. This paper focuses on rapid adaptation as a developing implementation science area to facilitate system responses during public health crises. We highlight frameworks to guide rapid adaptation for optimizing existing EBIs when responding to urgent public health issues. We also explore the economic implications of rapid adaptation. Resource limitations are frequently a central reason for implementation failure; thus, we consider the economic impacts of rapid adaptation. Finally, we provide examples and propose directions for future research and application.

Primary aim results of a clustered SMART for developing a school-level, adaptive implementation strategy to support CBT delivery at high schools in Michigan

BACKGROUND

Schools increasingly provide mental health services to students, but often lack access to implementation strategies to support school-based (and school professional [SP]) delivery of evidence-based practices. Given substantial heterogeneity in implementation barriers across schools, development of adaptive implementation strategies that guide which implementation strategies to provide to which schools and when may be necessary to support scale-up.

METHODS

A clustered, sequential, multiple-assignment randomized trial (SMART) of high schools across Michigan was used to inform the development of a school-level adaptive implementation strategy for supporting SP-delivered cognitive behavioral therapy (CBT). All schools were first provided with implementation support informed by Replicating Effective Programs (REP) and then were randomized to add in-person Coaching or not (phase 1). After 8 weeks, schools were assessed for response based on SP-reported frequency of CBT delivered to students and/or barriers reported. Responder schools continued with phase 1 implementation strategies. Slower-responder schools (not providing ≥ 3 CBT components to ≥10 students or >2 organizational barriers identified) were re-randomized to add Facilitation to current support or not (phase 2). The primary aim hypothesis was that SPs at schools receiving the REP + Coaching + Facilitation adaptive implementation strategy would deliver more CBT sessions than SPs at schools receiving REP alone. Secondary aims compared four implementation strategies (Coaching vs no Coaching × Facilitation vs no Facilitation) on CBT sessions delivered, including by type (group, brief and full individual). Analyses used a marginal, weighted least squares approach developed for clustered SMARTs.

RESULTS

SPs (n = 169) at 94 high schools entered the study. N = 83 schools (88%) were slower-responders after phase 1. Contrary to the primary aim hypothesis, there was no evidence of a significant difference in CBT sessions delivered between REP + Coaching + Facilitation and REP alone (111.4 vs. 121.1 average total CBT sessions; p = 0.63). In secondary analyses, the adaptive strategy that offered REP + Facilitation resulted in the highest average CBT delivery (154.1 sessions) and the non-adaptive strategy offering REP + Coaching the lowest (94.5 sessions).

CONCLUSIONS

The most effective strategy in terms of average SP-reported CBT delivery is the adaptive implementation strategy that (i) begins with REP, (ii) augments with Facilitation for slower-responder schools (schools where SPs identified organizational barriers or struggled to deliver CBT), and (iii) stays the course with REP for responder schools.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT03541317 , May 30, 2018.

Expanding access to medications for opioid use disorder in primary care clinics: an evaluation of common implementation strategies and outcomes

BACKGROUND

To combat the opioid epidemic in the USA, unprecedented federal funding has been directed to states and territories to expand access to prevention, overdose rescue, and medications for opioid use disorder (MOUD). Similar to other states, California rapidly allocated these funds to increase reach and adoption of MOUD in safety-net, primary care settings such as Federally Qualified Health Centers. Typical of current real-world implementation endeavors, a package of four implementation strategies was offered to all clinics. The present study examines (i) the pre-post effect of the package of strategies, (ii) whether/how this effect differed between new (start-up) versus more established (scale-up) MOUD practices, and (iii) the effect of clinic engagement with each of the four implementation strategies.

METHODS

Forty-one primary care clinics were offered access to four implementation strategies: (1) Enhanced Monitoring and Feedback, (2) Learning Collaboratives, (3) External Facilitation, and (4) Didactic Webinars. Using linear mixed effects models, RE-AIM guided outcomes of reach, adoption, and implementation quality were assessed at baseline and at 9 months follow-up.

RESULTS

Of the 41 clinics, 25 (61%) were at MOUD start-up and 16 (39%) were at scale-up phases. Pre-post difference was observed for the primary outcome of percent of patient prescribed MOUD (reach) (βtime = 3.99; 0.73 to 7.26; p = 0.02). The largest magnitude of change occurred in implementation quality (ES = 0.68; 95% CI = 0.66 to 0.70). Baseline MOUD capability moderated the change in reach (start-ups 22.60%, 95% CI = 16.05 to 29.15; scale-ups -4.63%, 95% CI = -7.87 to -1.38). Improvement in adoption and implementation quality were moderately associated with early prescriber engagement in Learning Collaboratives (adoption: ES = 0.61; 95% CI = 0.25 to 0.96; implementation quality: ES = 0.55; 95% CI = 0.41 to 0.69). Improvement in adoption was also associated with early prescriber engagement in Didactic Webinars (adoption: ES = 0.61; 95% CI = 0.20 to 1.05).

CONCLUSIONS

Rather than providing an all-clinics-get-all-components package of implementation strategies, these data suggest that it may be more efficient and effective to tailor the provision of implementation strategies based on the needs of clinic. Future implementation endeavors could benefit from (i) greater precision in the provision of implementation strategies based on contextual determinants, and (ii) the inclusion of strategies targeting engagement.

Developing a statewide network of coaches to support youth access to evidence-based practices

Background: A national shortage of mental health providers for youth exists in the United States. Implementation support for mental health services in schools, where students are most likely to access care, can help to fill these gaps. Coaching consists of in vivo modeling and support during service delivery and is effective in supporting the implementation of evidence-based practices (EBPs). This implementation report describes the recruitment and training of community providers to become coaches as a part of a modified train-the-trainer model of implementation support. Method: An EBP implementation program, Transforming Research into Action to Improve the Lives of Students (TRAILS), trained community providers in Michigan to increase knowledge of cognitive behavioral therapy (CBT) and position them as coaches in schools. The development of the coach network involved five stages prior to the initiation of a randomized controlled trial: (1) recruitment, (2) a one-day clinical training, (3) 12 weeks of individualized consultation, (4) evaluation, and (5) training in the coaching protocol. Results: A total of 347 individuals attended an initial training, and 187 were paired with a consultant. Eighty-six clinicians from 47 of Michigan's 83 counties successfully became coaches by completing all required elements of training. Coaches showed significant improvements in the use and knowledge of CBT across consultation. Conclusion: Statewide networks of trained mental health professionals can address gaps in mental healthcare for youth. This article demonstrates one strategy for strengthening and leveraging community expertise to support the implementation of EPBs in schools.

Plain Language Summary

Although youth face many barriers in accessing effective mental health care, schools are one setting where they can more easily receive treatment. Research shows that training and supports are needed for school mental health professionals to provide effective care. Coaching, which involves in vivo support for school mental health professionals, is one helpful strategy. We describe the process of recruiting and training community clinicians to become coaches. The Transforming Research into Action to Improve the Lives of Students (TRAILS) program successfully recruited and trained a network of 86 community clinicians to become coaches. Clinicians attended two day-long trainings and participated in 12 weeks of personalized consultation. Clients to whom these clinicians provided CBT showed significant symptom improvement. Clinicians also reported that they increased their knowledge and use of core CBT strategies, including psychoeducation, exposure, and behavioral activation. Community mental health professionals who are trained as coaches can address gaps in access to care for youth. We demonstrate one strategy for strengthening and leveraging clinicians’ knowledge of CBT to support the school-based implementation of CBT.

The Cognitive Behavioral Therapy Competence Scale (CCS): initial development and validation

Background:

Nearly one-third of youth are affected by a mental health disorder, and the majority do not receive adequate care. To improve clinical outcomes among youth, efforts have been made to train providers in evidence-based mental health practices, such as cognitive behavioral therapy (CBT). Such efforts call for valid assessment measures that can inform and evaluate training activities.

Aims:

This study presents the development and validation of the CBT Competence Scale (CCS), a brief self-report measure to assess provider competence for CBT delivery.

Method:

Participants were 387 school mental health professionals (SMHPs) working with students in Michigan, USA. Initial items (n=59) were developed to evaluate competence in delivering common elements of CBT, with competence conceptualized as covering domains of knowledge, perception, and use of CBT techniques. CCS validation proceeded in three steps: using item response theory to select the most important items for assessing knowledge, evaluating the factor structure using exploratory and then confirmatory factor analyses, and examining reliability and validity of the resultant measure.

Results:

The validated CCS measure consists of four dimensions of CBT competence across 33 items: Non-behavioral skills, Behavioral skills, Perceptions, and Knowledge. The CCS demonstrated excellent internal consistency and good construct-based validity.

Conclusions:

The CCS holds promise as a valid, informative measure of CBT competence appropriate for the school setting, with potential for application in other environments such as mental health clinics.

Key learning aims

(1)

Sequential multiple assignment randomized trial studies should report all key components: a systematic review

OBJECTIVE

Sequential Multiple Assignment Randomised Trial (SMART) designs allow multiple randomisations of participants; this allows assessment of stage-specific questions (individual randomisations) and adaptive interventions (i.e. treatment strategies). We assessed the quality of reporting of the information required to design SMART studies.

STUDY DESIGN AND SETTING

We systematically searched four databases (PubMed, Ovid, Web of Science and Scopus) for all trial reports, protocols, reviews, and methodological papers which mentioned SMART designs up to June 15, 2020.

RESULTS

Of the 157 selected records, 12 (7.64%) were trial reports, 24 (15.29%) were study protocols, 91 (58%) were methodological papers, and 30 (19.1%) were review papers. All these trials were powered using stage-specific aims. Only four (33.33%) of these trials reported parameters required for sample size calculations. A small number of the trials (16.67 %) were interested in determining the best embedded adaptive interventions. Most of the trials did not report information about multiple testing adjustment. Furthermore, most of records reported designs that were mainly focused on stage-specific aims.

CONCLUSIONS

Some features of SMART designs are seldomly reported and/or used. Furthermore, studies using this design tend to not adequately report information about all the design parameters, limiting their transparency and interpretability.

Optimal allocation to treatments in a sequential multiple assignment randomized trial

One of the main questions in the design of a trial is how many subjects should be assigned to each treatment condition. Previous research has shown that equal randomization is not necessarily the best choice. We study the optimal allocation for a novel trial design, the sequential multiple assignment randomized trial, where subjects receive a sequence of treatments across various stages. A subject's randomization probabilities to treatments in the next stage depend on whether he or she responded to treatment in the current stage. We consider a prototypical sequential multiple assignment randomized trial design with two stages. Within such a design, many pairwise comparisons of treatment sequences can be made, and a multiple-objective optimal design strategy is proposed to consider all such comparisons simultaneously. The optimal design is sought under either a fixed total sample size or a fixed budget. A Shiny App is made available to find the optimal allocations and to evaluate the efficiency of competing designs. As the optimal design depends on the response rates to first-stage treatments, maximin optimal design methodology is used to find robust optimal designs. The proposed methodology is illustrated using a sequential multiple assignment randomized trial example on weight loss management.

Replicating an effective VA program to train and support family caregivers: a hybrid type III effectiveness-implementation design

BACKGROUND

Caring for a growing aging population using existing long-term care resources while simultaneously supporting and educating family caregivers, is a public health challenge. We describe the application of the Replicating Effective Programs (REP) framework, developed by the Centers for Disease Control Prevention and used in public health program implementation, to scale up an evidence-based family caregiver training intervention in the Veterans Affairs (VA) healthcare system.

METHODS

From 2018 to 2020, clinicians at eight VA medical centers received REP-guided implementation including facilitation, technical assistance, and implementation tools to deliver the training program. The project team used the REP framework to develop activities across four distinct phases - (1) pre-conditions, (2) pre-implementation, (3) implementation, and (4) maintenance and evolution - and systematically tracked implementation facilitators, barriers, and adaptations.

RESULTS

Within the REP framework, results describe how each medical center adapted implementation approaches to fit local needs. We highlight examples of how sites balanced adaptations and intervention fidelity.

CONCLUSIONS

The REP framework shows promise for national expansion of the caregiver training intervention, including to non-VA systems of care, because it allows sites to adapt while maintaining intervention fidelity.

TRIAL REGISTRATION

NCT03474380 . Date registered: March 22, 2018.

Ascertaining properties of weighting in the estimation of optimal treatment regimes under monotone missingness

Dynamic treatment regimes operationalize precision medicine as a sequence of decision rules, one per stage of clinical intervention, that map up-to-date patient information to a recommended intervention. An optimal treatment regime maximizes the mean utility when applied to the population of interest. Methods for estimating an optimal treatment regime assume the data to be fully observed, which rarely occurs in practice. A common approach is to first use multiple imputation and then pool the estimators across imputed datasets. However, this approach requires estimating the joint distribution of patient trajectories, which can be high-dimensional, especially when there are multiple stages of intervention. We examine the application of inverse probability weighted estimating equations as an alternative to multiple imputation in the context of monotonic missingness. This approach applies to a broad class of estimators of an optimal treatment regime including both Q-learning and a generalization of outcome weighted learning. We establish consistency under mild regularity conditions and demonstrate its advantages in finite samples using a series of simulation experiments and an application to a schizophrenia study.

The Balanced Opioid Initiative: protocol for a clustered, sequential, multiple-assignment randomized trial to construct an adaptive implementation strategy to improve guideline-concordant opioid prescribing in primary care

BACKGROUND

Rates of opioid prescribing tripled in the USA between 1999 and 2015 and were associated with significant increases in opioid misuse and overdose death. Roughly half of all opioids are prescribed in primary care. Although clinical guidelines describe recommended opioid prescribing practices, implementing these guidelines in a way that balances safety and effectiveness vs. risk remains a challenge. The literature offers little help about which implementation strategies work best in different clinical settings or how strategies could be tailored to optimize their effectiveness in different contexts. Systems consultation consists of (1) educational/engagement meetings with audit and feedback reports, (2) practice facilitation, and (3) prescriber peer consulting. The study is designed to discover the most cost-effective sequence and combination of strategies for improving opioid prescribing practices in diverse primary care clinics.

METHODS/DESIGN

The study is a hybrid type 3 clustered, sequential, multiple-assignment randomized trial (SMART) that randomizes clinics from two health systems at two points, months 3 and 9, of a 21-month intervention. Clinics are provided one of four sequences of implementation strategies: a condition consisting of educational/engagement meetings and audit and feedback alone (EM/AF), EM/AF plus practice facilitation (PF), EM/AF + prescriber peer consulting (PPC), and EM/AF + PF + PPC. The study's primary outcome is morphine-milligram equivalent (MME) dose by prescribing clinicians within clinics. The study's primary aim is the comparison of EM/AF + PF + PPC versus EM/AF alone on change in MME from month 3 to month 21. The secondary aim is to derive cost estimates for each of the four sequences and compare them. The exploratory aim is to examine four tailoring variables that can be used to construct an adaptive implementation strategy to meet the needs of different primary care clinics.

DISCUSSION

Systems consultation is a practical blend of implementation strategies used in this case to improve opioid prescribing practices in primary care. The blend offers a range of strategies in sequences from minimally to substantially intensive. The results of this study promise to help us understand how to cost effectively improve the implementation of evidence-based practices.

TRIAL REGISTRATION

NCT04044521 (ClinicalTrials.gov). Registered 05 August 2019.

Designs and methods for implementation research: Advancing the mission of the CTSA program

INTRODUCTION

The US National Institutes of Health (NIH) established the Clinical and Translational Science Award (CTSA) program in response to the challenges of translating biomedical and behavioral interventions from discovery to real-world use. To address the challenge of translating evidence-based interventions (EBIs) into practice, the field of implementation science has emerged as a distinct discipline. With the distinction between EBI effectiveness research and implementation research comes differences in study design and methodology, shifting focus from clinical outcomes to the systems that support adoption and delivery of EBIs with fidelity.

METHODS

Implementation research designs share many of the foundational elements and assumptions of efficacy/effectiveness research. Designs and methods that are currently applied in implementation research include experimental, quasi-experimental, observational, hybrid effectiveness-implementation, simulation modeling, and configurational comparative methods.

RESULTS

Examples of specific research designs and methods illustrate their use in implementation science. We propose that the CTSA program takes advantage of the momentum of the field's capacity building in three ways: 1) integrate state-of-the-science implementation methods and designs into its existing body of research; 2) position itself at the forefront of advancing the science of implementation science by collaborating with other NIH institutes that share the goal of advancing implementation science; and 3) provide adequate training in implementation science.

CONCLUSIONS

As implementation methodologies mature, both implementation science and the CTSA program would greatly benefit from cross-fertilizing expertise and shared infrastructures that aim to advance healthcare in the USA and around the world.

Economic evaluation in implementation science: Making the business case for implementation strategies

Implementation researchers have made notable progress in developing and testing implementation strategies (i.e., highly-specified methods used to help providers improve uptake of mental health evidence-based practices: EBPs). Yet, implementation strategies are not widely applied in healthcare organizations to improve delivery of EBPs. Economic considerations are a key factor influencing the use of implementation strategies to deliver and sustain mental health evidence-based practices, in part because many health care leaders are reluctant to invest in ongoing implementation strategy support without knowing the return-on-investment. Comparative economic evaluation of implementation strategies provides critical information for payers, policymakers, and providers to make informed decisions if specific strategies are an efficient use of scarce organizational resources. Currently, few implementation studies include implementation cost data and even fewer conduct comparative economic analyses of implementation strategies. This summary will introduce clinicians, researchers and other health professionals to the economic evaluation in implementation science. We provide an overview of different economic evaluation methods, discuss differences between economic evaluation in health services and implementation science. We also highlight approaches and frameworks to guide economic evaluation of implementation, provide an example for a cognitive-behavioral therapy program and discuss recommendations.

Implementing integrated services in routine behavioral health care: primary outcomes from a cluster randomized controlled trial

BACKGROUND

An estimated 8.2 million adults in the United States live with co-occurring mental health and substance use disorders. Although the benefits of integrated treatment services for persons with co-occurring disorders has been well-established, gaps in access to integrated care persist. Implementation research can address this gap. We evaluated if the Network for the Improvement of Addiction Treatment (NIATx) implementation strategy was effective in increasing integrated services capacity among organizations treating persons with co-occurring disorders.

METHODS

This study employed a cluster randomized waitlist control group design. Forty-nine addiction treatment organizations from the State of Washington were randomized into one of two study arms: (1) NIATx strategy (active implementation strategy), or (2) waitlist (control). The primary outcome was a standardized organizational measure of integrated service capability: the Dual Diagnosis in Addiction Treatment (DDCAT) Index. Intent-to-treat analyses and per-protocol analyses were conducted to address the following questions: (1) Is NIATx effective in increasing integrated service capacity? and (2) Are there differences in organizations that actually use NIATx per-protocol versus those that do not?

RESULTS

From baseline to one-year post active implementation, both the NIATx strategy and waitlist arms demonstrated improvements over time in DDCAT Index total and DDCAT dimension scores. In intent-to-treat analyses, a moderate but statistically significant difference in improvement between study arms was seen only in the Program Milieu dimension (p = 0.020, Cohen's d = 0.54). In per-protocol analyses, moderate-to-large effects in Program Milieu (p = 0.002, Cohen's d = 0.91) and Continuity of Care (p = 0.026, Cohen's d = 0.63) dimensions, and in total DDCAT Index (p = 0.046, Cohen's d = 0.51) were found.

CONCLUSIONS

Overall, organizations in both study arms improved DDCAT Index scores over time. Organizations in the NIATx strategy arm with full adherence to the NIATx protocol had significantly greater improvements in the primary outcome measure of integrated service capacity for persons with co-occurring disorders.

TRAIL REGISTRATION

ClinicalTrials.gov, NCT03007940 . Retrospectively registered January 2017.