Assessing Implementation Strategy Reporting in the Mental Health Literature: A Narrative Review

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

BACKGROUND

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

METHODS

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms "implementation strategy", "intervention", "bundle", "support", and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

FINDINGS

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

CONCLUSION

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Building a sharable literature collection to advance the science and practice of implementation facilitation

Background

Implementation science seeks to produce generalizable knowledge on strategies that promote the adoption and sustained use of evidence-based innovations. Literature reviews on specific implementation strategies can help us understand how they are conceptualized and applied, synthesize findings, and identify knowledge gaps. Although rigorous literature reviews can advance scientific knowledge and facilitate theory development, they are time-consuming and costly to produce. Improving the efficiency of literature review processes and reducing redundancy of effort is especially important for this rapidly developing field. We sought to amass relevant literature on one increasingly used evidence-based strategy, implementation facilitation (IF), as a publicly available resource.

Methods

We conducted a rigorous systematic search of PubMed, CINAHL, and Web of Science citation databases for peer-reviewed, English-language articles with "facilitation" and a combination of other terms published from January 1996 to December 2021. We searched bibliographies of articles published from 1996 to 2015 and identified articles during the full text review that reported on the same study. Two authors screened 3,168 abstracts. After establishing inter-rater reliability, they individually conducted full-text review of 786 relevant articles. A multidisciplinary team of investigators provided recommendations for preparing and disseminating the literature collection.

Findings

The literature collection is comprised of 510 articles. It includes 277 empirical studies of IF and 77 other articles, including conceptual/theoretical articles, literature reviews, debate papers and descriptions of large-scale clinical initiatives. Over half of the articles were published between 2017 and 2021. The collection is publicly available as an Excel file and as an xml file that can be imported into reference management software.

Conclusion

We created a publicly accessible collection of literature about the application of IF to implement evidence-based innovations in healthcare. The comprehensiveness of this collection has the potential to maximize efficiency and minimize redundancy in scientific inquiry about this strategy. Scientists and practitioners can use the collection to more rapidly identify developments in the application of IF and to investigate a wide range of compelling questions on its use within and across different healthcare disciplines/settings, countries, and payer systems. We offer several examples of how this collection has already been used.

A feasibility study of provider-level implementation strategies to improve access to colorectal cancer screening for patients with schizophrenia: ACCESS2 (N-EQUITY 2104) trial

BACKGROUND

People with schizophrenia have a lower colorectal screening rate than the general population. A previous study reported an intervention using case management to encourage colorectal cancer screening for patients with schizophrenia in psychiatric outpatient settings. In this feasibility study, we developed provider-level implementation strategies and evaluated the feasibility of conducting a definitive trial in terms of the penetration of the intervention assessed at the patient level. Additionally, we examined the fidelity of strategies to implement the intervention at the provider level in a routine clinical psychiatric setting.

METHODS

This was a multicenter, single-arm feasibility study with medical staff at psychiatric hospitals in Japan. The provider-level implementation strategies developed in this study included three key elements (organizing an implementation team appointed by the facility director, interactive assistance using a clear guide that outlines who in the hospital should do what, and developing accessible educational materials) to overcome major barriers to implementation of the intervention and four additional elements (progress monitoring, joint meetings and information sharing among participating sites, adaptation of encouragement methods to specific contexts, and education of on-site staff), with reference to the elements identified in the Expert Recommendations for Implementing Change (ERIC). The feasibility of the strategies was evaluated by the proportion of patients who were eligible for encouragement (patients with schizophrenia aged 40, 50, or 60) who received encouragement. We set the goal of providing encouragement to at least 40% of eligible patients at each site.

RESULTS

Two public and four private psychiatric hospitals in Okayama and Shimane prefectures participated in this study. Regarding fidelity, all elements of the strategies were conducted as planned. Following the procedures in the guide, each team prepared and provided encouragement according to their own facility and region. Penetration, defined as the proportion of eligible patients who received encouragement, ranged from 33.3 to 100%; five of the six facilities achieved the target proportion.

CONCLUSIONS

The provider-level implementation strategies to implement encouragement were feasible in terms of penetration of the intervention assessed at the patient level. The results support the feasibility of proceeding with a future definitive trial of these strategies.

TRIAL REGISTRATION

jRCT, jRCT1060220026 . Registered on 06/04/2022.

Strategies for Enhancing the Implementation of Universal Mental Health Prevention Programs in Schools: A Systematic Review

A number of school-based mental health prevention programs have been found to be effective in research trials, but little is known about how to support implementation in real-life settings. To address this translational problem, this systematic review aims to identify effective strategies for enhancing the implementation of mental health prevention programs for children in schools. Four electronic databases were searched for empirical, peer-reviewed articles in English from January 2000 to October 2021 reporting the effects of implementation strategies for school-based universal mental health programs. Twenty-one articles were included in the narrative synthesis and assessed for quality using the Mixed Methods Appraisal Tool. Twenty-two strategies were found to be effective at improving program fidelity or adoption. The strategies with the strongest positive evidence base were those that involved monitoring and provision of feedback, engaging principals as program leaders, improving teachers' buy-in and organising school personnel implementation meetings. We recommend school-based practitioners trial strategies with positive findings from this review as part of their continuous quality improvement. This review highlights the pressing need for large-scale, randomised controlled trials to develop and trial more robust strategies to enhance adoption, as the five implementation studies found to measure adoption used qualitative methods limited by small samples sizes and case study designs.

Association between the number of adopted implementation strategies and contextual determinants: a mixed-methods study

BACKGROUND

The successful implementation of evidence-based innovations to improve healthcare delivery often requires a well-planned strategy to support their use. With a greater recognition of the importance of an implementation process, researchers have turned their attention to implementation strategies and their customization to target specific organizational barriers and facilitators. Further, there is a paucity of empirical evidence demonstrating the link between implementation determinants and the number of selected implementation strategies. The purpose of this mixed methods analysis is to examine how formatively assessed barriers and facilitators to implementation relate to the number and type of implementation strategies adopted to address context-specific factors.

METHODS

A mixed methods evaluation that included 15 rheumatology clinics throughout the United States that were planning for implementation of an evidence-based shared decision-making aid for patients with lupus. Quantitative data consisted of a count of the number of implementation strategies used by a clinic. Qualitative data collection was guided by the Consolidated Framework for Implementation Research (CFIR) and relied upon semi-structured interviews with 90 clinic members between November 2018 and August 2019.

RESULTS

Using the CFIR, we found that local clinic factors (Inner Setting Domain) resulted in different perceptions about Planning and Executing the DA (Process Domain); these domains were most likely to distinguish between the number and type of implementation strategies adopted by the clinics. In contrast, Intervention characteristics, Individual Characteristics, and the Outer Setting did not differentiate between the groups with different numbers of implementation strategies. The number and type of chosen strategies were not those associated with the context-specific factors.

CONCLUSIONS

Findings show that, despite recognition of the value of customizing implementation strategies for the contexts in which they are applied, they are too often chosen in a manner that fail to adequately reflect the diverse settings that may present unique factors associated with implementation. Our findings also highlight the importance of the inner context - both in terms of structural characteristics and existing work processes - as a driving factor for why some organizations select different numbers and types of implementation strategies.

A scoping review of frameworks in empirical studies and a review of dissemination frameworks

BACKGROUND

The field of dissemination and implementation (D&I) research has grown immensely in recent years. However, the field of dissemination research has not coalesced to the same degree as the field of implementation research. To advance the field of dissemination research, this review aimed to (1) identify the extent to which dissemination frameworks are used in dissemination empirical studies, (2) examine how scholars define dissemination, and (3) identify key constructs from dissemination frameworks.

METHODS

To achieve aims 1 and 2, we conducted a scoping review of dissemination studies published in D&I science journals. The search strategy included manuscripts published from 1985 to 2020. Articles were included if they were empirical quantitative or mixed methods studies about the dissemination of information to a professional audience. Studies were excluded if they were systematic reviews, commentaries or conceptual papers, scale-up or scale-out studies, qualitative or case studies, or descriptions of programs. To achieve aim 1, we compiled the frameworks identified in the empirical studies. To achieve aim 2, we compiled the definitions from dissemination from frameworks identified in aim 1 and from dissemination frameworks identified in a 2021 review (Tabak RG, Am J Prev Med 43:337-350, 2012). To achieve aim 3, we compile the constructs and their definitions from the frameworks.

FINDINGS

Out of 6017 studies, 89 studies were included for full-text extraction. Of these, 45 (51%) used a framework to guide the study. Across the 45 studies, 34 distinct frameworks were identified, out of which 13 (38%) defined dissemination. There is a lack of consensus on the definition of dissemination. Altogether, we identified 48 constructs, divided into 4 categories: process, determinants, strategies, and outcomes. Constructs in the frameworks are not well defined.

IMPLICATION FOR D&I RESEARCH

This study provides a critical step in the dissemination research literature by offering suggestions on how to define dissemination research and by cataloging and defining dissemination constructs. Strengthening these definitions and distinctions between D&I research could enhance scientific reproducibility and advance the field of dissemination research.

Scoping review of costs of implementation strategies in community, public health and healthcare settings

OBJECTIVES

To identify existing evidence concerning the cost of dissemination and implementation (D&I) strategies in community, public health and health service research, mapped with the 'Expert Recommendations for Implementing Change' (ERIC) taxonomy.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE, CINAHL, PsycINFO, Scopus and the Cochrane Library were searched to identify any English language reports that had been published between January 2008 and December 2019 concerning the cost of D&I strategies.

DATA EXTRACTION

We matched the strategies identified in each article using ERIC taxonomies; further classified them into five areas (eg, dissemination, implementation, integration, capacity building and scale-up); and extracted the corresponding costs (total costs and cots per action target and per evidence-based programme (EBP) participant). We also recorded the reported level of costing methodology used for cost assessment of D&I strategies.

RESULTS

Of the 6445 articles identified, 52 studies were eligible for data extraction. Lack of D&I strategy cost data was the predominant reason (55% of the excluded studies) for study exclusion. Predominant topic, setting, country and research design in the included studies were mental health (19%), primary care settings (44%), the US (35%) and observational (42%). Thirty-five (67%) studies used multicomponent D&I strategies (ranging from two to five discrete strategies). The most frequently applied strategies were Conduct ongoing training (50%) and Conduct educational meetings (23%). Adoption (42%) and reach (27%) were the two most frequently assessed outcomes. The overall costs of Conduct ongoing training ranged from $199 to $105 772 ($1-$13 973 per action target and $0.02-$412 per EBP participant); whereas the cost of Conduct educational meetings ranged from $987 to $1.1-$2.9 million/year ($33-$54 869 per action target and $0.2-$146 per EBP participant). The wide range of costs was due to the varying scales of the studies, intended audiences/diseases and the complexities of the strategy components. Most studies presented limited information on costing methodology, making interpretation difficult.

CONCLUSIONS

The quantity of published D&I strategy cost analyses is increasing, yet guidance on conducting and reporting of D&I strategy cost analysis is necessary to facilitate and promote the application of comparative economic evaluation in the field of D&I research.

Building organizational readiness: initial field testing of an expert-informed typology of implementation strategies

BACKGROUND

Organizational readiness is a known barrier to implementing evidence-based practices (EBPs) in community mental health services. A robust methodology for enhancing organizational readiness for implementation (ORI) has the potential to improve implementation outcomes of EBPs and ensure better services for people with a psychiatric disability. Prior work established a framework of implementation strategies targeting ORI enhancement by asking a group of implementation experts from various fields to categorize strategies from the "Expert Recommendations for Implementing Change" (ERIC) Project into three readiness stages, consistent with the pre-action stages of the Transtheoretical Model of behavioral change: Pre-contemplation, Contemplation, and Preparation. The current study provides initial confirmation and refinement to this expert-driven typology based on community mental health field experiences.

METHODS

We conducted in-depth interviews with stakeholders involved in a recent EBP implementation project. Participants included staff (n=9) from four community mental health agencies and the implementation team who facilitated the project (n=3). Their pre-implementation experiences were compared with the experts' typology to identify consistencies and discrepancies.

RESULTS

The participants' experiences were congruent with two thirds of the strategies identified by the experts for specific ORI stages. The refinements included 12 strategies used in additional stages beyond the experts' classification, four strategies from the ERIC list that were not included in the ORI typology, and five new strategies.

CONCLUSIONS

This study provides initial confirmation and refinements to the previously published ORI typology. The results offer guidance as to how ORI could be enhanced in the community mental health field.

Applying implementation science in mental health services: Technical assistance cases from the Mental Health Technology Transfer Center (MHTTC) network

OBJECTIVE

Critical gaps exist between implementation of effective interventions and the actual services delivered to people living with mental disorders. Many technical assistance (TA) efforts rely on one-time trainings of clinical staff and printed guidelines that alone are not effective in changing clinical practice. The Mental Health Technology Transfer Center (MHTTC) Network uses implementation science to accelerate the use of evidence-based practices (EBPs), improve performance, and bring about systems-level change.

METHOD

Four case examples illustrate how MHTTCs employ the Exploration-Preparation-Implementation-Sustainment (EPIS) implementation framework and intensive implementation strategies to educate clinicians, manage change, and improve processes. These examples include implementing motivational interviewing, cognitive-behavioral therapy for people with psychosis, strategies to decrease the no show rate for virtual appointments, and school mental health systems development.

RESULTS

From Preparation through Sustainment, MHTTCs successfully employed implementation strategies including learning communities, audit and feedback, and coaching to bring about change. Each project attended to inner and outer contexts to eliminate barriers. The examples also show the benefit of integrating process improvement alongside implementation.

CONCLUSIONS

The MHTTCs are a model for using implementation science to design technical assistance that leads to more successful practical execution of EBPs; thus reducing the gap between research and practice.

Specifying and Reporting Implementation Strategies Used in a School-Based Prevention Efficacy Trial

Background

Implementation strategies used to enhance the implementation of interventions during efficacy and effectiveness studies are rarely reported. Tracking and reporting implementation strategies during these phases has potential to improve future research studies and real-world implementation. We present an exemplar of how this might be executed by specifying and reporting the implementation strategies that were used during a school-based efficacy trial, Project POWER, which tested a trauma-informed prevention program delivered by a university research team, community members, and school staff facilitators in 29 schools.

Methods

Following the conclusion of the 4-year trial, core Project POWER research team members identified the implementation strategies that supported intervention delivery during the trial using an established taxonomy of school-based implementation strategies. The actors, actions, action targets, temporality, dose, and implementation outcomes were specified using established implementation strategies reporting guidelines.

Results

The research team identified 37 implementation strategies that were used during the Project POWER trial. Most strategies fell within the categories of Train and Educate Stakeholders, Use Evaluative and Iterative Strategies, and Develop Stakeholder Interrelationships. Actors included members of the research team and partner schools. Strategies were used multiple times during the preparation and implementation phases. Action targets were most often characteristics of individuals, implementation process, and characteristics of the inner setting. Strategies predominantly targeted the implementation outcomes of fidelity, acceptability, feasibility, and adoption.

Conclusions

This study provided evidence that implementation strategies are used and can be identified in efficacy research using a retrospective approach. Identifying and specifying implementation strategies used during the initial phases of the translational research pipeline can inform the implementation strategies that are carried forward, adapted, or discontinued in future trials and routine practice to improve implementation and effectiveness outcomes.

Strengthening methods for tracking adaptations and modifications to implementation strategies

BACKGROUND

Developing effective implementation strategies requires adequate tracking and reporting on their application. Guidelines exist for defining and reporting on implementation strategy characteristics, but not for describing how strategies are adapted and modified in practice. We built on existing implementation science methods to provide novel methods for tracking strategy modifications.

METHODS

These methods were developed within a stepped-wedge trial of an implementation strategy package designed to help community clinics adopt social determinants of health-related activities: in brief, an 'Implementation Support Team' supports clinics through a multi-step process. These methods involve five components: 1) describe planned strategy; 2) track its use; 3) monitor barriers; 4) describe modifications; and 5) identify / describe new strategies. We used the Expert Recommendations for Implementing Change taxonomy to categorize strategies, Proctor et al.'s reporting framework to describe them, the Consolidated Framework for Implementation Research to code barriers / contextual factors necessitating modifications, and elements of the Framework for Reporting Adaptations and Modifications-Enhanced to describe strategy modifications.

RESULTS

We present three examples of the use of these methods: 1) modifications made to a facilitation-focused strategy (clinics reported that certain meetings were too frequent, so their frequency was reduced in subsequent wedges); 2) a clinic-level strategy addition which involved connecting one study clinic seeking help with community health worker-related workflows to another that already had such a workflow in place; 3) a study-level strategy addition which involved providing assistance in overcoming previously encountered (rather than de novo) challenges.

CONCLUSIONS

These methods for tracking modifications made to implementation strategies build on existing methods, frameworks, and guidelines; however, as none of these were a perfect fit, we made additions to several frameworks as indicated, and used certain frameworks' components selectively. While these methods are time-intensive, and more work is needed to streamline them, they are among the first such methods presented to implementation science. As such, they may be used in research on assessing effective strategy modifications and for replication and scale-up of effective strategies. We present these methods to guide others seeking to document implementation strategies and modifications to their studies.

TRIAL REGISTRATION

clinicaltrials.gov ID: NCT03607617 (first posted 31/07/2018).

Strengthening methods for tracking adaptations and modifications to implementation strategies

BACKGROUND

Developing effective implementation strategies requires adequate tracking and reporting on their application. Guidelines exist for defining and reporting on implementation strategy characteristics, but not for describing how strategies are adapted and modified in practice. We built on existing implementation science methods to provide novel methods for tracking strategy modifications.

METHODS

These methods were developed within a stepped-wedge trial of an implementation strategy package designed to help community clinics adopt social determinants of health-related activities: in brief, an 'Implementation Support Team' supports clinics through a multi-step process. These methods involve five components: 1) describe planned strategy; 2) track its use; 3) monitor barriers; 4) describe modifications; and 5) identify / describe new strategies. We used the Expert Recommendations for Implementing Change taxonomy to categorize strategies, Proctor et al.'s reporting framework to describe them, the Consolidated Framework for Implementation Research to code barriers / contextual factors necessitating modifications, and elements of the Framework for Reporting Adaptations and Modifications-Enhanced to describe strategy modifications.

RESULTS

We present three examples of the use of these methods: 1) modifications made to a facilitation-focused strategy (clinics reported that certain meetings were too frequent, so their frequency was reduced in subsequent wedges); 2) a clinic-level strategy addition which involved connecting one study clinic seeking help with community health worker-related workflows to another that already had such a workflow in place; 3) a study-level strategy addition which involved providing assistance in overcoming previously encountered (rather than de novo) challenges.

CONCLUSIONS

These methods for tracking modifications made to implementation strategies build on existing methods, frameworks, and guidelines; however, as none of these were a perfect fit, we made additions to several frameworks as indicated, and used certain frameworks' components selectively. While these methods are time-intensive, and more work is needed to streamline them, they are among the first such methods presented to implementation science. As such, they may be used in research on assessing effective strategy modifications and for replication and scale-up of effective strategies. We present these methods to guide others seeking to document implementation strategies and modifications to their studies.

TRIAL REGISTRATION

clinicaltrials.gov ID: NCT03607617 (first posted 31/07/2018).