Systems antecedents for dissemination and implementation: a review and analysis of measures

Implementing point-of-care medical information systems into trauma and general surgeon practice in a middle-income country: a qualitative study utilizing the Consolidated Framework for Implementation Research

BACKGROUND

Point-of-care medical information systems (POCMIS) can provide an efficient and effective means of strengthening health systems globally through their encouragement of continued medical education. Using the Consolidated Framework for Implementation Research (CFIR) as a guide, this research provides suggestions for improved implementation of POCMIS in low- and middle-income countries informed by an intervention implemented across public and military hospitals in Lima, Peru.

METHODS

Analysis is based on qualitative interviews conducted with 12 Peruvian surgeons across eight public hospitals and one military hospital who received an intervention that provided free access to UpToDate and introduced Google Translate. The post-intervention interviews were transcribed, translated, and analyzed for themes overlapping with CFIR constructs to expose barriers to implementation and suggestions for improved implementation of future interventions.

RESULTS

Barriers included a lack of seniority buy-in and engaged leadership, an overabundance of personal preferences for multiple POCMIS, and a culture of assumption that inhibited open communication regarding access to and use of POCMIS. Suggestions for improved implementation focused on the adaptation of the intervention. Namely, surgeons discussed regionally-specific adaptations as well as adaptations specific to their surgical specialty including visual, rather than written, representation of the information available via POCMIS.

CONCLUSIONS

Results indicate necessary adaptations for implementing interventions including POCMIS in LMICs, mimicking much of the implementation science literature on intervention adaptation. In addition to explicit suggestions provided by surgeons, we also suggest actionable steps to adapt to barriers identified in our data. Rapid assessment procedures (RAP) are one established methodological technique useful for assessing organization culture prior to implementation, allowing for necessary cultural adaptations. Dynamic adaption process (DAP) is another useful and established method that breaks implementation into four phases allowing for adaptations based on the initial assessment of the intervention site.

Absorptive capacity in the adoption of innovations in health: a scoping review

OBJECTIVE

The objective of this scoping review was to explore how absorptive capacity has been conceptualized and measured in studies of innovation adoption in health care organizations.

INTRODUCTION

Current literature highlights the need to incorporate knowledge translation processes at the organizational and system level to enhance the adoption of new knowledge into practice. Absorptive capacity is a set of routines and processes characterized by knowledge acquisition, assimilation, transformation, and application. A key concept in organizational learning theory, absorptive capacity is thought to be critical to the adoption of new knowledge and innovations in organizations. To understand how absorptive capacity was conceptualized and measured in health care organizations, it was appropriate to conduct a scoping review to answer our research question.

INCLUSION CRITERIA

This scoping review included published and unpublished primary studies (ie, experimental, quasi-experimental, observational, and qualitative study designs), as well as reviews that broadly focused on the adoption of innovations at the organizational level in health care, and framed innovation adoption as processes that rely on organizational learning and absorptive or learning capacity.

METHODS

Searches included electronic databases (ie, MEDLINE, Embase, PsycINFO, CINAHL, and Scopus) and gray literature, as well as reference scanning of relevant studies. Study abstracts and full texts were screened for eligibility by two independent reviewers. Data extraction of relevant studies was also done independently by two reviewers. All discrepancies were addressed through discussion or adjudicated by a third reviewer. Synthesis of the extracted data focused on descriptive frequencies and counts of the results. This review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR).

RESULTS

The search strategies identified a total of 7433 citations. Sixteen papers were identified for inclusion, including a set of two companion papers, and data were extracted from 15 studies. We synthesized the objectives of the included studies and identified that researchers focused on at least one of the following aspects: i) exploring pre-existing capacity that affects improvement and innovation in health care settings; ii) describing factors influencing the spread and sustainability of organizations; iii) identifying measures and testing the knowledge application process; and iv) providing construct clarity. No new definitions were identified within this review; instead existing definitions were refined to suit the local context of the health care organization in which they were used.

CONCLUSIONS

Given the rapidly changing and evolving nature of health care, it is important to understand both current best practices and an organization's ability to acquire, assimilate, transform, and apply these practices to their specific organization. While much research has gone into developing ways to implement knowledge translation, understanding an organization's internal structures and framework for seeking out and implementing new evidence as it relates to absorptive capacity is still a relatively novel concept.

Quantitative Measurements for Factors Influencing Implementation in School Settings: Protocol for A Systematic Review and A Psychometric and Pragmatic Analysis

INTRODUCTION

In order to address the effectiveness and sustainability of school-based interventions, there is a need to consider the factors affecting implementation success. The rapidly growing field of implementation-focused research is struggling to determine how to assess and measure implementation-relevant constructs. Earlier research has identified the need for strong psychometric and pragmatic measures. The aims of this review are therefore to (i) systematically review the literature to identify measurements of the factors influencing implementations which have been developed or adapted in school settings, (ii) describe each measurement's psychometric and pragmatic properties, (iii) describe the alignment between each measurement and the corresponding domain and/or construct of the Consolidated Framework for Implementation Research (CFIR).

METHODS

Six databases (Medline, ERIC, PsycInfo, Cinahl, Embase, and Web of Science) will be searched for peer-reviewed articles reporting on school settings, published from the year 2000. The identified measurements will be mapped against the CFIR, and analyzed for their psychometric and pragmatic properties.

DISCUSSION

By identifying measurements that are psychometrically and pragmatically impactful in the field, this review will contribute to the identification of feasible, effective, and sustainable implementation strategies for future research in school settings.

A Longitudinal Mixed-Methods Examination of Positive Health Check: Implementation Results From a Type 1 Effectiveness-Implementation Hybrid Trial

BACKGROUND

Positive Health Check is an evidence-based video doctor intervention developed for improving the medication adherence, retention in care, and viral load suppression of people with HIV receiving clinical care.

SETTING

Four HIV primary care clinics within the United States.

METHODS

As part of a type 1 hybrid trial, a mixed-methods approach was used to longitudinally assess the following 3 key implementation constructs over a 23-month period: innovation-values fit (ie, the extent to which staff perceive innovation use will foster the fulfillment of their values), organizational readiness for change (ie, the extent to which organizational members are psychologically and behaviorally prepared to implement organizational change), and implementation climate (ie, the extent to which implementation is expected, supported, and rewarded). Quantitative mixed-effects regression analyses were conducted to assess changes over time in these constructs. Qualitative analyses were integrated to help provide validation and understanding.

RESULTS

Innovation-values fit and organizational readiness for change were found to be high and relatively stable. However, significant curvilinear change over time was found for implementation climate. Based on the qualitative data, implementation climate declined toward the end of implementation because of decreased engagement from clinic champions and differences in priorities between research and clinic staff.

CONCLUSIONS

The Positive Health Check intervention was found to fit within HIV primary care service settings, but there were some logistical challenges that needed to be addressed. Additionally, even within the context of an effectiveness trial, significant and nonlinear change in implementation climate should be expected over time.

Leading the charge in the education sector: development and validation of the School Implementation Leadership Scale (SILS)

Background

Strategic implementation leadership is a critical determinant of successful implementation, hypothesized to create a more supportive implementation climate conducive to the adoption and use of evidence-based practices. Implementation leadership behaviors may vary significantly across contexts, necessitating studies that examine the validity of established measurement tools in novel health service delivery sectors. The education sector is the most common site for delivering mental health services to children and adolescents in the USA, but research focused on implementation leadership in schools is in the early phases, and there is a need for adaptation and expansion of instruments in order to tailor to the school context. The current study adapted and validated the School Implementation Leadership Scale (SILS) (based on the Implementation Leadership Scale) in a sample of elementary school personnel from six school districts who were implementing one of two well-established prevention programs for supporting children’s mental health.

Methods

Participants were 441 public school teachers from 52 elementary schools in the Midwest and West Coast of the USA. Participants completed a survey that contained: (1) an adapted and expanded version of the SILS with additional items generated for four existing subscales as well as three new subscales (communication, vision/mission, and availability), and (2) additional tools to evaluate convergent and divergent validity (i.e., measures of general/molar leadership and teaching attitudes). Data underwent (1) examination of item characteristic curves to reduce items and ensure a pragmatic instrument, (2) confirmatory factor analyses to establish structural validity, and (3) evaluation of convergent and divergent validity.

Results

Item reduction analyses resulted in seven subscales of three items each. Results indicated acceptable fit for a seven-factor structural model (CFI = .995, TLI = .99, RMSEA = .07, SRMR = 0.02). Second-order factor loadings were high (λ = .89 to .96), suggesting that the SILS subscales comprise a higher-order implementation leadership factor. All subscales demonstrated good inter-item reliability (α = .91–.96). Convergent and divergent validity results were generally as hypothesized, with moderate to high correlations between SILS subscales and general leadership, moderate correlations with teaching attitudes, and low correlations with school demographics.

Conclusions

Overall, results provided strong structural, convergent, and divergent validity evidence for the 21-item, 7-factor SILS instrument. Implications for the measurement of implementation leadership in schools are discussed, as well as strategies to support leaders to enhance their strategic behaviors related to the implementation of mental health prevention programs (e.g., adaptation of existing leadership-focused implementation strategies).

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-022-01222-7.

Measuring capacity to use evidence-based interventions in community-based organizations: A comprehensive, scoping review

Introduction:

Community-based organizations (CBOs) are well-positioned to incorporate research evidence, local expertise, and contextual factors to address health inequities. However, insufficient capacity limits use of evidence-based interventions (EBIs) in these settings. Capacity-building implementation strategies are popular, but a lack of standard models and validated measures hinders progress in the field. To advance the literature, we conducted a comprehensive scoping review.

Methods:

With a reference librarian, we executed a comprehensive search strategy of PubMed/Medline, Web of Science Core Collection, and EBSCO Global Health. We included articles that addressed implementation science, capacity-building, and CBOs. Of 5527 articles, 99 met our inclusion criteria, and we extracted data using a double-coding process

Results:

Of the 99 articles, 47% defined capacity explicitly, 31% defined it indirectly, and 21% did not define it. Common concepts in definitions were skills, knowledge/expertise, and resources. Of the 57 articles with quantitative analysis, 48 (82%) measured capacity, and 11 (23%) offered psychometric data for the capacity measures. Of the 99 studies, 40% focused exclusively on populations experiencing inequities and 22% included those populations to some extent. The bulk of the studies came from high-income countries.

Conclusions:

Implementation scientists should 1) be explicit about models and definitions of capacity and strategies for building capacity, 2) specify expected multi-level implementation outcomes, 3) develop and use validated measures for quantitative work, and 4) integrate equity considerations into the conceptualization and measurement of capacity-building efforts. With these refinements, we can ensure that the necessary supports reach CBO practitioners and critical partners for addressing health inequities.

Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Background

Mental health is a critical component of wellness. Public policies present an opportunity for large-scale mental health impact, but policy implementation is complex and can vary significantly across contexts, making it crucial to evaluate implementation. The objective of this study was to (1) identify quantitative measurement tools used to evaluate the implementation of public mental health policies; (2) describe implementation determinants and outcomes assessed in the measures; and (3) assess the pragmatic and psychometric quality of identified measures.

Method

Guided by the Consolidated Framework for Implementation Research, Policy Implementation Determinants Framework, and Implementation Outcomes Framework, we conducted a systematic review of peer-reviewed journal articles published in 1995-2020. Data extracted included study characteristics, measure development and testing, implementation determinants and outcomes, and measure quality using the Psychometric and Pragmatic Evidence Rating Scale.

Results

We identified 34 tools from 25 articles, which were designed for mental health policies or used to evaluate constructs that impact implementation. Many measures lacked information regarding measurement development and testing. The most assessed implementation determinants were readiness for implementation, which encompassed training (n  =  20, 57%) and other resources (n  =  12, 34%), actor relationships/networks (n  =  15, 43%), and organizational culture and climate (n  =  11, 31%). Fidelity was the most prevalent implementation outcome (n  =  9, 26%), followed by penetration (n  =  8, 23%) and acceptability (n  =  7, 20%). Apart from internal consistency and sample norms, psychometric properties were frequently unreported. Most measures were accessible and brief, though minimal information was provided regarding interpreting scores, handling missing data, or training needed to administer tools.

Conclusions

This work contributes to the nascent field of policy-focused implementation science by providing an overview of existing measurement tools used to evaluate mental health policy implementation and recommendations for measure development and refinement. To advance this field, more valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap.

Plain Language Summary

Mental health is a critical component of wellness, and public policies present an opportunity to improve mental health on a large scale. Policy implementation is complex because it involves action by multiple entities at several levels of society. Policy implementation is also challenging because it can be impacted by many factors, such as political will, stakeholder relationships, and resources available for implementation. Because of these factors, implementation can vary between locations, such as states or countries. It is crucial to evaluate policy implementation, thus we conducted a systematic review to identify and evaluate the quality of measurement tools used in mental health policy implementation studies. Our search and screening procedures resulted in 34 measurement tools. We rated their quality to determine if these tools were practical to use and would yield consistent (i.e., reliable) and accurate (i.e., valid) data. These tools most frequently assessed whether implementing organizations complied with policy mandates and whether organizations had the training and other resources required to implement a policy. Though many were relatively brief and available at little-to-no cost, these findings highlight that more reliable, valid, and practical measurement tools are needed to assess and inform mental health policy implementation. Findings from this review can guide future efforts to select or develop policy implementation measures.

Absorptive capacity in the adoption of innovations in health care: a scoping review protocol

OBJECTIVE

To explore how absorptive capacity has been conceptualized and measured in studies of innovation adoption in health care organizations.

INTRODUCTION

Current literature highlights the need to incorporate knowledge translation processes at the organizational and system level to enhance the adoption of new knowledge into practice. Absorptive capacity is a set of routines and processes characterized by knowledge acquisition, assimilation, transformation, and application. Absorptive capacity, a key concept in organizational learning theory, is thought to be critical to the adoption of new knowledge and innovations in organizations.

INCLUSION CRITERIA

This scoping review will include primary studies (ie, experimental, quasi-experimental, observational, and qualitative study designs) and gray literature that broadly focus on the adoption of innovations at the organizational level in health care, and frame innovation adoption as processes that rely on organizational learning and absorptive or learning capacity.

METHODS

Data sources will include comprehensive searches of electronic databases (eg, MEDLINE, Embase, PsycINFO, CINAHL, and Scopus), gray literature, and reference scanning of relevant studies. Study abstracts and full texts will be screened for eligibility by two reviewers, independently. Data extraction of relevant studies will also be done independently by two reviewers. All discrepancies will be addressed through further discussion or adjudicated by a third reviewer. Synthesis of the extracted data will focus on descriptive frequencies, counts, and thematic analysis and the results will be reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR).

Quantitative approaches for the evaluation of implementation research studies

Implementation research necessitates a shift from clinical trial methods in both the conduct of the study and in the way that it is evaluated given the focus on the impact of implementation strategies. That is, the methods or techniques to support the adoption and delivery of a clinical or preventive intervention, program, or policy. As strategies target one or more levels within the service delivery system, evaluating their impact needs to follow suit. This article discusses the methods and practices involved in quantitative evaluations of implementation research studies. We focus on evaluation methods that characterize and quantify the overall impacts of an implementation strategy on various outcomes. This article discusses available measurement methods for common quantitative implementation outcomes involved in such an evaluation-adoption, fidelity, implementation cost, reach, and sustainment-and the sources of such data for these metrics using established taxonomies and frameworks. Last, we present an example of a quantitative evaluation from an ongoing randomized rollout implementation trial of the Collaborative Care Model for depression management in a large primary healthcare system.

Community-based behavioral health administrator perspectives on sustainability of Dialectical Behavior Therapy: a qualitative evaluation

BACKGROUND

Substantial resources have been invested in evidence-based practice (EBP) implementation in community settings; however, research suggests that EBPs do not always sustain over time.

METHOD

This qualitative study explored the perspectives of 13 community behavioral health agency leaders regarding the sustainability of an EBP 25 to 28 months following the original training period. Administrators from 10 agencies were interviewed to understand the complexities of the implementation process, sustainability of Dialectical Behavior Therapy, and their recommendations to enhance implementation and sustainability.

RESULTS

A content analysis revealed five emergent themes: treatment model opinions, resource concerns, staff selection/ turnover, population characteristics, and recommendations for future implementation.

CONCLUSIONS

These themes likely would be helpful in informing the design of future implementation and sustainability initiatives sensitive to the challenges of integrating EBPs in community settings.

Optimizing medication use through a synergistic technology testing process integrating implementation science to drive effectiveness and facilitate scale

OBJECTIVES

1) To describe a synergistic technology testing process (STTP) that integrates traditional technology assessment with implementation science principles to drive uptake, enhance outcomes, and facilitate scaling of medication optimization health information technology solutions; and 2) to illustrate the application of the STTP using an example that involves designing and testing a medication therapy problem (MTP) platform for use by pharmacists in primary care.

SUMMARY

Optimizing medication services requires supportive technologies that have been fully tested before release. Current testing approaches are not sufficient to produce the information needed to accelerate uptake and drive impact. Implementation science principles can supplement the traditional testing process by broadening its focus to include designing a truly usable technology, attending to contextual influences, studying the implementation process, and assessing the technology for its scalability. The STTP is an early attempt at outlining the integration of traditional technology testing with implementation science for pharmacy practice.

CONCLUSION

The potential impact of technology-supported medication optimization solutions to improve patient outcomes, enhance quality of care, and reduce costs could be substantial. Accelerating uptake, driving impact, and facilitating scaling will require innovative testing paradigms that result in evidence-based technologies that can feasibly be implemented in real-world settings.

German language questionnaires for assessing implementation constructs and outcomes of psychosocial and health-related interventions: a systematic review

Background

Over the past years, implementation science has gained more and more importance in German-speaking countries. Reliable and valid questionnaires are needed for evaluating the implementation of evidence-based practices. On an international level, several initiatives focused on the identification of questionnaires used in English-speaking countries but limited their search processes to mental health and public health settings. Our aim was to identify questionnaires used in German-speaking countries measuring the implementation of interventions in public health and health care settings in general and to assess their psychometric properties.

Methods

We searched five different bibliographic databases (from 1985 to August 2017) and used several other search strategies (e.g., reference lists, forward citation) to obtain our data. We assessed the instruments, which were identified in an independent dual review process, using 12 psychometric rating criteria. Finally, we mapped the instruments’ scales and subscales in regard to the constructs of the Consolidated Framework for Implementation Research (CFIR) and the Implementation Outcome Framework (IOF).

Results

We identified 31 unique instruments available for the assessment of implementation science constructs. Hospitals and other health care settings were the ones most often investigated (23 instruments), while education and childcare settings, workplace settings, and community settings lacked published instruments. Internal consistency, face and content validity, usability, and structural validity were the aspects most often described. However, most studies did not report on test-retest reliability, known-groups validity, predictive criterion validity, or responsiveness. Overall, the majority of studies did not reveal high-quality instruments, especially regarding the psychometric criteria internal consistency, structural validity, and criterion validity. In addition, we seldom detected instruments operationalizing the CFIR domains intervention characteristics, outer setting, and process, and the IOF constructs adoption, fidelity, penetration, and sustainability.

Conclusions

Overall, a sustained and continuous effort is needed to improve the reliability and validity of existing instruments to new ones. Instruments applicable to the assessment of implementation constructs in public health and community settings are urgently needed.

Trial registration

The systematic review protocol was registered in PROSPERO on October 19, 2017, under the following number: CRD42017075208.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0837-3) contains supplementary material, which is available to authorized users.

Process measures facilitate maturation of pediatric enhanced recovery protocols

BACKGROUND/PURPOSE

The role of process measures used to predict quality in pediatric colorectal surgery enhanced recovery protocols has not been described. The purpose of this study was to demonstrate the feasibility of abstracting and monitoring process measures over protocol improvement iteration.

METHODS

Patients enrolled in the Pediatric Colorectal Enhanced Recovery After Surgery pathway at our institution were grouped by stage of implementation. We used a quality improvement database to compare multistage enhanced recovery process measures and 30-day patient outcomes.

RESULTS

We identified 58 surgical patients with 28(48%) cases enrolled in the pathway. There was increased use of regional anesthesia techniques in pathway patients (83% versus 20%, p < 0.001). All preoperative process measures clinically improved between early and full implementation. Improvements included a dramatic increase in formal preoperative education (56% versus 0%, p = 0.004) and administration of preoperative medication (p = 0.025). Overall, 12 (21%) patients experienced postoperative complications, which were similarly distributed between implementation groups. Readmissions were highest during the early implementation phase (40%, p = 0.029). Children in the late implementation group experienced fewer complications, which clinically correlated with process measure adherence.

CONCLUSIONS

Process measures complement outcome measures in assessing quality and effectiveness of a pediatric colorectal recovery protocol. Adherence to processes may reduce complications.

LEVEL OF EVIDENCE

Treatment study, Level III.

Sustained Implementation Support Scale: Validation of a Measure of Program Characteristics and Workplace Functioning for Sustained Program Implementation

An evaluation measure of enablers and inhibitors to sustained evidence-based program (EBP) implementation may provide a useful tool to enhance organizations' capacity. This paper outlines preliminary validation of such a measure. An expert informant and consumer feedback approach was used to tailor constructs from two existing measures assessing key domains associated with sustained implementation. Validity and reliability were evaluated for an inventory composed of five subscales: Program benefits, Program burden, Workplace support, Workplace cohesion, and Leadership style. Exploratory and confirmatory factor analysis with a sample of 593 Triple P-Positive Parenting Program-practitioners led to a 28-item scale with good reliability and good convergent, discriminant, and predictive validity. Practitioners sustaining implementation at least 3 years post-training were more likely to have supervision/peer support, reported higher levels of program benefit, workplace support, and positive leadership style, and lower program burden compared to practitioners who were non-sustainers.

Opportunities for Epidemiologists in Implementation Science: A Primer

The field of epidemiology has been defined as the study of the spread and control of disease. However, epidemiology frequently focuses on studies of etiology and distribution of disease at the cost of understanding the best ways to control disease. Moreover, only a small fraction of scientific discoveries are translated into public health practice, and the process from discovery to translation is exceedingly slow. Given the importance of translational science, the future of epidemiologic training should include competency in implementation science, whose goal is to rapidly move evidence into practice. Our purpose in this paper is to provide epidemiologists with a primer in implementation science, which includes dissemination research and implementation research as defined by the National Institutes of Health. We describe the basic principles of implementation science, highlight key components for conducting research, provide examples of implementation studies that encompass epidemiology, and offer resources and opportunities for continued learning. There is a clear need for greater speed, relevance, and application of evidence into practice, programs, and policies and an opportunity to enable epidemiologists to conduct research that not only will inform practitioners and policy-makers of risk but also will enhance the likelihood that evidence will be implemented.

An updated protocol for a systematic review of implementation-related measures

BACKGROUND

Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript.

METHODS

Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence.

DISCUSSION

We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.

Building Capacity for Evidence-Based Public Health: Reconciling the Pulls of Practice and the Push of Research

Timely implementation of principles of evidence-based public health (EBPH) is critical for bridging the gap between discovery of new knowledge and its application. Public health organizations need sufficient capacity (the availability of resources, structures, and workforce to plan, deliver, and evaluate the preventive dose of an evidence-based intervention) to move science to practice. We review principles of EBPH, the importance of capacity building to advance evidence-based approaches, promising approaches for capacity building, and future areas for research and practice. Although there is general agreement among practitioners and scientists on the importance of EBPH, there is less clarity on the definition of evidence, how to find it, and how, when, and where to use it. Capacity for EBPH is needed among both individuals and organizations. Capacity can be strengthened via training, use of tools, technical assistance, assessment and feedback, peer networking, and incentives. Modest investments in EBPH capacity building will foster more effective public health practice.

Developing measures to assess constructs from the Inner Setting domain of the Consolidated Framework for Implementation Research

Background

Scientists and practitioners alike need reliable, valid measures of contextual factors that influence implementation. Yet, few existing measures demonstrate reliability or validity. To meet this need, we developed and assessed the psychometric properties of measures of several constructs within the Inner Setting domain of the Consolidated Framework for Implementation Research (CFIR).

Methods

We searched the literature for existing measures for the 7 Inner Setting domain constructs (Culture Overall, Culture Stress, Culture Effort, Implementation Climate, Learning Climate, Leadership Engagement, and Available Resources). We adapted items for the healthcare context, pilot-tested the adapted measures in 4 Federally Qualified Health Centers (FQHCs), and implemented the revised measures in 78 FQHCs in the 7 states (N = 327 respondents) with a focus on colorectal cancer (CRC) screening practices. To psychometrically assess our measures, we conducted confirmatory factor analysis models (CFA; structural validity), assessed inter-item consistency (reliability), computed scale correlations (discriminant validity), and calculated inter-rater reliability and agreement (organization-level construct reliability and validity).

Results

CFAs for most constructs exhibited good model fit (CFI > 0.90, TLI > 0.90, SRMR < 0.08, RMSEA < 0.08), with almost all factor loadings exceeding 0.40. Scale reliabilities ranged from good (0.7 ≤ α < 0.9) to excellent (α ≥ 0.9). Scale correlations fell below 0.90, indicating discriminant validity. Inter-rater reliability and agreement were sufficiently high to justify measuring constructs at the clinic-level.

Conclusions

Our findings provide psychometric evidence in support of the CFIR Inner Setting measures. Our findings also suggest the Inner Setting measures from individuals can be aggregated to represent the clinic-level. Measurement of the Inner Setting constructs can be useful in better understanding and predicting implementation in FQHCs and can be used to identify targets of strategies to accelerate and enhance implementation efforts in FQHCs.

Implementation successes and challenges in participating in a pragmatic study to improve colon cancer screening: perspectives of health center leaders

Little is known about the challenges faced by community clinics who must address clinical priorities first when participating in pragmatic studies. We report on implementation challenges faced by the eight community health centers that participated in Strategies and Opportunities to STOP Colon Cancer in Priority Populations (STOP CRC), a large comparative effectiveness cluster-randomized trial to evaluate a direct-mail program to increase the rate of colorectal cancer (CRC) screening. We conducted interviews, at the onset of implementation and 1 year later, with center leaders to identify challenges with implementing and sustaining an electronic medical record (EMR)-driven mailed program to increase CRC screening rates. We used the Consolidated Framework for Implementation Research to thematically analyze the content of meeting discussions and identify anticipated and experienced challenges. Common early concerns were patients' access to colonoscopy, patients' low awareness of CRC screening, time burden on clinic staff to carry out the STOP CRC program, inability to accurately identify eligible patients, and incompatibility of the program's approach with the patient population or organizational culture. Once the program was rolled out, time burden remained a primary concern and new organizational capacity and EMR issues were raised (e.g., EMR staffing resources and turnover in key leadership positions). Cited program successes were improved CRC screening processes and rates, more patients reached, reduced costs, and improved patient awareness, engagement, or satisfaction. These findings may inform any clinic considering mailed fecal testing programs and future pragmatic research efforts in community health centers.

Controlling Chronic Diseases Through Evidence-Based Decision Making: A Group-Randomized Trial

Introduction

Although practitioners in state health departments are ideally positioned to implement evidence-based interventions, few studies have examined how to build their capacity to do so. The objective of this study was to explore how to increase the use of evidence-based decision-making processes at both the individual and organization levels.

Methods

We conducted a 2-arm, group-randomized trial with baseline data collection and follow-up at 18 to 24 months. Twelve state health departments were paired and randomly assigned to intervention or control condition. In the 6 intervention states, a multiday training on evidence-based decision making was conducted from March 2014 through March 2015 along with a set of supplemental capacity-building activities. Individual-level outcomes were evidence-based decision making skills of public health practitioners; organization-level outcomes were access to research evidence and participatory decision making. Mixed analysis of covariance models was used to evaluate the intervention effect by accounting for the cluster randomized trial design. Analysis was performed from March through May 2017.

Results

Participation 18 to 24 months after initial training was 73.5%. In mixed models adjusted for participant and state characteristics, the intervention group improved significantly in the overall skill gap (P = .01) and in 6 skill areas. Among the 4 organizational variables, only access to evidence and skilled staff showed an intervention effect (P = .04).

Conclusion

Tailored and active strategies are needed to build capacity at the individual and organization levels for evidence-based decision making. Our study suggests several dissemination interventions for consideration by leaders seeking to improve public health practice.

Factors associated with an evidence-based measure of implementation for the Adolescent Community Reinforcement Approach

BACKGROUND

An evidence-based measure of implementation (EBMI) is an implementation outcome measure shown to have predictive validity with one or more future-measured constructs of importance. The current study sought to identify correlates and predictors of an EBMI called procedure exposure. Garner et al. (2016) found procedure exposure to be an EBMI for the Adolescent Community Reinforcement Approach (A-CRA).

METHODS

The dataset included 76 community-based substance use treatment organizations located across the United States. Organizational-level regression analyses, which were framed within the context of Chaudoir et al. (2013) framework for predicting implementation outcomes, were used to examine predictors of A-CRA procedure exposure RESULTS: The Washington Circle's treatment initiation performance measure (B=5.05 [SE=1.60], p=0.002), as well as session exposure (B=0.18 [SE=0.06], p=0.003), were significant predictors of A-CRA procedure exposure in the backward stepwise regression analysis (Adjusted R-square=0.55). The Washington Circle's treatment engagement performance measure (B=7.93 [SE=0.77], p<0.001), as well as time-to-proficiency (B=-0.04 [SE=0.02], p=0.02), each had significant bivariate relationships with A-CRA procedure exposure but were not retained in the final model.

CONCLUSIONS

Organizations implementing A-CRA are encouraged to make the following high priorities: (a) scheduling and completing a subsequent treatment session within 14days of their index session (treatment initiation) and (b) providing a targeted number of treatment sessions to each client (session exposure). To the extent organizations do this, they may be more likely to achieve higher levels of A-CRA procedure exposure.

Implementation outcome assessment instruments used in physical healthcare settings and their measurement properties: a systematic review protocol

INTRODUCTION

Over the past 10 years, research into methods that promote the uptake, implementation and sustainability of evidence-based interventions has gathered pace. However, implementation outcomes are defined in different ways and assessed by different measures; the extent to which these measures are valid and reliable is unknown. The aim of this systematic review is to identify and appraise studies that assess the measurement properties of quantitative implementation outcome instruments used in physical healthcare settings, to advance the use of precise and accurate measures.

METHODS AND ANALYSIS

The following databases will be searched from inception to March 2017: MEDLINE, EMBASE, PsycINFO, CINAHL and the Cochrane Library. Grey literature will be sought via HMIC, OpenGrey, ProQuest for theses and Web of Science Conference Proceedings Citation Index-Science. Reference lists of included studies and relevant reviews will be hand searched. Three search strings will be combined to identify eligible studies: (1) implementation literature, (2) implementation outcomes and (3) measurement properties. Screening of titles, abstracts and full papers will be assessed for eligibility by two reviewers independently and any discrepancies resolved via consensus with the wider team. The methodological quality of the studies will be assessed using the COnsensus-based Standards for the selection of health Measurement INstruments checklist. A set of bespoke criteria to determine the quality of the instruments will be used, and the relationship between instrument usability and quality will be explored.

ETHICS AND DISSEMINATION

Ethical approval is not necessary for systematic review protocols. Researchers and healthcare professionals can use the findings of this systematic review to guide the selection of implementation outcomes instruments, based on their psychometric quality, to assess the impact of their implementation efforts. The findings will also provide a useful guide for reviewers of papers and grants to determine the psychometric quality of the measures used in implementation research.

TRIAL REGISTRATION NUMBER

International Prospective Register of Systematic Reviews (PROSPERO): CRD42017065348.

Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework

BACKGROUND

The effectiveness of complex interventions, as well as their success in reaching relevant populations, is critically influenced by their implementation in a given context. Current conceptual frameworks often fail to address context and implementation in an integrated way and, where addressed, they tend to focus on organisational context and are mostly concerned with specific health fields. Our objective was to develop a framework to facilitate the structured and comprehensive conceptualisation and assessment of context and implementation of complex interventions.

METHODS

The Context and Implementation of Complex Interventions (CICI) framework was developed in an iterative manner and underwent extensive application. An initial framework based on a scoping review was tested in rapid assessments, revealing inconsistencies with respect to the underlying concepts. Thus, pragmatic utility concept analysis was undertaken to advance the concepts of context and implementation. Based on these findings, the framework was revised and applied in several systematic reviews, one health technology assessment (HTA) and one applicability assessment of very different complex interventions. Lessons learnt from these applications and from peer review were incorporated, resulting in the CICI framework.

RESULTS

The CICI framework comprises three dimensions-context, implementation and setting-which interact with one another and with the intervention dimension. Context comprises seven domains (i.e., geographical, epidemiological, socio-cultural, socio-economic, ethical, legal, political); implementation consists of five domains (i.e., implementation theory, process, strategies, agents and outcomes); setting refers to the specific physical location, in which the intervention is put into practise. The intervention and the way it is implemented in a given setting and context can occur on a micro, meso and macro level. Tools to operationalise the framework comprise a checklist, data extraction tools for qualitative and quantitative reviews and a consultation guide for applicability assessments.

CONCLUSIONS

The CICI framework addresses and graphically presents context, implementation and setting in an integrated way. It aims at simplifying and structuring complexity in order to advance our understanding of whether and how interventions work. The framework can be applied in systematic reviews and HTA as well as primary research and facilitate communication among teams of researchers and with various stakeholders.

Factors promoting and inhibiting sustained impact of a mental health task-shifting program for HIV providers in Ethiopia

BACKGROUND

Task-shifting mental health into general medical care requires more than brief provider training. Generalists need long-term support to master new skills and changes to work context are required to sustain change in the face of competing priorities. We examined program and context factors promoting sustainability of a mental health task-shifting training for hospital-based HIV providers in Ethiopia.

METHODS

Convergent mixed-methods quasi-experimental study. Sustained impact was measured by trained/not-trained provider differences in case detection and management 16 months following the end of formal support. Factors related to sustainability were examined through interviews with trained providers.

RESULTS

Extent of sustained impact: Trained providers demonstrated modest but better agreement with standardized screeners (greater sensitivity with similar specificity). They were more likely to request that patients with mental health problems return to see them v. making a referral. Factors promoting sustainability (reported in semi-structured interviews): provider belief that the treatments they had learned were effective. New interactions with on-site mental health staff were a source of ongoing learning and encouragement. Factors diminishing sustainability: providers feelings of isolation when mental health partners left for work elsewhere, failure to incorporate mental health indicators into administrative data, to re-stock staff education materials, and to build formal mechanisms for generalist-mental health staff interaction.

CONCLUSIONS

An intervention seen as feasible and effective, and promotion of relationships across professional lines, helped generalists sustain new skills. Failure to address key system context issues made use of the skills unsustainable as external supports ended.

The role of organizational structure in readiness for change: A conceptual integration

The purpose of this review is to extend extant conceptualizations of readiness for change as an individual-level phenomenon. This review-of-reviews focuses on existing conceptual frameworks from the dissemination, implementation, quality improvement, and organizational transformation literatures in order to integrate theoretical rationales for how organization structure, a key dimension of the organizational context, may impact readiness for change. We propose that the organization structure dimensions of differentiation and integration impact readiness for change at the individual level of analysis by influencing four key concepts of relevance, legitimacy, perceived need for change, and resource allocation. We identify future research directions that focus on these four key concepts.

Development and Pilot Test of the Workplace Readiness Questionnaire, a Theory-Based Instrument to Measure Small Workplaces' Readiness to Implement Wellness Programs

PURPOSE

To develop a theory-based questionnaire to assess readiness for change in small workplaces adopting wellness programs.

DESIGN

In developing our scale, we first tested items via "think-aloud" interviews. We tested the revised items in a cross-sectional quantitative telephone survey.

SETTING

The study setting comprised small workplaces (20-250 employees) in low-wage industries.

SUBJECTS

Decision-makers representing small workplaces in King County, Washington (think-aloud interviews, n = 9), and the United States (telephone survey, n = 201) served as study subjects.

MEASURES

We generated items for each construct in Weiner's theory of organizational readiness for change. We also measured workplace characteristics and current implementation of workplace wellness programs.

ANALYSIS

We assessed reliability by coefficient alpha for each of the readiness questionnaire subscales. We tested the association of all subscales with employers' current implementation of wellness policies, programs, and communications, and conducted a path analysis to test the associations in the theory of organizational readiness to change.

RESULTS

Each of the readiness subscales exhibited acceptable internal reliability (coefficient alpha range, .75-.88) and was positively associated with wellness program implementation ( p < .05). The path analysis was consistent with the theory of organizational readiness to change, except change efficacy did not predict change-related effort.

CONCLUSION

We developed a new questionnaire to assess small workplaces' readiness to adopt and implement evidence-based wellness programs. Our findings also provide empirical validation of Weiner's theory of readiness for change.

Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review

BACKGROUND

Recent reviews have synthesised the psychometric properties of measures developed to examine implementation science constructs in healthcare and mental health settings. However, no reviews have focussed primarily on the properties of measures developed to assess innovations in public health and community settings. This review identified quantitative measures developed in public health and community settings, examined their psychometric properties, and described how the domains of each measure align with the five domains and 37 constructs of the Consolidated Framework for Implementation Research (CFIR).

METHODS

MEDLINE, PsycINFO, EMBASE, and CINAHL were searched to identify publications describing the development of measures to assess implementation science constructs in public health and community settings. The psychometric properties of each measure were assessed against recommended criteria for validity (face/content, construct, criterion), reliability (internal consistency, test-retest), responsiveness, acceptability, feasibility, and revalidation and cross-cultural adaptation. Relevant domains were mapped against implementation constructs defined by the CFIR.

RESULTS

Fifty-one measures met the inclusion criteria. The majority of these were developed in schools, universities, or colleges and other workplaces or organisations. Overall, most measures did not adequately assess or report psychometric properties. Forty-six percent of measures using exploratory factor analysis reported >50 % of variance was explained by the final model; none of the measures assessed using confirmatory factor analysis reported root mean square error of approximation (<0.06) or comparative fit index (>0.95). Fifty percent of measures reported Cronbach's alpha of <0.70 for at least one domain; 6 % adequately assessed test-retest reliability; 16 % of measures adequately assessed criterion validity (i.e. known-groups); 2 % adequately assessed convergent validity (r > 0.40). Twenty-five percent of measures reported revalidation or cross-cultural validation. The CFIR constructs most frequently assessed by the included measures were relative advantage, available resources, knowledge and beliefs, complexity, implementation climate, and other personal resources (assessed by more than ten measures). Five CFIR constructs were not addressed by any measure.

CONCLUSIONS

This review highlights gaps in the range of implementation constructs that are assessed by existing measures developed for use in public health and community settings. Moreover, measures with robust psychometric properties are lacking. Without rigorous tools, the factors associated with the successful implementation of innovations in these settings will remain unknown.

Organizational Characteristics Influence Implementation of Worksite Health Protection and Promotion Programs: Evidence From Smaller Businesses

OBJECTIVE

We explored associations between organizational factors (size, sector, leadership support, and organizational capacity) and implementation of occupational safety and health (OSH) and worksite health promotion (WHP) programs in smaller businesses.

METHODS

We conducted a web-based survey of human resource managers of 117 smaller businesses (<750 employees) and analyzed factors associated with implementation of OSH and WHP among these sites using multivariate analyses.

RESULTS

Implementation of OSH, but not WHP activities, was related to industry sector (P = 0.003). Leadership support was positively associated with OSH activities (P < 0.001), but negatively associated with WHP implementation. Organizational capacity (budgets, staffing, and committee involvement) was associated with implementation of both OSH and WHP. Size was related to neither.

CONCLUSIONS

Leadership support and specifically allocated resources reflecting that support are important factors for implementing OSH and WHP in smaller organizations.

Changing practice to support self-management and recovery in mental illness: application of an implementation model

Health services introducing practice changes need effective implementation methods. Within the setting of a community mental health service offering recovery-oriented psychosocial support for people with mental illness, we aimed to: (i) identify a well-founded implementation model; and (ii) assess its practical usefulness in introducing a new programme for recovery-oriented self-management support. We reviewed the literature to identify implementation models applicable to community mental health organisations, and that also had corresponding measurement tools. We used one of these models to inform organisational change strategies. The literature review showed few models with corresponding tools. The Promoting Action on Research Implementation in Health Services (PARIHS) model and the related Organisational Readiness to Change Assessment (ORCA) tool were used. The PARIHS proposes prerequisites for health service change and the ORCA measures the extent to which these prerequisites are present. Application of the ORCA at two time points during implementation of the new programme showed strategy-related gains for some prerequisites but not for others, reflecting observed implementation progress. Additional strategies to address target prerequisites could be drawn from the PARIHS model. The PARIHS model and ORCA tool have potential in designing and monitoring practice change strategies in community mental health organisations. Further practical use and testing of implementation models appears justified in overcoming barriers to change.

Assessing and changing organizational social contexts for effective mental health services

Culture and climate are critical dimensions of a mental health service organization's social context that affect the quality and outcomes of the services it provides and the implementation of innovations such as evidence-based treatments (EBTs). We describe a measure of culture and climate labeled Organizational Social Context (OSC), which has been associated with innovation, service quality, and outcomes in national samples and randomized controlled trials (RCTs) of mental health and social service organizations. The article also describes an empirically supported organizational intervention model labeled Availability, Responsiveness, and Continuity (ARC), which has improved organizational social context, innovation, and effectiveness in five RCTs. Finally, the article outlines a research agenda for developing more efficient and scalable organizational strategies to improve mental health services by identifying the mechanisms that link organizational interventions and social context to individual-level service provider intentions and behaviors associated with innovation and effectiveness.

Study protocol for "Study of Practices Enabling Implementation and Adaptation in the Safety Net (SPREAD-NET)": a pragmatic trial comparing implementation strategies

BACKGROUND

Little research has directly compared the effectiveness of implementation strategies in any setting, and we know of no prior trials directly comparing how effectively different combinations of strategies support implementation in community health centers. This paper outlines the protocol of the Study of Practices Enabling Implementation and Adaptation in the Safety Net (SPREAD-NET), a trial designed to compare the effectiveness of several common strategies for supporting implementation of an intervention and explore contextual factors that impact the strategies' effectiveness in the community health center setting.

METHODS/DESIGN

This cluster-randomized trial compares how three increasingly hands-on implementation strategies support adoption of an evidence-based diabetes quality improvement intervention in 29 community health centers, managed by 12 healthcare organizations. The strategies are as follows: (arm 1) a toolkit, presented in paper and electronic form, which includes a training webinar; (arm 2) toolkit plus in-person training with a focus on practice change and change management strategies; and (arm 3) toolkit, in-person training, plus practice facilitation with on-site visits. We use a mixed methods approach to data collection and analysis: (i) baseline surveys on study clinic characteristics, to explore how these characteristics impact the clinics' ability to implement the tools and the effectiveness of each implementation strategy; (ii) quantitative data on change in rates of guideline-concordant prescribing; and (iii) qualitative data on the "how" and "why" underlying the quantitative results. The outcomes of interest are clinic-level results, categorized using the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework, within an interrupted time-series design with segmented regression models. This pragmatic trial will compare how well each implementation strategy works in "real-world" practices.

DISCUSSION

Having a better understanding of how different strategies support implementation efforts could positively impact the field of implementation science, by comparing practical, generalizable methods for implementing clinical innovations in community health centers. Bridging this gap in the literature is a critical step towards the national long-term goal of effectively disseminating and implementing effective interventions into community health centers.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02325531.

Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support

BACKGROUND

Little is known about how well or under what conditions health innovations are sustained and their gains maintained once they are put into practice. Implementation science typically focuses on uptake by early adopters of one healthcare innovation at a time. The later-stage challenges of scaling up and sustaining evidence-supported interventions receive too little attention. This project identifies the challenges associated with sustainability research and generates recommendations for accelerating and strengthening this work.

METHODS

A multi-method, multi-stage approach, was used: (1) identifying and recruiting experts in sustainability as participants, (2) conducting research on sustainability using concept mapping, (3) action planning during an intensive working conference of sustainability experts to expand the concept mapping quantitative results, and (4) consolidating results into a set of recommendations for research, methodological advances, and infrastructure building to advance understanding of sustainability. Participants comprised researchers, funders, and leaders in health, mental health, and public health with shared interest in the sustainability of evidence-based health care.

RESULTS

Prompted to identify important issues for sustainability research, participants generated 91 distinct statements, for which a concept mapping process produced 11 conceptually distinct clusters. During the conference, participants built upon the concept mapping clusters to generate recommendations for sustainability research. The recommendations fell into three domains: (1) pursue high priority research questions as a unified agenda on sustainability; (2) advance methods for sustainability research; (3) advance infrastructure to support sustainability research.

CONCLUSIONS

Implementation science needs to pursue later-stage translation research questions required for population impact. Priorities include conceptual consistency and operational clarity for measuring sustainability, developing evidence about the value of sustaining interventions over time, identifying correlates of sustainability along with strategies for sustaining evidence-supported interventions, advancing the theoretical base and research designs for sustainability research, and advancing the workforce capacity, research culture, and funding mechanisms for this important work.

Mixed picture of readiness for adoption of evidence-based prevention programs in communities: exploratory surveys of state program delivery systems

An emerging literature highlights the potential for broader dissemination of evidence-based prevention programs in communities through existing state systems, such as the land grant university Extension outreach system and departments of public education and health (DOE-DPH). This exploratory study entailed surveying representatives of the national Extension system and DOE-DPH, to evaluate dissemination readiness factors, as part of a larger project on an evidence-based program delivery model called PROSPER. In addition to assessing systems' readiness factors, differences among US regions and comparative levels of readiness between state systems were evaluated. The Extension web-based survey sample N was 958 and the DOE-DPH telephone survey N was 338, with response rates of 23 and 79 %, respectively. Extension survey results suggested only a moderate level of overall readiness nationally, with relatively higher perceived need for collaborative efforts and relatively lower perceived resource availability. There were significant regional differences on all factors, generally favoring the Northeast. Results from DOE-DPH surveys showed significantly higher levels for all readiness factors, compared with Extension systems. Overall, the findings present a mixed picture. Although there were clear challenges related to measuring readiness in complex systems, addressing currently limited dissemination resources, and devising strategies for optimizing readiness, all systems showed some readiness-related strengths.

Integration of Technology-based Behavioral Health Interventions in Substance Abuse and Addiction Services

The past decade has witnessed revolutionary changes to the delivery of health services, ushered in to a great extent by the introduction of electronic health record systems. More recently, a new class of technological advancements-technology-based behavioral health interventions, which involve the delivery of evidence-informed practices via computers, web-based applications, mobile phones, wearable sensors, or other technological platforms-has emerged and is primed to once again radically shift current models for behavioral healthcare. Despite the promise and potential of these new therapeutic approaches, a greater understanding of the impact of technology-based interventions on cornerstone issues of mental health and addiction services-namely access, quality, and cost-is needed. The current review highlights 1) relevant conceptual frameworks that guide this area of research, 2) key studies that inform the relevance of technology-based interventions for behavioral healthcare access, quality, and cost, 3) pressing methodological issues that require attention, 4) unresolved questions that warrant further investigation, and 5) practical implications that underscore important new directions for this emerging area of research.

Enhancing organizational capacity to provide cancer control programs among Latino churches: design and baseline findings of the CRUZA Study

Background

Faith-based organizations (FBOs) have been successful in delivering health promotion programs for African Americans, though few studies have been conducted among Latinos. Even fewer have focused on organizational change, which is required to sustain community-based initiatives. We hypothesized that FBOs serving Latinos would be more likely to offer evidence-based strategies (EBS) for cancer control after receiving a capacity enhancement intervention to implement health programs, and designed the CRUZA trial to test this hypothesis. This paper describes the CRUZA design and baseline findings.

Methods

We identified Catholic parishes in Massachusetts that provided Spanish-language mass (n = 65). A baseline survey assessed organizational characteristics relevant to adoption of health programs, including readiness for adoption, “fit” between innovation and organizational mission, implementation climate, and organizational culture. In the next study phase, parishes that completed the baseline assessment will be recruited to a randomized cluster trial, with the parish as the unit of analysis. Both groups will receive a Program Manual and Toolkit. Capacity Enhancement parishes will also be offered technical support, assistance forming health committees and building inter-institutional partnerships, and skills-based training.

Results

Of the 49 parishes surveyed at baseline (75%), one-third (33%) reported having provided at least one health program in the prior year. However, only two program offerings were cancer-specific. Nearly one-fifth (18%) had an active health ministry. There was a high level of organizational readiness to adopt cancer control programs, high congruence between parish missions and CRUZA objectives, moderately conducive implementation climates, and organizational cultures supportive of CRUZA programming. Having an existing health ministry was significantly associated with having offered health programs within the past year. Relationships between health program offerings and other organizational characteristics were not statistically significant.

Conclusions

Findings suggest that many parishes do not offer cancer control programs, yet many may be ready to do so. However, the perceptions about existing organizational practices and policies may not be conducive to program initiation. A capacity enhancement intervention may hold promise as a means of increasing health programming. The efficacy of such an intervention will be tested in phase two of this study.

Context and implementation: A concept analysis towards conceptual maturity

Context and implementation of health interventions have received increasing attention over the past decade, in particular with respect to their influence on the effectiveness and reach of complex interventions. The underlying concepts are both considered partially mature, limiting their operationalization in research and practice. We conducted systematic literature searches and pragmatic utility (PU) concept analyses to provide a state-of-the-art assessment of the concepts of "context" and "implementation" in the health sciences to create a common understanding for their use within systematic reviews and HTA. We performed two separate searches, one for context (EMBASE, MEDLINE) and the other for implementation (Google Scholar) to identify relevant models, theories and frameworks. 17 publications on context and 35 articles on implementation met our inclusion criteria. PU concept analysis comprises three guiding principles: selection of the literature, organization and structuring of the literature, and asking analytic questions of the literature. Both concepts were analyzed according to four features of conceptual maturity, i.e., consensual definitions, clear characteristics, fully described preconditions and outcomes, and delineated boundaries. Context and implementation are highly intertwined, with both concepts influencing and interacting with each other. Context is defined as a set of characteristics and circumstances that surround the implementation effort. Implementation is conceptualized as a planned and deliberately initiated effort with the intention to put an intervention into practice. The concept of implementation presents largely consensual definitions and relatively well-defined boundaries, while distinguishing features, preconditions and outcomes are not yet fully articulated. In contrast, definitions of context vary widely, and boundaries with neighbouring concepts, such as setting and environment, are blurred; characteristics, preconditions and outcomes are ill-defined. Therefore, the maturity of both concepts should be further improved to facilitate operationalization in systematic reviews and HTAs.

Perceived Characteristics of Intervention Scale

The Perceived Characteristics of Intervention Scale (PCIS), a 20-item assessment measure, was developed to assess health care providers’ views of interventions. Two hundred and fifteen Department of Veterans Affairs’ residential treatment providers from 38 programs across the United States completed an online survey that included the PCIS as well as self-reported use of two evidence-based treatments. The PCIS was anchored to ask about two evidence-based psychotherapies for posttraumatic stress disorder, prolonged exposure, and cognitive processing therapy. The PCIS is a reliable measure of perceived characteristics of interventions, with some preliminary support for its validity. Consideration of providers’ perceptions of particular evidence-based treatments may serve as an aid to improve their dissemination, implementation, and sustained use.

Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation

There has been a growing impetus to bridge the gap between basic science discovery, development of evidence-based practices (EBPs), and the availability and delivery of EBPs in order to improve the public health impact of such practices. To capitalize on factors that support implementation and sustainment of EBPs, it is important to consider that health care is delivered within the outer context of public health systems and the inner context of health care organizations and work groups. Leaders play a key role in determining the nature of system and organizational contexts. This article addresses the role of leadership and actions that leaders can take at and across levels in developing a strategic climate for EBP implementation within the outer (i.e., system) and inner (i.e., organization, work group) contexts of health care. Within the framework of Edgar Schein's "climate embedding mechanisms," we describe strategies that leaders at the system, organization, and work group levels can consider and apply to develop strategic climates that support the implementation and sustainment of EBP in health care and allied health care settings.

Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS)

BACKGROUND

Although the importance of the organizational environment for implementing evidence-based practices (EBP) has been widely recognized, there are limited options for measuring implementation climate in public sector health settings. The goal of this research was to develop and test a measure of EBP implementation climate that would both capture a broad range of issues important for effective EBP implementation and be of practical use to researchers and managers seeking to understand and improve the implementation of EBPs.

METHODS

Participants were 630 clinicians working in 128 work groups in 32 US-based mental health agencies. Items to measure climate for EBP implementation were developed based on past literature on implementation climate and other strategic climates and in consultation with experts on the implementation of EBPs in mental health settings. The sample was randomly split at the work group level of analysis; half of the sample was used for exploratory factor analysis (EFA), and the other half was used for confirmatory factor analysis (CFA). The entire sample was utilized for additional analyses assessing the reliability, support for level of aggregation, and construct-based evidence of validity.

RESULTS

The EFA resulted in a final factor structure of six dimensions for the Implementation Climate Scale (ICS): 1) focus on EBP, 2) educational support for EBP, 3) recognition for EBP, 4) rewards for EBP, 5) selection for EBP, and 6) selection for openness. This structure was supported in the other half of the sample using CFA. Additional analyses supported the reliability and construct-based evidence of validity for the ICS, as well as the aggregation of the measure to the work group level.

CONCLUSIONS

The ICS is a very brief (18 item) and pragmatic measure of a strategic climate for EBP implementation. It captures six dimensions of the organizational context that indicate to employees the extent to which their organization prioritizes and values the successful implementation of EBPs. The ICS can be used by researchers to better understand the role of the organizational context on implementation outcomes and by organizations to evaluate their current climate as they consider how to improve the likelihood of implementation success.

Leadership networks in Catholic parishes: implications for implementation research in health

Through two case studies of Catholic parishes in Massachusetts, this study explores the implications of leader-centered versus distributed leadership in Catholic parishes for the implementation of evidence-based health interventions. The two parishes involved in the study differ from each other in several ways. In the first, parishioners are less engaged in leadership activities at the decision-making level in the parish. A small group of lay volunteers work with the parish priest and other ordained leaders on parish activities. In the second parish, a large and active lay volunteer leadership have forged an organizational structure that allows more independence from the pastor's direct oversight. In this parish, lay volunteer leaders are the prime drivers of organizational programs and events. In 2012-2013, three types of networks were assessed at each parish: discussion, collaboration, and outside-of-parish ties. The contrasts between each parish include differences in density of collaboration, in frequency of discussion, and network centrality of the respective parish priests. We further identified key actors in the network structures at each parish. We discuss the implications of these findings for understanding organizational capacity in the context of health program implementation.

Organizations disseminating health messages: the roles of organizational identification and HITs

Research into the dissemination of health information now includes more focus on how various organizations (e.g., beauty shops, schools, workplaces, and churches) and health information technologies (HITs) reach and affect audiences. One relational feature of organizations is identification--the feeling of belongingness. Our study explores how it influences audiences, especially in combination with HITs such as e-mail, websites, and social media. We use social identity theory to predict how organizational identification and social media might function in health communication. Using a 3 × 2 experimental design, we find that people's identification with a message source mediates the effect of social media on outcomes. These findings improve our understanding of when organizations might be most helpful for disseminating health information.

Dissemination and Implementation Research Funded by the US National Institutes of Health, 2005-2012

Dissemination and implementation (D&I) research is a growing area of science focused on overcoming the science-practice gap by targeting the distribution of information and adoption of interventions to public health and clinical practice settings. This study examined D&I research projects funded under specific program announcements by the US National Institutes of Health (NIH) from 2005 to 2012. The authors described the projects' D&I strategies, funding by NIH Institute, focus, characteristics of the principal investigators (PIs) and their organizations, and other aspects of study design and setting. Results showed 46 R01s, 6 R03s, and 24 R21s funded totaling $79.2 million. The top funders were the National Cancer Institute and the National Institute of Mental Health, together providing 61% of funding. The majority of PIs were affiliated with Schools of Medicine or large, nonprofit research organizations and think tanks. Only 4% of projects were to PIs with appointments at Schools of Nursing, with 7% of the funding. The most commonly funded projects across all of the studies focused on cancer control and screening, substance abuse prevention and treatment, and mental health services. Typically implemented in community and organizational settings, D&I research provides an excellent opportunity for team science, including nurse scientists and interdisciplinary collaborators.

What, why, and how care protocols are implemented in Ontario nursing homes

The aim of this study was to better understand care protocol implementation, including the influence of organizational-contextual factors on implementation approaches, in long-term care homes operating in Ontario. We surveyed directors of care employed in all 547 Ontario LTC homes, and combined survey data with secondary organizational data on rural/urban location, nursing home size, chain membership, type of ownership, and accreditation status. Motivations for the use/selection of care protocols in nursing homes primarily derived from beliefs in continuous improvement and in evidence-based care. Protocol selection was largely participative, involving management and staff. External information sources were important for protocol implementation, and in-service education was the chief means of training and educating staff. Significant differences in approaches to implementation were evident in association with differences in ownership. Three key success factors for implementation were identified: contextualizing the practice change, adequately resourcing for implementation, and demonstrating connections between practice change and outcomes.

Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures

BACKGROUND

Two of the current methodological barriers to implementation science efforts are the lack of agreement regarding constructs hypothesized to affect implementation success and identifiable measures of these constructs. In order to address these gaps, the main goals of this paper were to identify a multi-level framework that captures the predominant factors that impact implementation outcomes, conduct a systematic review of available measures assessing constructs subsumed within these primary factors, and determine the criterion validity of these measures in the search articles.

METHOD

We conducted a systematic literature review to identify articles reporting the use or development of measures designed to assess constructs that predict the implementation of evidence-based health innovations. Articles published through 12 August 2012 were identified through MEDLINE, CINAHL, PsycINFO and the journal Implementation Science. We then utilized a modified five-factor framework in order to code whether each measure contained items that assess constructs representing structural, organizational, provider, patient, and innovation level factors. Further, we coded the criterion validity of each measure within the search articles obtained.

RESULTS

Our review identified 62 measures. Results indicate that organization, provider, and innovation-level constructs have the greatest number of measures available for use, whereas structural and patient-level constructs have the least. Additionally, relatively few measures demonstrated criterion validity, or reliable association with an implementation outcome (e.g., fidelity).

DISCUSSION

In light of these findings, our discussion centers on strategies that researchers can utilize in order to identify, adapt, and improve extant measures for use in their own implementation research. In total, our literature review and resulting measures compendium increases the capacity of researchers to conceptualize and measure implementation-related constructs in their ongoing and future research.

Implementation science and its application to population health

Implementation science studies the use of strategies to adapt and use evidence-based interventions in targeted settings (e.g., schools, workplaces, health care facilities, public health departments) to sustain improvements to population health. This nascent field of research is in the early stages of developing theories of implementation and evaluating the properties of measures. Stakeholder engagement, effectiveness studies, research synthesis, and mathematical modeling are some of the methods used by implementation scientists to identify strategies to embed evidence-based interventions in clinical and public health programs. However, for implementation science to reach its full potential to improve population health the existing paradigm for how scientists create evidence, prioritize publications, and synthesize research needs to shift toward greater stakeholder input and improved reporting on external validity. This shift will improve the relevance of the research that is produced and provide information that will help guide decision makers in their selection of research-tested interventions.

Fostering more-effective public health by identifying administrative evidence-based practices: a review of the literature

CONTEXT

The aim of evidence-based decision-making in public health involves the integration of science-based interventions with community preferences to improve population health. Although considerable literature is available on the development and adoption of evidence-based guidelines and barriers to their implementation, the evidence base specific to public health administration is less developed. This article reviews the literature from public health and related disciplines to identify administrative evidence-based practices (A-EBPs; i.e., agency-level structures and activities that are positively associated with performance measures).

EVIDENCE ACQUISITION

A "review of reviews" was carried out to assess the evidence for the effectiveness of A-EBPs covering the time frame January 2000 through March 2012. The following steps were used: (1) select databases; (2) determine search parameters and conduct the search; (3) screen titles and abstracts; (4) obtain selected documents; (5) perform initial synthesis; (6) abstract data; and (7) synthesize evidence.

EVIDENCE SYNTHESIS

In both the reviews and original empiric studies, the most common outcome reported was performance of the local health department or local public health system. On the basis of a synthesis of data from 20 reviews, a total of 11 high-priority A-EBPs were identified (i.e., practices that local public health systems potentially can modify within a few years). The A-EBPs covered five major domains of workforce development, leadership, organizational climate and culture, relationships and partnerships, and financial processes.

CONCLUSIONS

As the body of practice-based research continues to grow and the ability to measure administrative evidence-based practices improves, this initial list can be further developed and improved.