Pragmatic adaptation of implementation research measures for a novel context and multiple professional roles: a factor analysis study

Promoting Evidence-Based Practice: The Influence of Novel Structural Change to Accelerate Translational Rehabilitation

OBJECTIVE

To evaluate changes in clinicians' use of evidence-based practice (EBP), openness toward EBP, and their acceptance of organizational changes after a rehabilitation hospital transitioned to a new facility designed to accelerate clinician-researcher collaborations.

DESIGN

Three repeated surveys of clinicians before, 7-9 months, and 2.5 years after transition to the new facility.

SETTING

Inpatient rehabilitation hospital.

PARTICIPANTS

Physicians, nurses, therapists, and other health care professionals (n=410, 442, and 448 respondents at Times 1, 2, and 3, respectively).

INTERVENTIONS

Implementation of physical (architecture, design) and team-focused (champions, leaders, incentives) changes in a new model of care to promote clinician-researcher collaborations.

MAIN OUTCOME MEASURES

Adapted versions of the Evidence-Based Practice Questionnaire (EBPQ), the Evidence-Based Practice Attitudes Scale (EBPAS), and the Organizational Change Recipients' Beliefs Scale (OCRBS) were used. Open-ended survey questions were analyzed through exploratory content analysis.

RESULTS

Response rates at Times 1, 2, and 3 were 67% (n=410), 69% (n=422), and 71% (n=448), respectively. After accounting for familiarity with the model of care, there was greater reported use of EBP at Time 3 compared with Time 2 (adjusted meant2=3.51, standard error (SE)=0.05; adj. meant3=3.64, SE=0.05; P=.043). Attitudes toward EBPs were similar over time. Acceptance of the new model of care was lower at Time 2 compared with Time 1, but rebounded at Time 3 (adjusted meant1=3.44, SE=0.04; adj. meant2=3.19, SE=0.04; P<.0001; adj. meant3=3.51, SE=0.04; P<.0001). Analysis of open-ended responses suggested that clinicians' optimism for the model of care was greater over time, but continued quality improvement should focus on cultivating communication between clinicians and researchers.

CONCLUSIONS

Accelerating clinician-researcher collaborations in a rehabilitation setting requires sustained effort for successful implementation beyond novel physical changes. Organizations must be responsive to clinicians' changing concerns to adapt and sustain a collaborative translational medicine model and allow sufficient time, probably years, for such transitions to occur.

Implementation Leadership in the Point of Care Nursing Context: A Systematic Review Comparing Two Measurement Tools

Introduction

Implementation leadership (IL) are effective point of care (POC) nursing leadership behaviors that facilitate contexts conducive to the successful implementation of evidence-based practices (EBPs). However, no systematic evaluation of IL tools validated for the nursing context existed.

Aims

The purpose of this systematic review was to compare iterations of two IL measurement tools, the Implementation Leadership Scale (ILS) and the iLEAD, for application in a nursing context; and to critically appraise and summarize the methodological quality of studies assessing their psychometric properties.

Methods

A comprehensive search was conducted in four databases. Two reviewers independently screened titles and abstracts, reviewed full-text articles, and performed extraction into data tables. Statisticians appraised the quality control aspects. Findings were narratively summarized.

Results

A total of 247 records were included, where 10 for the ILS (including different versions) and one for the iLEAD met the inclusion criteria. Three studies evaluated the psychometric properties of the ILS in nursing, and its translations into Chinese and Greek. Content validity was deemed to be doubtful for both tools, but the ILS had adequate rating for comprehensiveness; methodological quality was very good for structural validity, internal consistency, hypothesis testing, and responsiveness where applicable for both scales, with the exception of cross-cultural validity which had ratings of adequate and inadequate for versions of the scales. Several study findings met the criteria for good measurement properties. No studies for either tool formally assessed feasibility.

Conclusion

Applying validated and contextually relevant tools to evaluate the capacity of nursing leadership to engage in IL in real-world contexts are needed. The ILS shows promise but requires further validation for contexts with diverse and multiple nursing leaders at the POC. Feasibility needs to be further studied.

Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research

The pragmatic (i.e., practical) quality of quantitative implementation measures has received increased attention in the implementation science literature in recent years. Implementation measures that are judged to be pragmatic by implementation stakeholders are thought to be more likely to be applied in research and practice. Despite the need for pragmatic implementation measures, ambiguity and uncertainty regarding what constitutes a pragmatic measure remains. This study sought to identify and critically appraise the published literature to understand (i) how pragmatism is defined as a measurement construct/quality of implementation determinants and outcome instruments; (ii) how pragmatic qualities of instruments are evaluated; (iii) identify key gaps and limitations of the current evidence-base and (iv) identify recommendations for future research. We conducted a scoping review of the literature also employing methods of critical review. PubMed and PsycINFO databases, using the OVID interface, were searched for relevant articles published between January 2010 and September 2020. Articles that contained a definition and/or described characteristics of "pragmatism" as a measurement construct of quantitative implementation outcomes (as defined by Proctor's Implementation Outcomes taxonomy) and/or implementation determinants were eligible for inclusion. Nine articles met inclusion criteria. A degree of overlap in definitions and terms used to describe the pragmatic qualities of quantitative implementation determinant and outcome instruments were found. The most frequently cited descriptors of pragmatism were "not burdensome", "brief", "reliable", "valid" and "sensitive to change". 3 of the 9 included articles involved international implementation stakeholders in defining and conceptualizing pragmatism and employed specific methods to do so, including a systematic literature review, stakeholder interviews, concept mapping, and a Delphi process. All other articles defined pragmatism, with or without citing relevant literature. One article objectively assessed the pragmatic qualities, above and beyond the psychometric qualities, of implementation measures, using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The evidence base within the implementation instrumentation literature on what pragmatism is and how it might be assessed is limited. Some of the research identified in the review provides a strong foundation to build upon, by testing its applicability in other settings (including healthcare areas and countries) and among a more diverse group of stakeholders. We discuss directions for further development of the concept of pragmatism relating to the measurement of implementation determinants and outcomes.

Don't Get Lost in Translation: Integrating Developmental and Implementation Sciences to Accelerate Real-World Impact on Children's Development, Health, and Wellbeing

Translation of developmental science discoveries is impeded by numerous barriers at different stages of the research-to-practice pipeline. Actualization of the vast potential of the developmental sciences to improve children's health and development in the real world is imperative but has not yet been fully realized. In this commentary, we argue that an integrated developmental-implementation sciences framework will result in a translational mindset essential for accelerating real world impact. We delineate key principles and methods of implementation science of salience to the developmental science audience, lay out a potential synthesis between implementation and developmental sciences, provide an illustration of the Mental Health, Earlier Partnership (MHE-P), and set actionable steps for realization. Blending these approaches along with wide-spread adoption of the translational mindset has transformative potential for population-level impact of developmental science discovery.

Planned Evaluation of the Healthy Weight Clinic Pediatric Weight Management and Implementation: Massachusetts-CORD 3.0

Background: Despite evidence that offering multidisciplinary treatment for children with obesity is effective, access to evidence-based pediatric weight management interventions (PWMIs) is limited. The Healthy Weight Clinic PWMI is a multidisciplinary approach in primary care that improves BMI among children with a BMI ≥ 85th percentile. Objective: To describe the method by which we will evaluate the adoption, acceptability, and feasibility of integrating and implementing a multidisciplinary Healthy Weight Clinic (HWC) into primary care. Design/Methods: We used the Consolidated Framework for Implementation Research (CFIR) domains and constructs to inform our implementation strategies. We will use a Type III hybrid effectiveness-implementation design to test our implementation strategies and improvement in BMI. Sources of data collection will include qualitative interviews with patient caregivers, HWC staff and surveys with HWC staff, patient caregivers, and electronic health record data. Our outcomes are guided by the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework. Results: We identified all five CFIR domains as integral for successful implementation. Some strategies to address barriers within these domains include online self-paced training modules for the HWC staff, a virtual learning collaborative, and engagement of site leadership. Outcomes will be measured at the patient and pilot site levels, and they will include patients reached, patient health outcomes such as BMI and quality of life, level of adoption, acceptability, feasibility, and sustainability of the PWMI. Conclusion: Our use of implementation science frameworks in the planning of Healthy Weight Clinic PWMI could create a sustainable and effective program for dissemination.

Collaborative care for depression management in primary care: A randomized roll-out trial using a type 2 hybrid effectiveness-implementation design

Background

The Collaborative Care Model (CoCM) is a well-established treatment for depression in primary care settings. The critical drivers and specific strategies for improving implementation and sustainment are largely unknown. Rigorous pragmatic research is needed to understand CoCM implementation processes and outcomes.

Methods

This study is a hybrid Type 2 randomized roll-out effectiveness-implementation trial of CoCM in 11 primary care practices affiliated with an academic medical center. The Collaborative Behavioral Health Program (CBHP) was developed as a means of improving access to effective mental health services for depression. Implementation strategies are provided to all practices. Using a sequential mixed methods approach, we will assess key stakeholders’ perspectives on barriers and facilitators of implementation and sustainability of CBHP. The speed and quantity of implementation activities completed over a 30-month period for each practice will be assessed. Economic analyses will be conducted to determine the budget impact and cost offset of CBHP in the healthcare system. We hypothesize that CBHP will be effective in reducing depressive symptoms and spillover effects on chronic health conditions. We will also examine differential outcomes among racial/ethnic minority patients.

Discussion

This study will elucidate critical drivers of successful CoCM implementation. It will be among the first to conduct economic analyses on a fee-for-service model utilizing billing codes for CoCM. Data may inform ways to improve implementation efficiency with an optimization approach to successive practices due to the roll-out design. Changes to the protocol and current status of the study are discussed.

The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects

BACKGROUND

Numerous models, frameworks, and theories exist for specific aspects of implementation research, including for determinants, strategies, and outcomes. However, implementation research projects often fail to provide a coherent rationale or justification for how these aspects are selected and tested in relation to one another. Despite this need to better specify the conceptual linkages between the core elements involved in projects, few tools or methods have been developed to aid in this task. The Implementation Research Logic Model (IRLM) was created for this purpose and to enhance the rigor and transparency of describing the often-complex processes of improving the adoption of evidence-based interventions in healthcare delivery systems.

METHODS

The IRLM structure and guiding principles were developed through a series of preliminary activities with multiple investigators representing diverse implementation research projects in terms of contexts, research designs, and implementation strategies being evaluated. The utility of the IRLM was evaluated in the course of a 2-day training to over 130 implementation researchers and healthcare delivery system partners.

RESULTS

Preliminary work with the IRLM produced a core structure and multiple variations for common implementation research designs and situations, as well as guiding principles and suggestions for use. Results of the survey indicated a high utility of the IRLM for multiple purposes, such as improving rigor and reproducibility of projects; serving as a "roadmap" for how the project is to be carried out; clearly reporting and specifying how the project is to be conducted; and understanding the connections between determinants, strategies, mechanisms, and outcomes for their project.

CONCLUSIONS

The IRLM is a semi-structured, principle-guided tool designed to improve the specification, rigor, reproducibility, and testable causal pathways involved in implementation research projects. The IRLM can also aid implementation researchers and implementation partners in the planning and execution of practice change initiatives. Adaptation and refinement of the IRLM are ongoing, as is the development of resources for use and applications to diverse projects, to address the challenges of this complex scientific field.

Quantitative approaches for the evaluation of implementation research studies

Implementation research necessitates a shift from clinical trial methods in both the conduct of the study and in the way that it is evaluated given the focus on the impact of implementation strategies. That is, the methods or techniques to support the adoption and delivery of a clinical or preventive intervention, program, or policy. As strategies target one or more levels within the service delivery system, evaluating their impact needs to follow suit. This article discusses the methods and practices involved in quantitative evaluations of implementation research studies. We focus on evaluation methods that characterize and quantify the overall impacts of an implementation strategy on various outcomes. This article discusses available measurement methods for common quantitative implementation outcomes involved in such an evaluation-adoption, fidelity, implementation cost, reach, and sustainment-and the sources of such data for these metrics using established taxonomies and frameworks. Last, we present an example of a quantitative evaluation from an ongoing randomized rollout implementation trial of the Collaborative Care Model for depression management in a large primary healthcare system.