Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services

Does a Long-Term Quality and Safety Curriculum for Health Care Professionals Improve Clinical Practice? An Evaluation of Quality Improvement Projects

INTRODUCTION

Little is known about the effects on clinical practice of continuing education quality and safety curricula. The aim of this study is to gain insight into learning outcomes on the fourth level of the Kirkpatrick evaluation model for systematically deployed quality improvement projects performed by health care professionals during a Masters in Healthcare Quality and Safety in the Netherlands.

METHODS

The researchers reviewed 35 projects led by health care professionals in 16 different hospitals to determine their scopes and effects. Afterward, professionals took an online survey to determine the extent of their project's sustainability and spread.

RESULTS

Improving health care safety was the most prevalent quality dimension (n = 11, 31%). A positive change was measured by professionals for 64% (n = 35) of the primary outcomes. Statistical significance was measured in 19 (35%) of the outcomes, of which nine (47%) were found to have a statistically significant effect. A minority of professionals (17%) judged their project as sustained by the department, while some stated that the intervention (37%) or the results of the project (11%) had spread.

DISCUSSION

Although most projects indicated an improvement in their primary outcomes, only a few resulted in statistically significant changes. Teaching professionals in using evaluation methods that take into account the complex context where these projects are performed and teaching them leadership skills is needed to reduce the likelihood of unmeasured outcomes. Analyzing learning experiences of professionals in performing the project is important to see what they learned from performing quality improvement projects, providing experiences that may lead to sustainable effects in future projects.

Pediatric Primary Care Perspectives on Integrated Mental Health Care for Autism

OBJECTIVE

Timely identification of mental health needs and linkage to services is critical to provide comprehensive care for children with autism spectrum disorder (ASD). Pediatric primary care is well-positioned to facilitate this process through integrated care approaches. As a first step toward mental health integration, this study applied the Exploration, Preparation, Implementation, and Sustainment framework to characterize determinants of implementing integrated care practices for ASD.

METHODS

Sixty pediatric primary care providers and leaders from 3 organizations completed focus groups and surveys about identification of mental health needs in children with ASD and access to mental health services. Findings were integrated to examine convergence (ie, do the 2 methods confirm or find similar results) and expansion (ie, do the 2 methods provide insights beyond either method alone).

RESULTS

Results converged regarding 3 primary influences to integrated care practices for ASD: 1) limited specialized mental health referral options for ASD, 2) unique structural characteristics of the mental health system act as barriers to accessing care, and 3) caregivers differ in the degree to which they understand co-occurring mental health conditions and pursue recommended services. Qualitative results provided expansion by highlighting unique implementation considerations (eg, alignment with health care delivery priorities and values) based on primary care characteristics.

CONCLUSIONS

Findings confirm need for a tailored approach for linking children with ASD to appropriate mental health treatment. Results yield insight into the needs for organizational capacity to support integrated care and provide direction toward adapting an integrated mental health care model for children with ASD.

Reprint of: Qualitative methods in implementation research: An introduction

Qualitative methods are a valuable tool in implementation research because they help to answer complex questions such as how and why efforts to implement best practices may succeed or fail, and how patients and providers experience and make decisions in care. This article orients the novice implementation scientist to fundamentals of qualitative methods and their application in implementation research, describing: 1) implementation-related questions that can be addressed by qualitative methods; 2) qualitative methods commonly used in implementation research; 3) basic sampling and data collection procedures; and 4) recommended practices for data analysis and ensuring rigor. To illustrate qualitative methods decision-making, a case example is provided of a study examining implementation of a primary care-based collaborative care management model for women Veterans with anxiety, depression, and PTSD.

Significance of Leaders for Sustained Use of Evidence-Based Practices: A Qualitative Focus-Group Study with Mental Health Practitioners

Evidence-based practices that are implemented in mental health services are often challenging to sustain. In this focus-group study, 26 mental health practitioners with high fidelity scores were interviewed regarding their experiences with implementing the illness management and recovery, an evidence-based practice for people with severe mental disorders, in their services and how this could influence further use. Findings indicate that high fidelity is not equivalent to successful implementation. Rather, to sustain the practice in services, the practitioners emphasized the importance of their leaders being positive and engaged in the intervention, and hold clear goals and visions for the intervention in the clinic. In addition, the practitioners' understanding of outcome monitoring as a resource for practice improvement must be improved to avoid random patient experiences becoming the decisive factor in determining further use.Trial registration: ClinicalTrials.gov NCT02077829. Registered 25 February 2014.

Qualitative methods in implementation research: An introduction

Qualitative methods are a valuable tool in implementation research because they help to answer complex questions such as how and why efforts to implement best practices may succeed or fail, and how patients and providers experience and make decisions in care. This article orients the novice implementation scientist to fundamentals of qualitative methods and their application in implementation research, describing: 1) implementation-related questions that can be addressed by qualitative methods; 2) qualitative methods commonly used in implementation research; 3) basic sampling and data collection procedures; and 4) recommended practices for data analysis and ensuring rigor. To illustrate qualitative methods decision-making, a case example is provided of a study examining implementation of a primary care-based collaborative care management model for women Veterans with anxiety, depression, and PTSD.

Training in Implementation Practice Leadership (TRIPLE): evaluation of a novel practice change strategy in behavioral health organizations

Background

Effective leadership for organizational change is critical to the implementation of evidence-based practices (EBPs). As organizational leaders in behavioral health organizations often are promoted from within the agency for their long-standing, effective work as counselors, they may lack formal training in leadership, management, or practice change. This study assesses a novel implementation leadership training designed to promote leadership skills and successful organizational change specific to EBP implementation.

Methods

We conducted a pre-post outcome evaluation of the Training in Implementation Practice Leadership (TRIPLE), delivered via three in-person, half-day training sessions, with interim coaching and technical support. Sixteen mid-level leaders (75% female, 94% Caucasian, mean age 37 years) from 8 substance abuse treatment agencies participated. Professional roles included clinical managers, quality improvement coordinators, and program directors. Participants completed surveys prior to the first and following the final session. At both time points, measures included the Implementation Leadership Scale, Implementation Climate Scale, and Organizational Readiness for Implementing Change Scale. At post-test, we added the Training Acceptability and Appropriateness Scale (TAAS), assessing participant satisfaction with the training. Qualitative interviews were conducted 6 to 8 months after the training.

Results

Most participants (86% and 79%, respectively) reported increased implementation leadership skills and implementation climate; paired samples t tests indicated these pre-post increases were statistically significant. Implementation leadership scores improved most markedly on the Proactive and Knowledgeable subscales. For implementation climate, participants reported the greatest increases in educational support and recognition for using EBP. Post-test scores on the TAAS also indicated that participants found the training program to be highly acceptable and appropriate for their needs. Qualitative results supported positive outcomes of training that resulted in both increased organizational implementation as well as leadership skills of participants.

Conclusions

This training program represents an innovative, effective, and well-received implementation strategy for emerging behavioral healthcare leaders seeking to adopt or improve the delivery of EBPs. Reported implementation leadership skills and implementation climate improved following the training program, suggesting that TRIPLE may have helped fulfill a critical need for emerging behavioral healthcare leaders.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0906-2) contains supplementary material, which is available to authorized users.

The evaluation of nurse navigators in chronic and complex care

AIM

With increasing age and chronicity in populations, the need to reduce the costs of care while enhancing quality and hospital avoidance, is important. Nurse-led co-ordination is one such model of care that supports this approach. The aim of this research was to assess the impact that newly appointed Navigators have on service provision; social and economic impact; nurses' professional quality of life and compassion fatigue; and analysis of the change that has occurred to models of care and service delivery.

DESIGN

A concurrent mixed-method approach was selected to address the research aims.

METHODS

The research project was funded in July 2018 and will conclude in December 2020. Several cohorts will be studied including; patients assigned to a navigator, patients not assigned to a navigator, family members of patients assigned a navigator; and a sample sized estimated at 140 navigators.

DISCUSSION

This study provides a comprehensive international longitudinal and mixed method framework for evaluating the impact of nurse navigators on quality of care outcomes for patients with chronic conditions. IMPACT-WHAT PROBLEM WILL THE STUDY ADDRESS?: Even with specialty focused co-ordinated care, patients get lost in the system, increasing the incidence of non-compliance and exacerbation of condition. Navigators work with patients across service boundaries allowing for care that is patient responsive, and permitting variables in clinical, social and practical elements of care to be addressed in a timely manner. This novel nurse-led approach, supports hospital avoidance and patient self-management, while encouraging expansion and opportunity for the nursing and midwifery workforce.

Perceptions Among Child Welfare Staff when Modifying A Child Mental Health Intervention to be Implemented in Child Welfare Services

In order to increase access to child mental health evidence-based interventions (EBIs) for vulnerable and hard-to-engage families involved in the child welfare (CW) system, innovative approaches coupled with input from service providers are needed. One potential solution involves utilizing task-shifting strategies and implementation science theoretical frameworks to implement such EBIs in CW settings. This study examined perceptions among CW staff who were members of a collaborative advisory board involved in the implementation of the 4Rs and 2Ss Strengthening Families Program (4R2S) in CW placement prevention settings, utilizing task-shifting strategies and the Practical, Robust, Implementation, and Sustainability Model. Advisory board members reported difficulties in engaging families, heavy workloads, and conflicting implementation initiatives. While 4R2S was perceived as generally aligned with their organization's mission, modifications to the intervention and to agency procedures were recommended to promote implementation success. Suggested modifications to the existing 4R2S training and supervision are discussed. Findings underscore the importance of understanding the experiences of CW service providers, which can inform future efforts to implement child mental health EBIs in CW services.

Evaluation of the social validity and implementation process of a psychoeducational program for parents of a child with Autism Spectrum Disorder

The aim of this study was to assess the social validity and document the implementation of a psychoeducational program designed to support parents after their child's diagnosis, from both parents' and professionals' perspectives. A complete version (five workshops and five individual follow-ups) and a shortened version (five workshops only) of the program were evaluated. Parents filled in satisfaction questionnaires after every workshop and at the end of the program. Professionals who facilitated the program filled in a specially designed questionnaire to rate the quality of the program and of its implementation, the fidelity of implementation and the parents' responsiveness. In addition, video recordings of the workshops allowed an objective assessment of the fidelity of implementation. Attendance rates were high; parents were satisfied and felt they had made progress. Professionals evaluated positively the quality of the program and of its implementation, and felt parents were responsive. However, minor adaptations had to be made. The program has good social validity, which allowed easy and satisfying implementation, but it does require some flexibility. Overall, this study provides useful insight on the implementation process that may help clinical services to use this evidence-based program effectively.

Therapists' Adaptations to an Intervention to Reduce Challenging Behaviors in Children with Autism Spectrum Disorder in Publicly Funded Mental Health Services

Publicly funded mental health services play an important role in serving children with autism spectrum disorder (ASD). Previous research indicates a high likelihood of adaptations when therapists deliver evidence based practices to non-ASD populations, though less is known about therapists' use of adaptations for children with ASD receiving mental health services. The current study uses a mixed quantitative and qualitative approach to characterize the types and reasons therapists adapted a clinical intervention [An Individualized Mental Health Intervention for Children with ASD (AIM HI)] for delivery with clinically complex children with ASD served in publicly funded mental health settings and identify therapist characteristics that predict use of adaptations. The most common adaptations were characterized as augmenting AIM HI and were done to individualize the intervention to fit with therapeutic style, increase caregiver participation, and address clients' and caregivers' needs and functioning. No therapist characteristics emerged as significant predictors of adaptations. Results suggest that therapists' adaptations were largely consistent with the AIM HI protocol while individualizing the model to address the complex needs of youth with ASD.

A mixed methods study to adapt and implement integrated mental healthcare for children with autism spectrum disorder

BACKGROUND

There is a critical need for effective implementation of integrated healthcare systems for children with autism spectrum disorder (ASD). Children with ASD have many service needs, including the need to access effective mental healthcare, given high rates of co-occurring psychiatric conditions. Pediatric primary care is an ongoing point of healthcare that is well positioned to identify mental health concerns and facilitate linkage to mental health services for children with ASD. However, identifying mental health problems in children with ASD by primary care providers is complex, subject to being overlooked and may significantly vary based on primary care organizational characteristics. Efforts targeting integrated primary-mental healthcare implementation require a tailored approach for children with ASD.

METHODS

This mixed methods, community-partnered study will apply the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons et al., 2011; Moullin et al., in press) to adapt and implement an integrated care model, "Access to Tailored Autism INtegrated Care" (ATTAIN), in pediatric practices within three diverse healthcare settings for children ages 4-18 years. Key inner context factors from the Exploration, Preparation, and Implementation phases of the EPIS framework will guide three objectives of this study: (1) to identify targets to improve mental health screening and linkage to mental health services in primary care for children with ASD, (2) to adapt integrated care procedures to facilitate identification of mental health problems and linkage to evidence-based care for children with ASD, and (3) to examine feasibility, acceptability, and uptake of the adapted integrated mental healthcare model through a pilot study in pediatric primary care.

DISCUSSION

Improving integrated mental healthcare for children with ASD could have a significant public health impact on mental healthcare access, child clinical outcomes, and reduction in healthcare costs. Results from this mixed methods study will inform selection of implementation strategies to conduct larger-scale implementation of tailored integrated mental healthcare for children with ASD that will ultimately help to address the high unmet mental health needs for these children.

Adapting COMPASS for youth with ASD to improve transition outcomes using implementation science

Implementation science provides guidance on adapting existing evidence based practices (EBPs) by incorporating implementation concerns from the start. Focus-group methodology was used to understand barriers and facilitators of transition planning and implementation for students with autism spectrum disorder (ASD) who often experience disparate postsecondary outcomes compared to peers. Results were used to modify an evidence-based consultation intervention originally applied to young students with ASD, called the Collaborative Model for Promoting Competence and Success (COMPASS; Ruble, Dalrymple, & McGrew, 2012). Because consultation is a multilevel EBP, two existing implementation science frameworks were used to guide adaptation: the Framework for Evidence Based Implementation and Intervention Practices (Dunst & Trivette, 2012) and the Consolidated Framework for Implementation Research (Damschroder et al., 2009). The purpose of this article is to describe a process of adaptation of COMPASS that may be useful for other implementation science studies of consultation interventions, teacher acceptability, feasibility, and burden, and parent/student satisfaction with the adapted intervention. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Mixed-methods approach to evaluate an mHealth intervention to increase adherence to triage of human papillomavirus-positive women who have performed self-collection (the ATICA study): study protocol for a hybrid type I cluster randomized effectiveness-implementation trial

BACKGROUND

Cervical cancer is one of the leading causes of cancer death among women worldwide, with more than 85% of cases occurring in low- and middle-income countries. Human papillomavirus (HPV) screening allows for self-collection with the potential to increase coverage, but still requires triage to identify which HPV+ women need diagnostic and treatment procedures. However, achieving high levels of triage adherence can be challenging, especially among socially vulnerable women. This paper describes the ATICA protocol (Application of Communication and Information Technologies to Self-Collection, for its initials in Spanish), aimed at evaluating the implementation strategy and the effectiveness of a multi-component mobile health (mHealth) intervention to increase adherence to triage among women with HPV+ self-collected tests.

METHODS

We will use an effectiveness-implementation hybrid type I trial with a mixed-methods evaluation approach. A cluster randomized trial design including 200 community health workers (CHWs) will evaluate whether the mHealth intervention increases adherence to triage among HPV+ women who self-collected at home during a CHW visit within 120 days after a positive result. The intervention includes an initial mobile phone text message (SMS) alert and subsequent reminders sent to HPV+ women. For those who do not adhere to triage within 60 days of a positive HPV test, an email and SMS will be sent to the CHWs to promote contact with these women during home visits. We will use the Consolidated Framework for Implementation Research (CFIR) as an organizing and analytic framework to evaluate the implementation of the intervention while also drawing on Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM). We will conduct a self-administered, semi-structured survey of CHWs, semi-structured interviews with local health authorities, and a survey of HPV+ women. Combining both qualitative and quantitative data will provide rich insights into local implementation challenges and successes.

DISCUSSION

Findings from the implementation evaluation will be applicable to programs that use or are planning to incorporate HPV self-collection and/or mHealth interventions in different settings and countries. This innovative study will also serve as a model for using implementation science in the region.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT03478397 . Registered on 20 March 2018.

Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda

The field of implementation science was developed to better understand the factors that facilitate or impede implementation and generate evidence for implementation strategies. In this article, we briefly review progress in implementation science, and suggest five priorities for enhancing the impact of implementation strategies. Specifically, we suggest the need to: (1) enhance methods for designing and tailoring implementation strategies; (2) specify and test mechanisms of change; (3) conduct more effectiveness research on discrete, multi-faceted, and tailored implementation strategies; (4) increase economic evaluations of implementation strategies; and (5) improve the tracking and reporting of implementation strategies. We believe that pursuing these priorities will advance implementation science by helping us to understand when, where, why, and how implementation strategies improve implementation effectiveness and subsequent health outcomes.

Procedures to Select Digital Sensing Technologies for Passive Data Collection With Children and Their Caregivers: Qualitative Cultural Assessment in South Africa and Nepal

BACKGROUND

Populations in low-resource settings with high childhood morbidity and mortality increasingly are being selected as beneficiaries for interventions using passive sensing data collection through digital technologies. However, these populations often have limited familiarity with the processes and implications of passive data collection. Therefore, methods are needed to identify cultural norms and family preferences influencing the uptake of new technologies.

OBJECTIVE

Before introducing a new device or a passive data collection approach, it is important to determine what will be culturally acceptable and feasible. The objective of this study was to develop a systematic approach to determine acceptability and perceived utility of potential passive data collection technologies to inform selection and piloting of a device. To achieve this, we developed the Qualitative Cultural Assessment of Passive Data collection Technology (QualCAPDT). This approach is built upon structured elicitation tasks used in cultural anthropology.

METHODS

We piloted QualCAPDT using focus group discussions (FGDs), video demonstrations of simulated technology use, attribute rating with anchoring vignettes, and card ranking procedures. The procedure was used to select passive sensing technologies to evaluate child development and caregiver mental health in KwaZulu-Natal, South Africa, and Kathmandu, Nepal. Videos were produced in South Africa and Nepal to demonstrate the technologies and their potential local application. Structured elicitation tasks were administered in FGDs after showing the videos. Using QualCAPDT, we evaluated the following 5 technologies: home-based video recording, mobile device capture of audio, a wearable time-lapse camera attached to the child, proximity detection through a wearable passive Bluetooth beacon attached to the child, and an indoor environmental sensor measuring air quality.

RESULTS

In South Africa, 38 community health workers, health organization leaders, and caregivers participated in interviews and FGDs with structured elicitation tasks. We refined the procedure after South Africa to make the process more accessible for low-literacy populations in Nepal. In addition, the refined procedure reduced misconceptions about the tools being evaluated. In Nepal, 69 community health workers and caregivers participated in a refined QualCAPDT. In both countries, the child's wearable time-lapse camera achieved many of the target attributes. Participants in Nepal also highly ranked a home-based environmental sensor and a proximity beacon worn by the child.

CONCLUSIONS

The QualCAPDT procedure can be used to identify community norms and preferences to facilitate the selection of potential passive data collection strategies and devices. QualCAPDT is an important first step before selecting devices and piloting passive data collection in a community. It is especially important for work with caregivers and young children for whom cultural beliefs and shared family environments strongly determine behavior and potential uptake of new technology.

Innovations in Mixed Methods Evaluations

Mixed methods research-i.e., research that draws on both qualitative and quantitative methods in varying configurations-is well suited to address the increasing complexity of public health problems and their solutions. This review focuses specifically on innovations in mixed methods evaluations of intervention, program or policy (i.e., practice) effectiveness, and implementation. The article begins with an overview of the structure, function, and process of different mixed methods designs and then provides illustrations of their use in effectiveness studies, implementation studies, and combined effectiveness-implementation hybrid studies. The article then examines four specific innovations: procedures for transforming (or "quantitizing") qualitative data, application of rapid assessment and analysis procedures in the context of mixed methods studies, development of measures to assess implementation outcomes, and strategies for conducting both random and purposive sampling, particularly in implementation-focused evaluation research. The article concludes with an assessment of challenges to integrating qualitative and quantitative data in evaluation research.

Reducing distress and promoting resilience: a preliminary trial of a CBT skills intervention among recently HIV-diagnosed MSM in China

People living with HIV/AIDS (PLWHA) in China experience significant psychological distress, due to high rates of stigma and low availability of mental health resources. Recently diagnosed Chinese PLWHA who are men who have sex with men (MSM) are particularly vulnerable to distress, facing both HIV and sexual orientation stigma. Reducing distress and enhancing psychological resilience is critical in promoting wellbeing. However, no research to date has examined evidence-based interventions to reduce psychological symptoms and improve resilience in this population. Based on qualitative research on their mental health needs, we developed a culturally tailored, brief 3-session CBT skills-based intervention for integration into primary care [Yang, J. P., Simoni, J., Cheryan, S., Shiu, C., Chen, W., Zhao, H., & Lu, H. (2018). The development of a brief distress reduction intervention for individuals recently diagnosed with HIV in China. Cognitive Behavioral Practice, 25(2), 319-334. doi: 10.1016/j.cbpra.2017.08.002 ]. The intervention includes cognitive restructuring to address depressive thought patterns, behavioral activation to decrease isolation, and paced breathing to reduce anxiety. We conducted a pilot Type 1 hybrid effectiveness-implementation trial assessing pre-post mental health outcomes as well as feasibility, acceptability, and appropriateness information. Ten recently diagnosed MSM completed the research protocol of three individual weekly sessions. Paired-samples t tests demonstrated significant reduction in HIV-related distress, depression, problems with adjustment, as well as improvements in resilience, and perceived social support. Participants and community advisory board members found the intervention highly acceptable, appropriate, and feasible. Preliminary data from the first known study examining a psychological intervention with evidence-based components for recently diagnosed Chinese MSM suggests that this brief intervention may be useful for reducing distress and promoting resilience.

Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework

Background

Effective implementation of evidence-based practices (EBPs) remains a significant challenge. Numerous existing models and frameworks identify key factors and processes to facilitate implementation. However, there is a need to better understand how individual models and frameworks are applied in research projects, how they can support the implementation process, and how they might advance implementation science. This systematic review examines and describes the research application of a widely used implementation framework, the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.

Methods

A systematic literature review was performed to identify and evaluate the use of the EPIS framework in implementation efforts. Citation searches in PubMed, Scopus, PsycINFO, ERIC, Web of Science, Social Sciences Index, and Google Scholar databases were undertaken. Data extraction included the objective, language, country, setting, sector, EBP, study design, methodology, level(s) of data collection, unit(s) of analysis, use of EPIS (i.e., purpose), implementation factors and processes, EPIS stages, implementation strategy, implementation outcomes, and overall depth of EPIS use (rated on a 1–5 scale).

Results

In total, 762 full-text articles were screened by four reviewers, resulting in inclusion of 67 articles, representing 49 unique research projects. All included projects were conducted in public sector settings. The majority of projects (73%) investigated the implementation of a specific EBP. The majority of projects (90%) examined inner context factors, 57% examined outer context factors, 37% examined innovation factors, and 31% bridging factors (i.e., factors that cross or link the outer system and inner organizational context). On average, projects measured EPIS factors across two of the EPIS phases (M = 2.02), with the most frequent phase being Implementation (73%). On average, the overall depth of EPIS inclusion was moderate (2.8 out of 5).

Conclusion

This systematic review enumerated multiple settings and ways the EPIS framework has been applied in implementation research projects, and summarized promising characteristics and strengths of the framework, illustrated with examples. Recommendations for future use include more precise operationalization of factors, increased depth and breadth of application, development of aligned measures, and broadening of user networks. Additional resources supporting the operationalization of EPIS are available.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0842-6) contains supplementary material, which is available to authorized users.

The games they play: Observations of children with autism spectrum disorder on the school playground

The playground may be an important context to examine the social functioning of children with autism spectrum disorder. Previous literature on playground peer engagement has used quantitative methods, but there is limited research using qualitative observations to understand the nuances of playground behavior. Using a mixed-methods approach, 55 elementary school-aged children with autism spectrum disorder who are primarily included in general education settings were observed on the school playground using the Playground Observation of Peer Engagement. Quantitative and qualitative data were examined using a mixed-methods approach. The results showed that children with autism spectrum disorder: engage in solitary and peripheral activities; demonstrate appropriate initiations and responses to peers; display self-stimulatory, motoric behaviors most frequently during solitary activities; and often have neutral affect on the playground. These findings suggest that intervention and supports for children with autism spectrum disorder may be important to deliver at recess to address peer engagement.

Mindfulness-Based Stress Reduction for Parents of Children with Developmental Delays: Understanding the Experiences of Latino Families

Parents of children with developmental delays (DD) report elevated mental health difficulties compared to parents of children with typical development, which appear largely associated with child behavior problems. Latino parents of children with DD may experience heightened risk for poor mental health outcomes due additional stressors associated with minority status. Mindfulness-Based Stress Reduction (MBSR) appears to be efficacious for reducing parenting stress and improving wellbeing in families of children with DD; however, studies have rarely focused on families from racial and ethnic minority backgrounds. The current study employed a mixed-methods, waitlist-control design to examine the efficacy of MBSR for improving parent and child outcomes in Latino and non-Latino families. Results indicate that MBSR is similarly efficacious for Latino and non-Latino families in improving parent mental health (parenting stress, depressive symptomatology, and life satisfaction) and reducing parent-reported child behavior problems. Qualitative analyses further highlight avenues for improving the efficacy of MBSR for Latino families by providing intervention directly in Spanish, rather than using translation services, for Spanish-speaking families. Findings reveal the efficacy of standard MSBR for Latino parents of children with DD and underscore the potential benefits of disseminating this practice to traditionally underrepresented families.

A Mixed Methods Approach to Evaluate Partnerships and Implementation of the Massachusetts Prevention and Wellness Trust Fund

Background: Strong partnerships are critical to integrate evidence-based prevention interventions within clinical and community-based settings, offering multilevel and sustainable solutions to complex health issues. As part of Massachusetts' 2012 health reform, The Prevention and Wellness Trust Fund (PWTF) funded nine local partnerships throughout the state to address hypertension, pediatric asthma, falls among older adults, and tobacco use. The initiative was designed to improve health outcomes through prevention and disease management strategies and reduce healthcare costs.

Purpose: Describe the mixed-methods study design for investigating PWTF implementation.

Methods: The Consolidated Framework for Implementation Research guided the development of this evaluation. First, the study team conducted semi-structured qualitative interviews with leaders from each of nine partnerships to document partnership development and function, intervention adaptation and delivery, and the influence of contextual factors on implementation. The interview findings were used to develop a quantitative survey to assess the implementation experiences of 172 staff from clinical and community-based settings and a social network analysis to assess changes in the relationships among 72 PWTF partner organizations. The quantitative survey data on ratings of perceived implementation success were used to purposively select 24 staff for interviews to explore the most successful experiences of implementing evidence-based interventions for each of the four conditions.

Conclusions: This mixed-methods approach for evaluation of implementation of evidence-based prevention interventions by PWTF partnerships can help decision-makers set future priorities for implementing and assessing clinical-community partnerships focused on prevention.

RE-AIM in Clinical, Community, and Corporate Settings: Perspectives, Strategies, and Recommendations to Enhance Public Health Impact

The RE-AIM Framework is a planning and evaluation model that has been used in a variety of settings to address various programmatic, environmental, and policy innovations for improving population health. In addition to the broad application and diverse use of the framework, there are lessons learned and recommendations for the future use of the framework across clinical, community, and corporate settings. The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how the RE-AIM concepts and elements within each dimension can be applied by researchers and practitioners in diverse settings, among diverse populations and for diverse health topics.

The impact of inter-organizational alignment (IOA) on implementation outcomes: evaluating unique and shared organizational influences in education sector mental health

BACKGROUND

Integrated healthcare delivered by work groups in nontraditional service settings is increasingly common, yet contemporary implementation frameworks typically assume a single organization-or organizational unit-within which system-level processes influence service quality and implementation success. Recent implementation frameworks predict that inter-organizational alignment (i.e., similarity in values, characteristics, activities related to implementation across organizations) may facilitate the implementation of evidence-based practices (EBP), but few studies have evaluated this premise. This study's aims examine the impact of overlapping organizational contexts by evaluating the implementation contexts of externally employed mental health clinicians working in schools-the most common integrated service delivery setting for children and adolescents. Aim 1 is to estimate the effects of unique intra-organizational implementation contexts and combined inter-organizational alignment on implementation outcomes. Aim 2 is to examine the underlying mechanisms through which inter-organizational alignment facilitates or hinders EBP implementation.

METHODS/DESIGN

This study will conduct sequential, exploratory mixed-methods research to evaluate the intra- and inter-organizational implementation contexts of schools and the external community-based organizations that most often employ school-based mental health clinicians, as they relate to mental health EBP implementation. Aim 1 will involve quantitative surveys with school-based, externally-employed mental health clinicians, their supervisors, and proximal school-employed staff (total n = 120 participants) to estimate the effects of each organization's general and implementation-specific organizational factors (e.g., climate, leadership) on implementation outcomes (fidelity, acceptability, appropriateness) and assess the moderating role of the degree of clinician embeddedness in the school setting. Aim 2 will explore the mechanisms through which inter-organizational alignment influences implementation outcomes by presenting the results of Aim 1 surveys to school-based clinicians (n = 30) and conducting semi-structured qualitative interviews. Qualitative data will be evaluated using an integrative inductive and deductive approach.

DISCUSSION

The study aims are expected to identify intra- and inter-organizational constructs that are most instrumental to EBP implementation success in school-based integrated care settings and illuminate mechanisms that may account for the influence of inter-organizational alignment. In addition to improving school-based mental health, these findings will spur future implementation science that considers the relationships across organizations and optimize the capacity of implementation science to guide practice in increasingly complex systems of care.

WE'RE THE EYES AND THE EARS, BUT WE DON'T HAVE A VOICE: PERSPECTIVES OF HOSPICE AIDES

Hospice aides are integral to the delivery of home hospice care and play an essential role in the effective communication and collaboration of the hospice interdisciplinary team. Despite their critical role, little is known about hospice aides' perceptions of their role in providing hospice services. This qualitative study explores hospice aides' perspectives of their work, their relationships with patients, families and interdisciplinary team members, and their contributions to end-of-life care. Two focus groups of hospice aides (N = 13), and 16 hours of observation of in-home caregiving were conducted. Focus group data were inductively and iteratively coded, and three themes were identified: 1) "We're the eyes and ears", 2) "We're kept out of the loop", and 3) "We have no voice." Aides described themselves as front-line providers based on the close relationships they formed with patients and family members, yet felt under-recognized by members of the hospice team. Observational field notes were reexamined with themes to confirm convergence of qualitative and observational data. Better understanding of their role could enhance integration of aides into patient- and family-centered team interactions, leading to both improved patient and family outcomes and retention and support of this critical component of the hospice workforce.

Factors Associated with Clinician Participation in TF-CBT Post-workshop Training Components

For proficiency in an evidence-based treatment (EBT), mental health professionals (MHPs) need training activities extending beyond a one-time workshop. Using data from 178 MHPs participating in a statewide TF-CBT dissemination project, we used five variables assessed at the workshop, via multiple and logistic regression, to predict participation in three post-workshop training components. Perceived in-workshop learning and client-treatment mismatch were predictive of consultation call participation and case presentation respectively. Attitudes toward EBTs were predictive of trauma assessment utilization, although only with non-call participants removed from analysis. Productivity requirements and confidence in TF-CBT skills were not associated with participation in post-workshop activities.

Contracting and Procurement for Evidence-Based Interventions in Public-Sector Human Services: A Case Study

Sustainment of evidence-based interventions (EBIs) in human services depends on the inner context of community-based organizations (CBOs) that provide services and the outer context of their broader environment. Increasingly, public officials are experimenting with contracting models from for-profit industries to procure human services. In this case study, we conducted qualitative interviews with key government and CBO stakeholders to examine implementation of the Best Value-Performance Information Procurement System to contract for EBIs in a child welfare system. Findings suggest that stakeholder relationships may be compromised when procurement disregards local knowledge, communication, collaboration, and other factors supporting EBIs and public health initiatives.

Leading for the long haul: a mixed-method evaluation of the Sustainment Leadership Scale (SLS)

Background

Despite our progress in understanding the organizational context for implementation and specifically the role of leadership in implementation, its role in sustainment has received little attention. This paper took a mixed-method approach to examine leadership during the sustainment phase of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Utilizing the Implementation Leadership Scale as a foundation, we sought to develop a short, practical measure of sustainment leadership that can be used for both applied and research purposes.

Methods

Data for this study were collected as a part of a larger mixed-method study of evidence-based intervention, SafeCare®, sustainment. Quantitative data were collected from 157 providers using web-based surveys. Confirmatory factor analysis was used to examine the factor structure of the Sustainment Leadership Scale (SLS). Qualitative data were collected from 95 providers who participated in one of 15 focus groups. A framework approach guided qualitative data analysis. Mixed-method integration was also utilized to examine convergence of quantitative and qualitative findings.

Results

Confirmatory factor analysis supported the a priori higher order factor structure of the SLS with subscales indicating a single higher order sustainment leadership factor. The SLS demonstrated excellent internal consistency reliability. Qualitative analyses offered support for the dimensions of sustainment leadership captured by the quantitative measure, in addition to uncovering a fifth possible factor, available leadership.

Conclusions

This study found qualitative and quantitative support for the pragmatic SLS measure. The SLS can be used for assessing leadership of first-level leaders to understand how staff perceive leadership during sustainment and to suggest areas where leaders could direct more attention in order to increase the likelihood that EBIs are institutionalized into the normal functioning of the organization.

The good coach: implementation and sustainment factors that affect coaching as evidence-based intervention fidelity support

PURPOSE

Evidence-based interventions (EBIs) for human services unfold within complicated social and organizational circumstances and are influenced by the attitudes and behaviors of diverse stakeholders situated within these environments. Coaching is commonly regarded as an effective strategy to support service providers in delivering EBIs and attaining high levels of fidelity over time. The purpose of this paper is to address a lacuna in research examining the factors influencing coaching, an important EBI support component.

METHODOLOGY

The authors use the Exploration, Preparation, Implementation, and Sustainment framework to consider inner- and outer-context factors that affect coaching over time. This case study of coaching draws from a larger qualitative data set from three iterative investigations of implementation and sustainment of a home visitation program, SafeCare®. SafeCare is an EBI designed to reduce child neglect.

FINDINGS

The authors elaborate on six major categories of findings derived from an iterative data coding and analysis process: perceptions of "good" and "bad" coaches by system sustainment status; coach as peer; in-house coaching capacity; intervention developer requirements vs other outer-context needs; outer- context support; and inner-context support.

PRACTICAL IMPLICATIONS

Coaching is considered a key component for effective implementation of EBIs in public-sector systems, yet is under-studied. Understanding inner- and outer-context factors illuminates the ways they affect the capacity of coaches to support service delivery.

ORIGINALITY

This paper demonstrates that coaching can accomplish more than provision of EBI fidelity support. Stakeholders characterized coaches as operating as boundary spanners who link inner and outer contexts to enable EBI implementation and sustainment.

Implementation outcome assessment instruments used in physical healthcare settings and their measurement properties: a systematic review protocol

INTRODUCTION

Over the past 10 years, research into methods that promote the uptake, implementation and sustainability of evidence-based interventions has gathered pace. However, implementation outcomes are defined in different ways and assessed by different measures; the extent to which these measures are valid and reliable is unknown. The aim of this systematic review is to identify and appraise studies that assess the measurement properties of quantitative implementation outcome instruments used in physical healthcare settings, to advance the use of precise and accurate measures.

METHODS AND ANALYSIS

The following databases will be searched from inception to March 2017: MEDLINE, EMBASE, PsycINFO, CINAHL and the Cochrane Library. Grey literature will be sought via HMIC, OpenGrey, ProQuest for theses and Web of Science Conference Proceedings Citation Index-Science. Reference lists of included studies and relevant reviews will be hand searched. Three search strings will be combined to identify eligible studies: (1) implementation literature, (2) implementation outcomes and (3) measurement properties. Screening of titles, abstracts and full papers will be assessed for eligibility by two reviewers independently and any discrepancies resolved via consensus with the wider team. The methodological quality of the studies will be assessed using the COnsensus-based Standards for the selection of health Measurement INstruments checklist. A set of bespoke criteria to determine the quality of the instruments will be used, and the relationship between instrument usability and quality will be explored.

ETHICS AND DISSEMINATION

Ethical approval is not necessary for systematic review protocols. Researchers and healthcare professionals can use the findings of this systematic review to guide the selection of implementation outcomes instruments, based on their psychometric quality, to assess the impact of their implementation efforts. The findings will also provide a useful guide for reviewers of papers and grants to determine the psychometric quality of the measures used in implementation research.

TRIAL REGISTRATION NUMBER

International Prospective Register of Systematic Reviews (PROSPERO): CRD42017065348.

Organizational theory for dissemination and implementation research

Background

Even under optimal internal organizational conditions, implementation can be undermined by changes in organizations’ external environments, such as fluctuations in funding, adjustments in contracting practices, new technology, new legislation, changes in clinical practice guidelines and recommendations, or other environmental shifts. Internal organizational conditions are increasingly reflected in implementation frameworks, but nuanced explanations of how organizations’ external environments influence implementation success are lacking in implementation research. Organizational theories offer implementation researchers a host of existing, highly relevant, and heretofore largely untapped explanations of the complex interaction between organizations and their environment. In this paper, we demonstrate the utility of organizational theories for implementation research.

Discussion

We applied four well-known organizational theories (institutional theory, transaction cost economics, contingency theories, and resource dependency theory) to published descriptions of efforts to implement SafeCare, an evidence-based practice for preventing child abuse and neglect. Transaction cost economics theory explained how frequent, uncertain processes for contracting for SafeCare may have generated inefficiencies and thus compromised implementation among private child welfare organizations. Institutional theory explained how child welfare systems may have been motivated to implement SafeCare because doing so aligned with expectations of key stakeholders within child welfare systems’ professional communities. Contingency theories explained how efforts such as interagency collaborative teams promoted SafeCare implementation by facilitating adaptation to child welfare agencies’ internal and external contexts. Resource dependency theory (RDT) explained how interagency relationships, supported by contracts, memoranda of understanding, and negotiations, facilitated SafeCare implementation by balancing autonomy and dependence on funding agencies and SafeCare developers.

Summary

In addition to the retrospective application of organizational theories demonstrated above, we advocate for the proactive use of organizational theories to design implementation research. For example, implementation strategies should be selected to minimize transaction costs, promote and maintain congruence between organizations’ dynamic internal and external contexts over time, and simultaneously attend to organizations’ financial needs while preserving their autonomy. We describe implications of applying organizational theory in implementation research for implementation strategies, the evaluation of implementation efforts, measurement, research design, theory, and practice. We also offer guidance to implementation researchers for applying organizational theory.

Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study

BACKGROUND

Examining the role of modifiable barriers and facilitators is a necessary step toward developing effective implementation strategies. This study examines whether both general (organizational culture, organizational climate, and transformational leadership) and strategic (implementation climate and implementation leadership) organizational-level factors predict therapist-level determinants of implementation (knowledge of and attitudes toward evidence-based practices).

METHODS

Within the context of a system-wide effort to increase the use of evidence-based practices (EBPs) and recovery-oriented care, we conducted an observational, cross-sectional study of 19 child-serving agencies in the City of Philadelphia, including 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators. Organizational variables included characteristics such as EBP initiative participation, program size, and proportion of independent contractor therapists; general factors such as organizational culture and climate (Organizational Social Context Measurement System) and transformational leadership (Multifactor Leadership Questionnaire); and strategic factors such as implementation climate (Implementation Climate Scale) and implementation leadership (Implementation Leadership Scale). Therapist-level variables included demographics, attitudes toward EBPs (Evidence-Based Practice Attitudes Scale), and knowledge of EBPs (Knowledge of Evidence-Based Services Questionnaire). We used linear mixed-effects regression models to estimate the associations between the predictor (organizational characteristics, general and strategic factors) and dependent (knowledge of and attitudes toward EBPs) variables.

RESULTS

Several variables were associated with therapists' knowledge of EBPs. Clinicians in organizations with more proficient cultures or higher levels of transformational leadership (idealized influence) had greater knowledge of EBPs; conversely, clinicians in organizations with more resistant cultures, more functional organizational climates, and implementation climates characterized by higher levels of financial reward for EBPs had less knowledge of EBPs. A number of organizational factors were associated with the therapists' attitudes toward EBPs. For example, more engaged organizational cultures, implementation climates characterized by higher levels of educational support, and more proactive implementation leadership were all associated with more positive attitudes toward EBPs.

CONCLUSIONS

This study provides evidence for the importance of both general and strategic organizational determinants as predictors of knowledge of and attitudes toward EBPs. The findings highlight the need for longitudinal and mixed-methods studies that examine the influence of organizational factors on implementation.

Inconsistent staffing and its impact on service delivery in ASD early-intervention

BACKGROUND

Inconsistent staffing (high turnover, casual workforce) is problematic in organisations, with the potential to impact both staff and services provided. Research has primarily focused on the impacts of inconsistent staffing in child welfare and community services with little evidence surrounding their ability to impact the outcomes for children with Autism Spectrum Disorders attending early-intervention services.

AIM

The aim of this study was to explore staff views regarding the impact of staff turnover on the delivery of group based early intervention for children with ASD.

METHODS AND PROCEDURES

We conducted five focus groups involving 29 professional (e.g. teachers, behaviour therapists), para-professional (e.g. child care workers), and managerial staff to explore their views. Audio recordings were transcribed verbatim for use in thematic analysis.

OUTCOMES AND RESULTS

Two central themes, comprising five categories emerged to encompass participants' views. "Impacts on Staff" accounted for the challenges existing staff felt when working with new and untrained staff in the centre. Participants also expressed concerns for the "Impacts on Service Delivery" that resulted from dynamic staffing, affecting fidelity of interventions and the program itself.

CONCLUSIONS AND IMPLICATIONS

The findings suggest that lacking a consistent staffing structure is problematic when attempting to provide high quality early intervention services to children with ASD and suggest that future research should investigate the extent of inconsistent staffing, impacts of inconsistent staffing on providing intervention, and develop a range of tools to help measure these effects.

Is It Time to Share Qualitative Research Data?

Policies by the National Institutes of Health and the National Science Foundation, as well as scandals surrounding failures to reproduce the findings of key studies in psychology, have generated increased calls for sharing research data. Most of these discussions have focused on quantitative, rather than qualitative, research data. This paper examines scientific, ethical, and policy issues surrounding sharing qualitative research data. We consider advantages of sharing data, including enabling verification of findings, promoting new research in an economical manner, supporting research education, and fostering public trust in science. We then examine standard procedures for archiving and sharing data, such as anonymizing data and establishing data use agreements. Finally, we engage a series of concerns with sharing qualitative research data such as the importance of relationships in interpreting data, the risk of re-identifying participants, issues surrounding consent and data ownership, and the burden of data documentation and depositing on researchers. For each concern, we identify options that enable data sharing or describe conditions under which select data might be withheld from a data repository. We conclude by suggesting that the default assumption should be that qualitative data will be shared unless concerns exist that cannot be addressed through standard data depositing practices such as anonymizing data or through data use agreements.

The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study

If evidence-based interventions (EBIs) are not sustained, investments are wasted and public health impact is limited. Leadership has been suggested as a key determinant of implementation and sustainment; however, little empirical work has examined this factor. This mixed-methods study framed using the Exploration, Preparation, Implementation, Sustainment (EPIS) conceptual framework examines leadership in both the outer service system context and inner organizational context in eleven system-wide implementations of the same EBI across two U.S. states and 87 counties. Quantitative data at the outer context (i.e., system) and inner context (i.e., team) levels demonstrated that leadership predicted future sustainment and differentiated between sites with full, partial, or no sustainment. In the outer context positive sustainment leadership was characterized as establishing a project's mission and vision, early and continued planning for sustainment, realistic project plans, and having alternative strategies for project survival. Inner context frontline transformational leadership predicted sustainment while passive-avoidant leadership predicted non-sustainment. Qualitative results found that sustainment was associated with outer context leadership characterized by engagement in ongoing supportive EBI championing, marketing to stakeholders; persevering in these activities; taking action to institutionalize the EBI with funding, contracting, and system improvement plans; and fostering ongoing collaboration between stakeholders at state and county, and community stakeholder levels. For frontline leadership the most important activities included championing the EBI and providing practical support for service providers. There was both convergence and expansion that identified unique contributions of the quantitative and qualitative methods. Greater attention to leadership in both the outer system and inner organizational contexts is warranted to enhance EBI implementation and sustainment.

Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol

BACKGROUND

Reducing youth suicide in the United States (U.S.) is a national public health priority, and lesbian, gay, bisexual, transgender, and queer or questioning (LGBTQ) youth are at elevated risk. The Centers for Disease Control and Prevention (CDC) endorses six evidence-based (EB) strategies that center on meeting the needs of LGBTQ youth in schools; however, fewer than 6 % of U.S. schools implement all of them. The proposed intervention model, "RLAS" (Implementing School Nursing Strategies to Reduce LGBTQ Adolescent Suicide), builds on the Exploration, Preparation, Implementation, and Sustainment (EPIS) conceptual framework and the Dynamic Adaptation Process (DAP) to implement EB strategies in U.S. high schools. The DAP accounts for the multilevel context of school settings and uses Implementation Resource Teams (IRTs) to facilitate appropriate expertise, advise on acceptable adaptations, and provide data feedback to make schools implementation ready and prepared to sustain changes.

METHODS/DESIGN

Mixed methods will be used to examine individual, school, and community factors influencing both implementation process and youth outcomes. A cluster randomized controlled trial will assess whether LGBTQ students and their peers in RLAS intervention schools (n = 20) report reductions in suicidality, depression, substance use, bullying, and truancy related to safety concerns compared to those in usual care schools (n = 20). Implementation progress and fidelity for each EB strategy in RLAS intervention schools will be examined using a modified version of the Stages of Implementation Completion checklist. During the implementation and sustainment phases, annual focus groups will be conducted with the 20 IRTs to document their experiences identifying and advancing adaptation supports to facilitate use of EB strategies and their perceptions of the DAP.

DISCUSSION

The DAP represents a data-informed, collaborative, multiple stakeholder approach to progress from exploration to sustainment and obtain fidelity during the implementation of EB strategies in school settings. This study is designed to address the real-world implications of enabling the use of EB strategies by school nurses with the goal of decreasing suicide and youth risk behaviors among LGBTQ youth. Through its participatory processes to refine and sustain EB strategies in high schools, the RLAS represents a novel contribution to implementation science.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02875535.

A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools

Background

The significant lifelong impairments associated with autism spectrum disorder (ASD), combined with the growing number of children diagnosed with ASD, have created urgency in improving school-based quality of care. Although many interventions have shown efficacy in university-based research, few have been effectively implemented and sustained in schools, the primary setting in which children with ASD receive services. Individual- and organizational-level factors have been shown to predict the implementation of evidence-based interventions (EBIs) for the prevention and treatment of other mental disorders in schools, and may be potential targets for implementation strategies in the successful use of autism EBIs in schools. The purpose of this study is to examine the individual- and organizational-level factors associated with the implementation of EBIs for children with ASD in public schools.

Methods

We will apply the Domitrovich and colleagues (2008) framework that examines the influence of contextual factors (i.e., individual- and organizational-level factors) on intervention implementation in schools. We utilize mixed methods to quantitatively test whether the factors identified in the Domitrovich and colleagues (2008) framework are associated with the implementation of autism EBIs, and use qualitative methods to provide a more comprehensive understanding of the factors associated with successful implementation and sustainment of these interventions with the goal of tailoring implementation strategies.

Discussion

The results of this study will provide an in-depth understanding of individual- and organizational-level factors that influence the successful implementation of EBIs for children with ASD in public schools. These data will inform potential implementation targets and tailoring of strategies that will help schools overcome barriers to implementation and ultimately improve the services and outcomes for children with ASD.

Theater-Based Community Engagement Project for Veterans Recovering From Substance Use Disorders

In this study, we examine the feasibility and acceptability of a 6-wk, interdisciplinary, occupation-based theater project for facilitating community engagement and substance use disorder (SUD) recovery in veterans. All data were collected at baseline, postintervention, and 6-wk and 6-mo follow-up intervals. Of the invited veterans, 24% consented to participate (n = 14), and 50% were retained (n = 7). Average attendance was 91%. Considerable improvements in social and occupational participation were noted at postintervention and at 6-wk follow-up but were not retained at 6 mo. No important change in self-efficacy was noted. Of the participants, 86% remained abstinent for 6 wk following the intervention. Theater provides a feasible and acceptable resource for potentially facilitating SUD recovery. Larger controlled effectiveness studies of theater are needed to examine whether robust and notable recovery outcomes in people with SUDs can be linked to participation in theater.

Feasibility of improving child behavioral health using task-shifting to implement the 4Rs and 2Ss program for strengthening families in child welfare

Background

Children whose families are involved with child welfare services manifest disproportionately high levels of behavioral difficulties, which could be addressed in community-based organizations providing services to prevent out-of-home placement. Unfortunately, few evidence-based practices have been successfully implemented in child welfare settings, especially those originally delivered by mental health providers. Given that such settings typically employ caseworkers who lack prior mental health training, this is a significant barrier to implementation. Consequently, the overall aim of the current study is to test the feasibility of shifting a mental health intervention from specialized services to community-based organizations. It uses task-shifting and the Practical, Robust, Implementation, and Sustainability model (PRISM) to implement an evidence-based intervention to reduce child behavior difficulties, originally provided by mental health practitioners, so that it can be delivered by caseworkers providing placement prevention services to child welfare-involved families. Task-shifting involves (1) modifying the intervention for provision by non-mental health providers, (2) training non-mental health providers in the modified intervention, and (3) establishing regular supervision and monitoring by mental health specialists.

Methods/design

This study uses the 4Rs and 2Ss Program for Strengthening Families, a multiple family group service delivery model to reduce child behavior difficulties, as the example intervention. This intervention has had prior success with child welfare-involved families. The proposed study objectives are (1) to tailor the content, training, and supervision of the intervention for delivery by caseworkers serving child welfare-involved families and (2) to assess the feasibility and acceptability of the modified intervention. Mixed quantitative and qualitative methods will assess feasibility and acceptability from key stakeholders (caseworkers, supervisors, administrators, caregivers). In phase I, a collaborative advisory board will be convened (1) to modify the intervention to be delivered by caseworkers in placement prevention service settings and (2) to develop training and supervision protocols for caseworkers. In phase 2, the modified intervention will be pilot-tested for delivery by n = 4 caseworkers to n = 20 families receiving placement preventive services (where children manifest behavior problems). Mixed quantitative/qualitative methods will be used to assess feasibility and acceptability.

Discussion

This protocol will be of particular interest to agency administrators, program managers, and researchers interested in developing and testing cross-setting implementation guidelines for similar evidence-based practices.

Evaluation of Boys Town In-Home Family Services with Families Referred by Child Welfare

This study evaluated the Boys Town In-Home Family Services model with families referred by child welfare for issues related to maltreatment. Participants were 135 parents (mean age = 32.15 years, SD = 9.13) who completed intake and discharge assessments. The target child ranged in age from one month to 17 years (M = 4.54, SD = 4.38). We had a high-risk sample (e.g., 57% and 41% of parents reported being victims of physical and sexual abuse, respectively; 24% of parents reported attempting suicide in their lifetimes). The intervention was implemented with a degree of fidelity consistent with model standards. Reduced levels of perceived stressors were found for several domains of functioning with the largest effects observed for family safety, parental capabilities, and environmental factors. Results serve as an important step in building the evidence base of a widely disseminated intervention.

Learning from bottom-up dissemination: Importing an evidence-based trauma intervention for infants and young children to Israel

This article describes a pilot study of a "bottom up" dissemination process of a new evidence based intervention for treating early childhood trauma. Clinicians applied to learn Child-Parent Psychotherapy (CPP), imported to Israel from the U.S. A focus group of six graduates of a CPP training program responded to questions concerning their experiences learning and using CPP. All 39 CPP graduates from two cohorts also completed a cross sectional survey related to their use of CPP. Within the focus group, the openness of the workplace and the intervention's characteristics were considered major factors impacting CPP use; the training program was perceived to promote CPP implementation, and lack of supervision and secondary traumatic stress were the major inhibiting factors. Using CPP-informed therapy, as opposed to CPP with fidelity, was perceived to be one of the main outcomes of the training. Survey results showed that 53% of graduates were using CPP in over three cases, and almost all intended to use CPP within the next year. Ninety-five percent were using CPP principles in their therapeutic work. The implications of importing a new evidence based intervention to a foreign country that utilizes a different dissemination system within a different professional culture are discussed.

The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis

BACKGROUND

Nurses are increasingly expected to engage in evidence-informed decision-making (EIDM) to improve client and system outcomes. Despite an improved awareness about EIDM, there is a lack of use of research evidence and understanding about the effectiveness of interventions to promote EIDM. This project aimed to discover if knowledge translation (KT) interventions directed to nurses in tertiary care are effective for improving EIDM knowledge, skills, behaviours, and, as a result, client outcomes. It also sought to understand contextual factors that affect the impact of such interventions.

METHODS

A systematic review funded by the Canadian Institutes of Health Research (PROSPERO registration: CRD42013003319) was conducted. Included studies examined the implementation of any KT intervention involving nurses in tertiary care to promote EIDM knowledge, skills, behaviours, and client outcomes or studies that examined contextual factors. Study designs included systematic reviews, quantitative, qualitative, and mixed method studies. The search included electronic databases and manual searching of published and unpublished literature to November 2012; key databases included MEDLINE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Excerpta Medica (EMBASE). Two reviewers independently performed study selection, risk of bias assessment, and data extraction. Studies with quantitative data determined to be clinically homogeneous were synthesized using meta-analytic methods. Studies with quantitative data not appropriate for meta-analysis were synthesized narratively by outcome. Studies with qualitative data were synthesized by theme.

RESULTS

Of the 44,648 citations screened, 30 citations met the inclusion criteria (18 quantitative, 10 qualitative, and 2 mixed methods studies). The quality of studies with quantitative data ranged from very low to high, and quality criteria was generally met for studies with qualitative data. No studies evaluated the impact on knowledge and skills; they primarily investigated the effectiveness of multifaceted KT strategies for promoting EIDM behaviours and improving client outcomes. Almost all studies included an educational component. A meta-analysis of two studies determined that a multifaceted intervention (educational meetings and use of a mentor) did not increase engagement in a range of EIDM behaviours [mean difference 2.7, 95 % CI (-1.7 to 7.1), I (2) = 0 %]. Among the remaining studies, no definitive conclusions could be made about the relative effectiveness of the KT interventions due to variation of interventions and outcomes, as well as study limitations. Findings from studies with qualitative data identified the organizational, individual, and interpersonal factors, as well as characteristics of the innovation, that influence the success of implementation.

CONCLUSIONS

KT interventions are being implemented and evaluated on nurses' behaviour and client outcomes. This systematic review may inform the selection of KT interventions and outcomes among nurses in tertiary care and decisions about further research.

Identifying priorities in knowledge translation from the perspective of trainees: results from an online survey

Background

The need to identify priorities to help shape future directions for research and practice increases as the knowledge translation (KT) field advances. Since many KT trainees are developing their research programs, understanding their concerns and KT research and practice priorities is important to supporting the development and advancement of KT as a field. Our purpose was to identify research and practice priorities in the KT field from the perspectives of KT researcher/practitioner trainees.

Findings

Survey response rate was 62 % (44/71). Participants were mostly Canadian graduate students, post-doctoral fellows, residents, and learners from various disciplines; the majority was from Ontario (44 %) and Quebec (20 %). Seven percent (5/71) were from other countries including USA, UK, and Switzerland. Seven main KT priority themes were identified: determining the effectiveness of KT strategies, technology use, increased key stakeholder involvement, context, theory, expand ways of inquiry, and sustainability.

Conclusions

Overall, the priorities identified by the trainees correspond with KT literature and with KT experts’ views. The trainees appeared to push the boundaries of current KT literature with respect to creative use of communication technologies research.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-015-0282-5) contains supplementary material, which is available to authorized users.

From a "perfect storm" to "smooth sailing": policymaker perspectives on implementation and sustainment of an evidence-based practice in two states

Policymakers shape implementation and sustainment of evidence-based practices (EBPs), whether they are developing or responding to legislation and policies or negotiating public sector resource constraints. As part of a large mixed-method study, we conducted qualitative interviews with 24 policymakers involved in delivery of the same EBP in two U.S. states. We analyzed transcripts via open and focused coding techniques to identify the commonality, diversity, and complexity of implementation challenges; approaches to overcoming those challenges; and the importance of system-level contextual factors in ensuring successful implementation. Key findings centered on building support and leadership for EBPs; funding and contractual strategies; partnering with stakeholders; tackling challenges via proactive planning and problem solving; and the political, legal, and systemic pressures affecting EBP longevity. The policymaker perspectives offer guidance on nurturing system and organizational practice environments to achieve positive outcomes and for optimally addressing macro-level influences that bear upon the instantiation of EBPs in public sector child welfare systems.

Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation

BACKGROUND

Leadership is important in the implementation of innovation in business, health, and allied health care settings. Yet there is a need for empirically validated organizational interventions for coordinated leadership and organizational development strategies to facilitate effective evidence-based practice (EBP) implementation. This paper describes the initial feasibility, acceptability, and perceived utility of the Leadership and Organizational Change for Implementation (LOCI) intervention. A transdisciplinary team of investigators and community stakeholders worked together to develop and test a leadership and organizational strategy to promote effective leadership for implementing EBPs.

METHODS

Participants were 12 mental health service team leaders and their staff (n = 100) from three different agencies that provide mental health services to children and families in California, USA. Supervisors were randomly assigned to the 6-month LOCI intervention or to a two-session leadership webinar control condition provided by a well-known leadership training organization. We utilized mixed methods with quantitative surveys and qualitative data collected via surveys and a focus group with LOCI trainees.

RESULTS

Quantitative and qualitative analyses support the LOCI training and organizational strategy intervention in regard to feasibility, acceptability, and perceived utility, as well as impact on leader and supervisee-rated outcomes.

CONCLUSIONS

The LOCI leadership and organizational change for implementation intervention is a feasible and acceptable strategy that has utility to improve staff-rated leadership for EBP implementation. Further studies are needed to conduct rigorous tests of the proximal and distal impacts of LOCI on leader behaviors, implementation leadership, organizational context, and implementation outcomes. The results of this study suggest that LOCI may be a viable strategy to support organizations in preparing for the implementation and sustainment of EBP.

Does a 3-week critical research appraisal course affect how students perceive their appraisal skills and the relevance of research for clinical practice? A repeated cross-sectional survey

BACKGROUND

Research utilisation is essential in developing evidence-based practices; although many students may be generally able to adopt such skills, there are reports of barriers related to critical appraisal skills.

OBJECTIVES

To explore how students perceive the relevance of research to future clinical practice and patients, and to what extent they read research (including reading pattern). Additionally, the objective was to explore whether a three-week intensive course in critical appraisal of research could affect these variables.

DESIGN

A cross-sectional survey design, with a pre- and post-test.

SETTINGS

One large university college in Southeastern Norway.

PARTICIPANTS

196 multidisciplinary healthcare students at baseline and 147 after three weeks.

METHODS

A purposely-designed 21 item questionnaire was used to quantify students' attitudes towards using research and critical thinking. The questionnaire was based on themes emerging from prior focus group interviews with 10 nursing and social educator students as well as from the existing literature.

RESULTS

At baseline, 6.1% and 7.1% of respondents perceived the research to be of little or very little importance for their future work and patients, respectively. Furthermore, 83.2% reported that they seldom or very seldom read scientific papers. At baseline, 40 different patterns of reading a scientific paper were identified. Additionally, 7.1% of respondents reported to read the introduction, methods and conclusion in combination. Significantly improved scores were found after completing the three-week course related to a) relevance of research for future work (p<0.01), b) self-perceived skills in critical appraisal (p<0.001), c) ability to find scientific papers (p≤0.01), and d) relevance of research for patients and users (p≤0.001).

CONCLUSIONS

Teaching students' practical critical appraisal skills improved their view of the relevance of research for patients, future work as well as their own critical appraisal skills. Prospective studies are warranted to explore the effects of such teaching modules in the long-term.

Qualitative and mixed methods research in dissemination and implementation science: introduction to the special issue

This special issue provides examples of how qualitative and mixed methods research approaches can be used in dissemination and implementation science. In this introductory article, we provide a brief rationale for why and how qualitative and mixed methods approaches can be useful in moving the field forward. Specifically, we provide a brief primer on common qualitative methods, including a review of guidelines provided by the National Institutes of Health. Next, we introduce the six articles in the issue. The first of the articles by Palinkas represents a more thorough and authoritative discussion related to qualitative methods, using the other five articles in the issue (and other published works) as examples. The remaining five articles are empirical and/or descriptive articles of recently completed or ongoing qualitative or mixed methods studies related to dissemination and implementation of evidence-based practices for children and adolescents.

Mixed-method study of a conceptual model of evidence-based intervention sustainment across multiple public-sector service settings

BACKGROUND

This study examines sustainment of an EBI implemented in 11 United States service systems across two states, and delivered in 87 counties. The aims are to 1) determine the impact of state and county policies and contracting on EBI provision and sustainment; 2) investigate the role of public, private, and academic relationships and collaboration in long-term EBI sustainment; 3) assess organizational and provider factors that affect EBI reach/penetration, fidelity, and organizational sustainment climate; and 4) integrate findings through a collaborative process involving the investigative team, consultants, and system and community-based organization (CBO) stakeholders in order to further develop and refine a conceptual model of sustainment to guide future research and provide a resource for service systems to prepare for sustainment as the ultimate goal of the implementation process.

METHODS

A mixed-method prospective and retrospective design will be used. Semi-structured individual and group interviews will be used to collect information regarding influences on EBI sustainment including policies, attitudes, and practices; organizational factors and external policies affecting model implementation; involvement of or collaboration with other stakeholders; and outer- and inner-contextual supports that facilitate ongoing EBI sustainment. Document review (e.g., legislation, executive orders, regulations, monitoring data, annual reports, agendas and meeting minutes) will be used to examine the roles of state, county, and local policies in EBI sustainment. Quantitative measures will be collected via administrative data and web surveys to assess EBI reach/penetration, staff turnover, EBI model fidelity, organizational culture and climate, work attitudes, implementation leadership, sustainment climate, attitudes toward EBIs, program sustainment, and level of institutionalization. Hierarchical linear modeling will be used for quantitative analyses. Qualitative analyses will be tailored to each of the qualitative methods (e.g., document review, interviews). Qualitative and quantitative approaches will be integrated through an inclusive process that values stakeholder perspectives.

DISCUSSION

The study of sustainment is critical to capitalizing on and benefiting from the time and fiscal investments in EBI implementation. Sustainment is also critical to realizing broad public health impact of EBI implementation. The present study takes a comprehensive mixed-method approach to understanding sustainment and refining a conceptual model of sustainment.

Ethnographic process evaluation in primary care: explaining the complexity of implementation

Background

The recent growth of implementation research in care delivery systems has led to a renewed interest in methodological approaches that deliver not only intervention outcome data but also deep understanding of the complex dynamics underlying the implementation process. We suggest that an ethnographic approach to process evaluation, when informed by and integrated with quantitative data, can provide this nuanced insight into intervention outcomes. The specific methods used in such ethnographic process evaluations are rarely presented in detail; our objective is to stimulate a conversation around the successes and challenges of specific data collection methods in health care settings. We use the example of a translational clinical trial among 11 community clinics in Portland, OR that are implementing an evidence-based, health-information technology (HIT)-based intervention focused on patients with diabetes.

Discussion

Our ethnographic process evaluation employed weekly diaries by clinic-based study employees, observation, informal and formal interviews, document review, surveys, and group discussions to identify barriers and facilitators to implementation success, provide insight into the quantitative study outcomes, and uncover lessons potentially transferable to other implementation projects. These methods captured the depth and breadth of factors contributing to intervention uptake, while minimizing disruption to clinic work and supporting mid-stream shifts in implementation strategies. A major challenge is the amount of dedicated researcher time required.

Summary

The deep understanding of the ‘how’ and ‘why’ behind intervention outcomes that can be gained through an ethnographic approach improves the credibility and transferability of study findings. We encourage others to share their own experiences with ethnography in implementation evaluation and health services research, and to consider adapting the methods and tools described here for their own research.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-014-0607-0) contains supplementary material, which is available to authorized users.