Concordance Between Administrator and Clinician Ratings of Organizational Culture and Climate

An evaluation of implementation climate in inpatient maternity care: a cross-sectional survey study

OBJECTIVE

Researchers in obstetrics and gynecology are continuously generating new evidence to inform clinical care delivery. Yet, much of this newly emerging evidence fails to be rapidly and effectively integrated into routine clinical practice. Implementation climate refers to clinicians' perceptions of to what degree organizations support and reward use of an evidence-based practice (EBP) and is an important construct in the science of implementation in healthcare. Little is known about implementation climate for EBPs in maternity care. Thus, we aimed to (a) determine the reliability of the Implementation Climate Scale (ICS) for use in inpatient maternity care, (b) describe implementation climate in inpatient maternity care overall, and (c) compare individual perceptions of implementation climate between physician and nursing clinicians on these units.

STUDY DESIGN

We performed a cross-sectional survey of clinicians in inpatient maternity units across 2 urban, academic hospitals in the northeastern United States in 2020. Clinicians completed the 18-question validated ICS [scored 0-4]. Scale reliability by role was evaluated using Cronbach's α. Subscale and total scores were described overall and compared by physician versus nursing role using independent t-tests, as well as linear regression to control for confounders.

RESULTS

111 clinicians completed the survey (physicians = 65; nursing = 46). Physicians were less likely to identify as female (75.4% vs. 100.0%, p < .001), but were of similar age and years of experience as nursing clinicians. Reliability of the ICS was excellent, with Cronbach's α of 0.91 and 0.86 among physicians and nursing clinicians, respectively. Scores were notably low for implementation climate in maternity care overall and for all subscales. ICS total scores were also higher among physicians as compared to nurses (2.18(±0.56) vs. 1.92(±0.50), p = .02), which remained significant in multivariable modeling (p = .02). Unadjusted subscale scores were higher among physicians in Recognition for EBP (2.68(±0.89) vs. 2.30(±0.86), p = .03) and Selection for EBP (2.24(±0.93) vs. 1.62(±1.04), p = .002). After adjustment for potential confounders, subscale scores for Focus on EBP (p = .04) and Selection for EBP (p = .002) were all higher among physicians.

CONCLUSIONS

This study supports the ICS as a reliable scale for measuring implementation climate in the inpatient maternity care setting. Notably lower implementation climate scores across subscales and roles compared to other settings may underlie the vast evidence to practice gap in obstetrics. In order to successfully implement practices that reduce maternal morbidity, we may need to focus on building educational support and rewarding EBP utilization on labor and delivery units, with an emphasis on nursing clinicians.

Therapists' Perceived Competence in Delivering Trauma-Focused Cognitive Behavioral Therapy During Statewide Learning Collaboratives

The learning collaborative (LC), a multi-component training and implementation model, is one promising approach to address the need for increased availability of trauma-focused evidence-based practices. The current study used data from four cohorts of a statewide LC on Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) to 1) evaluate pre- to post-LC changes in therapists' perceived competence in delivering TF-CBT and 2) explore therapist and contextual factors related to therapists' perceived TF-CBT competence. Therapists (N = 237) completed pre- and post-LC measures of practice information, interprofessional collaboration, organizational climate, and TF-CBT knowledge, perceived competence, and use. Findings indicated therapists' perceived TF-CBT competence significantly increased, pre- to post-LC (d = 1.31), with greater use of trauma-focused practices at pre-training and more TF-CBT training cases completed predicting greater pre- to post-LC gains in perceived TF-CBT competence. These findings highlight the need to assist therapists in identifying and completing training cases to promote competence and implementation.

A Comparison of the Readiness of Youth Service Agencies to Implement a Technology-Based Toolkit to Support Treatment Delivery

Implementation initiatives and technology-based resources aim to address barriers to Evidence-Based Practice (EBP) use by creating generalizable techniques that can be used for a variety of youth-serving agencies. However, research has not carefully examined unique differences between agency types or individual programs in readiness to use such technologies and implementation strategies. The current study explored differences between community mental health clinics and child advocacy centers on organizational cultural factors (e.g., ability to change and commitment for change) to implement a novel technology-based toolkit to support delivery of Trauma Focused Cognitive Behavioral Therapy (TF-CBT). Results indicated that TF-CBT providers from child advocacy centers reported greater commitment to change and more support to use the technology-based system than those from community mental health centers. Findings suggest that implementation initiatives should address the needs of individual agencies and service settings and adaptations should be explored to best meet the needs of these settings.

Provider perspectives and reach of an evidence-based intervention in community services for toddlers

Best-practice recommendations for young children at high likelihood of autism include active involvement of caregivers in intervention. However, the use of evidence-based parent-mediated interventions in community practice remains limited. Preliminary evidence suggests that Project ImPACT for Toddlers demonstrates positive parent and child outcomes in community settings. Project ImPACT for Toddlers was adapted specifically for toddlers and teaches parents of young children strategies to build their child’s social, communication, and play skills in daily routines. This study reports implementation outcomes from the initial community rollout of Project ImPACT for Toddlers and examines the system-wide intervention reach, with the goal of informing continued community sustainment and scale-up. Participants include 38 community providers who participated in a Project ImPACT for Toddlers’ training study who completed an implementation survey and semi-structured interviews after approximately 3 months of community implementation. Participants perceived the training model as acceptable and appropriate, and identified several strengths of the approach. Interview themes also supported the feasibility, acceptability, and utility of the intervention in community settings. Quantitative findings complemented the thematic results from interviews. Intervention reach data indicate an increasing number of agencies delivering and families receiving Project ImPACT for Toddlers. Efforts to scale-up evidence-based interventions in early intervention should continue to build upon the model of the Bond, Regulate, Interact, Develop, Guide, and Engage Collaborative.

Lay abstract

Expert recommendations for toddlers who are likely to develop autism include caregivers being actively involved in the services children receive. However, many services available in the community may not follow these recommendations. Evidence suggests that an intervention named Project ImPACT for Toddlers demonstrates positive parent and child outcomes for families in the community. Project ImPACT for Toddlers was designed specifically for toddlers by a group of parents, clinicians, researchers, and funders. It teaches parents of young children strategies to support their child’s development in daily routines. This study reports the perspectives of early intervention providers who learned to use Project ImPACT for Toddlers on whether the intervention was a good fit for their practice and easy to use. The study also examines how many agencies are using Project ImPACT for Toddlers and how many families have received the intervention in the community. The goal of the study is to inform the continued use of Project ImPACT for Toddlers in the community and support offering the intervention in other regions. Participants include 38 community providers who participated in a training study of Project ImPACT for Toddlers and completed a survey and semi-structured interview after approximately 3 months of using Project ImPACT for Toddlers with families. Participants perceived the training model as acceptable and appropriate, and identified the group-based model of training, comprehensive materials, and agency support as strengths of the approach. Survey findings complemented the results from the interviews. Data indicate an increasing number of agencies and families accessing Project ImPACT for Toddlers. Efforts to expand evidence-based intervention in early intervention should continue to build upon the model used for Project ImPACT for Toddlers.

Nursing Home Senior Managers and Direct Care Staff: Are There Differences in Their Perceptions of Safety Climate?

OBJECTIVES

Improving nursing home safety is important to the quality of resident care. Increasing evidence points to the relationship between actual safety and a strong safety climate, i.e., staff agreement about safety norms. This national study focused on Veterans Health Administration nursing homes (Community Living Centers [CLCs]), assessing direct care staff and senior managers' agreement about safety norms.

METHODS

We recruited all 134 CLCs to participate in the previously validated CLC Employee Survey of Attitudes about Resident Safety. To assess whether safety climate domains (7) differed by management level and by direct care staff occupation, we estimated multilevel linear regression models with random effects clustered by CLCs, medical center, Department of Veterans Affairs 2017 integrated service network (n = 20), and region.

RESULTS

Of the 5288 individuals we e-mailed, 1397 (25.7%) completed surveys, with participation from 56 CLCs or 41.8% of 134 CLCs. In our analysis of 1316 nurses, nursing assistants, clinicians/specialists, and senior managers, senior managers rated co-worker interactions around safety (P < 0.0013) and overall safety in their CLC (P < 0.0001) more positively than did direct care staff. In contrast, on these same two domains, direct care groups had similar perceptions, though differing significantly in safety priorities, safety attitudes, and senior management commitment to safety.

CONCLUSIONS

In this national sample of nursing homes in one of the largest integrated U.S. healthcare systems, direct care staff generally perceived weaker safety processes than did senior managers, pointing to future targets for interventions to strengthen safety climate.

Sustainment of Trauma-Focused and Evidence-Based Practices Following Learning Collaborative Implementation

Given the need to develop and validate effective implementation models that lead to sustainable improvements, we prospectively examined changes in attitudes, behaviors, and perceived organizational support during and after statewide Community-Based Learning Collaboratives (CBLCs) promoting trauma-focused evidence-based practices (EBPs). Participants (N = 857; i.e., 492 clinicians, 218 brokers, and 139 senior leaders) from 10 CBLCs completed surveys pre- and post-CBLC; a subsample (n = 146) completed a follow-up survey approximately two years post-CBLC. Results indicated (a) medium, sustained increases in clinician-reported use of trauma-focused EBPs, (b) medium to large, sustained increases in perceived organizational support for trauma-focused EBPs, and (c) trivial to small, sustained increases in perceived organizational support for EBPs broadly. In contrast, clinician-reported overall attitudes towards EBPs decreased to a trivial degree pre- to post-CBLC, but then increased to a small, statistically significant degree from post-CBLC to follow-up. Notably, the degree of perceived improvements in organizational support for general and trauma-focused EBPs varied by professional role. Findings suggest the CBLC implementation strategies may both increase and sustain provider practices and organizational support towards EBPs, particularly those EBPs a CBLC explicitly targets.

Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form

The Knowledge of Evidence-Based Services Questionnaire (KEBSQ) is an objective measure of therapist knowledge of practices derived from the evidence base for the treatment of youth psychopathology. However, the length of this measure (i.e., 40 items) and respondent demands associated with each item makes it burdensome for researchers and clinicians. This study developed and validated a Short Form of the KEBSQ using Item Response Theory measurement models. The Short Form consists of 17 items and generates two separate scores: Correct Endorsements and Correct Rejections. The Short Form was found to correlate highly with and perform similarly to the Full Form, providing preliminary validity evidence.

Making Implementation Last: The Impact of Training Design on the Sustainability of an Evidence-Based Treatment in a Randomized Controlled Trial

Although advances have been made in facilitating the implementation of evidence-based treatments, little is known about the most effective way to sustain their use over time. The current study examined the sustainability of one evidence-based treatment, Parent-Child Interaction Therapy (PCIT), following a statewide implementation trial testing three training methods: Cascading Model, Learning Collaborative, and Distance Education. Participants included 100 clinicians and 50 administrators from 50 organizations across Pennsylvania. Clinicians and administrators reported on sustainability at 24-months, as measured by the number of clients receiving PCIT and the continued use of the PCIT protocol. Multi-level path analysis was utilized to examine the role of training on sustainability. Clinicians and administrators reported high levels of sustainability at 24-months. Clinicians in the Cascading Model reported greater average PCIT caseloads at 24-months, whereas clinicians in the Learning Collaborative reported greater full use of the PCIT protocol at 24-months. Attending consultation calls was associated with delivering PCIT to fewer families. Implications for the sustainable delivery of PCIT beyond the training year as well as for the broader field of implementation science are discussed.

Leadership profiles associated with the implementation of behavioral health evidence-based practices for autism spectrum disorder in schools

Implementation of evidence-based practice (EBP) for autism spectrum disorder (ASD) in the education system is a public health priority. Leadership is a critical driver of EBP implementation but little is known about the types of leadership behaviors exhibited by school leaders and how this influences the context of EBP implementation, particularly for students with ASD. The objectives of this study were to determine (1) the leadership profiles of principals involved in EBP implementation for students with ASD and (2) how these leadership profiles related to school characteristics and implementation climate. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework was used to guide the design and analysis of this study. Participants (n = 296) included principals, teachers, and classroom support staff. They provided demographic information and completed the Multifactor Leadership Questionnaire and Implementation Climate Scale. Using latent profile analysis, a three-pattern solution was identified: Disengaged (6% of sample), Undifferentiated (23% of sample), and Optimal (71% of sample). Principals in schools with higher proportions of students with an individualized education program were more likely to be classified as Undifferentiated than Optimal. The Optimal group was associated with more positive implementation climate than the Undifferentiated or Disengaged groups. Findings suggest that leadership behaviors rated by principals and their staff involved in implementation of common autism EBPs can be meaningfully clustered into three discernible profiles that are shaped by organizational context and linked to strategic implementation climate. Our study findings have implications for leadership training and service delivery in schools by underscoring the critical nature of school leadership during implementation of EBPs for children with autism and the interplay between specific leadership behaviors and strategic implementation climate.

Agreement and Discrepancy Between Supervisor and Clinician Alliance: Associations with Clinicians' Perceptions of Psychological Climate and Emotional Exhaustion

Despite increasing interest in supervision as a leverage point for bolstering public mental health services, the potential influence of supervisory alliance on organizations and direct service providers remains understudied, particularly in the context of supporting evidence-based treatment (EBT) use. This study examined agreement and discrepancy between supervisor and clinician ratings of alliance associated with clinicians' perceptions of psychological climate and emotional exhaustion. Results indicated that discrepancies in alliance ratings were common and associated with clinicians' perceptions of psychological climate. These findings have important implications for collaboration among supervisors and clinicians within a community mental health organizational context and the provision of EBTs.

Comparing Agency Leader and Therapist Perspectives on Evidence-Based Practices: Associations with Individual and Organizational Factors in a Mental Health System-Driven Implementation Effort

Agency leaders and therapists are essential stakeholders in implementation of evidence-based practices (EBPs) within publicly-funded mental health services. Little is known about how these stakeholders differ in their perceptions of specific EBPs and which individual and organizational factors differentially influence these perceptions. Within the context of a system-driven implementation of multiple EBPs, survey data from 160 leaders and 720 therapists were examined to assess differences in perceptions of six EBPs. Findings indicated that leaders and therapists have unique perspectives and preferences regarding EBPs that are shaped by distinct sociodemographic and professional characteristics and aspects of organizational functioning.

Evaluation of the "Foundations in Knowledge Translation" training initiative: preparing end users to practice KT

BACKGROUND

Current knowledge translation (KT) training initiatives are primarily focused on preparing researchers to conduct KT research rather than on teaching KT practice to end users. Furthermore, training initiatives that focus on KT practice have not been rigorously evaluated and have focused on assessing short-term outcomes and participant satisfaction only. Thus, there is a need for longitudinal training evaluations that assess the sustainability of training outcomes and contextual factors that may influence outcomes.

METHODS

We evaluated the KT training initiative "Foundations in KT" using a mixed-methods longitudinal design. "Foundations in KT" provided training in KT practice and included three tailored in-person workshops, coaching, and an online platform for training materials and knowledge exchange. Two cohorts were included in the study (62 participants, including 46 "Foundations in KT" participants from 16 project teams and 16 decision-maker partners). Participants completed self-report questionnaires, focus groups, and interviews at baseline and at 6, 12, 18, and 24 months after the first workshop.

RESULTS

Participant-level outcomes include survey results which indicated that participants' self-efficacy in evidence-based practice (F(1,8.9) = 23.7, p = 0.001, n = 45), KT activities (F(1,23.9) = 43.2, p < 0.001, n = 45), and using evidence to inform practice increased over time (F(1,11.0) = 6.0, p = 0.03, n = 45). Interviews and focus groups illustrated that participants' understanding of and confidence in using KT increased from baseline to 24 months after the workshop. Interviews and focus groups suggested that the training initiative helped participants achieve their KT project objectives, plan their projects, and solve problems over time. Contextual factors include teams with high self-reported organizational capacity and commitment to implement at the start of their project had buy-in from upper management that resulted in secured funding and resources for their project. Training initiative outcomes include participants who applied the KT knowledge and skills they learned to other projects by sharing their knowledge informally with coworkers. Sustained spread of KT practice was observed with five teams at 24 months.

CONCLUSIONS

We completed a longitudinal evaluation of a KT training initiative. Positive participant outcomes were sustained until 24 months after the initial workshop. Given the emphasis on implementing evidence and the need to train implementers, these findings are promising for future KT training.