Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services

A complementary marriage of perspectives: understanding organizational social context using mixed methods

Background

Organizational factors impact the delivery of mental health services in community settings. Mixed-methods analytic approaches have been recommended, though little research within implementation science has explicitly compared inductive and deductive perspectives to understand their relative value in understanding the same constructs. The purpose of our study is to use two different paradigmatic approaches to deepen our understanding of organizational social context. We accomplish this by using a mixed-methods approach in an investigation of organizational social context in community mental health clinics.

Methods

Nineteen agencies, representing 23 sites, participated. Enrolled participants included 130 therapists, 36 supervisors, and 22 executive administrators. Quantitative data was obtained via the Organizational Social Context (OSC) measure. Qualitative data, comprised of direct observation with spot sampling generated from agency visits, was coded using content analysis and grounded theory. The present study examined elements of organizational social context that would have been missed if only quantitative data had been obtained and utilized mixed methods to investigate if stratifying observations based on quantitative ratings from the OSC resulted in the emergence of differential themes.

Results

Four of the six OSC constructs were commonly observed in field observations (i.e., proficiency, rigidity, functionality, stress), while the remaining two constructs were not frequently observed (i.e., resistance, engagement). Constructs emerged related to organizational social context that may have been missed if only quantitative measurement was employed, including those around the physical environment, commentary about evidence-based practice initiatives, leadership, cultural diversity, distrust, and affect. Stratifying agencies by “best,” “average,” and “worst” organizational social context impacted interpretation for three constructs (affect, stress, and leadership).

Conclusions

Results support the additive value of integrating inductive and deductive perspectives in implementation science research. This synthesis of approaches facilitated a more comprehensive understanding and interpretation of the findings than would have been possible if either methodology had been employed in isolation.

Avaliando a fidelidade de intervenções psicossociais: uma revisão sistemática da literatura

Objetivo Intervenções psicossociais têm tido impacto positivo na vida das pessoas com transtorno mental grave, porém tais intervenções devem ser implementadas fielmente ao protocolo. A fidelidade refere-se à medida que uma intervenção adere ao modelo original e sua avaliação é essencial para que os desfechos possam ser creditados à intervenção. O objetivo deste estudo foi realizar uma revisão sistemática da literatura mundial das intervenções psicossociais destinadas a pacientes com transtornos mentais que possuem um instrumento ou método de avaliação de fidelidade ao modelo original. Métodos Pesquisas bibliográficas sistemáticas foram realizadas para encontrar estudos relevantes ao tema nas seguintes bases de dados: Embase, Medline, Scopus e SciELO. Foram incluídos estudos de intervenções psicossociais, realizados na comunidade, dirigidos a pacientes diagnosticados com transtornos mentais. Intervenções de comparação poderiam incluir tanto o tratamento-padrão como uma intervenção de comparação ativa. A qualidade dos estudos foi avaliada de forma independente por dois revisores, utilizando critérios adaptados de instrumentos validados. Resultados Trinta estudos preencheram os critérios de inclusão. Os estudos mostraram a eficácia da avaliação da fidelidade em diferenciar diferentes modelos de tratamento, sua validade preditiva para os desfechos e a confiabilidade dos instrumentos utilizados, bem como os fatores facilitadores e os obstáculos para a obtenção de alta fidelidade nas intervenções avaliadas. Conclusão Além de documentar a adesão ao modelo original, a fidelidade fornece informações relativas à população-alvo e aos desfechos esperados, o que contribui para que seja alcançada excelência no processo de implementação das intervenções psicossociais.

Importance of mixed methods in pragmatic trials and dissemination and implementation research

With increased attention to the importance of translating research to clinical practice and policy, recent years have seen a proliferation of particular types of research, including pragmatic trials and dissemination and implementation research. Such research seeks to understand how and why interventions function in real-world settings, as opposed to highly controlled settings involving conditions not likely to be repeated outside the research study. Because understanding the context in which interventions are implemented is imperative for effective pragmatic trials and dissemination and implementation research, the use of mixed methods is critical to understanding trial results and the success or failure of implementation efforts. This article discusses a number of dimensions of mixed methods research, utilizing at least one qualitative method and at least one quantitative method, that may be helpful when designing projects or preparing grant proposals. Although the strengths and emphases of qualitative and quantitative approaches differ substantially, methods may be combined in a variety of ways to achieve a deeper level of understanding than can be achieved by one method alone. However, researchers must understand when and how to integrate the data as well as the appropriate order, priority, and purpose of each method. The ability to demonstrate an understanding of the rationale for and benefits of mixed methods research is increasingly important in today's competitive funding environment, and many funding agencies now expect applicants to include mixed methods in proposals. The increasing demand for mixed methods research necessitates broader methodological training and deepened collaboration between medical, clinical, and social scientists. Although a number of challenges to conducting and disseminating mixed methods research remain, the potential for insight generated by such work is substantial.

A mixed methods multiple case study of implementation as usual in children's social service organizations: study protocol

BACKGROUND

Improving quality in children's mental health and social service settings will require implementation strategies capable of moving effective treatments and other innovations (e.g., assessment tools) into routine care. It is likely that efforts to identify, develop, and refine implementation strategies will be more successful if they are informed by relevant stakeholders and are responsive to the strengths and limitations of the contexts and implementation processes identified in usual care settings. This study will describe: the types of implementation strategies used; how organizational leaders make decisions about what to implement and how to approach the implementation process; organizational stakeholders' perceptions of different implementation strategies; and the potential influence of organizational culture and climate on implementation strategy selection, implementation decision-making, and stakeholders' perceptions of implementation strategies.

METHODS/DESIGN

This study is a mixed methods multiple case study of seven children's social service organizations in one Midwestern city in the United States that compose the control group of a larger randomized controlled trial. Qualitative data will include semi-structured interviews with organizational leaders (e.g., CEOs/directors, clinical directors, program managers) and a review of documents (e.g., implementation and quality improvement plans, program manuals, etc.) that will shed light on implementation decision-making and specific implementation strategies that are used to implement new programs and practices. Additionally, focus groups with clinicians will explore their perceptions of a range of implementation strategies. This qualitative work will inform the development of a Web-based survey that will assess the perceived effectiveness, relative importance, acceptability, feasibility, and appropriateness of implementation strategies from the perspective of both clinicians and organizational leaders. Finally, the Organizational Social Context measure will be used to assess organizational culture and climate. Qualitative, quantitative, and mixed methods data will be analyzed and interpreted at the case level as well as across cases in order to highlight meaningful similarities, differences, and site-specific experiences.

DISCUSSION

This study is designed to inform efforts to develop more effective implementation strategies by fully describing the implementation experiences of a sample of community-based organizations that provide mental health services to youth in one Midwestern city.

Implementation of an efficacious intervention for high risk women in Mexico: protocol for a multi-site randomized trial with a parallel study of organizational factors

BACKGROUND

Studies of implementation of efficacious human immunodeficiency virus (HIV) prevention interventions are rare, especially in resource-poor settings, but important, because they have the potential to increase the impact of interventions by improving uptake and sustainability. Few studies have focused on provider and organizational factors that may influence uptake and fidelity to core intervention components. Using a hybrid design, we will study the implementation of an efficacious intervention to reduce sexually transmitted infections (STIs) among female sex workers (FSWs) in 12 cities across Mexico. Our protocol will test a 'train-the-trainer' implementation model for transporting the Mujer Segura (Healthy Woman) intervention into community-based organizations (CBOs).

METHODS

We have partnered with Mexican Foundation for Family Planning (Mexfam), a non-governmental organization that has CBOs throughout Mexico. At each CBO, trained ethnographers will survey CBO staff on characteristics of their organization and on their attitudes toward their CBO and toward the implementation of evidence-based interventions (EBIs). Then, after CBO staff recruit a sample of 80 eligible FSWs and deliver a standard-care, didactic intervention to 40 women randomly selected from that pool, a Mexfam staff person will be trained in the Mujer Segura intervention and will then train other counselors to deliver Mujer Segura to the 40 remaining participating FSWs. FSW participants will receive a baseline behavioral assessment and be tested for HIV and STIs (syphilis, gonorrhea, and chlamydia); they will be reassessed at six months post-intervention to measure for possible intervention effects. At the same time, both qualitative and quantitative data will be collected on the implementation process, including measures of counselors' fidelity to the intervention model. After data collection at each CBO is complete, the relative efficacy of the Mujer Segura intervention will be analyzed, and across CBOs, correlations will be examined between individual and organizational provider characteristics and intervention efficacy.

DISCUSSION

This cooperative, bi-national research study will provide critical insights into barriers and facilitating factors associated with implementing interventions in CBOs using the 'train the trainer' model. Our work builds on similar scale-up strategies that have been effective in the United States. This study has the potential to increase our knowledge of the generalizability of such strategies across health issues, national contexts, and organizational contexts.

TRIAL REGISTRATION

NCT01465607.

A concept mapping approach to guide and understand dissemination and implementation

Many efforts to implement evidence-based programs do not reach their full potential or fail due to the variety of challenges inherent in dissemination and implementation. This article describes the use of concept mapping-a mixed method strategy-to study implementation of behavioral health innovations and evidence-based practice (EBP). The application of concept mapping to implementation research represents a practical and concise way to identify and quantify factors affecting implementation, develop conceptual models of implementation, target areas to address as part of implementation readiness and active implementation, and foster communication among stakeholders. Concept mapping is described and a case example is provided to illustrate its use in an implementation study. Implications for the use of concept mapping methods in both research and applied settings towards the dissemination and implementation of behavioral health services are discussed.

The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study

ABSTBACKGROUND: Evidence-based practices have not been routinely adopted in community mental health organizations despite the support of scientific evidence and in some cases even legislative or regulatory action. We examined the association of clinician attitudes toward evidence-based practice with organizational culture, climate, and other characteristics in a nationally representative sample of mental health organizations in the United States.

METHODS

In-person, group-administered surveys were conducted with a sample of 1,112 mental health service providers in a nationwide sample of 100 mental health service institutions in 26 states in the United States. The study examines these associations with a two-level Hierarchical Linear Modeling (HLM) analysis of responses to the Evidence-Based Practice Attitude Scale (EBPAS) at the individual clinician level as a function of the Organizational Social Context (OSC) measure at the organizational level, controlling for other organization and clinician characteristics.

RESULTS

We found that more proficient organizational cultures and more engaged and less stressful organizational climates were associated with positive clinician attitudes toward adopting evidence-based practice.

CONCLUSIONS

The findings suggest that organizational intervention strategies for improving the organizational social context of mental health services may contribute to the success of evidence-based practice dissemination and implementation efforts by influencing clinician attitudes.

Dynamic adaptation process to implement an evidence-based child maltreatment intervention

Background

Adaptations are often made to evidence-based practices (EBPs) by systems, organizations, and/or service providers in the implementation process. The degree to which core elements of an EBP can be maintained while allowing for local adaptation is unclear. In addition, adaptations may also be needed at the system, policy, or organizational levels to facilitate EBP implementation and sustainment. This paper describes a study of the feasibility and acceptability of an implementation approach, the Dynamic Adaptation Process (DAP), designed to allow for EBP adaptation and system and organizational adaptations in a planned and considered, rather than ad hoc, way. The DAP involves identifying core elements and adaptable characteristics of an EBP, then supporting implementation with specific training on allowable adaptations to the model, fidelity monitoring and support, and identifying the need for and solutions to system and organizational adaptations. In addition, this study addresses a secondary concern, that of improving EBP model fidelity assessment and feedback in real-world settings.

Methods

This project examines the feasibility, acceptability, and utility of the DAP; tests the degree to which fidelity can be maintained using the DAP compared to implementation as usual (IAU); and examines the feasibility of using automated phone or internet-enabled, computer-based technology to assess intervention fidelity and client satisfaction. The study design incorporates mixed methods in order to describe processes and factors associated with variations in both how the DAP itself is implemented and how the DAP impacts fidelity, drift, and adaptation. The DAP model is to be examined by assigning six regions in California (USA) to either the DAP (n = 3) or IAU (n = 3) to implement an EBP to prevent child neglect.

Discussion

The DAP represents a data-informed, collaborative, multiple stakeholder approach to maintain intervention fidelity during the implementation of EBPs in the field by providing support for intervention, system, and organizational adaptation and intervention fidelity to meet local needs. This study is designed to address the real-world implications of EBP implementation in public sector service systems and is relevant for national, state, and local service systems and organizations.

Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation

OBJECTIVE

Leadership is important in practice change, yet there are few studies addressing this issue in mental health and social services. This study examined the differential roles of transformational (i.e., charismatic) leadership and leader-member exchange (i.e., the relationship between a supervisor and their direct service providers) on team innovation climate (i.e., openness to new innovations) and provider attitudes toward adopting evidence-based practice (EBP) during a statewide evidence-based practice implementation (EBPI) of an intervention to reduce child neglect.

METHOD

Participants were 140 case-managers in 30 teams providing home-based services to families in a statewide child-welfare system. Teams were assigned by region to EBPI or services as usual (SAU) conditions. Multiple group path analysis was used to examine associations of transformational leadership and leader-member exchange with innovation climate and attitudes toward adoption and use of EBP.

RESULTS

Transformational leadership predicted higher innovation climate during implementation, whereas leader-member exchange predicted higher innovation climate during SAU. Innovation climate was, in turn, associated with more positive attitudes toward EBP for the EBPI group.

CONCLUSIONS

Strategies designed to enhance supervisor transformational leadership have the potential to facilitate implementation efforts by promoting a strong climate for EBPI and positive provider attitudes toward adoption and use of EBP.