Implementation of evidence-based services for youth: assessing provider knowledge

Mental Health Provider Reach and Engagement in a Countywide Training Initiative

Many training initiatives are underway to increase implementation of evidence-based practice (EBPs) in mental healthcare. However, little is known about what types of trainings and supports yield the highest reach and engagement. Supported by a tax-funded, countywide initiative to improve access to quality care for youths, the current mixed methods study evaluates mental health (MH) provider reach, or registering for the training initiative, and engagement, or participation in training activities, for several EBP training and implementation supports. MH providers were offered free 1) formal EBP workshops, 2) a biweekly learning community, 3) individual case consultation, and 4) confidential online clinical feedback system. To register, interested providers (N = 698) completed a web-based assessment measuring clinical practice information, organizational implementation climate, and EBP knowledge, attitudes, and practices. Thirteen providers, selected via purposeful sampling stratified by level of participation, completed semi-structured qualitative interviews. While the training initiative achieved high reach (66% of county agencies had a provider register), far fewer providers engaged substantially in training. Quantitative results indicated that providers whose professional discipline was not psychology, had higher baseline EBP knowledge, more extensive use of common evidence-based strategies, and less extensive use of other therapy strategies, engaged in more training. Rapid qualitative analysis of interviews expanded upon these findings and illuminated provider, organizational, system, practical, and training activity-specific barriers and facilitators to engagement. Findings suggest the importance of identifying strategies for improving provider engagement in training activities beyond workshops. Implications for future research and training initiatives are discussed.

Principle-Guided Psychotherapy for Children and Adolescents (FIRST): study protocol for a randomized controlled effectiveness trial in outpatient clinics

BACKGROUND

Hundreds of youth psychotherapy randomized trials have generated scores of helpful empirically supported treatments (ESTs). However, the standardized structure of many ESTs and their focus on a single disorder or homogeneous cluster of problems may not be ideal for clinically referred youths who have comorbidity and whose treatment needs may shift from week to week. This concern has prompted development of flexible transdiagnostic, modular youth psychotherapies. One of these, designed for efficient training and implementation, is FIRST-a transdiagnostic intervention built on five empirically supported principles of change (i.e., feeling calm, increasing motivation, repairing thoughts, solving problems, and trying the opposite) and targeting common internalizing and externalizing youth mental health disorders and problems. FIRST has shown promise in improving youth mental health in three open trials. Now, in a more rigorous test, we seek to (1) conduct a randomized controlled trial comparing FIRST to usual care in real-world clinical practice settings; (2) examine a promising candidate mediator of change-regulation of negative emotions; and (3) explore variables that may influence clinicians' treatment implementation.

METHODS

This is an assessor-naïve randomized controlled effectiveness trial in youth outpatient community clinics in New England and Texas. Using double randomization, clinic-employed clinicians and treatment-referred youths (7-15 years old) are independently randomly allocated (1:1) to FIRST or usual care. We aim to recruit 212 youth participants, all referred through normal community pathways, with elevated symptoms of anxiety, depression, conduct problems, or post-traumatic stress. This study will test the effectiveness of FIRST compared to usual care on mental health outcomes, examine whether those outcomes are mediated by regulation of negative emotions, and explore clinician factors that may be associated with FIRST implementation and outcomes. Session recordings are coded to assess treatment fidelity.

DISCUSSION

This study will evaluate the effectiveness of FIRST in youth community mental health settings, relative to the care usually provided in those settings. If FIRST is found to be effective, it could offer an efficient and practical method to increase use of empirically supported treatment principles in real-world practice contexts.

TRIAL REGISTRATION

NIH Clinical Trials Registry, NCT04725721. Registered 27 January 2021, https://clinicaltrials.gov/ct2/show/study/NCT04725721.

Principle-Guided Psychotherapy for Children and Adolescents (FIRST): Study Protocol for a Randomized Controlled Effectiveness Trial in Outpatient Clinics

Background

Hundreds of youth psychotherapy randomized trials have generated scores of helpful empirically supported treatments (ESTs). However, the standardized structure of many ESTs and their focus on a single disorder or homogeneous cluster of problems may not be ideal for clinically referred youths who have comorbidity and whose treatment needs may shift from week to week. This concern has prompted development of flexible transdiagnostic, modular youth psychotherapies. One of these, designed for efficient training and implementation, is FIRST-a transdiagnostic intervention built on five empirically supported principles of change (i.e., feeling calm, increasing motivation, repairing thoughts, solving problems, and trying the opposite) and targeting common internalizing and externalizing youth mental health disorders and problems. FIRST has shown promise in improving youth mental health in three open trials. Now, in a more rigorous test, we seek to (1) conduct a randomized controlled trial comparing FIRST to usual care in real-world clinical practice settings; (2) examine a promising candidate mediator of change-regulation of negative emotions; and (3) explore variables that may influence clinicians' treatment implementation.

Methods

This is an assessor-naïve randomized controlled effectiveness trial in youth outpatient community clinics in New England and Texas. Using double randomization, clinic-employed clinicians and treatment-referred youths (7-15 years old) are independently randomly allocated (1:1) to FIRST or usual care. We aim to recruit 212 youth participants, all referred through normal community pathways, with elevated symptoms of anxiety, depression, conduct problems, or post-traumatic stress. This study will test the effectiveness of FIRST compared to usual care on mental health outcomes, examine whether those outcomes are mediated by regulation of negative emotions, and explore clinician factors that may be associated with FIRST implementation and outcomes. Session recordings are coded to assess treatment fidelity.

Discussion

This study will evaluate the effectiveness of FIRST in youth community mental health settings, relative to the care usually provided in those settings. If FIRST is found to be effective, it could offer an efficient and practical method to increase use of empirically supported treatment principles in real-world practice contexts.

Trial registration

NIH Clinical Trials Registry, NCT04725721. Registered 27 January 2021, https://clinicaltrials.gov/ct2/show/study/NCT04725721.

How low can you go? Examining the effects of brief online training and post-training consultation dose on implementation mechanisms and outcomes for measurement-based care

BACKGROUND

Initial training and ongoing post-training consultation (i.e., ongoing support following training, provided by an expert) are among the most common implementation strategies used to change clinician practice. However, extant research has not experimentally investigated the optimal dosages of consultation necessary to produce desired outcomes. Moreover, the degree to which training and consultation engage theoretical implementation mechanisms-such as provider knowledge, skills, and attitudes-is not well understood. This study examined the effects of a brief online training and varying dosages of post-training consultation (BOLT+PTC) on implementation mechanisms and outcomes for measurement-based care (MBC) practices delivered in the context of education sector mental health services.

METHODS

A national sample of 75 clinicians who provide mental health interventions to children and adolescents in schools were randomly assigned to BOLT+PTC or control (services as usual). Those in BOLT+PTC were further randomized to 2-, 4-, or 8-week consultation conditions. Self-reported MBC knowledge, skills, attitudes, and use (including standardized assessment, individualized assessment, and assessment-informed treatment modification) were collected for 32 weeks. Multilevel models were used to examine main effects of BOLT+PTC versus control on MBC use at the end of consultation and over time, as well as comparisons among PTC dosage conditions and theorized mechanisms (skills, attitudes, knowledge).

RESULTS

There was a significant linear effect of BOLT+PTC over time on standardized assessment use (b = .02, p < .01), and a significant quadratic effect of BOLT+PTC over time on individualized assessment use (b = .04, p < .001), but no significant effect on treatment modification. BOLT + any level of PTC resulted in higher MBC knowledge and larger growth in MBC skill over the intervention period as compared to control. PTC dosage levels were inconsistently predictive of outcomes, providing no clear evidence for added benefit of higher PTC dosage.

CONCLUSIONS

Online training and consultation in MBC had effects on standardized and individualized assessment use among clinicians as compared to services as usual with no consistent benefit detected for increased consultation dosage. Continued research investigating optimal dosages and mechanisms of these established implementation strategies is needed to ensure training and consultation resources are deployed efficiently to impact clinician practices.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05041517 . Retrospectively registered on 10 September 2021.

Assessing provider perceptions of training: Initial evaluation of the Acceptability, Feasibility, and Appropriateness Scale

There is a well-documented gap between research and practice in the treatment of mental health problems. One promising approach to bridging this gap is training community-based providers in evidence-based practices (EBPs). However, a paucity of valid, reliable measures to assess a range of outcomes of such trainings impedes our ability to evaluate and improve training toward this end. The current study examined the factor structure of the Acceptability, Feasibility, Appropriateness Scale (AFAS), a provider-report measure that assesses three perceptual implementation outcomes of trainings that may be leading indicators of training success (i.e., acceptability, feasibility, and appropriateness). Providers who attended half-day EBP trainings for common mental health problems reported on the acceptability, feasibility, and appropriateness of these trainings using the AFAS (N  =  298). Confirmatory factor analysis indicates good fit to the hypothesized three-factor structure (RMSEA  =  .058, CFI  =  .990, TLI  =  .987). Acceptability, feasibility, and appropriateness were three distinct but related constructs. Cronbach's alpha ranged from .86 to .91, indicating acceptable internal consistency for the three subscales. Acceptability and feasibility, but not appropriateness, scores varied between workshops, though variability across workshops was generally limited. This initial evaluation of the AFAS is in line with recent efforts to enhance psychometric reporting practices for implementation outcome measures and provides future directions for further development and refinement of the AFAS.

Plain Language Summary

Clinician training in evidence-based practices is often used to increase implementation of evidence-based practices in mental health service settings. However, one barrier to evaluating the success of clinician trainings is the lack of measures that reliably and accurately assess clinician training outcomes. This study was the initial evaluation of the Acceptability, Feasibility, Appropriateness Scale (AFAS), a measure that assesses the immediate outcomes of clinician trainings. This study found some evidence supporting the AFAS reliability and its three subscales. With additional item refinement and psychometric testing, the AFAS could become a useful measure of a training's immediate impact on providers.

The Cognitive Behavioral Therapy Competence Scale (CCS): initial development and validation

Background:

Nearly one-third of youth are affected by a mental health disorder, and the majority do not receive adequate care. To improve clinical outcomes among youth, efforts have been made to train providers in evidence-based mental health practices, such as cognitive behavioral therapy (CBT). Such efforts call for valid assessment measures that can inform and evaluate training activities.

Aims:

This study presents the development and validation of the CBT Competence Scale (CCS), a brief self-report measure to assess provider competence for CBT delivery.

Method:

Participants were 387 school mental health professionals (SMHPs) working with students in Michigan, USA. Initial items (n=59) were developed to evaluate competence in delivering common elements of CBT, with competence conceptualized as covering domains of knowledge, perception, and use of CBT techniques. CCS validation proceeded in three steps: using item response theory to select the most important items for assessing knowledge, evaluating the factor structure using exploratory and then confirmatory factor analyses, and examining reliability and validity of the resultant measure.

Results:

The validated CCS measure consists of four dimensions of CBT competence across 33 items: Non-behavioral skills, Behavioral skills, Perceptions, and Knowledge. The CCS demonstrated excellent internal consistency and good construct-based validity.

Conclusions:

The CCS holds promise as a valid, informative measure of CBT competence appropriate for the school setting, with potential for application in other environments such as mental health clinics.

Key learning aims

(1)

The effect of caregiver key opinion leaders on increasing caregiver demand for evidence-based practices to treat youth anxiety: protocol for a randomized control trial

Background

Research has identified cognitive behavioral therapy with exposures (CBT) as an effective treatment for youth anxiety. Despite implementation efforts, few anxious youth receive CBT. Direct-to-consumer marketing offers a different approach to address the unmet need for youth receiving effective treatments. Involving a local caregiver key opinion leader in direct-to-consumer initiatives may be an effective strategy to increase caregiver demand for CBT. Research indicates that key opinion leaders improve health promotion campaigns, but key opinion leaders have not been studied in the context of increasing caregiver demand for evidence-based treatments.

Method

Project CHAT (Caregivers Hearing about Anxiety Treatments) will test the role of key opinion leader participation in conducting outreach presentations to increase caregiver desire to seek CBT for their youth’s anxiety. Caregiver attendees (N = 180) will be cluster randomized by school to receive one of two different approaches for presentations on CBT for youth anxiety. Both approaches will involve community outreach presentations providing information on recognizing youth anxiety, strategies caregivers can use to decrease youth anxiety, and how to seek CBT for youth anxiety. The researcher-only condition will be co-facilitated by two researchers. In the key opinion leader condition, a caregiver key opinion leader from each local community will be involved in tailoring the content of the presentation to the context of the community, co-facilitating the presentation with a researcher, and endorsing strategies in the presentation that they have found to be helpful. In line with the theory of planned behavior, caregiver attendees will complete measures assessing their knowledge of, attitudes towards, perceived subjective norms about, and intention to seek CBT pre- and post-presentation; they will indicate whether they sought CBT for their youth at 3-month follow-up. Results will be analyzed using a mixed method approach to assess the effectiveness of a key opinion leader to increase caregiver demand for CBT.

Discussion

This study will be the first to examine the potential of key opinion leaders to increase caregiver demand for CBT. If proven effective, the use of key opinion leaders could serve as a scalable dissemination strategy to increase the reach of evidence-based treatments.

Trial registration

This trial was registered on clinicaltrials.gov (NCT04929262) on June 18, 2021. At the time of trial registration, pre/post-presentation data had been collected from 17 participants; thus, it was retrospectively registered.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00213-x.

Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form

The Knowledge of Evidence-Based Services Questionnaire (KEBSQ) is an objective measure of therapist knowledge of practices derived from the evidence base for the treatment of youth psychopathology. However, the length of this measure (i.e., 40 items) and respondent demands associated with each item makes it burdensome for researchers and clinicians. This study developed and validated a Short Form of the KEBSQ using Item Response Theory measurement models. The Short Form consists of 17 items and generates two separate scores: Correct Endorsements and Correct Rejections. The Short Form was found to correlate highly with and perform similarly to the Full Form, providing preliminary validity evidence.

Therapist and Youth Predictors of Specific Practices Derived from the Evidence-Base in Community Mental Health

Over the past several years, youth treatment research has moved toward understanding the dissemination and implementation of evidence-based practices (EBPs). As a result, studies have focused on identifying predictors that aid in successful adoption and sustainment of EBPs. Theories of behavior change posit that therapist knowledge and attitudes play a fundamental role in EBP adoption; however, studies have produced mixed findings, which may be an artifact of broad definitions of both EBP knowledge and EBP itself. The current study was an examination of 46 youth community therapists and the extent to which varying types of knowledge and attitudes as well as youth characteristics predicted specific practices derived from the evidence-base. Results suggested that specific EBP knowledge predicted specific practices, highlighting the need for more specificity when examining predictors of EBP use. Therapists' attitudes, demographic characteristics, and youth characteristics were also significant predictors of EBP use. Future research should consider examining discrete and specific practices to better understand and predict therapists' future behavior.

Factors Associated with Community-Partnered School Behavioral Health Clinicians' Adoption and Implementation of Evidence-Based Practices

Community-partnered school behavioral health (CP-SBH) is a model whereby schools partner with local community agencies to deliver services. This mixed-methods study examined 80 CP-SBH clinicians' adoption and implementation of evidence-based practice (EBP) approaches following mandated training. Forty-four clinicians were randomly assigned to one of two training conditions for a modular common elements approach to EBPs; 36 clinicians were preselected for training in a non-modular EBP. EBP knowledge improved for all training conditions at 8-month follow-up and practice element familiarity improved for modular approach training conditions, but the modular condition including ongoing consultation did not yield better results. Qualitative interviews (N = 17) highlighted multi-level influences of the CP-SBH service system and individual clinician characteristics on adoption and implementation.

Provider general attitudes versus specific perceptions of evidence-based psychotherapies for PTSD

This study examined the role of attitudes toward evidence-based psychotherapies (EBPs) in predicting use of prolonged exposure (PE) and cognitive processing therapy (CPT), two EBPs for posttraumatic stress disorder (PTSD) among PTSD treatment providers within the Department of Veterans Affairs. Providers' general attitudes toward EBPs, as well as their specific perceptions of PE and CPT, were examined as potential predictors of use. One hundred fifty-nine providers from 38 Department of Veterans Affairs' residential PTSD programs across the United States completed an online survey that included the predictors listed as well as self-reported use of PE on an individual basis and CPT on an individual and on a group basis. Although general attitudes toward EBPs were related to use of individually administered CPT, they were not related to use of PE or group-administered CPT. For each of the 3 treatments, however, specific positive perceptions were related to use. In examination of other training, skill, and delivery-related variables, general attitudes appear more in line with perceptions and delivery of CPT than PE. Perhaps this is because of the unique exposure component of PE. Assessing provider perceptions of specific EBPs may help providers in guiding their own practice as well as aid treatment developers, trainers, and administrators to more effectively tailor dissemination and implementation efforts. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

What Therapy Practices Do Providers Value in Youth Behavioral Health? A Measure Development Study

Existing measures of attitudes toward evidence-based practices (EBPs) assess attitudes toward manualized or research-based treatments. Providers of youth behavioral health (N = 282) completed the Valued Practices Inventory (VPI), a new measure of provider attitudes toward specific practices for youth that avoids mention of EBPs by listing specific therapies-some of which are drawn from EBPs (e.g., problem solving) and some of which are not included in EBPs (e.g., dream interpretation). Exploratory factor analysis revealed two factors: practices derived from the evidence base (PDEB) and alternative techniques (AT). The PDEB scale was significantly correlated with scales on the Evidence-Based Practice Attitude Scale-50 (Aarons et al. in Administration and Policy in Mental Health and Mental Health Services Research, 39(5): 331-340, 2012), whereas the AT scale was not. Attitudes toward PDEB and AT were also related to provider characteristics such as years of experience and work setting. The VPI offers a complementary approach to existing measures of attitudes because it avoids mention of EBPs, which may help prevent biases in responses.

Annual Research Review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field

BACKGROUND

Scientists have developed evidence-based interventions that improve the symptoms and functioning of youth with psychiatric disorders; however, these interventions are rarely used in community settings. Eliminating this research-to-practice gap is the purview of implementation science, the discipline devoted to the study of methods to promote the use of evidence-based practices in routine care.

METHODS

We review studies that have tested factors associated with implementation in child psychology and psychiatry, explore applications of social science theories to implementation, and conclude with recommendations to advance implementation science through the development and testing of novel, multilevel, causal theories.

RESULTS

During its brief history, implementation science in child psychology and psychiatry has documented the implementation gap in routine care, tested training approaches and found them to be insufficient for behavior change, explored the relationships between variables and implementation outcomes, and initiated randomized controlled trials to test implementation strategies. This research has identified targets related to implementation (e.g., clinician motivation, organizational culture) and demonstrated the feasibility of activating these targets through implementation strategies. However, the dominant methodological approach has been atheoretical and predictive, relying heavily on a set of variables from heuristic frameworks.

CONCLUSIONS

Optimizing the implementation of effective treatments in community care for youth with psychiatric disorders is a defining challenge of our time. This review proposes a new direction focused on developing and testing integrated causal theories. We recommend implementation scientists: (a) move from observational studies of implementation barriers and facilitators to trials that include causal theory; (b) identify a core set of implementation determinants; (c) conduct trials of implementation strategies with clear targets, mechanisms, and outcomes; (d) ensure that behaviors that are core to EBPs are clearly defined; and (e) agree upon standard measures. This agenda will help fulfill the promise of evidence-based practice for improving youth behavioral health.

Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework

Background

Effective implementation of evidence-based practices (EBPs) remains a significant challenge. Numerous existing models and frameworks identify key factors and processes to facilitate implementation. However, there is a need to better understand how individual models and frameworks are applied in research projects, how they can support the implementation process, and how they might advance implementation science. This systematic review examines and describes the research application of a widely used implementation framework, the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.

Methods

A systematic literature review was performed to identify and evaluate the use of the EPIS framework in implementation efforts. Citation searches in PubMed, Scopus, PsycINFO, ERIC, Web of Science, Social Sciences Index, and Google Scholar databases were undertaken. Data extraction included the objective, language, country, setting, sector, EBP, study design, methodology, level(s) of data collection, unit(s) of analysis, use of EPIS (i.e., purpose), implementation factors and processes, EPIS stages, implementation strategy, implementation outcomes, and overall depth of EPIS use (rated on a 1–5 scale).

Results

In total, 762 full-text articles were screened by four reviewers, resulting in inclusion of 67 articles, representing 49 unique research projects. All included projects were conducted in public sector settings. The majority of projects (73%) investigated the implementation of a specific EBP. The majority of projects (90%) examined inner context factors, 57% examined outer context factors, 37% examined innovation factors, and 31% bridging factors (i.e., factors that cross or link the outer system and inner organizational context). On average, projects measured EPIS factors across two of the EPIS phases (M = 2.02), with the most frequent phase being Implementation (73%). On average, the overall depth of EPIS inclusion was moderate (2.8 out of 5).

Conclusion

This systematic review enumerated multiple settings and ways the EPIS framework has been applied in implementation research projects, and summarized promising characteristics and strengths of the framework, illustrated with examples. Recommendations for future use include more precise operationalization of factors, increased depth and breadth of application, development of aligned measures, and broadening of user networks. Additional resources supporting the operationalization of EPIS are available.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0842-6) contains supplementary material, which is available to authorized users.

Overlapping and Non-overlapping Practices in Usual and Evidence-Based Care for Youth Anxiety

This study compared consistencies and discrepancies in usual care with practices derived from the evidence-base (PDEB) for youth anxiety in a public mental health system. Youth-level factors (diagnosis, functional impairment) as predictors of the discrepancies were also examined. Psychosocial and service data from 2485 youth with an anxiety disorder and/or receiving services for an anxiety treatment target were extracted. Therapists (N = 616) identified the treatment targets and practices youth received. Although many PDEB for youth anxiety were used by therapists in this sample, Exposure was only used in 15% of cases. Practices not consistent with youth anxiety treatment were also reported and included: PDEB for other conditions, practices common to all therapies, and practices that are not consistent with evidence-based care. Age and diagnosis predicted the delivery of PDEB for youth anxiety. Usual care incorporated many components of evidence-based care but was more diffuse and less focused on well-supported practices.

Group CBT for Externalizing Disorders in Urban Schools: Effect of Training Strategy on Treatment Fidelity and Child Outcomes

Public schools are an ideal setting for the delivery of mental health services to children. Unfortunately, services provided in schools, and more so in urban schools, have been found to lead to little or no significant clinical improvements. Studies with urban school children seldom report on the effects of clinician training on treatment fidelity and child outcomes. This study examines the differential effects of two levels of school-based counselor training: training workshop with basic consultation (C) vs. training workshop plus enhanced consultation (C+) on treatment fidelity and child outcomes. Fourteen school staff members (counselors) were randomly assigned to C or C+. Counselors implemented a group cognitive behavioral therapy protocol (Coping Power Program, CPP) for children with or at risk for externalizing behavior disorders. Independent coders coded each CPP session for content and process fidelity. Changes in outcomes from pre to post were assessed via a parent psychiatric interview and interviewer-rated severity of illness and global impairment. Counselors in C+ delivered CPP with significantly higher levels of content and process fidelity compared to counselors in C. Both C and C+ resulted in significant improvement in interviewer-rated impairment; the conditions did not differ from each other with regard to impairment. Groups did not differ with regard to pre- to- posttreatment changes in diagnostic severity level. School-based behavioral health staff in urban schools are able to implement interventions with fidelity and clinical effectiveness when provided with ongoing consultation. Enhanced consultation resulted in higher fidelity. Enhanced consultation did not result in better student outcomes compared to basic consultation. Implications for resource allocation decisions with staff training in EBP are discussed.

Knowledge, Attitudes, Social Desirability, and Organizational Characteristics in Youth Mental Health Services

This project investigated the extent to which knowledge of evidence-based practices (EBPs) and organizational characteristics predicted attitudes toward EBPs, while controlling for social desirability and organization membership. Participants were 167 public sector youth practitioners. Hierarchical multiple regression analyses were conducted to determine which factors significantly predicted EBP attitudes. Findings suggested that social desirability, organization membership, and various organizational characteristics predicted EBP attitudes. Results are discussed as they relate to the importance of including social desirability in future research and identifying different factors that influence EBP attitudes across various organizations.

The relationship between consumer, clinician, and organizational characteristics and use of evidence-based and non-evidence-based therapy strategies in a public mental health system

We investigated the relationship between consumer, clinician, and organizational factors and clinician use of therapy strategies within a system-wide effort to increase the use of cognitive-behavioral therapy. Data from 247 clinicians in 28 child-serving organizations were collected. Clinicians participating in evidence-based practice training initiatives were more likely to report using cognitive-behavioral therapy when they endorsed more clinical experience, being salaried clinicians, and more openness to evidence-based practice. Clinicians participating in evidence-based practice initiatives were more likely to use psychodynamic techniques when they had older clients, less knowledge about evidence-based practice, more divergent attitudes toward EBP, higher financial strain, and worked in larger organizations. In clinicians not participating in evidence-based training initiatives; depersonalization was associated with higher use of cognitive-behavioral; whereas clinicians with less knowledge of evidence-based practices were more likely to use psychodynamic techniques. This study suggests that clinician characteristics are important when implementing evidence-based practices; and that consumer, clinician, and organizational characteristics are important when de-implementing non evidence-based practices. This work posits potential characteristics at multiple levels to target with implementation and deimplementation strategies.

Predictors of clinician use of exposure therapy in community mental health settings

Exposure therapy is recognized as the key component of cognitive-behavioral treatment for anxiety. However, exposure is the least used evidence-based treatment in community mental health settings and is the most challenging technique for clinicians to adopt within the context of effectiveness and implementation trials. Little work has examined clinician and organizational characteristics that predict use of exposure, which is important for identifying implementation strategies that may increase its use. In a large sample of community health clinicians (N=335) across 31 clinical practice sites, this study characterized clinician and organizational predictors of exposure use and relaxation for anxiety. Mixed effects regression analyses indicated that both clinician attitudes and an organization's implementation climate may be important levers for interventions seeking to increase clinician exposure use. Greater clinician use of relaxation strategies was also associated with less exposure use. Results point to important implications for implementing cognitive-behavioral therapy for anxiety, including de-emphasizing relaxation and attending to organizational climate.

Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study

BACKGROUND

Examining the role of modifiable barriers and facilitators is a necessary step toward developing effective implementation strategies. This study examines whether both general (organizational culture, organizational climate, and transformational leadership) and strategic (implementation climate and implementation leadership) organizational-level factors predict therapist-level determinants of implementation (knowledge of and attitudes toward evidence-based practices).

METHODS

Within the context of a system-wide effort to increase the use of evidence-based practices (EBPs) and recovery-oriented care, we conducted an observational, cross-sectional study of 19 child-serving agencies in the City of Philadelphia, including 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators. Organizational variables included characteristics such as EBP initiative participation, program size, and proportion of independent contractor therapists; general factors such as organizational culture and climate (Organizational Social Context Measurement System) and transformational leadership (Multifactor Leadership Questionnaire); and strategic factors such as implementation climate (Implementation Climate Scale) and implementation leadership (Implementation Leadership Scale). Therapist-level variables included demographics, attitudes toward EBPs (Evidence-Based Practice Attitudes Scale), and knowledge of EBPs (Knowledge of Evidence-Based Services Questionnaire). We used linear mixed-effects regression models to estimate the associations between the predictor (organizational characteristics, general and strategic factors) and dependent (knowledge of and attitudes toward EBPs) variables.

RESULTS

Several variables were associated with therapists' knowledge of EBPs. Clinicians in organizations with more proficient cultures or higher levels of transformational leadership (idealized influence) had greater knowledge of EBPs; conversely, clinicians in organizations with more resistant cultures, more functional organizational climates, and implementation climates characterized by higher levels of financial reward for EBPs had less knowledge of EBPs. A number of organizational factors were associated with the therapists' attitudes toward EBPs. For example, more engaged organizational cultures, implementation climates characterized by higher levels of educational support, and more proactive implementation leadership were all associated with more positive attitudes toward EBPs.

CONCLUSIONS

This study provides evidence for the importance of both general and strategic organizational determinants as predictors of knowledge of and attitudes toward EBPs. The findings highlight the need for longitudinal and mixed-methods studies that examine the influence of organizational factors on implementation.

An Exploratory Factor Analysis of the Knowledge of Evidence-Based Services Questionnaire

Recent developments for disseminating and implementing evidence-based practices (EBPs) have emphasized the importance of increasing therapists' knowledge of such techniques. However, systematic efforts to measure knowledge in therapists serving youth have mostly relied on poorly researched study-specific measures. This study investigated the structure of EBP knowledge in a large sample of youth community therapists (N = 240) via a therapist report instrument emphasizing therapeutic commonalities across various types of treatment approaches. Findings supported a three-factor structure of knowledge with scores on factors varying by therapists' primary practice setting and education level. The relationships between therapist knowledge of and attitudes towards EBPs are also discussed.

Determinants and Functions of Standardized Assessment Use Among School Mental Health Clinicians: A Mixed Methods Evaluation

The current study evaluated why and how school mental health clinicians use standardized assessment tools in their work with youth and families. Quantitative and qualitative (focus group) data were collected prior to and following a training and consultation sequence as part of a trial program to assess school clinician's (n = 15) experiences administering standardized tools to youth on their caseloads (n = 191). Findings indicated that, although assessment use was initially somewhat low, clinicians used measures to conduct initial assessments with the bulk of their caseloads (average = 62.2%) during the implementation period. Clinicians also reported on factors influencing their use of assessments at the client, provider, and system levels; perceived functions of assessment; student responses to assessment use; and use of additional sources of clinically-relevant information (primarily educational data) for the purposes of assessment and progress monitoring. Implications for the contextual appropriateness of standardized assessment and training in assessment tools are discussed.

Lessons learned while building a trauma-informed public behavioral health system in the City of Philadelphia

Exposure to traumatic experiences among youth is a serious public health concern. A trauma-informed public behavioral health system that emphasizes core principles such as understanding trauma, promoting safety, supporting consumer autonomy, sharing power, and ensuring cultural competence, is needed to support traumatized youth and the providers who work with them. This article describes a case study of the creation and evaluation of a trauma-informed publicly funded behavioral health system for children and adolescents in the City of Philadelphia (the Philadelphia Alliance for Child Trauma Services; PACTS) using the Exploration, Preparation, Implementation, and Sustainment (EPIS) as a guiding framework. We describe our evaluation of this effort with an emphasis on implementation determinants and outcomes. Implementation determinants include inner context factors, specifically therapist knowledge and attitudes (N=114) towards evidence-based practices. Implementation outcomes include rate of PTSD diagnoses in agencies over time, number of youth receiving TF-CBT over time, and penetration (i.e., number of youth receiving TF-CBT divided by the number of youth screening positive on trauma screening). We describe lessons learned from our experiences building a trauma-informed public behavioral health system in the hopes that this case study can guide other similar efforts.

Independent Contractors in Public Mental Health Clinics: Implications for Use of Evidence-Based Practices

OBJECTIVES

Community mental health clinics are increasingly utilizing independent contractors to provide clinical services. At the same time, many organizations are participating in initiatives intended to increase implementation of evidence-based practices (EBPs). The primary aim of this study was to understand the associations of utilizing independent contractors with clinician knowledge and attitudes toward EBPs and organizational culture and climate. The study also sought to understand the potential impact of using independent contractors on mental health services delivery from the perspective of organizational leadership.

METHODS

Quantitative data were collected from 130 therapists in 23 organizations; qualitative data were collected from executive administrators in nine of the 16 organizations participating in EBP initiatives sponsored by the City of Philadelphia. Regression with random effects was used to estimate the associations between worker status (contractor or employee) and clinician attitudes toward EBPs, knowledge of EBPs, and organizational culture and climate. Qualitative inquiry was used to understand the impact of reliance on independent contractors on organizational participation in EBP initiatives.

RESULTS

Independent contractors endorsed less positive attitudes toward EBPs and scored lower on knowledge of EBPs. Interviews revealed four main themes: reasons for using independent contractors, general consequences of using independent contractors, specific impact of independent contractors on participation in EBP initiatives, and suggestions for alternatives.

CONCLUSIONS

A growing number of community mental health clinics rely on independent contractors. There may be consequences of this shift that deserve exploration.

Cross-Cultural Validation of the Modified Practice Attitudes Scale: Initial Factor Analysis and a New Factor Model

OBJECTIVE

The objective of this study was to create the Korean version of the Modified Practice Attitudes Scale (K-MPAS) to measure clinicians' attitudes toward evidence-based treatments (EBTs) in the Korean mental health system.

METHOD

Using 189 U.S. therapists and 283 members from the Korean mental health system, we examined the reliability and validity of the MPAS scores. We also conducted the first exploratory and confirmatory factor analysis on the MPAS and compared EBT attitudes across U.S. and Korean therapists.

RESULTS

Results revealed that the inclusion of both "reversed-worded" and "non-reversed-worded" items introduced significant method effects that compromised the integrity of the one-factor MPAS model. Problems with the one-factor structure were resolved by eliminating the "non-reversed-worded" items. Reliability and validity were adequate among both Korean and U.S. therapists. Korean therapists also reported significantly more negative attitudes toward EBTs on the MPAS than U.S. therapists.

CONCLUSIONS

The K-MPAS is the first questionnaire designed to measure Korean service providers' attitudes toward EBTs to help advance the dissemination of EBTs in Korea. The current study also demonstrated the negative impacts that can be introduced by incorporating oppositely worded items into a scale, particularly with respect to factor structure and detecting significant group differences.

Predictors of community therapists' use of therapy techniques in a large public mental health system

IMPORTANCE

Few studies have examined the effects of individual and organizational characteristics on the use of evidence-based practices in mental health care. Improved understanding of these factors could guide future implementation efforts to ensure effective adoption, implementation, and sustainment of evidence-based practices.

OBJECTIVE

To estimate the relative contribution of individual and organizational factors on therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques within the context of a large-scale effort to increase use of evidence-based practices in an urban public mental health system serving youth and families.

DESIGN, SETTING, AND PARTICIPANTS

In this observational, cross-sectional study of 23 organizations, data were collected from March 1 through July 25, 2013. We used purposive sampling to recruit the 29 largest child-serving agencies, which together serve approximately 80% of youth receiving publically funded mental health care. The final sample included 19 agencies with 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators.

MAIN OUTCOMES AND MEASURES

Therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques, as measured by the Therapist Procedures Checklist-Family Revised.

RESULTS

Individual factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 16%; family therapy techniques, 7%; and psychodynamic therapy techniques, 20%. Organizational factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 23%; family therapy techniques, 19%; and psychodynamic therapy techniques, 7%. Older therapists and therapists with more open attitudes were more likely to endorse use of cognitive-behavioral therapy techniques, as were those in organizations that had spent fewer years participating in evidence-based practice initiatives, had more resistant cultures, and had more functional climates. Women were more likely to endorse use of family therapy techniques, as were those in organizations employing more fee-for-service staff and with more stressful climates. Therapists with more divergent attitudes and less knowledge about evidence-based practices were more likely to use psychodynamic therapy techniques.

CONCLUSIONS AND RELEVANCE

This study suggests that individual and organizational factors are important in explaining therapist behavior and use of evidence-based practices, but the relative importance varies by therapeutic technique.

Perceived Characteristics of Intervention Scale

The Perceived Characteristics of Intervention Scale (PCIS), a 20-item assessment measure, was developed to assess health care providers’ views of interventions. Two hundred and fifteen Department of Veterans Affairs’ residential treatment providers from 38 programs across the United States completed an online survey that included the PCIS as well as self-reported use of two evidence-based treatments. The PCIS was anchored to ask about two evidence-based psychotherapies for posttraumatic stress disorder, prolonged exposure, and cognitive processing therapy. The PCIS is a reliable measure of perceived characteristics of interventions, with some preliminary support for its validity. Consideration of providers’ perceptions of particular evidence-based treatments may serve as an aid to improve their dissemination, implementation, and sustained use.

"If it's worth my time, i will make the time": school-based providers' decision-making about participating in an evidence-based psychotherapy consultation program

This study evaluated influences on school-based clinicians' decision-making surrounding participation in a modular psychotherapy training and consultation program lasting one academic year. Clinicians were recruited from three participation groups: those who never engaged, those who engaged and then discontinued, and those who participated fully. Qualitative interviews explored influences on initial and continued participation, as well as differences in decision-making by participation group, knowledge about evidence-based practices, and attitudes toward evidence-based practices. Eight major themes were identified: time, practice utility, intervention/training content, training process, attitudes toward training, social influences, commitment to training, and expectations. Some themes were discussed universally across all comparison groups, while others varied in frequency or content. Recommendations for increasing participation are presented, based on the findings.

A cluster randomized trial to evaluate external support for the implementation of positive behavioral interventions and supports by school personnel

Background

Urban schools lag behind non-urban schools in attending to the behavioral health needs of their students. This is especially evident with regard to the level of use of evidence-based interventions with school children. Increased used of evidence-based interventions in urban schools would contribute to reducing mental health services disparities in low-income communities. School-wide positive behavioral interventions and supports (SWPBIS) is a service delivery framework that can be used to deliver universal preventive interventions and evidence-based behavioral health treatments, such as group cognitive behavioral therapy. In this article, we describe our ongoing research on creating internal capacity for program implementation. We also examine the cost-effectiveness and resulting school climate when two different levels of external support are provided to personnel as they implement a two-tier SWPBIS program.

Methods/Design

The study follows six K – 8 schools in the School District of Philadelphia randomly assigned to consultation support or consultation-plus-coaching support. Participants are: approximately 48 leadership team members, 180 school staff and 3,900 students in Tier 1, and 12 counselors, and 306 child participants in Tier 2. Children who meet inclusion criteria for Tier 2 will participate in group cognitive behavioral therapy for externalizing or anxiety disorders. The study has three phases, baseline/training, implementation, and sustainability. We will measure implementation outcomes, service outcomes, child outcomes, and cost.

Discussion

Findings from this study will provide evidence as to the appropriateness of school-wide prevention and treatment service delivery models for addressing services disparities in schools. The effectiveness and cost-effectiveness analyses of the two levels of training and consultation should help urban school districts and policymakers with the planning and deployment of cost-effective strategies for the implementation of evidence-based interventions for some of the most common behavioral health problems in school children.

Trial registration

ClinicalTrials.gov identifier: NCT01941069

Using modular psychotherapy in school mental health: provider perspectives on intervention-setting fit

The "fit" or appropriateness of well-researched interventions within usual care contexts is among the most commonly cited, but infrequently researched, factors in the successful implementation of new practices. The current study was initiated to address two exploratory research questions: How do clinicians describe their current school mental health service delivery context? and How do clinicians describe the fit between modular psychotherapy and multiple levels of the school mental health service delivery context? Following a year-long training and consultation program in an evidence-based, modular approach to psychotherapy, semistructured qualitative interviews were conducted with 17 school-based mental health providers to evaluate their perspectives on the appropriateness of implementing the approach within a system of school-based health centers. Interviews were transcribed and coded for themes using conventional and directed content analysis. Findings identified key elements of the school mental health context including characteristics of the clinicians, their practices, the school context, and the service recipients. Specific evaluation of intervention-setting appropriateness elicited many comments about both practical and value-based (e.g., cultural considerations) aspects at the clinician and client levels but fewer comments at the school or organizational levels. Results suggest that a modular approach may fit well with the school mental health service context, especially along practical aspects of appropriateness. Future research focused on the development of methods for routinely assessing appropriateness at different stages of the implementation process is recommended.

Engineering youth service system infrastructure: Hawaii's continued efforts at large-scale implementation through knowledge management strategies

Hawaii's Child and Adolescent Mental Health Division provides a unique illustration of a youth public mental health system with a long and successful history of large-scale quality improvement initiatives. Many advances are linked to flexibly organizing and applying knowledge gained from the scientific literature and move beyond installing a limited number of brand-named treatment approaches that might be directly relevant only to a small handful of system youth. This article takes a knowledge-to-action perspective and outlines five knowledge management strategies currently under way in Hawaii. Each strategy represents one component of a larger coordinated effort at engineering a service system focused on delivering both brand-named treatment approaches and complimentary strategies informed by the evidence base. The five knowledge management examples are (a) a set of modular-based professional training activities for currently practicing therapists, (b) an outreach initiative for supporting youth evidence-based practices training at Hawaii's mental health-related professional programs, (c) an effort to increase consumer knowledge of and demand for youth evidence-based practices, (d) a practice and progress agency performance feedback system, and (e) a sampling of system-level research studies focused on understanding treatment as usual. We end by outlining a small set of lessons learned and a longer term vision for embedding these efforts into the system's infrastructure.

Policy to implementation: evidence-based practice in community mental health--study protocol

Background

Evidence-based treatments (EBTs) are not widely available in community mental health settings. In response to the call for implementation of evidence-based treatments in the United States, states and counties have mandated behavioral health reform through policies and other initiatives. Evaluations of the impact of these policies on implementation are rare. A systems transformation about to occur in Philadelphia, Pennsylvania, offers an important opportunity to prospectively study implementation in response to a policy mandate.

Methods/design

Using a prospective sequential mixed-methods design, with observations at multiple points in time, we will investigate the responses of staff from 30 community mental health clinics to a policy from the Department of Behavioral Health encouraging and incentivizing providers to implement evidence-based treatments to treat youth with mental health problems. Study participants will be 30 executive directors, 30 clinical directors, and 240 therapists. Data will be collected prior to the policy implementation, and then at two and four years following policy implementation. Quantitative data will include measures of intervention implementation and potential moderators of implementation (i.e., organizational- and leader-level variables) and will be collected from executive directors, clinical directors, and therapists. Measures include self-reported therapist fidelity to evidence-based treatment techniques as measured by the Therapist Procedures Checklist-Revised, organizational variables as measured by the Organizational Social Context Measurement System and the Implementation Climate Assessment, leader variables as measured by the Multifactor Leadership Questionnaire, attitudes towards EBTs as measured by the Evidence-Based Practice Attitude Scale, and knowledge of EBTs as measured by the Knowledge of Evidence- Based Services Questionnaire. Qualitative data will include semi-structured interviews with a subset of the sample to assess the implementation experience of high-, average-, and low-performing agencies. Mixed methods will be integrated through comparing and contrasting results from the two methods for each of the primary hypotheses in this study.

Discussion

Findings from the proposed research will inform both future policy mandates around implementation and the support required for the success of these policies, with the ultimate goal of improving the quality of treatment provided to youth in the public sector.

Effects of workshop trainings on evidence-based practice knowledge and attitudes among youth community mental health providers

Enhancing the public health impact of evidence-based practices (EBPs) in usual care settings is a key priority of the National Institute of Mental Health. Longitudinal data from community mental health providers (N = 268) participating in a series of state-sponsored workshops in modular approaches to EBPs for youth are presented. EBP workshop attendance for youth anxiety resulted in increased knowledge for EBPs for anxiety (and not other conditions) and EBP workshop attendance for youth disruptive behaviors resulted in increased knowledge for EBPs for disruptive behaviors (and not other conditions). Providers' tendencies toward incorrectly classifying non-EBP therapies as evidence-based increased over time, suggesting that providers over-generalize the EBP label as a result of attending these types of workshops. Regarding EBP attitudes, most measures of attitudes improved when providers attended a workshop. Additionally, an overly inclusive view of what constitutes an EBP at intake was related to significant decreases in openness to trying EBPs over time, whereas more positive attitudes at intake was related to achieving a more refined view of what constitutes an EBP over the course of attending trainings. Study limitations and implications for implementation of EBPs in usual care settings are discussed.

Therapist training in empirically supported treatments: a review of evaluation methods for short- and long-term outcomes

Therapist training efforts have been assessed using several outcomes. A model for evaluating therapist training in empirically supported treatments is presented, adapted from Kirkpatrick's (in: Craig and Bittel (eds.) Training and development handbook, 1967) training evaluation model. The adapted framework includes short-term outcomes, such as reactions to training and changes in attitude, knowledge, or skills, and longer-term outcomes, such as changes in therapist behavior in practice or client outcomes. Evaluation methods for these outcomes are reviewed, with information on their validity, reliability, and feasibility. An agenda for further research to improve therapist training evaluation is presented, with discussion of how evaluation can inform other areas of the field.