State implementation of evidence-based practice for youths, part II: recommendations for research and policy

References to Evidence-based Program Registry (EBPR) websites for behavioral health in U.S. state government statutes and regulations

BACKGROUND AND AIM

U.S. state governments have the responsibility to regulate and license behavioral healthcare interventions, such as for addiction and mental illness, with increasing emphasis on implementing evidence-based programs (EBPs). A serious obstacle to this is lack of clarity or agreement about what constitutes "evidence-based." The study's purpose was to determine the extent to which and in what contexts web-based Evidence-based Program Registries (EBPRs) are referenced in state government statutes and regulations ("mandates") concerning behavioral healthcare. Examples are: What Works Clearinghouse; National Register of Evidence-based Programs and Practices; Cochrane Database of Systematic Reviews.

METHODS

The study employed the Westlaw Legal Research Database to search for 30 known EBPR websites relevant to behavioral healthcare within the statutes and regulations of all 50 states.

RESULTS

There was low prevalence of EBPR references in state statutes and regulations pertaining to behavioral healthcare; 20 states had a total of 33 mandates that referenced an EBPR. These mandates usually do not rely on an EBPR as the sole acceptable source for classifying a program or practice as "evidence-based." Instead, EBPRs were named in conjunction with internal state or external sources of information about putative program effectiveness, which may be less valid than EBPRs, to determine what is "evidence-based."

CONCLUSION

Greater awareness of scientifically - based EBPRs and greater understanding of their advantages need to be fostered among state legislators and regulators charged with making policy to increase or improve the use of evidence-based programs and practices in behavioral healthcare in the U.S.

Academic-Policy Partnerships in Evidence-Based Practice Implementation and Policy Maker Use of Child Mental Health Research

OBJECTIVE

Strategies are needed to improve policy makers' evidence-informed decision making and the availability of evidence-based, state-supported services. This study examined whether academic-policy partnerships could promote these outcomes.

METHODS

Data from two national surveys of state mental health agency representatives were used to compare barriers to implementation of evidence-based practices (EBPs) and policy makers' use of child mental health research in states with strong academic-policy partnerships in workforce training or in program implementation/evaluation (IE) with barriers in states with no or limited partnerships in these areas.

RESULTS

Strong IE partnerships were associated with more confidence in research use and fewer issues with provider readiness and capacity but with more issues with EBP fidelity. Strong training partnerships were associated with fewer endorsements of lack of time as a barrier to research use.

CONCLUSIONS

Academic-policy partnerships had some benefit for states' research use and EBP implementation. Because these partnerships may reduce barriers, further research should explore characteristics of effective collaborations.

State behavioral health agency website references to evidence-based program registers

PURPOSE

Evidence-based program registers (EBPRs) are important tools for facilitating the use of evidence-based practices or programs (EBPs) by state statutory agencies responsible for behavioral healthcare, broadly defined as substance misuse, mental health, HIV/AIDS prevention, child welfare, and offender rehabilitation. There are currently no data on the purposes for which such state agencies reference EBPRs on their official websites.

METHOD

A webscraping method was used to identify and classify relevant "hits", defined as a state behavioral health webpage with single or multiple references to a study EBPR. A total of 778 hits (unique combinations of webpage and register) were coded. Up to three codes were applied to each hit for the "reasons for the EBPR reference" (EBPR use) dimension, one code was applied to each hit for the "purpose of the EBPR reference" and "intended audience of the webpage containing the hit" dimensions, and up to two codes were applied to each hit for the "funding mentions" dimension.

RESULTS

Three EBPRs out of 28 accounted for 73.6% of the hits. The most frequent reason for referencing EBPRs were as a resource for selecting EBPs or validating existing programs and practices. The references tended to appear in reports from the state, in training materials, or guidelines. The references tended to address broad groups of behavioral healthcare professionals. EBPRs were frequently referenced in the context of federal block grants or other federal funding.

CONCLUSIONS

Increasing state agencies' awareness and use of the entire range of existing EBPRs may improve implementation of EBPs nationally.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare

Background

Despite consistent recognition of their influence, empirical study of how outer setting factors (e.g., policies, financing, stakeholder relationships) influence public systems’ investment in and adoption of evidence-based treatment (EBT) is limited. This study examined associations among unmodifiable (e.g., demographic, economic, political, structural factors) and modifiable (e.g., allocation of resources, social processes, policies, and regulations) outer setting factors and adoption of behavioral health EBT by US states.

Methods

Multilevel models examined relationships between state characteristics, an array of funding and policy variables, and state adoption of behavioral health EBTs for adults and children across years 2002–2012, using data from the National Association for State Mental Health Program Directors Research Institute and other sources.

Results

Several unmodifiable state factors, including per capita income, controlling political party, and Medicaid expansion, predicted level of state fiscal investments in EBT. By contrast, modifiable factors, such as interagency collaboration and investment in research centers, were more predictive of state policies supportive of EBT. Interestingly, level of adult EBT adoption was associated with state fiscal supports for EBT, while child EBT adoption was predicted more by supportive policies. State per capita debt and direct state operation of services (versus contracting for services) predicted both child and adult EBT adoption.

Conclusions

State-level EBT adoption and associated implementation support is associated with an interpretable array of policy, financing, and oversight factors. Such information expands our knowledge base of the role of the outer setting in implementation and may provide insight into how best to focus efforts to promote EBT for behavioral health disorders.

Impact of a Web-Based Electronic Health Record on Behavioral Health Service Delivery for Children and Adolescents: Randomized Controlled Trial

Background

Electronic health records (EHRs) have been widely proposed as a mechanism for improving health care quality. However, rigorous research on the impact of EHR systems on behavioral health service delivery is scant, especially for children and adolescents.

Objective

The current study evaluated the usability of an EHR developed to support the implementation of the Wraparound care coordination model for children and youth with complex behavioral health needs, and impact of the EHR on service processes, fidelity, and proximal outcomes.

Methods

Thirty-four Wraparound facilitators working in two programs in two states were randomized to either use the new EHR (19/34, 56%) or to continue to implement Wraparound services as usual (SAU) using paper-based documentation (15/34, 44%). Key functions of the EHR included standard fields such as youth and family information, diagnoses, assessment data, and progress notes. In addition, there was the maintenance of a coordinated plan of care, progress measurement on strategies and services, communication among team members, and reporting on services, expenditures, and outcomes. All children and youth referred to services for eight months (N=211) were eligible for the study. After excluding those who were ineligible (69/211, 33%) and who declined to participate (59/211, 28%), a total of 83/211 (39%) children and youth were enrolled in the study with 49/211 (23%) in the EHR condition and 34/211 (16%) in the SAU condition. Facilitators serving these youth and families and their supervisors completed measures of EHR usability and appropriateness, supervision processes and activities, work satisfaction, and use of and attitudes toward standardized assessments. Data from facilitators were collected by web survey and, where necessary, by phone interviews. Parents and caregivers completed measures via phone interviews. Related to fidelity and quality of behavioral health care, including Wraparound team climate, working alliance with providers, fidelity to the Wraparound model, and satisfaction with services.

Results

EHR-assigned facilitators from both sites demonstrated the robust use of the system. Facilitators in the EHR group reported spending significantly more time reviewing client progress (P=.03) in supervision, and less time overall sending reminders to youth/families (P=.04). A trend toward less time on administrative tasks (P=.098) in supervision was also found. Facilitators in both groups reported significantly increased use of measurement-based care strategies overall, which may reflect cross-group contamination (given that randomization of staff to the EHR occurred within agencies and supervisors supervised both types of staff). Although not significant at P<.05, there was a trend (P=.10) toward caregivers in the EHR group reporting poorer shared agreement on tasks on the measure of working alliance with providers. No other significant between-group differences were found.

Conclusions

Results support the proposal that use of EHR systems can promote the use of client progress data and promote efficiency; however, there was little evidence of any impact (positive or negative) on overall service quality, fidelity, or client satisfaction. The field of children’s behavioral health services would benefit from additional research on EHR systems using designs that include larger sample sizes and longer follow-up periods.

Trial Registration

ClinicalTrials.gov NCT02421874; https://clinicaltrials.gov/ct2/show/NCT02421874 (Archived by WebCite at http://www.webcitation.org/6yyGPJ3NA)

Leadership, Organizational Climate, and Perceived Burden of Evidence-Based Practice in Mental Health Services

The use of evidence-based practices (EBPs) is associated with favorable client outcomes, yet perceived burden of using EBPs may affect the adoption and implementation of such practices. Multilevel path analysis was used to examine the associations of transformational leadership with organizational climate, and their associations with perceived burden of using EBPs. Results indicated significant relationships between transformational leadership and empowering and demoralizing climates, and between demoralizing climate and perceived burden of EBPs. We found significant indirect associations of leadership and perceived burden through organizational climate. Findings suggest that further research is needed to examine the extent to which improving leadership and organizational climate may reduce perceived burden and use of EBPs with the ultimate goal of enhancing quality of care.

Contracting and Procurement for Evidence-Based Interventions in Public-Sector Human Services: A Case Study

Sustainment of evidence-based interventions (EBIs) in human services depends on the inner context of community-based organizations (CBOs) that provide services and the outer context of their broader environment. Increasingly, public officials are experimenting with contracting models from for-profit industries to procure human services. In this case study, we conducted qualitative interviews with key government and CBO stakeholders to examine implementation of the Best Value-Performance Information Procurement System to contract for EBIs in a child welfare system. Findings suggest that stakeholder relationships may be compromised when procurement disregards local knowledge, communication, collaboration, and other factors supporting EBIs and public health initiatives.

The relationship between consumer, clinician, and organizational characteristics and use of evidence-based and non-evidence-based therapy strategies in a public mental health system

We investigated the relationship between consumer, clinician, and organizational factors and clinician use of therapy strategies within a system-wide effort to increase the use of cognitive-behavioral therapy. Data from 247 clinicians in 28 child-serving organizations were collected. Clinicians participating in evidence-based practice training initiatives were more likely to report using cognitive-behavioral therapy when they endorsed more clinical experience, being salaried clinicians, and more openness to evidence-based practice. Clinicians participating in evidence-based practice initiatives were more likely to use psychodynamic techniques when they had older clients, less knowledge about evidence-based practice, more divergent attitudes toward EBP, higher financial strain, and worked in larger organizations. In clinicians not participating in evidence-based training initiatives; depersonalization was associated with higher use of cognitive-behavioral; whereas clinicians with less knowledge of evidence-based practices were more likely to use psychodynamic techniques. This study suggests that clinician characteristics are important when implementing evidence-based practices; and that consumer, clinician, and organizational characteristics are important when de-implementing non evidence-based practices. This work posits potential characteristics at multiple levels to target with implementation and deimplementation strategies.

The transformation of behavioral healthcare in New Mexico

Since 1997, public-sector behavioral healthcare in New Mexico has remained under continual transition. We have conducted qualitative research to examine recent efforts in NM to establish a recovery-oriented behavioral healthcare system, focusing on comprehensive community support services, clinical homes, and core service agencies. We examine how decisions made in the outer context (e.g., the system level) shaped the implementation of each initiative within the inner context of service provision (e.g., provider agencies). We also clarify how sociopolitical factors, as exemplified in changes instituted by one gubernatorial administration and undone by its successor, can undermine implementation efforts and create crises within fragile behavioral healthcare systems. Finally, we discuss findings in relation to efforts to promote wraparound service planning and to establish medical home models under national healthcare reform.

Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001-2012

OBJECTIVE

Empirical study of public behavioral health systems' use of data and their investment in evidence-based treatments (EBTs) is limited. This study describes trends in state-level EBT investment and research supports from 2001 to 2012.

METHODS

Data were from National Association for State Mental Health Program Directors Research Institute (NRI) surveys, which were completed by representatives of state mental health authorities (SMHAs). Multilevel models examined change over time related to state adoption of EBTs, numbers of clients served, and penetration rates for six behavioral health EBTs for adults and children: supported housing, supported employment, assertive community treatment, therapeutic foster care, multisystemic therapy, and functional family therapy. State supports related to research, evaluation, and information management were also examined.

RESULTS

Increasing percentages of states reported funding an external research center, promoting the adoption of EBTs through provider contracts, and providing financial incentives for EBTs. Decreasing percentages of states reported promoting EBT adoption through stakeholder mobilization, monitoring fidelity, and specific budget requests. There was greater reported use of adult-focused EBTs (65%-80%) compared with youth-focused EBTs (25%-50%). Overall penetration rates of EBTs were low (1%-3%) and EBT adoption by states showed flat or declining trends. SMHAs' investment in data systems and use of research showed little change.

CONCLUSIONS

SMHA investment in EBTs, implementation infrastructure, and use of research has declined. More systematic measurement and examination of these metrics may provide a useful approach for setting priorities, evaluating success of health reform efforts, and making future investments.

An ARC-Informed Family Centered Care Intervention for Children's Community Based Mental Health Programs

The experience of parents in helping their children access and use mental health services is linked to service outcomes. Parent peer support service, based on the principles of family-centered care, is one model to improve parent experience and engagement in services. Yet, little is known about how best to integrate this service into the existing array of mental health services. Integration is challenged by philosophical differences between family-centered services and traditional children's treatment services, and is influenced by the organizational social contexts in which these services are embedded. We describe an organizational and frontline team intervention that draws on research in behavior change, technology transfer, and organizational social context for youth with serious emotional disturbance. The two-pronged intervention, called FAMILY (FCC and ARC Model to Improve the Lives of Youth) is guided by the evidence-based Availability, Responsiveness, and Continuity (ARC) organizational intervention, targeted primarily at program and upper management leadership and includes a family-centered care (FCC) intervention, targeted at frontline providers. The approach employs multilevel implementation strategies to promote the uptake, implementation and sustainability of new practices. We include examples of exercises and tools, and highlight implementation challenges and lessons learned in facilitating program and staff level changes in family-centered service delivery.

Parent Depression and Anger in Peer-Delivered Parent Support Services

Knowledge about parents who seek peer-delivered parent support services in children's mental health is limited. In this prospective study, characteristics of 124 parents who sought peer parent advocate services related to their children's behavioral difficulties are described. This urban sample consisted primarily of low-income mothers of color, 80% of whom were caring for children with clinically significant behavioral problems. Of these parents, 64% endorsed clinically significant levels of depressive symptoms at baseline. Linear mixed effects models were used to examine associations between parent depression and anger expression with working alliances with peer advocates. No independent or combined effects of parent depression or anger expression on working alliance were found. However, adjusting for family demographic factors, caregiver strain and child symptoms, parent depression interacted with anger expression to influence working alliances, primarily around agreement and mutual engagement on goals. Among parents who endorsed clinically significant depressive symptoms, anger expression did not influence working alliance but among non-depressed parents, anger expression was negatively associated with working alliance. Implications for training peer parent advocates to more effectively engage low income parents are discussed.

The Critical Role of State Agencies in the Age of Evidence-Based Approaches: The Challenge of New Expectations

Evidence-based approaches only benefit individuals when fully and effectively implemented. Since funding and monitoring alone will not ensure the full and effective implementation of effective strategies, state agencies have the opportunity to assess and modify current roles, functions, and policies to align with the requirements of evidence-based strategies. Based on a growing body of knowledge to guide effective implementation processes, state agencies, or designated partner organizations, can develop the capacity, mechanisms, and infrastructure to effectively implement evidence-based strategies. This article describes a framework that can guide this process. Informed by the literature and shaped by "real-world experience," the Active Implementation Frameworks provide a stage-matched approach to purposeful, active, and effective implementation.

From a "perfect storm" to "smooth sailing": policymaker perspectives on implementation and sustainment of an evidence-based practice in two states

Policymakers shape implementation and sustainment of evidence-based practices (EBPs), whether they are developing or responding to legislation and policies or negotiating public sector resource constraints. As part of a large mixed-method study, we conducted qualitative interviews with 24 policymakers involved in delivery of the same EBP in two U.S. states. We analyzed transcripts via open and focused coding techniques to identify the commonality, diversity, and complexity of implementation challenges; approaches to overcoming those challenges; and the importance of system-level contextual factors in ensuring successful implementation. Key findings centered on building support and leadership for EBPs; funding and contractual strategies; partnering with stakeholders; tackling challenges via proactive planning and problem solving; and the political, legal, and systemic pressures affecting EBP longevity. The policymaker perspectives offer guidance on nurturing system and organizational practice environments to achieve positive outcomes and for optimally addressing macro-level influences that bear upon the instantiation of EBPs in public sector child welfare systems.

Mixed-method study of a conceptual model of evidence-based intervention sustainment across multiple public-sector service settings

BACKGROUND

This study examines sustainment of an EBI implemented in 11 United States service systems across two states, and delivered in 87 counties. The aims are to 1) determine the impact of state and county policies and contracting on EBI provision and sustainment; 2) investigate the role of public, private, and academic relationships and collaboration in long-term EBI sustainment; 3) assess organizational and provider factors that affect EBI reach/penetration, fidelity, and organizational sustainment climate; and 4) integrate findings through a collaborative process involving the investigative team, consultants, and system and community-based organization (CBO) stakeholders in order to further develop and refine a conceptual model of sustainment to guide future research and provide a resource for service systems to prepare for sustainment as the ultimate goal of the implementation process.

METHODS

A mixed-method prospective and retrospective design will be used. Semi-structured individual and group interviews will be used to collect information regarding influences on EBI sustainment including policies, attitudes, and practices; organizational factors and external policies affecting model implementation; involvement of or collaboration with other stakeholders; and outer- and inner-contextual supports that facilitate ongoing EBI sustainment. Document review (e.g., legislation, executive orders, regulations, monitoring data, annual reports, agendas and meeting minutes) will be used to examine the roles of state, county, and local policies in EBI sustainment. Quantitative measures will be collected via administrative data and web surveys to assess EBI reach/penetration, staff turnover, EBI model fidelity, organizational culture and climate, work attitudes, implementation leadership, sustainment climate, attitudes toward EBIs, program sustainment, and level of institutionalization. Hierarchical linear modeling will be used for quantitative analyses. Qualitative analyses will be tailored to each of the qualitative methods (e.g., document review, interviews). Qualitative and quantitative approaches will be integrated through an inclusive process that values stakeholder perspectives.

DISCUSSION

The study of sustainment is critical to capitalizing on and benefiting from the time and fiscal investments in EBI implementation. Sustainment is also critical to realizing broad public health impact of EBI implementation. The present study takes a comprehensive mixed-method approach to understanding sustainment and refining a conceptual model of sustainment.

Statewide implementation of the 4 Rs and 2 Ss for strengthening families

Embedding evidence-informed practices for children with mental health needs into "real-world" community settings has proven challenging. In this article, we discuss how the Practical, Robust, Implementation, and Sustainability Model (PRISM) guided statewide (New York) implementation of an evidence-informed intervention targeting families and youth with oppositional defiant and conduct disorders, collectively referred to as disruptive behavior disorders. Additionally, we present qualitative findings based on provider reports of integrating a novel, evidence-informed intervention within their respective community mental health settings. Finally, we discuss a process of modifying the intervention to address agency-level factors, as well as inform more widespread implementation efforts.

Testing use of payers to facilitate evidence-based practice adoption: protocol for a cluster-randomized trial

Background

More effective methods are needed to implement evidence-based findings into practice. The Advancing Recovery Framework offers a multi-level approach to evidence-based practice implementation by aligning purchasing and regulatory policies at the payer level with organizational change strategies at the organizational level.

Methods

The Advancing Recovery Buprenorphine Implementation Study is a cluster-randomized controlled trial designed to increase use of the evidence-based practice buprenorphine medication to treat opiate addiction. Ohio Alcohol, Drug Addiction, and Mental Health Services Boards (ADAMHS), who are payers, and their addiction treatment organizations were recruited for a trial to assess the effects of payer and treatment organization changes (using the Advancing Recovery Framework) versus treatment organization changes alone on the use of buprenorphine. A matched-pair randomization, based on county characteristics, was applied, resulting in seven county ADAMHS boards and twenty-five treatment organizations in each arm. Opioid dependent patients are nested within cluster (treatment organization), and treatment organization clusters are nested within ADAMHS county board. The primary outcome is the percentage of individuals with an opioid dependence diagnosis who use buprenorphine during the 24-month intervention period and the 12-month sustainability period. The trial is currently in the baseline data collection stage.

Discussion

Although addiction treatment providers are under increasing pressure to implement evidence-based practices that have been proven to improve patient outcomes, adoption of these practices lags, compared to other areas of healthcare. Reasons frequently cited for the slow adoption of EBPs in addiction treatment include, regulatory issues, staff, or client resistance and lack of resources. Yet the way addiction treatment is funded, the payer’s role—has not received a lot of attention in research on EBP adoption.

This research is unique because it investigates the role of payers in evidence-based practice implementation using a randomized controlled design instead of case examples. The testing of the Advancing Recovery Framework is designed to broaden the understanding of the impact payers have on evidence-based practice (EBP) adoption.

Trial registration

http://NCT01702142 (ClinicalTrials.gov registry, USA)

Randomized trial of the Availability, Responsiveness and Continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs

OBJECTIVES

The primary objective of the study was to assess whether the Availability, Responsiveness and Continuity (ARC) organizational intervention improved youth outcomes in community based mental health programs. The second objective was to assess whether programs with more improved organizational social contexts following the 18-month ARC intervention had better youth outcomes than programs with less improved social contexts.

METHOD

Eighteen community mental health programs that serve youth between the ages of 5 and 18 were randomly assigned to ARC or control conditions. Clinicians (n = 154) in the participating programs completed the Organizational Social Context (OSC) measure at baseline and following the 18-month ARC organizational intervention. Caregivers of 393 youth who were served by the 18 programs (9 in ARC and 9 in control) completed the Shortform Assessment for Children (SAC) once a month for six months beginning at intake.

RESULTS

Hierarchical linear models (HLM) analyses indicated that youth outcomes were significantly better in the programs that completed the 18 month ARC intervention. HLM analyses also showed that youth outcomes were best in the programs with the most improved organizational social contexts following the 18 month ARC intervention.

CONCLUSIONS

Youth outcomes in community mental health programs can be improved with the ARC organizational intervention and outcomes are best in programs that make the most improvements in organizational social context. The relationships linking ARC, organizational social context, and youth outcomes suggest that service improvement efforts will be more successful if those efforts include strategies to improve the organizational social contexts in which the services are embedded.

Development of a Multilevel Framework to Increase Use of Targeted Evidence-Based Practices in Addiction Treatment Clinics

Implementing specific evidence-based practices (EBPs) across a set of addiction treatment providers have been a persistent challenge. In the Advancing Recovery(AR) demonstration project, single state agencies, the entities that distribute federal funds for substance use disorder prevention and treatment services, worked in partnership with providers to increase the use of EBPs in the treatment of addiction. The project supported two cohorts of six 2-year awards. Field observations from the first year of implementation guided development of a multilevel framework (the Advancing Recovery Framework). Government entities and other payers can use the framework as a guide for implementing evidence-based clinical practices within treatment networks. The Advancing Recover Framework calls for a combination of policy and organizational changes at both the payer (government agency) and provider levels. Using the Advancing Recovery Framework, 11 of the 12 AR payer/provider partnerships increased use of clinical EPBs. This article identifies key payer policy changes applied during different phases of EBP program implementation. The public health benefit of the demonstration project was broader use of medication-assisted therapy and continuing care in addiction treatment services.

Improving community-based mental health care for children: translating knowledge into action

There is urgent need for improvement in community-based mental health care for children and families. Multiple studies have documented serious limitations in the effectiveness of "usual care." Fortunately, many empirically-supported strategies to improve care have been developed, and thus there is now a great deal of knowledge available to address this significant public health problem. The goal of this selective review is to highlight and synthesize that empirically-supported knowledge to stimulate and facilitate the needed translation of knowledge into action. The review provides a sound foundation for constructing improved services by consolidating descriptive data on the status quo in children's mental health care, as well as evidence for an array of promising strategies to improve (a) Service access and engagement; (b) Delivery of evidence-based practices; and (c) Outcome accountability. A multi-level framework is used to highlight recommended care improvement targets.

Expanding the domains of attitudes towards evidence-based practice: the evidence based practice attitude scale-50

Mental health and social service provider attitudes toward evidence-based practice have been measured through the development and validation of the Evidence-Based Practice Attitude Scale (EBPAS; Aarons, Ment Health Serv Res 6(2):61–74, 2004). Scores on the EBPAS scales are related to provider demographic characteristics, organizational characteristics, and leadership. However, the EBPAS assesses only four domains of attitudes toward EBP. The current study expands and further identifies additional domains of attitudes towards evidence-based practice. A qualitative and quantitative mixed-methods approach was used to: (1) generate items from multiples sources (researcher, mental health program manager, clinician/therapist), (2) identify potential content domains, and (3) examine the preliminary domains and factor structure through exploratory factor analysis. Participants for item generation included the investigative team, a group of mental health program managers (n = 6), and a group of clinicians/therapists (n = 8). For quantitative analyses a sample of 422 mental health service providers from 65 outpatient programs in San Diego County completed a survey that included the new items. Eight new EBPAS factors comprised of 35 items were identified. Factor loadings were moderate to large and internal consistency reliabilities were fair to excellent. We found that the convergence of these factors with the four previously identified evidence-based practice attitude factors (15 items) was small to moderate suggesting that the newly identified factors represent distinct dimensions of mental health and social service provider attitudes toward adopting EBP. Combining the original 15 items with the 35 new items comprises the EBPAS 50-item version (EBPAS-50) that adds to our understanding of provider attitudes toward adopting EBPs. Directions for future research are discussed.

The role of a state-level prevention support system in promoting high-quality implementation and sustainability of evidence-based programs

Although numerous evidence-based programs (EBPs) have been proven effective in research trials and are being widely promoted through federal, state, and philanthropic dollars, few have been "scaled up" in a manner likely to have a measurable impact on today's critical social problems. The Interactive Systems Framework for Dissemination and Implementation (ISF) explicates three systems that are critical in addressing the barriers that prevent these programs from having their intended public health impact. In this article we describe the relevance of these systems in a real-world context with a specific focus on the Prevention Support System (PSS). We expand on the ISF model by presenting funders and policy-makers as active and engaged stakeholders, and demonstrate how a state-level PSS has used empirical evidence to inform general and program-specific capacity-building and support interactions among researchers, funders, and practitioners in Pennsylvania. By embracing this expanded ISF framework as a conceptual model for the wide-scale dissemination and support of EBPs, and recognizing the need for a distinct state-level PSS, Pennsylvania has created an infrastructure to effectively address the primary barriers to moving from lists of EBPs to achieving population-level public health improvement.

Outcome domains in child mental health research since 1996: have they changed and why does it matter?

OBJECTIVE

Child mental health treatment and services research yields more immediate public health benefit when they focus on outcomes of relevance to a broader group of stakeholders. We reviewed all experimental studies of child and adolescent treatment and service effectiveness published in the last 15 years (1996-2011) and compared the distribution and types of outcome domains to a prior review that focused on studies from the prior 15 years (1980-1995).

METHOD

Studies were included if they focused on children from birth to 18 years of age with specific or general psychiatric conditions, employed randomized designs, and examined intervention effects with a six-month or longer post-treatment assessment in treatment studies or a 6-month or longer post-baseline assessment for services studies. Two hundred (n=200) studies met criteria. Reported outcome measures were coded into conceptual categories drawn from the 1980-1995 review.

RESULTS

There was a five-fold increase in the total number of studies (38 versus 200) across the two 15-year time periods, with the largest increase in the number of studies that focused on consumer-oriented outcomes (from eight to 47 studies, an almost sixfold increase); two new domains, parent symptoms and health-related outcomes, were identified. The majority of studies (more than 95%) continued to focus on symptoms and diagnoses as an outcome. Impact ratings were higher among studies examining four or more outcomes versus one to two outcomes in all categories with the exception of Posttraumatic Stress Disorder.

CONCLUSIONS

Given major shifts in health care policy affecting mental health services, the emergence of health and parent-related outcomes as well as greater attention to consumer perspectives parallels emerging priorities in health care and can enhance the relevance of child outcome studies for implementation in the real world.

A state agency-university partnership for translational research and the dissemination of evidence-based prevention and intervention

This article describes a decade-long partnership between the Prevention Research Center at Penn State and the Pennsylvania Commission on Crime and Delinquency. This partnership has evolved into a multi-agency initiative supporting the implementation of nearly 200 replications of evidence-based prevention and intervention programs, and a series of studies indicating a significant and sustained impact on youth outcomes and more efficient utilization of system resources. We describe how the collaboration has developed into a sophisticated prevention support infrastructure, discuss the partnership and policy lessons learned throughout this journey, and identify remaining issues in promoting this type of research-policy partnership.

Randomized trial of the Availability, Responsiveness, and Continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth

OBJECTIVE

Evidence-based Practice (EBP) implementation is likely to be most efficient and effective in organizations with positive social contexts (i.e., organizational culture, climate, and work attitudes of clinicians). The study objective was to test whether an organizational intervention labeled Availability, Responsiveness and Continuity (ARC) could improve the organizational social contexts of community-based mental health programs for youth.

METHOD

The study randomly assigned 26 community-based mental health programs for youth to ARC or control conditions. The organizational cultures, climates, and work attitudes of clinicians (n = 197) in the programs were assessed with the Organizational Social Context (OSC) measure for mental health services at baseline and following the 18-month ARC intervention.

RESULTS

Hierarchical linear models (HLM) analyses indicated that organizational culture, climate, and work attitudes were significantly improved in the ARC condition after 18 months. Clinicians in programs assigned to ARC reported less rigid, less centralized and less apathetic organizational cultures, more engaged and functional organizational climates with less role conflict, and work attitudes with improved morale, job satisfaction, and organizational commitment.

CONCLUSIONS

ARC improved the organizational social contexts of clinicians in community-based mental health programs for youth. Results suggest that organizational intervention strategies can be used to create the types of organizational social contexts that are believed to be necessary for EBP implementation and other service innovations in mental health programs.

Employing policy and purchasing levers to increase the use of evidence-based practices in community-based substance abuse treatment settings: reports from single state authorities

State public health authorities are critical to the successful implementation of science based addiction treatment practices by community-based providers. The literature to date, however, lacks examples of state level policy strategies that promote evidence-based practices (EBPs). This mixed-methods study documents changes in two critical state-to-provider strategies aimed at accelerating use of evidence-based practices: purchasing levers (financial incentives and mechanisms) and policy or regulatory levers. A sample of 51 state representatives was interviewed. Single State Authorities for substance abuse treatment (SSAs) that fund providers directly or through managed care were significantly more likely to have contracts that required or encouraged evidence-based interventions, as compared to SSAs that fund providers indirectly through sub-state entities. Policy levers included EBP-related legislation, language in rules and regulations, and evidence-based criteria in state plans and standards. These differences in state policy are likely to result in significant state level variations regarding both the extent to which EBPs are implemented by community-based treatment providers and the quality of implementation.

Statewide CBT Training for Clinicians and Supervisors Treating Youth: The New York State Evidence Based Treatment Dissemination Center

In recent years, several states have undertaken efforts to disseminate evidence-based treatments to agencies and clinicians in their children's service system. In New York, the Evidence Based Treatment Dissemination Center adopted a unique translation-based training and consultation model in which an initial 3-day training was combined with a year of clinical consultation with specific clinician and supervisor elements. This model has been used by the New York State Office of Mental Health for the past 3 years to train 1,210 clinicians and supervisors statewide. This article describes the early adoption and initial implementation of a statewide training program in cognitive-behavioral therapy for youth. The training and consultation model and descriptive findings are presented; lessons learned are described. Future plans include a focus on sustainability and measurement feedback of youth outcomes to enhance the continuity of this program and the quality of the clinical services.

Effects of workshop training for providers under mandated use of an evidence-based practice

Workshops are a common strategy for fostering the adoption of evidence-based practices (EBP), but workshops alone may not change provider behavior. This study investigates the impact of a two-day training combined with an existing mandate for EBP use. Providers attending regional workshops showed improved attitudes toward the behavioral parent training model, but not EBPs in general. Participants were more accepting of behavioral techniques shortly after training, but the effect was not maintained. Examination of youth served prior to and after the training showed that providers increased their use of the EBP but overall outcomes were not improved.

Developing the mental health workforce: review and application of training approaches from multiple disciplines

Strategies specifically designed to facilitate the training of mental health practitioners in evidence-based practices (EBPs) have lagged behind the development of the interventions themselves. The current paper draws from an interdisciplinary literature (including medical training, adult education, and teacher training) to identify useful training and support approaches as well as important conceptual frameworks that may be applied to training in mental health. Theory and research findings are reviewed, which highlight the importance of continued consultation/support following training workshops, congruence between the training content and practitioner experience, and focus on motivational issues. In addition, six individual approaches are presented with careful attention to their empirical foundations and potential applications. Common techniques are highlighted and applications and future directions for mental health workforce training and research are discussed.

Advancing a conceptual model of evidence-based practice implementation in public service sectors

Implementation science is a quickly growing discipline. Lessons learned from business and medical settings are being applied but it is unclear how well they translate to settings with different historical origins and customs (e.g., public mental health, social service, alcohol/drug sectors). The purpose of this paper is to propose a multi-level, four phase model of the implementation process (i.e., Exploration, Adoption/Preparation, Implementation, Sustainment), derived from extant literature, and apply it to public sector services. We highlight features of the model likely to be particularly important in each phase, while considering the outer and inner contexts (i.e., levels) of public sector service systems.

Kernels vs. Ears, and Other Questions for a Science of Treatment Dissemination

Combining intervention diffusion with change in clinical practice and public policy is an ambitious agenda. The impressive effort in Hawaii can be instructive, highlighting questions for a science of treatment dissemination. Among these questions, some of the most important are the following: (a) Who should be targeted for change? (e.g., "downstream" clinicians in practice, "upstream" clinicians in training, consumers, "brokers," policy makers, or payers?); (b) What should be disseminated? (e.g., full evidence-based protocols, specific treatment elements or "kernels"?); and (c) Which procedures maximize change? (e.g., what combination and duration of teaching, supervision, consultation, and other support?). Ultimately, change efforts need to assess what aspects of practice were actually altered, what measurable impact the changes had on clinical outcomes, and what changes in practices and outcomes can be sustained over time.

Evaluation of the sustainability and clinical outcome of Alternatives for Families: A Cognitive-Behavioral Therapy (AF-CBT) in a child protection center

This paper examines the sustainability and outcome of Alternatives for Families: A Cognitive-Behavioral Therapy (AF-CBT) as delivered by practitioners in a community-based child protection program who had received training in the model several years earlier. Formerly described as Abuse-Focused CBT, AF-CBT is an evidence-based treatment (EBT) for child physical abuse and family aggression/conflict that was included in the National Child Traumatic Stress Network's initial EBT dissemination efforts in 2002. Seven practitioners participated in a year-long learning collaborative in AF-CBT and in similar training programs for 4 other EBTs. The agency's routine data collection system was used to document the clinical and adjustment outcomes of 52 families presenting with a physically abused child who received their services between 2 and 5 years after the AF-CBT training had ended. Measures of the use of all 5 EBTs documented their frequency, internal consistency, and intercorrelations. Controlling for the unique content of the other four EBTs, the amount of AF-CBT Abuse-specific content delivered was related to improvements on standardized parent rating scales (i.e., child externalizing behavior, anger, anxiety, social competence) and both parent and clinician ratings of the child's adjustment at discharge (i.e., child more safe, less scared/sad, more appropriate with peers). The amount of AF-CBT General content was related to a few discharge ratings (better child prognosis, helpfulness to parents). These novel data provide suggestive evidence for the sustainability and clinical benefits of AF-CBT in an existing community clinic serving physically abused children and their families, and are discussed in the context of key developments in the treatment model and dissemination literature.

Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS)

The Evidence-Based Practice Attitude Scale (EBPAS) assesses mental health and social service provider attitudes toward adopting evidence-based practices. Scores on the EBPAS derive from 4 subscales (i.e., Appeal, Requirements, Openness, and Divergence) as well as the total scale, and preliminary studies have linked EBPAS scores to clinic structure and policies, organizational culture and climate, and first-level leadership. EBPAS scores are also related to service provider characteristics, including age, education level, and level of professional development. The present study examined the factor structure, reliability, and norms of EBPAS scores in a sample of 1,089 mental health service providers from a nationwide sample drawn from 100 service institutions in 26 states in the United States. The study also examined associations of provider demographic characteristics with EBPAS subscale and total scores. Confirmatory factor analysis supported a second-order factor model, and reliability coefficients for the subscales ranged from .91 to .67 (total scale = .74). The study establishes national norms for the EBPAS so that comparisons can be drawn for U.S. local as well as international studies of attitudes toward evidence-based practices. The results suggest that the factor structure and reliability are likely generalizable to a variety of service provider contexts and different service settings and that the EBPAS subscales are associated with provider characteristics. Directions for future research are discussed.

The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations

Evidence-based treatments (EBT) are underutilized in community settings, where consumers are often seen for treatment. Underutilization of EBTs may be related to a lack of empirically informed and supported training strategies. The goals of this review are to understand the state of the literature for training therapists in psychotherapy skills and to offer recommendations to improve research in this area. Results of this review of 55 studies evaluating six training methods indicate that multi-component trainings have been studied most often and have most consistently demonstrated positive training outcomes relative to other training methods. Studies evaluating utility of reading, self-directed trainings, and workshops have documented that these methods do not routinely produce positive outcomes. Workshop follow-ups help to sustain outcomes. Little is known about the impact of train-the-trainer methods. Methodological flaws and factors that may influence training outcome and future directions are also reviewed.

Extended Day Treatment: A Comprehensive Model of After School Behavioral Health Services for Youth

This paper describes Extended Day Treatment (EDT), an innovative intermediate-level service for children and adolescents with serious emotional and behavioral disorders, delivered in the after school hours. The current paper describes the core components of the EDT model of care within the context of statewide systems of care, including its theoretical foundations, core service components, relation to evidence-based practices, workforce composition and staff training, and data collection and reporting mechanisms. Recommendations are provided for statewide implementation, followed by discussion of model development as an approach to systems reform for the treatment of children and youth with emotional and behavioral disorders.

Treatment research for children and youth exposed to traumatic events: moving beyond efficacy to amp up public health impact

OBJECTIVE

Population-based demands for trauma services have accelerated interest in the rapid deployment of efficacious interventions to address the diverse mental health consequences of traumatic experiences. However, optimal strategies for supporting either implementation or dissemination of trauma-focused interventions within healthcare or mental healthcare systems are underdeveloped.

METHODS

This work offers suggestions for adapting treatment research parameters in order to advance the science on the implementable and practical use of trauma-focused interventions within a public health framework. To this end, we briefly examine the current status of research evidence in this area and discuss efficacy and effectiveness treatment research parameters with specific attention to the implications for developing the research base on the implementation and dissemination of effective trauma practices for children and adolescents.

RESULTS

Examples from current studies are used to identify approaches for developing, testing and enhancing strategies to roll out effective treatment practices in real-world settings.

CONCLUSIONS

New approaches that reflect the contexts in which these practices are implemented may enhance the feasibility, acceptability, replicability and sustainability of trauma treatments and services, and thus improve outcomes for a broader population of youth and families.

Beyond outcomes monitoring: measurement feedback systems in child and adolescent clinical practice

PURPOSE OF REVIEW

To review the literature published during the past year relevant to identifying the best measures for monitoring progress in the treatment of child and adolescent clients and their families.

RECENT FINDINGS

The current literature shows an increasing focus on clinical utility in measure development as demonstrated by the recent emphasis on evidence-based assessment. However, there is very little research on how the inclusion of monitoring might enhance clinician practice and ultimately youth and family outcomes. There is great promise in expanding our thinking beyond mere outcome measurement to a measurement feedback system that provides timely feedback that is comprehensive and concurrent with treatment.

SUMMARY

Investment in the development of measurement feedback systems is needed to enhance clinical judgment and increase effectiveness of treatment. Clinical utility and consumer appeal need to be key considerations for measures intended to be used in everyday clinical practice. Most importantly, we must harness the power of technology and advances in measurement to provide clinicians with the tools to use effectively the systematic data provided through frequent measurement with measurement feedback systems.

Second opinions improve ADHD prescribing in a medicaid-insured community population

OBJECTIVE

The appropriate use of psychotropic medications in youths is an important public health concern. In this article, we describe a review process developed to monitor the use of stimulants and atomoxetine for attention-deficit/hyperactivity syndrome (ADHD) in youths receiving fee-for-service Medicaid services.

METHOD

Washington State Medicaid developed threshold safety parameters for ADHD medications through a process involving the community. A second opinion was mandated when safety thresholds based on dose, combination therapies, or age was exceeded. Use and cost were compared 2 years before and after the program began.

RESULTS

From May 2006 to April 2008, 5.35% of ADHD prescriptions exceeded safety thresholds, resulting in 1,046 second-opinion reviews. Of those, 538 (51.4%) resulted in a prescription adjustment. Adjustments were made to primary care physician (52%), psychiatrist (50%), nurse practitioner (54%), and physician assistant-written (51%) prescriptions. When the preperiod and postperiod were compared, second opinions reduced ADHD medication at high doses (53%), in combinations (44%), and for patients 5 years of age and younger (23%). The review process resulted in a savings of $1.2 million, with 538 fewer patients exceeding safety thresholds. This was a 10:1 return over administrative costs; however, the overall Medicaid expenditures for ADHD medication still increased because of higher unit costs and the preferential use by clinicians of newer brands entering the market.

CONCLUSIONS

A statewide second-opinion process reduced outlier ADHD medication prescription practices and was cost-effective. Suggestions for process and quality improvements in prescribing to children diagnosed with ADHD are discussed.

Child welfare recommendations to improve mental health services for children who have experienced abuse and neglect: a national perspective

This study uses a key informant approach to understand the nature, extent, and quality of outpatient mental health services for children in the child welfare system (CWS) in the United States. We interviewed 89 county child welfare administrators to determine the status of outpatient mental health services and provide recommendations for enhancing care and service delivery. Developed for this study (Caring for Children in Child Welfare), the interview was incorporated in the second formal data collection wave (i.e., 18 months after study baseline assessment) of the National Survey of Child and Adolescent Well-Being. The results highlighted general variability in the degree to which these agencies used evidence-based interventions within outpatient services, demonstrated clinical expertise with this population, and met the needs of their families. Community agency use of evidence-based interventions was found to predict their effectiveness in improving clients' mental health problems. Proposed suggestions for service improvement varied across domains and reflected the need for more communication/coordination, service access, options and resources, and practice refinements to accommodate families' needs. We discuss the implications of these recommendations from CWS stakeholders for enhancing the service delivery system.