From data to wisdom: quality improvement strategies supporting large-scale implementation of evidence-based services

Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

BACKGROUND

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

METHOD

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability and inclusion (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

RESULTS

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

DISCUSSION

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

The Content of Usual Clinical Care for Youth with Primary Anxiety Problems

The current study was designed to describe usual clinical care for youth with primary anxiety problems in community mental health centers. The observer-rated Therapy Process Observational Coding System for Child Psychotherapy - Revised Strategies scale (TPOCS-RS), designed to assess therapeutic techniques from five theory-based domains, was used to code sessions (N = 403) from the usual clinical care group of two randomized effectiveness trials: (a) Youth Anxiety Study (YAS) with 21 youth (M age = 10.44 years, SD = 1.91; 49.2% Latinx; 46.6%, 53.4% male) and 16 clinicians (77.5% female; 43.8% White), and (b) Child STEPS Multisite Trial with 17 youth (M age = 10.00 years, SD = 1.87; 58.8% male; 41.2% White) and 13 clinicians (M age = 40.00 years; SD = 9.18; 76.9% female; 61.5% White). The average number of TPOCS-RS items observed per treatment session was more than 10, and multiple techniques were used together in each session. All TPOCS-RS items were observed at least once throughout a clinical case, and most items reoccurred (i.e., observed in two or more sessions). The dosage of TPOCS-RS in all items was below 5 on a 7-point scale. In conclusion, clinicians in both usual care samples used a wide range of techniques from several theory-based domains at a low to medium dose. However, the type and dosage of the techniques used did vary across the two samples.

Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Provider Use of, Attitudes Towards, and Self-efficacy with Key Measurement-based Care Practices in Youth Mental Health Treatment: A Multi-site Examination

Measurement-based care (MBC) is an underutilized evidence-based practice, and current implementation efforts demonstrate limited success in increasing MBC use. A better understanding of MBC implementation determinants is needed to improve these efforts, particularly from studies examining the full range of MBC practices and that span multiple samples of diverse providers using different MBC systems. This study addressed these limitations by conducting a multi-site survey examining MBC predictors and use in youth treatment. Participants were 159 clinicians and care coordinators working in youth mental health care settings across the United States. Participants were drawn from three program evaluations of MBC implementation. Providers completed measures assessing use of five MBC practices (administering measures, viewing feedback, reviewing feedback in supervision, sharing feedback with clients in session, and using feedback to plan treatment), MBC self-efficacy, and MBC attitudes. Despite expectations that MBC should be standard care for all clients, providers reported only administering measures to 40-60% of clients on average, with practices related to the use of feedback falling in the 1-39% range. Higher MBC self-efficacy and more positive views of MBC practicality predicted higher MBC use, although other attitude measures were not significant predictors. Effects of predictors were not moderated by site, suggesting consistent predictors across implementation settings. Implications of study findings for future research and for the implementation of MBC are discussed.

Mapping evidence-based interventions to the care of unaccompanied minor refugees using a group formulation approach

BACKGROUND

How we adapt treatment algorithms to complex, clinically untested, difficult-to-engage patient groups without losing evidence base in everyday practice is a clinical challenge. Here we describe process and reasoning for fast, pragmatic, context-relevant and service-based adaptations of a group intervention for unaccompanied minor asylum seekers (UASC) arriving in Europe. We employed a distillation-matching model and deployment-focused process in a mixed-method, top-down (theory-driven) and bottom-up (participant-informed) approach. Prevalence of mental disorders amongst UASC is extremely high. They also represent a marginalised and hard-to-engage group with limited evidence for effective treatments.

METHOD

Content and process adaptations followed four steps: (1) descriptive local group characterisation and theoretical formulation of problems; (2) initial adaptation of evidenced treatment, based on problem-to-component grid; (3) iterative adaptation using triangulated feedback; and (4) small-scale pilot evaluation.

RESULTS

Based on evidence and participant feedback, adaptations included minimising verbal demands, facilitating in-session inductive learning, fostering social connectedness via games, enhancing problem-solving skills, accounting for multi-traumatisation, uncertainty and deportation. Quantitative evaluation suggested improved feasibility, with increased attendance, low drop-out and symptom improvement on depression and trauma scores.

CONCLUSIONS

By describing the principles under-pinning development of a group intervention for severely traumatised UASC, we contribute to the literature supporting dynamic adaptations of psychological interventions, without losing reference to evidence base. Complex and difficult-to-reach clinical groups are often those in most need of care, yet least researched and most affected by inequality of care. Pragmatic adaptations of proven programs are often necessary to increase feasibility.

Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): A modified Delphi study

Background

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

Method

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

Results

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

Discussion

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

Assessing practical implementation of modular psychotherapy for youth in community-based settings using benchmarking

Background

Achieving high quality outcomes in a community context requires the strategic coordination of many activities in a service system, involving families, clinicians, supervisors, and administrators. In modern implementation trials, the therapy itself is guided by a treatment manual; however, structured supports for other parts of the service system may remain less well-articulated (e.g., supervision, administrative policies for planning and review, information/feedback flow, resource availability). This implementation trial investigated how a psychosocial intervention performed when those non-therapy supports were not structured by a research team, but were instead provided as part of a scalable industrial implementation, testing whether outcomes achieved would meet benchmarks from published research trials.

Method

In this single-arm observational benchmarking study, a total of 59 community clinicians were trained in the Modular Approach to Therapy for Children (MATCH) treatment program. These clinicians delivered MATCH treatment to 166 youth ages 6 to 17 naturally presenting for psychotherapy services. Clinicians received substantially fewer supports from the treatment developers or research team than in the original MATCH trials and instead relied on explicit process management tools to facilitate implementation. Prior RCTs of MATCH were used to benchmark the results of the current initiative. Client improvement was assessed using the Top Problems Assessment and Brief Problem Monitor.

Results

Analysis of client symptom change indicated that youth experienced improvement equal to or better than the experimental condition in published research trials. Similarly, caregiver-reported outcomes were generally comparable to those in published trials.

Conclusions

Although results must be interpreted cautiously, they support the feasibility of using process management tools to facilitate the successful implementation of MATCH outside the context of a formal research or funded implementation trial. Further, these results illustrate the value of benchmarking as a method to evaluation industrial implementation efforts. Plain Language Summary: Randomized effectiveness trials are inclusive of clinicians and cases that are routinely encountered in community-based settings, while continuing to rely on the research team for both clinical and administrative guidance. As a result, the field still struggles to understand what might be needed to support sustainable implementation and how interventions will perform when brought to scale in community settings without those clinical trial supports. Alternative approaches are needed to delineate and provide the clinical and operational support needed for implementation and to efficiently evaluate how evidence-based treatments perform. Benchmarking findings in the community against findings of more rigorous clinical trials is one such approach. This paper offers two main contributions to the literature. First, it provides an example of how benchmarking is used to evaluate how the Modular Approach to Therapy for Children (MATCH) treatment program performed outside the context of a research trial. Second, this study demonstrates that MATCH produced comparable symptom improvements to those seen in the original research trials and describes the implementation strategies associated with this success. In particular, although clinicians in this study had less rigorous expert clinical supervision as compared with the original trials, clinicians were provided with process management tools to support implementation. This study highlights the importance of evaluating the performance of intervention programs when brought to scale in community-based settings. This study also provides support for the use of process management tools to assist providers in effective implementation.

What Gets Measured Gets Done: How Mental Health Agencies can Leverage Measurement-Based Care for Better Patient Care, Clinician Supports, and Organizational Goals

Mental health clinicians and administrators are increasingly asked to collect and report treatment outcome data despite numerous challenges to select and use instruments in routine practice. Measurement-based care (MBC) is an evidence-based practice for improving patient care. We propose that data collected from MBC processes with patients can be strategically leveraged by agencies to also support clinicians and respond to accountability requirements. MBC data elements are outlined using the Precision Mental Health Framework (Bickman et al. in Adm Policy Mental Health Mental Health Serv Res 43:271-276, 2016), practical guidance is provided for agency administrators, and conceptual examples illustrate strategic applications of one or more instruments to meet various needs throughout the organization.

Where Does It Begin? Community-Based Therapists' Intentions for Treating Non-comorbid Youth

This study examined patterns and predictors of 79 public sector therapists' practice element (PE) intentions for treating hypothetical single problem area youth with either anxiety or disruptive behavior problems. Analyses of intention profiles suggested that PE intentions varied by diagnosis and included both a majority of practices derived from the evidence base (PDEBs) and a large number of practices with minimal evidence support (PMESs) for each problem area. Hierarchical multiple regression results indicated the only significant predictor of higher PDEB intentions for both problem areas was reporting more PMES intentions. Implications for community-based research and implementation efforts are discussed.

The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare

Background

Despite consistent recognition of their influence, empirical study of how outer setting factors (e.g., policies, financing, stakeholder relationships) influence public systems’ investment in and adoption of evidence-based treatment (EBT) is limited. This study examined associations among unmodifiable (e.g., demographic, economic, political, structural factors) and modifiable (e.g., allocation of resources, social processes, policies, and regulations) outer setting factors and adoption of behavioral health EBT by US states.

Methods

Multilevel models examined relationships between state characteristics, an array of funding and policy variables, and state adoption of behavioral health EBTs for adults and children across years 2002–2012, using data from the National Association for State Mental Health Program Directors Research Institute and other sources.

Results

Several unmodifiable state factors, including per capita income, controlling political party, and Medicaid expansion, predicted level of state fiscal investments in EBT. By contrast, modifiable factors, such as interagency collaboration and investment in research centers, were more predictive of state policies supportive of EBT. Interestingly, level of adult EBT adoption was associated with state fiscal supports for EBT, while child EBT adoption was predicted more by supportive policies. State per capita debt and direct state operation of services (versus contracting for services) predicted both child and adult EBT adoption.

Conclusions

State-level EBT adoption and associated implementation support is associated with an interpretable array of policy, financing, and oversight factors. Such information expands our knowledge base of the role of the outer setting in implementation and may provide insight into how best to focus efforts to promote EBT for behavioral health disorders.

Disaster Exposure and Mental Health Among Puerto Rican Youths After Hurricane Maria

IMPORTANCE

Quantifying the magnitude of disaster exposure and trauma-related symptoms among youths is critical for deployment of psychological services in underresourced settings. Hurricane Maria made landfall in Puerto Rico on September 20, 2017, resulting in massive destruction and unprecedented mortality.

OBJECTIVE

To determine the magnitude of disaster exposure and mental health outcomes among Puerto Rican youths after Hurricane Maria.

DESIGN, SETTING, AND PARTICIPANTS

Survey study in which a school-based survey was administered to each public school student at all schools in Puerto Rico between February 1 and June 29, 2018 (5-9 months after Hurricane Maria). Of the 226 808 students eligible to participate, 96 108 students completed the survey.

MAIN OUTCOMES AND MEASURES

Participants were assessed for exposure to hurricane-related stressors, posttraumatic stress disorder (PTSD), and depressive symptoms, using standardized self-report measures administered in Spanish. Descriptive statistics were compiled for all outcome variables, as was the frequency of individuals reporting clinically elevated symptoms of PTSD or depression. Differences in these statistics across sexes were also examined via t tests. Correlations between demographic, geographic, and main outcome variables were also calculated, and regressions were conducted to examine their association with symptoms of PTSD.

RESULTS

A total of 96 108 students participated in the study (42.4% response rate; 50.3% female), representative of grades 3 to 12 across all 7 educational regions of Puerto Rico. As a result of the hurricane, 83.9% of youths saw houses damaged, 57.8% had a friend or family member leave the island, 45.7% reported damage to their own homes, 32.3% experienced shortages of food or water, 29.9% perceived their lives to be at risk, and 16.7% still had no electricity 5 to 9 months after the hurricane. Overall, 7.2% of youths (n = 6900) reported clinically significant symptoms of PTSD; comparison of the frequency of reporting clinically elevated symptoms of PTSD across sex yielded a significant difference (t = 12.77; 95% CI of the difference, 0.018-0.025; P < .001), with girls (8.2%) exceeding the clinical cutoff score more often than boys (6.1%). Finally, similar analysis of differences in depression between sexes was also significant (t = 17.56; 95% CI of the difference, 0.31-0.39; P < .001), with girls displaying higher mean (SD) scores (2.72 [3.14]) than boys (2.37 [2.93]). Demographic and risk variables accounted for approximately 20% of variance in symptoms of PTSD (r2 = 0.195; 95% CI, 0.190-0.200).

CONCLUSIONS AND RELEVANCE

Survey results indicate that Hurricane Maria exposed Puerto Rican youths to high levels of disaster-related stressors, and youths reported high levels of PTSD and depressive symptoms. Results are currently being used by the Puerto Rico Department of Education to inform targeted and sustainable evidence-based practices aimed at improving mental health outcomes for Puerto Rico's youths.

Patterns and predictors of compliance with utilization management guidelines supporting a state policy to improve the quality of youth mental health services

Despite a need to improve community mental health services for youths, little is known about compliance with state policies created to improve the quality of services in these settings. This study examined rates, patterns, and predictors of compliance with utilization management guidelines developed by the state of Texas to support a public health policy based on empirical evidence of effective mental health services (i.e., an evidence-based policy). Compliance was defined as authorizing policy-recommended service packages, whereas policy "overrides" occurred when recommended service packages were not authorized. The study sample consisted of 688 youths from ethnically and economically diverse backgrounds. Clinics reported that forty-six percent of youths were not authorized the policy-recommended service package. Overrides were primarily based on level of intensity. Most often, authorized services were less intensive than those recommended by the state guidelines. Higher severity at intake across multiple indicators was associated with authorizing less intensive services than what the policy guidelines recommended. Future studies evaluating system-level efforts such as state mental health policies should pay close attention to levels of service intensity, and their relation to the needs of youth in community settings.

Training in Evidence-Based Psychological Practice at the Master's Level

Although a wealth of evidence supports the use of evidence-based psychological practice (EBPP) for clients with diverse difficulties, from anxiety and depression to behavioral problems, the majority of training in EBPP takes place at the doctoral and postdoctoral level. This is problematic given that there are many more master's level mental health professionals than doctoral ones, especially in rural and low-income areas. This article outlines a model for a 60-hour training program that focuses on preparing master's students to become competent in the practice of EBPP in a relatively short period of time, while at the same time meeting requirements for licensure in most states. Course sequence, course content, clinical experience, and supervision issues are all addressed, as are challenges to implementation of the model.

Facilitators and Barriers of Implementing a Measurement Feedback System in Public Youth Mental Health

This study examines implementation facilitators and barriers of a statewide roll-out of a measurement feedback system (MFS) in a youth public mental health system. 76 % of all state care coordinators (N = 47) completed interviews, which were coded via content analysis until saturation. Facilitators (e.g., recognition of the MFS's clinical utility) and barriers (e.g., MFS's reliability and validity) emerged paralleling the Exploration, Adoption/Preparation, Implementation, and Sustainment framework outlined by Aarons et al. (Adm Policy Mental Health Mental Health Serv Res, 38:4-23, 2011). Sustainment efforts may leverage innovation fit, individual adopter, and system related facilitators.

The Price per Prospective Consumer of Providing Therapist Training and Consultation in Seven Evidence-Based Treatments within a Large Public Behavioral Health System: An Example Cost-Analysis Metric

OBJECTIVE

Public-sector behavioral health systems seeking to implement evidence-based treatments (EBTs) may face challenges selecting EBTs given their limited resources. This study describes and illustrates one method to calculate cost related to training and consultation to assist system-level decisions about which EBTs to select.

METHODS

Training, consultation, and indirect labor costs were calculated for seven commonly implemented EBTs. Using extant literature, we then estimated the diagnoses and populations for which each EBT was indicated. Diagnostic and demographic information from Medicaid claims data were obtained from a large behavioral health payer organization and used to estimate the number of covered people with whom the EBT could be used and to calculate implementation-associated costs per consumer.

RESULTS

Findings suggest substantial cost to therapists and service systems related to EBT training and consultation. Training and consultation costs varied by EBT, from Dialectical Behavior Therapy at $238.07 to Cognitive Behavioral Therapy at $0.18 per potential consumer served. Total cost did not correspond with the number of prospective consumers served by an EBT.

CONCLUSION

A cost-metric that accounts for the prospective recipients of a given EBT within a given population may provide insight into how systems should prioritize training efforts. Future policy should consider the financial burden of EBT implementation in relation to the context of the population being served and begin a dialog in creating incentives for EBT use.

Distilling the Core Elements of Family Therapy for Adolescent Substance Use: Conceptual and Empirical Solutions

This article describes several barriers to widespread dissemination of manualized family-based treatments for adolescent substance use (ASU). We then offer a highly promising solution for adopting and sustaining family therapy in usual care: distilling the core practice elements of empirically validated family therapy models for ASU. We present a conceptual distillation of family therapy for ASU grounded in existing observational fidelity measures for three manualized models, a process that yielded four core elements: Family Engagement, Relational Reframing, Family Behavior Change, and Family Restructuring. We then introduce an innovative empirical method for distilling core elements that can serve as a template for rigorous distillation of other treatment approaches. Finally, we discuss how core elements can enhance family therapy services within the diverse workforce of usual care for ASU.

At the Intersection of Health Information Technology and Decision Support: Measurement Feedback Systems... and Beyond

We select and comment on concepts and examples from the target articles in this special issue on measurement feedback systems, placing them in the context of some of our own insights and ideas about measurement feedback systems, and where those systems lie at the intersection of technology and decision making. We contend that, connected to the many implementation challenges relevant to many new technologies, there are fundamental design challenges that await a more elaborate specification of the clinical information and decision models that underlie these systems. Candidate features of such models are discussed, which include referencing multiple evidence bases, facilitating observed and expected value comparisons, fostering collaboration, and allowing translation across multiple ontological systems. We call for a new metaphor for these technologies that goes beyond measurement feedback and encourages a deeper consideration of the increasingly complex clinical decision models needed to manage the uncertainty of delivering clinical care.

Determinants and Functions of Standardized Assessment Use Among School Mental Health Clinicians: A Mixed Methods Evaluation

The current study evaluated why and how school mental health clinicians use standardized assessment tools in their work with youth and families. Quantitative and qualitative (focus group) data were collected prior to and following a training and consultation sequence as part of a trial program to assess school clinician's (n = 15) experiences administering standardized tools to youth on their caseloads (n = 191). Findings indicated that, although assessment use was initially somewhat low, clinicians used measures to conduct initial assessments with the bulk of their caseloads (average = 62.2%) during the implementation period. Clinicians also reported on factors influencing their use of assessments at the client, provider, and system levels; perceived functions of assessment; student responses to assessment use; and use of additional sources of clinically-relevant information (primarily educational data) for the purposes of assessment and progress monitoring. Implications for the contextual appropriateness of standardized assessment and training in assessment tools are discussed.

Factor Analysis of Therapist-Identified Treatment Targets in Community-Based Children's Mental Health

The present study used exploratory and confirmatory factor analyses to identify underlying latent factors affecting variation in community therapists' endorsement of treatment targets. As part of a statewide practice management program, therapist completed monthly reports of treatment targets (up to 10 per month) for a sample of youth (n = 790) receiving intensive in-home therapy. Nearly 75 % of youth were diagnosed with multiple co-occurring disorders. Five factors emerged: Disinhibition, Societal Rules Evasion, Social Engagement Deficits, Emotional Distress, and Management of Biodevelopmental Outcomes. Using logistic regression, primary diagnosis predicted therapist selection of Disinhibition and Emotional Distress targets. Client age predicted endorsement of Societal Rules Evasion targets. Practice-to-research implications are discussed.

User-Centered Design for Psychosocial Intervention Development and Implementation

The current paper articulates how common difficulties encountered when attempting to implement or scale-up evidence-based treatments are exacerbated by fundamental design problems, which may be addressed by a set of principles and methods drawn from the contemporary field of user-centered design. User-centered design is an approach to product development that grounds the process in information collected about the individuals and settings where products will ultimately be used. To demonstrate the utility of this perspective, we present four design concepts and methods: (a) clear identification of end users and their needs, (b) prototyping/rapid iteration, (c) simplifying existing intervention parameters/procedures, and (d) exploiting natural constraints. We conclude with a brief design-focused research agenda for the developers and implementers of evidence-based treatments.

Future Directions for the Implementation and Dissemination of Statewide Developmental-Behavioral Pediatric Integrated Health Care

The integration of mental health and pediatric health care services has long been a goal for both research and practice. With the advent of federal policies developed to mandate clinical efficiency across the health care spectrum, this issue is becoming more salient. Applied literature on this topic is only recently emerging, however, and there are limited contextual examples to guide program development, research, and refinement. This article presents background information relevant to the development of such a program (the Center for Advancement of Youth). The cultural and organizational contexts for the project are discussed, with particular emphasis on models for cooperation among several institutions of varying size and scope. The implications for the future of tangible research in this area are also discussed, with attention to extending lessons learned to diverse settings motivated to integrate various aspects of health care service provision.

Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001-2012

OBJECTIVE

Empirical study of public behavioral health systems' use of data and their investment in evidence-based treatments (EBTs) is limited. This study describes trends in state-level EBT investment and research supports from 2001 to 2012.

METHODS

Data were from National Association for State Mental Health Program Directors Research Institute (NRI) surveys, which were completed by representatives of state mental health authorities (SMHAs). Multilevel models examined change over time related to state adoption of EBTs, numbers of clients served, and penetration rates for six behavioral health EBTs for adults and children: supported housing, supported employment, assertive community treatment, therapeutic foster care, multisystemic therapy, and functional family therapy. State supports related to research, evaluation, and information management were also examined.

RESULTS

Increasing percentages of states reported funding an external research center, promoting the adoption of EBTs through provider contracts, and providing financial incentives for EBTs. Decreasing percentages of states reported promoting EBT adoption through stakeholder mobilization, monitoring fidelity, and specific budget requests. There was greater reported use of adult-focused EBTs (65%-80%) compared with youth-focused EBTs (25%-50%). Overall penetration rates of EBTs were low (1%-3%) and EBT adoption by states showed flat or declining trends. SMHAs' investment in data systems and use of research showed little change.

CONCLUSIONS

SMHA investment in EBTs, implementation infrastructure, and use of research has declined. More systematic measurement and examination of these metrics may provide a useful approach for setting priorities, evaluating success of health reform efforts, and making future investments.

Fostering SMART partnerships to develop an effective continuum of behavioral health services and supports in schools

The education sector offers compelling opportunities to address the shortcomings of traditional mental health delivery systems and to prevent and treat youth mental, emotional, and behavioral (MEB) problems. Recognizing that social and emotional wellness is intrinsically related to academic success, schools are moving to adopt multi-tier frameworks based on the public health model that provide a continuum of services to all children, including services to address both academic and MEB problems. In this article, we review the potential value of multi-tier frameworks in facilitating access to, and increasing the effectiveness of, mental health services in schools, and review the empirical support for school-based mental health interventions by tier. We go on to describe a community-academic partnership between the Seattle Public Schools and the University of Washington School Mental Health Assessment, Research, and Training (SMART) Center that exemplifies how multi-tier educational frameworks, research and evidence, and purposeful collaboration can combine to improve development and implementation of a range of school-based strategies focused on MEB needs of students. Finally, we present a set of 10 recommendations that may help guide other research and practice improvement efforts to address MEB problems in youth through effective school mental health programming. (PsycINFO Database Record

Student Perceptions of the Acceptability and Utility of Standardized and Idiographic Assessment in School Mental Health

Evidence-based assessment (EBA) comprises the use of research and theory to select methods and processes that have demonstrated reliability, validity, and clinical usefulness for prescribed populations. EBA can lead to positive clinical change, and recent work has suggested that it is perceived to be useful by school mental health providers. However, virtually nothing is known about student perceptions of assessment use. Semi-structured interviews were conducted with 31 ethnically diverse middle and high school students (71% female) receiving mental health services in school-based health centers. Findings indicated that the majority of students found assessments to be useful, and perceived three primary functions of assessments: structuring the therapy session, increasing students' self-awareness, and improving communication with the provider. Barriers to acceptability were also found for a minority of respondents. Some students found the nature of standardized assessments to be confining, and others expressed that they wanted more feedback from their counselors about their responses. Idiographic assessments demonstrated especially high acceptability in this sample, with students reporting that tracking idiographic outcomes increased self-awareness, spurred problem-solving, and helped them to reach behavioral goals. Implications for school mental health service improvements are discussed.

Using health information technology to improve adherence to opioid prescribing guidelines in primary care

Objective:

To evaluate the impact of a clinical dashboard for opioid analgesic management on opioid prescribing and adherence to opioid practice guidelines in primary care.

Methods:

A pre/postimplementation evaluation using electronic health record (EHR) data from patients receiving chronic opioid therapy (COT) between April 1, 2011 and March 31, 2013. Measures include annual proportions of COT patients who received urine drug testing, signed an opioid treatment agreement, had a documented assessment of pain-related functional status, and had at least 1 visit with a behavioral health provider.

Results:

Adherence to several opioid prescribing guidelines improved in the postimplementation year compared with the preimplementation year: (1) the proportions of COT patients with a signed opioid treatment agreement and urine drug testing increased from 49% to 63% and 66% to 86%, respectively. The proportion of COT patients with a documented assessment of functional status increased from 33% to 46% and those with a behavioral health visit increased from 24% to 28%. However, there was a small decline in the proportion of patients prescribed COT from 3.4% to 3.1%.

Discussion:

Implementation of an opioid dashboard led to increased adherence to certain opioid practice guidelines and a decline in COT. This may be attributable to more efficient team-based pain management facilitated by the dashboard and increased transparency of opioid prescription practices. Health Information Technology solutions such as clinical dashboards can increase adherence to practice guidelines.

The Old Solutions Are the New Problem: How Do We Better Use What We Already Know About Reducing the Burden of Mental Illness?

Kazdin and Blase (2011) propose that traditional models of delivering therapy require more resources than are available to address the scope of mental illness. We argue that finding new platforms and avenues for our existing treatments is a good start but that it is not enough. We contend that the field also needs to develop formal strategies to reorganize its increasing abundance of knowledge to address the scarcity of resources for its application. If we can better utilize our existing knowledge, treatment delivery and service resource allocation can become more efficient and effective. If the field continues with its almost singular emphasis on knowledge proliferation (e.g., developing new treatments), as opposed to knowledge management (e.g., developing new ways to design, apply, and organize existing treatments), the problem outlined by Kazdin and Blase cannot be solved.

Monitoring Client Progress and Feedback in School-Based Mental Health

Research in children's mental health has suggested that emotional and behavioral problems in are inextricably tied to academic difficulties. However, evidence-based programs implemented in school-based mental health tend to focus primarily on treatment practices, with less explicit emphasis on components of evidence-based assessment (EBA), such as progress monitoring and feedback. The current paper describes two studies that incorporated standardized assessment and progress monitoring/feedback into school-based mental health programs. Barriers to implementation are identified, recommendations for clinicians implementing EBA in the school setting are provided, and examples of mental health and academic indicators are discussed.

Clinician use of standardized assessments following a common elements psychotherapy training and consultation program

Despite increasing emphasis on the implementation of evidence-based treatments in community service settings, little attention has been paid to supporting the use of evidence-based assessment (EBA) methods and processes, a parallel component of evidence-based practice. Standardized assessment (SA) tools represent a key aspect of EBA and are central to data-driven clinical decision making. The current study evaluated the impact of a statewide training and consultation program in a common elements approach to psychotherapy. Practitioner attitudes toward, skill applying, and use of SA tools across four time points (pre-training, post-training, post-consultation, and follow-up) were assessed. Results indicated early increases in positive SA attitudes, with more gradual increases in self-reported SA skill and use. Implications for supporting the sustained use of SA tools are discussed, including the use of measurement feedback systems, reminders, and SA-supportive supervision practices.

A case for modular design: Implications for implementing evidence-based interventions with culturally-diverse youth

Community-based therapists are frequently faced with the complex task of applying existing research knowledge to clients who may differ markedly from those enrolled in controlled outcome studies. The current paper examines the utility of modular psychotherapy design as one method of facilitating the flexible delivery of evidence-based mental health services to ethnically and culturally diverse children and families. Modularity complements existing approaches to the provision of culturally-sensitive, empirically-informed treatment through its ability to balance the prioritization of research evidence and local practitioner cultural knowledge. Specific applications of modular principles to clinical work with diverse youth are highlighted. Special considerations and limitations relevant to modular psychotherapy and the overall mental health services research literature are discussed, as well as the continued importance of individual clinicians' cultural competence and use of treatment progress monitoring, both of which should be combined with identified treatment modules to support the delivery of high-quality care.

Family voice with informed choice: coordinating wraparound with research-based treatment for children and adolescents

The wraparound process is a type of individualized, team-based care coordination that has become central to many state and system efforts to reform children's mental health service delivery for youths with the most complex needs and their families. Although the emerging wraparound research base is generally positive regarding placements and costs, effect sizes are smaller for clinical and functional outcomes. This article presents a review of literature on care coordination and wraparound models, with a focus on theory and research that indicates the need to better connect wraparound-enrolled children and adolescents to evidence-based treatment (EBT). The article goes on to describe how recently developed applications of EBT that are based on quality improvement and flexible application of "common elements" of research-based care may provide a more individualized approach that better aligns with the philosophy and procedures of the wraparound process. Finally, this article presents preliminary studies that show the feasibility and potential effectiveness of coordinating wraparound with the Managing and Adapting Practice system, and discusses intervention development and research options that are currently under way.

Validity of Clinician's Self-Reported Practice Elements on the Monthly Treatment and Progress Summary

An increased demand for accountability in community mental health systems has resulted in a need for valid, reliable measures of therapeutic practice. The Monthly Treatment and Progress Summary (MTPS), developed through the Hawaii Child and Adolescent Mental Health Division, is a clinician-report measure that describes therapeutic practices, treatment targets, and progress ratings for each treatment case. The current study evaluated the validity of the therapeutic strategies reported on the MTPS by comparing coder- and clinician-reported use of practices. Using 47 audio recordings from 19 youths' therapy sessions, trained observers reliably coded 12 discrete practices. Four of the 12 practices were found to be valid according to clinician-coder agreement (intraclass correlations ≥0.60). The coding system was revised, utilizing extensiveness and experiential scales, and 100 % of practices were valid according to clinician-coder agreement. Practical use of the MTPS, implications for service systems, and future directions for research on the MTPS are discussed.

From distal to proximal: Routine educational data monitoring in school-based mental health

Research and practice in school-based mental health (SBMH) typically includes educational variables only as distal outcomes, resulting from improvements in mental health symptoms rather than directly from mental health intervention. Although sometimes appropriate, this approach also has the potential to inhibit the integration of mental health and schools. The current paper applies an existing model of data-driven decision making (Daleiden & Chorpita, 2005) to detail how SBMH can better integrate routine monitoring of school and academic outcomes into four evidence bases: general services research evidence, case histories, local aggregate, and causal mechanisms. The importance of developing new consultation protocols specific to data-driven decision making in SBMH as well as supportive infrastructure (e.g., measurement feedback systems) to support the collection and use of educational data is also described.

Structuring the collaboration of science and service in pursuit of a shared vision

The enduring needs of our society highlight the importance of a shared vision to improve human functioning and yield better lives for families and communities. Science offers a powerful strategy for managing the inevitable uncertainty in pursuit of these goals. This article presents ideas and examples of methods that could preserve the strengths of the two major paradigms in children's mental health, evidence-based treatments and individualized care models, but that also have the potential to extend their applicability and impact. As exemplified in some of the articles throughout this issue, new models to connect science and service will likely emerge from novel consideration of better ways to structure and inform collaboration within mental health systems. We contend that the future models for effective systems will involve increased attention to (a) client and provider developmental pathways, (b) explicit frameworks for coordinating people and the knowledge and other resources they use, and (c) a balance of evidence-based planning and informed adaptation. We encourage the diverse community of scientists, providers, and administrators in our field to come together to enhance our collective wisdom through consideration of and reflection on these concepts and their illustrations.

Engineering youth service system infrastructure: Hawaii's continued efforts at large-scale implementation through knowledge management strategies

Hawaii's Child and Adolescent Mental Health Division provides a unique illustration of a youth public mental health system with a long and successful history of large-scale quality improvement initiatives. Many advances are linked to flexibly organizing and applying knowledge gained from the scientific literature and move beyond installing a limited number of brand-named treatment approaches that might be directly relevant only to a small handful of system youth. This article takes a knowledge-to-action perspective and outlines five knowledge management strategies currently under way in Hawaii. Each strategy represents one component of a larger coordinated effort at engineering a service system focused on delivering both brand-named treatment approaches and complimentary strategies informed by the evidence base. The five knowledge management examples are (a) a set of modular-based professional training activities for currently practicing therapists, (b) an outreach initiative for supporting youth evidence-based practices training at Hawaii's mental health-related professional programs, (c) an effort to increase consumer knowledge of and demand for youth evidence-based practices, (d) a practice and progress agency performance feedback system, and (e) a sampling of system-level research studies focused on understanding treatment as usual. We end by outlining a small set of lessons learned and a longer term vision for embedding these efforts into the system's infrastructure.

Evidence base update for psychosocial treatments for pediatric obsessive-compulsive disorder

Pediatric obsessive-compulsive disorder (OCD) is a chronic and impairing condition that often persists into adulthood. Barrett, Farrell, Pina, Peris, and Piacentini (2008), in this journal, provided a detailed review of evidence-based psychosocial treatments for youth with OCD. The current review provides an evidence base update of the pediatric OCD psychosocial treatment literature with particular attention to advances in the field as well as to the methodological challenges inherent in evaluating such findings. Psychosocial treatment studies conducted since the last review are described and evaluated according to methodological rigor and evidence-based classification using the JCCAP evidence-based treatment evaluation criteria (Southam-Gerow & Prinstein, this issue). Findings from this review clearly converge in support of cognitive-behavioral therapy as an effective and appropriate first line treatment for youth with OCD (either alone or in combination with medication). Although no treatment for pediatric OCD has yet to be designated as "well-established," both individual and individual family-based treatments have been shown to be "probably efficacious." Moderators and predictors of treatment outcome are discussed as are the areas where we have advanced the field and the areas where we have room to grow. The methodological and clinical challenges inherent in a review of the evidence base are reviewed. Finally, future research directions are outlined.

Advances in Applying Treatment Integrity Research for Dissemination and Implementation Science: Introduction to Special Issue

This special series focuses upon the ways in which research on treatment integrity, a multidimensional construct including assessment of the content and quality of a psychosocial treatment delivered to a client as well as relational elements, can inform dissemination and implementation science. The five articles for this special series illustrate how treatment integrity concepts and methods can be applied across different levels of the mental health service system to advance dissemination and implementation science. In this introductory article, we provide an overview of treatment integrity research and describe three broad conceptual models that are relevant to the articles in the series. We conclude with a brief description of each of the five articles in the series.

Beyond context to the skyline: thinking in 3D

Sweeping and profound structural, regulatory, and fiscal changes are rapidly reshaping the contours of health and mental health practice. The community-based practice contexts described in the excellent review by Garland and colleagues are being fundamentally altered with different business models, regional networks, accountability standards, and incentive structures. If community-based mental health services are to remain viable, the two-dimensional and flat research and practice paradigm has to be replaced with three-dimensional thinking. Failure to take seriously the changes that are happening to the larger healthcare context and respond actively through significant system redesign will lead to the demise of specialty mental health services.

Improving community-based mental health care for children: translating knowledge into action

There is urgent need for improvement in community-based mental health care for children and families. Multiple studies have documented serious limitations in the effectiveness of "usual care." Fortunately, many empirically-supported strategies to improve care have been developed, and thus there is now a great deal of knowledge available to address this significant public health problem. The goal of this selective review is to highlight and synthesize that empirically-supported knowledge to stimulate and facilitate the needed translation of knowledge into action. The review provides a sound foundation for constructing improved services by consolidating descriptive data on the status quo in children's mental health care, as well as evidence for an array of promising strategies to improve (a) Service access and engagement; (b) Delivery of evidence-based practices; and (c) Outcome accountability. A multi-level framework is used to highlight recommended care improvement targets.

Using agency data for evidence-based programming: a university-agency collaboration

Agencies providing residential treatment are encouraged, or even mandated, to collect outcomes data and to implement evidence based practices, yet little guidance has been provided on how to do so using agency administrative data that are collected on an ongoing basis. We examined data on Child and Adolescent Functional Assessment Scale (CAFAS) scores for 1608 admissions to a residential treatment center from 2002 through 2008. CAFAS scores were measured every 90 days, providing multiple CAFAS scores for each individual. Results demonstrated that on average residents improved in functioning over time. Sensitive to the evolving needs of residents who had been entering the program, the treatment center made significant program changes in 2006 to attempt to better serve residents through a broad array of specialized programming. Compared to the overall results, the analysis suggested that residents who entered the program since October 2006 appeared to have made larger improvements in their CAFAS scores. Results were derived by employing multilevel models appropriate for estimating growth trajectories with repeated measures data. Conversations with agency staff suggested that using administrative data, and advanced statistical models, were extremely helpful for organizational decision making and evidence-based programming.

Developing the mental health workforce: review and application of training approaches from multiple disciplines

Strategies specifically designed to facilitate the training of mental health practitioners in evidence-based practices (EBPs) have lagged behind the development of the interventions themselves. The current paper draws from an interdisciplinary literature (including medical training, adult education, and teacher training) to identify useful training and support approaches as well as important conceptual frameworks that may be applied to training in mental health. Theory and research findings are reviewed, which highlight the importance of continued consultation/support following training workshops, congruence between the training content and practitioner experience, and focus on motivational issues. In addition, six individual approaches are presented with careful attention to their empirical foundations and potential applications. Common techniques are highlighted and applications and future directions for mental health workforce training and research are discussed.