Theories related to changing clinician practice

The Connections Matter: Bi-Directional Learning in Program Evaluation and Practice-Oriented Research

Intended for researchers and clinical leaders, this article suggests that embedded program evaluation is a good fit with the desired features of practice-oriented research. The systematic nature of evaluation that is built into the operational workflow of a practice setting may increase the diversity of methods available to explore processes and outcomes of interest. We propose a novel conceptual framework that uses a human-centered systems lens to foster such embedded evaluation in clinical routine. This approach emphasizes the evaluator-practitioner partnership to build confidence in the bi-directional learning of practice-based evidence with evidence-based practice. The iterative cycles inherent to design thinking are aimed at developing better evaluation questions. The attention to structure and context inherent to systems thinking is intended to support meaningful perspectives in the naturally complex world of health care. Importantly, the combined human-centered systems lens can create greater awareness of the influence of individual and systemic biases that exist in any endeavor or institution that involves people. Recommended tools and strategies include systems mapping, program theory development, and visual facilitation using a logic model to represent the complexity of mental health treatment for communication, shared understanding, and connection to the broader evidence base. To illustrate elements of the proposed conceptual framework, two case examples are drawn from routine outcome monitoring (ROM) and progress feedback. We conclude with questions for future collaboration and research that may strengthen the partnership of evaluators and practitioners as a community of learners in service of local and system-level improvement.

Can Measurement-Based Care Reduce Burnout in Mental Health Clinicians?

Burnout is a syndrome characterized by mental and emotional fatigue or exhaustion, depersonalization, and a lessened sense of personal accomplishment and efficacy. Burnout leads to negative consequences for mental health clinicians and for mental health care organizations. Measurement-based care (MBC) is a clinical process in which clinicians and clients use patient-generated data, also called treatment feedback, to collaboratively monitor mental health care and to inform goal-setting and treatment planning. We propose that MBC may improve the experience of care for both clients and clinicians, and ultimately protect against each of the three components of burnout. When combined with other organizational changes, adoption of MBC may support organizational level efforts to reduce burnout in mental health services.

Therapist use of cognitive behavior therapy and eye movement desensitization and reprocessing components for the treatment of posttraumatic stress disorder in practice settings

Introduction

Treatment practice guidelines for posttraumatic stress disorder (PTSD) recommend both Eye Movement Desensitization and Reprocessing (EMDR) and Cognitive Behavior Therapy (CBT); however, implementation in practice setting remains challenging. Here we aim to foster implementation efforts for PTSD by identifying the relative use of the various components of empirically supported treatments by therapists and the characteristics that predict their use.

Methods

Surveyed 346 therapists (84.07% female) of whom 272 participants (78.61%) were trained primarily in CBT and 135 participants (39.02%) were trained in primarily in EMDR. Assessed relative use of various EMDR and CBT components as well as several training and personality factors.

Results

Psychoeducation about trauma was the most common element used. "Off label" use of components was also identified with application of EMDR techniques to other diagnoses. Findings also suggest underutilization of in vivo exposure techniques across therapists. EMDR therapists reported relatively high use of core EMDR techniques (i.e., greater use of EMDR core techniques). Big five personality factors, therapy efficacy, and anxiety were associated with differential component use.

Discussion

Results identify trends in empirically supported component use and therapist characteristics that are associated with the use of various techniques for PTSD. The findings suggest implementation efforts could foster training in underused techniques, address barriers to their utilization and develop knowledge of effective packages of components.

Taking a Magnifying Glass to Measurement-Based Care Consultation Sessions: with What Issues Do Mental Health Clinicians Struggle?

PURPOSE

Ongoing consultation following initial training is one of the most commonly deployed implementation strategies to facilitate uptake of evidence-based practices, such as measurement-based care (MBC). Group consultation provides an interactive experience with an expert and colleagues to get feedback on actual issues faced, yet there is little research that unpacks the questions raised in consultation and what types of issues are important to address.

METHODS

The current study characterized the questions and concerns raised by community mental health clinicians (N = 38 across six clinics) during group consultation sessions completed as part of an MBC implementation trial. We conducted a qualitative content analysis of consultation forms completed by clinicians before each MBC consultation session.

RESULTS

Clinicians sought MBC consultation for clients across a range of ages and levels of depression severity. Qualitative results revealed five main questions and concerns in consultation sessions: (1) how to administer the PHQ-9, (2) how to review PHQ-9 scores, (3) how to respond to PHQ-9 score, (4) the types of clients for whom MBC would be appropriate, and (5) how MBC could impact a clinician's usual care.

CONCLUSION

Findings highlight the need for ongoing consultation and limitations of workshop training alone. Practical recommendations for addressing the common questions and concerns identified are presented to support MBC use.

The results of clinician-focused implementation strategies on uptake and outcomes of Measurement-Based Care (MBC) in general mental health care

BACKGROUND

Measurement-Based Care (MBC) is the routine administration of measures, clinicians' review of the feedback and discussion of the feedback with their clients, and collaborative evaluation of the treatment plan. Although MBC is a promising way to improve outcomes in clinical practice, the implementation of MBC faces many barriers, and its uptake by clinicians is low. The purpose of this study was to investigate whether implementation strategies that were developed with clinicians and aimed at clinicians had an effect on (a) clinicians' uptake of MBC and (b) clients' outcomes of MBC.

METHODS

We used an effectiveness-implementation hybrid design based on Grol and Wensing's implementation framework to assess the impact of clinician-focused implementation strategies on both clinicians' uptake of MBC and outcomes obtained with MBC for clients in general mental health care. We hereby focused on the first and second parts of MBC, i.e., the administration of measures and use of feedback. Primary outcome measures were questionnaire completion rate and discussion of the feedback with clients. Secondary outcomes were treatment outcome, treatment length, and satisfaction with treatment.

RESULTS

There was a significant effect of the MBC implementation strategies on questionnaire completion rate (one part of clinicians' uptake), but no significant effect on the amount of discussion of the feedback (the other part of clinicians' uptake). Neither was there a significant effect on clients' outcomes (treatment outcome, treatment length, and satisfaction with treatment). Due to various study limitations, the results should be viewed as exploratory.

CONCLUSIONS

Establishing and sustaining MBC in real-world general mental health care is complex. This study helps to disentangle the effects of MBC implementation strategies on differential clinician uptake, but the effects of MBC implementation strategies on client outcomes need further examination.

Experiences and results from using a novel clinical feedback system in routine stoma care nurse follow-up of patients with an ostomy: a longitudinal study

BACKGROUND

A faecal or urinary ostomy may be lifesaving. However, it involves significant bodily change, and the adjustment process to life with an ostomy includes a broad spectre of physical and psychosocial challenges. Thus, new interventions are needed to improve adaptation to living with an ostomy. This study aimed to examine experiences and outcomes using a new clinical feedback system with patient-reported outcome measures in ostomy care.

METHODS

In this longitudinal explorative study, 69 ostomy patients were followed by a stoma care nurse in an outpatient clinic, using a clinical feedback system postoperatively at 3, 6 and 12 months. The patients responded electronically to the questionnaires before each consultation. The Generic Short Patient Experiences Questionnaire was used to measure patient experiences and satisfaction with follow-up. The Ostomy Adjustment Scale (OAS) measured adjustment to life with an ostomy, and the Short Form-36 (SF-36) assessed the patient's health-related quality of life. Longitudinal regression models with time as an explanatory (categorical) variable were used to analyse changes. The STROBE guideline was applied.

RESULTS

The patients were satisfied with their follow-up (96%). Especially, they felt they received sufficient and individualised information, were involved in treatment decisions, and benefited from the consultations. The OAS subscale scores for 'daily activities', 'knowledge and skills' and 'health' improved over time (all p < 0.05), as did the physical and mental component summary scores of the SF-36 (all p < 0.05). Effect sizes of changes were small (0.20-0.40). Sexuality was the most challenging factor reported.

CONCLUSIONS

The clinical feedback system could be helpful because outpatient follow-ups for ostomy patients may be more tailored when clinicians use clinical feedback systems. However, further development and testing are needed.

Characterizing measurement-based care implementation using therapist report

The effectiveness of measurement-based care (MBC), an evidence-based practice that uses regularly collected assessment data to guide clinical decision-making, is impacted by whether and how therapists use information from MBC tools in treatment. Improved characterization of how therapists use MBC in treatment sessions with youth is needed to guide implementation and understand variability in MBC effectiveness. To meet this need, this study examined therapists' sharing and discussion of MBC in treatment sessions. Thirty therapists were randomly assigned to the MBC condition as part of a comparative effectiveness trail of treatments for adolescent anxiety and depression. A qualitative content analysis was conducted on therapists' written explanations of changes made to the session based on the MBC data. Therapists reported sharing data with youth and caregivers in an average of 34.6% and 27.4% of sessions, respectively. Therapists reported incorporating MBC data in an average of 21.1% of sessions. When data were used, therapists predominately focused changes on short-term (e.g., current symptoms, treatment skill) rather than long-term (e.g., symptom progress, treatment goals) decision-making. Therapists inconsistently used MBC data, highlighting the need for improved training in and monitoring of how therapists use MBC in session to guide collaborative treatment decision-making with youth and caregivers.

Youth Top Problems and Early Treatment Response to the Unified Protocols for Transdiagnostic Treatment of Emotional Disorders in Children and Adolescents

The Top Problems assessment is an idiographic measure of client concerns that may allow clinicians to identify early treatment response. Few studies have examined early response to evidence-based therapies using Top Problems. We collected weekly Top Problems ratings from 95 youth with emotional disorders who received treatment using the Unified Protocols for Transdiagnostic Treatment of Emotional Disorders in Children and Adolescents (UP-C/UP-A). We assessed Top Problems rating change from session 1 to session 4, the role of pre-treatment variables in early Top Problems rating change, and the role of early Top Problems rating change in post-treatment symptom outcomes. Top Problems ratings decreased significantly from session 1 to session 4. Younger child age and higher parent cognitive flexibility were associated with early Top Problems improvement. Controlling for pre-treatment, early Top Problems rating change did not explain the variance in post-treatment outcomes. Future research should examine Top Problems trajectories over treatment course.

Inside the Clockwork of the ECHO Factorial Trial: A Conceptual Model With Proposed Mediators for Prevention of Emotional Problems in Children

Having interventions that are not only evidence-based and effective but also cost-effective and efficient is important for the prevention and treatment of child and adolescent emotional problems. A randomized clinical trial (RCT) tests the total interventions effect but does not address specific components of the intervention. In this article the hypothesis and a conceptual model of the ECHO study are presented and discussed. The ECHO intervention consists of three different components each containing two levels of intervention. By using a cluster randomized factorial design, children aged 8-12 at 40 schools across Norway will be randomized to eight different experimental conditions investigating the optimal balance between effect, cost-effectiveness, and efficiency. The article presents the design and the different components being tested and discusses how optimalization can be reached through this innovative design. The article also discusses how interventions can be improved by investigating and understanding the mechanisms of change within psychological interventions. For each of the three components in the study we consider the mediators that could be active within the intervention and how the study investigates such mediation. The results will contribute to a better understanding of how psychological interventions work and how we intend to optimize the EMOTION intervention.

Measurement-based care as a practice improvement tool: Clinical and organizational applications in youth mental health

Measurement-based care (MBC) is systematic, ongoing assessment to monitor treatment progress and inform clinical decision-making. MBC is considered an evidence-based practice, with extensive research support in adult clinical populations and emerging evidence in youth populations. This paper describes both clinical (e.g., enhancing therapy alliance, informing case conceptualization) and organizational (e.g., informing quality improvement efforts) applications of MBC, and illustrates the clinical applications through two case examples from publicly-funded mental health agency settings. The paper concludes by detailing future research needed to enhance the clinical and organizational utility of MBC.

Pediatric Consultation-Liaison: Patient Characteristics and Considerations for Training in Evidence-Based Practices

Consultation-liaison services are an integral part of many pediatric hospital settings, yet characteristics of this patient population have not been extensively documented. The current study is a retrospective one-year chart review of the consultation-liaison service at a large pediatric hospital in the Southwestern United States. The purpose of this study is twofold: (1) to characterize this hospital's CL population and (2) to use these characteristics to identify preliminary evidence-based practices that should be considered for CL provider training. Identifying evidence-based practice elements that align with the characteristics of consultation-liaison patient populations may inform trainings for consultation-liaison staff. This would help to ensure that youth seen in hospital consultation-liaison services are getting the best available services, which is critical given the shortened time frame available to work with this patient population.

Effects of Fidelity-Focused Consultation on Clinicians' Implementation: An Exploratory Multiple Baseline Design

Identification of effective consultation models could inform implementation efforts. This study examined the effects of a fidelity-focused consultation model among community-based clinicians implementing Attachment and Biobehavioral Catch-up. Fidelity data from 1217 sessions from 7 clinicians were examined in a multiple baseline design. In fidelity-focused consultation, clinicians received feedback from consultants' fidelity coding, and also coded their own fidelity. Clinicians' fidelity increased after fidelity-focused consultation began, but did not increase during other training periods. Fidelity was sustained for 30 months after consultation ended. Findings suggest that consultation procedures involving fidelity coding feedback and self-monitoring of fidelity may promote implementation outcomes.

An Innovator and a Disruptor: Leonard Bickman on Program Theory, Null Findings, and Advice to Future Child Mental Health Services Researchers

With a career spanning more than half a century, Leonard Bickman has contributed to improving children's mental health through research on innovative interventions, such as measurement-based care, and service-level initiatives, including systems of care. Len's highly productive career in children's mental health services research is founded in his influence on the science of program evaluation, particularly in the area of program theory. This article provides an abridged and combined version of a video interview with Len dated April 16, 2019 and written responses to a series of questions posed to Len in advance of the festschrift held in his honor at Johns Hopkins Bloomberg School of Public Health on May 11, 2018.

A new model of patient-reported outcome monitoring with a clinical feedback system in ostomy care: rationale, description and evaluation protocol

BACKGROUND

Living with an ostomy can be challenging and adapting to life with an ostomy can be particularly complex, with regard to both the physical and psychosocial aspects. Follow-up with a stoma care nurse (SCN) is usually performed after surgery to support the adaptation process. In the present paper, we describe a new model of ostomy care, where a clinical feedback system (CFS) is implemented in order to improve the adaption process of patients with an ostomy. We also present a plan for evaluating patients experience with the CFS and their clinical outcomes.

METHODS

In this study, we include patients who had recently performed colostomy, ileostomy, or urostomy surgery. The intervention includes self-reported measures for adaptation to life with an ostomy and health-related quality of life (HRQoL), as well as patient experiences and satisfaction recorded by the clinical feedback system. The measures are electronically assessed before each clinical consultation at 3, 6, and 12 months after surgery. The scores are instantly analysed and graphically presented for use during the consultation and the patient and the SCN can discuss the findings. Patient experiences and satisfaction with care will be measured with the Generic Short Patient Experiences Questionnaire. Adaptation to the life with ostomy will be measured with the Ostomy Adjustment Scale, and HRQoL with the Short Form 36.

DISCUSSION

This study presents a novel approach that could lead to improved consultation, more patient involvement, and better adaptation to life with an ostomy.

TRIAL REGISTER

ClinicalTrials.gov Registration Number: NCT03841071. Date 18. February 2019 retrospectively registered.

Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET): A comparative effectiveness trial of a transdiagnostic treatment and a measurement feedback system

Emotional disorders, encompassing a range of anxiety and depressive disorders, are the most prevalent and comorbid psychiatric disorders in adolescence. Unfortunately, evidence-based psychosocial therapies typically focus on single disorders, are rarely adopted by community mental health center clinicians, and effect sizes are modest. This article describes the protocol for a comparative effectiveness study of two novel interventions designed to address these challenges. The first intervention is a transdiagnostic treatment (the Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents, UP-A), a promising new approach that uses a small number of common strategies to treat a broad range of emotional disorders, and their underlying shared emotional vulnerabilities. The second intervention is a standardized measurement feedback system, the Youth Outcomes Questionnaire (YOQ), designed to improve clinical decision making using weekly symptom and relational data. The three study arms are treatment as usual (TAU), TAU plus the YOQ (TAU+), and UP-A (used in combination with the YOQ). The primary aims of the study are to [1] compare the effects of the UP-A and TAU+ to TAU in community mental health clinics, [2] to isolate the effects of measurement and feedback by comparing the UP-A and TAU+ condition, and [3] to examine the mechanisms of action of both interventions. Design considerations and study methods are provided to inform future effectiveness research.

Routine Outcome Monitoring in CAMHS: How Can We Enable Implementation in Practice?

BACKGROUND

Many CAMHS teams across the UK are now required to use Routine Outcome Monitoring (ROM). However, some clinicians hold negative attitudes towards ROM and various practical implementation issues have been identified.

METHOD

The aim of this study was to explore clinician experiences of using ROM in the context of an 'enforced' initial implementation initiative. Twenty clinicians were surveyed at the beginning (T1) and end (T2) of the six-month period in a large UK CAMHS network adopting CYP-IAPT practice. Changes in the use of and attitudes towards ROM were investigated, as well as barriers to implementation.

RESULTS

Overall, a small but significant increase in clinician use of ROM was observed from T1 to T2, but attitudes towards ROM did not change significantly. On the whole, clinicians were more positive than negative about ROM during the implementation period, but key implementation challenges included clinician concerns about the value and (mis)use of ROM data, poor technological support and additional workload demands.

CONCLUSIONS

CAMHS clinicians will not necessarily become more receptive to ROM simply as a by-product of being asked to use it more. Seeking clinician feedback at the early stages of implementation can help provide a focus for improvement efforts. Ideas for future research and important limitations of the study are discussed.

The Expanding Relevance of Routinely Collected Outcome Data for Mental Health Care Decision Making

Evidence shows that routine outcome monitoring (ROM) and feedback using standardized measurement tools enhances the outcomes of individual patients. When outcome data from a large number of patients and clinicians are collected, patterns can be tracked and comparisons can be made at multiple levels. Variability in skills and outcomes among clinicians and service settings has been documented, and the relevance of ROM for decision making is rapidly expanding alongside the transforming health care landscape. In this article, we highlight several developing core implications of ROM for mental health care, and frame points of future work and discussion.

Practitioner Review: Empirical evolution of youth psychotherapy toward transdiagnostic approaches

BACKGROUND

Psychotherapy for children and adolescents (herein, 'youths') has grown more precise and focused over the decades, shifting toward empirically supported treatments standardized via therapist manuals. The manuals have increasingly emphasized precise targeting of single disorders or problems, or homogenous clusters. These focal treatments represent a valuable advance, with intervention benefit documented in hundreds of studies. However, relatively few of these treatments are widely used in everyday clinical practice, and their level of benefit may not be ideal, particularly in practice contexts and when compared to usual clinical care.

THESIS

These limitations may be due, in part, to a mismatch between focal treatment design and the young people treated in real-world clinical care, who are diagnostically heterogeneous, and very frequently present with comorbidity. Improved fit may be achieved via transdiagnostic treatment approaches designed to address multiple disorders and problems, if these approaches can retain the benefits of manualization and the substantive clinical strength that has generated empirical support to date.

SCOPE

Here we review the evolution of empirically-based youth psychotherapy from focal treatment manuals toward transdiagnostic approaches, and we describe and illustrate three transdiagnostic treatment strategies: (a) a core dysfunction approach, (b) a common elements and modular design approach, and (c) a principle-guided approach.

CLINICAL APPLICATIONS

We complement research findings with a clinical perspective based on our use of manual-guided transdiagnostic intervention in clinical care settings, and we propose directions for research and practice.

Maximizing Psychotherapy Outcome beyond Evidence-Based Medicine

Despite evidence that psychotherapy has a positive impact on psychological disorders, 30% of patients fail to respond during clinical trials, and as many as 65% of patients in routine care leave treatment without a measured benefit. In addition, therapists appear to overestimate positive outcomes in their patients relative to measured outcomes and are particularly poor at identifying patients at risk for a negative outcome. These problems suggest the need for measuring and monitoring patient treatment response over the course of treatment while applying standardized methods of identifying at-risk cases. Computer-assisted methods for measuring, monitoring, identifying potential deteriorators, and providing feedback to clinicians are described along with a model that explains why feedback is likely to be beneficial to patients. The results of 12 clinical trials are summarized and suggest that deterioration rates can be substantially reduced in at-risk cases (from baseline rates of 21% down to 13%) and that recovery rates are substantially increased in this subgroup of cases (from a baseline of 20% up to 35%) when therapists are provided this information. When problem-solving methods are added to feedback, deterioration in at-risk cases is further reduced to 6% while recovery/improvement rates rise to about 50%. It is suggested that the feedback methods become a standard of practice. Such a change in patterns of care can be achieved through minimal modification to routine practice but may require discussions with patients about their clinical progress.

An initial waitlist-controlled trial of the unified protocol for the treatment of emotional disorders in adolescents

A substantial proportion of adolescents are non-responders to well-established treatments for anxiety and depression, and many existent approaches do not adequately address comorbidity. There is a need to develop and evaluate unified treatments for adolescents that flexibly address higher order factors shared among internalizing or emotional disorders. The Unified Protocol for the Treatment of Emotional Disorders in Adolescents (UP-A) is a transdiagnostic treatment that targets shared vulnerability and maintenance factors in a flexible format. This study examined initial outcomes of a randomized, waitlist-controlled trial of the UP-A. The UP-A outperformed waitlist at mid-treatment with respect to disorder severity and functional impairment, and there was a significant treatment effect in favor of the UP-A on all outcome measures at post-treatment. Within-subjects analyses collapsing across participants revealed significant improvements on outcome measures over time. Results support further study of the UP-A and its potential efficacy in treating adolescent anxiety and depression.

Deriving Implementation Strategies for Outcome Monitoring Feedback from Theory, Research and Practice

Outcome monitoring feedback is a promising intervention to enhance outcomes of clinical practice. However, effective implementation can be tough and research suggests that feedback is not equally effective under all circumstances. In this article, feedback theory, research and experience from clinical practice is used to provide implementation strategies. Factors that moderate the effectiveness of outcome monitoring feedback, including feedback, recipient and organization characteristics are discussed. It is important to pay attention to implementation processes, such as providing sufficient training for clinicians, in order for feedback to be capable of enhancing outcomes.

Capabilities and Characteristics of Digital Measurement Feedback Systems: Results from a Comprehensive Review

Measurement feedback systems (MFS) are a class of health information technology (HIT) that function as an implementation support strategy for integrating measurement based care or routine outcome monitoring into clinical practice. Although many MFS have been developed, little is known about their functions. This paper reports findings from an application of health information technology-academic and commercial evaluation (HIT-ACE), a systematic and consolidated evaluation method, to MFS designed for use in behavioral healthcare settings. Forty-nine MFS were identified and subjected to systematic characteristic and capability coding. Results are presented with respect to the representation of characteristics and capabilities across MFS.

Implementing a Measurement Feedback System: A Tale of Two Sites

A randomized experiment was conducted in two outpatient clinics evaluating a measurement feedback system called contextualized feedback systems. The clinicians of 257 Youth 11-18 received feedback on progress in mental health symptoms and functioning either every 6 months or as soon as the youth's, clinician's or caregiver's data were entered into the system. The ITT analysis showed that only one of the two participating clinics (Clinic R) had an enhanced outcome because of feedback, and only for the clinicians' ratings of youth symptom severity on the SFSS. A dose-response effect was found only for Clinic R for both the client and clinician ratings. Implementation analyses showed that Clinic R had better implementation of the feedback intervention. Clinicians' questionnaire completion rate and feedback viewing at Clinic R were 50 % higher than clinicians at Clinic U. The discussion focused on the differences in implementation at each site and how these differences may have contributed to the different outcomes of the experiment.

Patient reported outcome measures in child and adolescent mental health services: associations between clinician demographic characteristics, attitudes and efficacy

BACKGROUND

Policy recommends using patient reported outcome measures (PROMs), yet their use is persistently low. Our aim was to examine the association between PROM use and clinician demographic characteristics, attitudes and efficacy.

METHOD

A sample of N = 109 clinicians completed an online survey.

RESULTS

Clinicians who reported higher levels of use of cognitive behaviour or humanistic approaches had higher levels of PROM use than clinicians who reported lower levels of use of these approaches. Clinicians who reported having received training had higher levels of self-efficacy regarding PROMs than clinicians who reported not having received training, but the effects of training on PROM attitudes and use were not significant. Still, clinicians with more positive attitudes or self-efficacy regarding PROMs had higher levels of PROM use than clinicians with less positive attitudes or self-efficacy regarding PROMs.

CONCLUSION

Clinicians should be supported to have the knowledge, skills and confidence to effectively use PROMs in their clinical practice.

Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems

Background

Health information technologies (HIT) have become nearly ubiquitous in the contemporary healthcare landscape, but information about HIT development, functionality, and implementation readiness is frequently siloed. Theory-driven methods of compiling, evaluating, and integrating information from the academic and commercial sectors are necessary to guide stakeholder decision-making surrounding HIT adoption and to develop pragmatic HIT research agendas. This article presents the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology, a structured, theory-driven method for compiling and evaluating information from multiple sectors. As an example demonstration of the methodology, we apply HIT-ACE to mental and behavioral health measurement feedback systems (MFS). MFS are a specific class of HIT that support the implementation of routine outcome monitoring, an evidence-based practice.

Results

HIT-ACE is guided by theories and frameworks related to user-centered design and implementation science. The methodology involves four phases: (1) coding academic and commercial materials, (2) developer/purveyor interviews, (3) linking putative implementation mechanisms to hit capabilities, and (4) experimental testing of capabilities and mechanisms. In the current demonstration, phase 1 included a systematic process to identify MFS in mental and behavioral health using academic literature and commercial websites. Using user-centered design, implementation science, and feedback frameworks, the HIT-ACE coding system was developed, piloted, and used to review each identified system for the presence of 38 capabilities and 18 additional characteristics via a consensus coding process. Bibliometic data were also collected to examine the representation of the systems in the scientific literature. As an example, results are presented for the application of HIT-ACE phase 1 to MFS wherein 49 separate MFS were identified, reflecting a diverse array of characteristics and capabilities.

Conclusions

Preliminary findings demonstrate the utility of HIT-ACE to represent the scope and diversity of a given class of HIT beyond what can be identified in the academic literature. Phase 2 data collection is expected to confirm and expand the information presented and phases 3 and 4 will provide more nuanced information about the impact of specific HIT capabilities. In all, HIT-ACE is expected to support adoption decisions and additional HIT development and implementation research.

The Effect of a Change Agent on Use of Evidence-Based Mental Health Practices

Children's service systems are faced with a critical need to disseminate evidence-based mental health interventions. Despite the proliferation of comprehensive implementation models, little is known about the key active processes in effective implementation strategies. This proof of concept study focused on the effect of change agent interactions as conceptualized by Rogers' diffusion of innovation theory on providers' (N = 57) use of a behavioral intervention in a child welfare agency. An experimental design compared use for providers randomized to training as usual or training as usual supplemented by change agent interactions after the training. Results indicate that the enhanced condition increased use of the intervention, supporting the positive effect of change agent interactions on use of new practices. Change agent types of interaction may be a key active process in implementation strategies following training.

Assessing mental health clinicians' intentions to adopt evidence-based treatments: reliability and validity testing of the evidence-based treatment intentions scale

Background

Intentions play a central role in numerous empirically supported theories of behavior and behavior change and have been identified as a potentially important antecedent to successful evidence-based treatment (EBT) implementation. Despite this, few measures of mental health clinicians’ EBT intentions exist and available measures have not been subject to thorough psychometric evaluation or testing. This paper evaluates the psychometric properties of the evidence-based treatment intentions (EBTI) scale, a new measure of mental health clinicians’ intentions to adopt EBTs.

Methods

The study evaluates the reliability and validity of inferences made with the EBTI using multi-method, multi-informant criterion variables collected over 12 months from a sample of 197 mental health clinicians delivering services in 13 mental health agencies. Structural, predictive, and discriminant validity evidence is assessed.

Results

Findings support the EBTI’s factor structure (χ² = 3.96, df = 5, p = .556) and internal consistency reliability (α = .80). Predictive validity evidence was provided by robust and significant associations between EBTI scores and clinicians’ observer-reported attendance at a voluntary EBT workshop at a 1-month follow-up (OR = 1.92, p < .05), self-reported EBT adoption at a 12-month follow-up (R² = .17, p < .001), and self-reported use of EBTs with clients at a 12-month follow-up (R² = .25, p < .001). Discriminant validity evidence was provided by small associations with clinicians’ concurrently measured psychological work climate perceptions of functionality (R² = .06, p < .05), engagement (R² = .06, p < .05), and stress (R² = .00, ns).

Conclusions

The EBTI is a practical and theoretically grounded measure of mental health clinicians’ EBT intentions. Scores on the EBTI provide a basis for valid inferences regarding mental health clinicians’ intentions to adopt EBTs. Discussion focuses on research and practice applications.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-016-0417-3) contains supplementary material, which is available to authorized users.

Feedback from Outcome Measures and Treatment Effectiveness, Treatment Efficiency, and Collaborative Practice: A Systematic Review

Due to recent increases in the use of feedback from outcome measures in mental health settings, we systematically reviewed evidence regarding the impact of feedback from outcome measures on treatment effectiveness, treatment efficiency, and collaborative practice. In over half of 32 studies reviewed, the feedback condition had significantly higher levels of treatment effectiveness on at least one treatment outcome variable. Feedback was particularly effective for not-on-track patients or when it was provided to both clinicians and patients. The findings for treatment efficiency and collaborative practice were less consistent. Given the heterogeneity of studies, more research is needed to determine when and for whom feedback is most effective.

Feedback mechanisms of change: How problem alerts reported by youth clients and their caregivers impact clinician-reported session content

OBJECTIVE

This study explored how clinician-reported content addressed in treatment sessions was predicted by clinician feedback group and multi-informant cumulative problem alerts that appeared in computerized feedback reports for 299 clients aged 11-18 years receiving home-based community mental health treatment.

METHOD

Measures included a clinician report of content addressed in sessions and additional measures of treatment progress and process (e.g., therapeutic alliance) completed by clinicians, clients, and their caregivers. Item responses in the top 25th percentile in severity from these measures appeared as "problem alerts" on corresponding computerized feedback reports. Clinicians randomized to the feedback group received feedback weekly while the control group did not. Analyses were conducted using the Cox proportional hazards regression for recurrent events.

RESULTS

For all content domains, the results of the survival analyses indicated a robust effect of the feedback group on addressing specific content in sessions, with feedback associated with shorter duration to first occurrence and increased likelihood of addressing or focusing on a topic compared to the non-feedback group.

CONCLUSION

There appears to be an important relationship between feedback and cumulative problem alerts reported by multiple informants as they influence session content.

Monitoring Client Progress and Feedback in School-Based Mental Health

Research in children's mental health has suggested that emotional and behavioral problems in are inextricably tied to academic difficulties. However, evidence-based programs implemented in school-based mental health tend to focus primarily on treatment practices, with less explicit emphasis on components of evidence-based assessment (EBA), such as progress monitoring and feedback. The current paper describes two studies that incorporated standardized assessment and progress monitoring/feedback into school-based mental health programs. Barriers to implementation are identified, recommendations for clinicians implementing EBA in the school setting are provided, and examples of mental health and academic indicators are discussed.

A systematic appraisal of allegiance effect in randomized controlled trials of psychotherapy

BACKGROUND

Experimenter's allegiance (EA) refers to a personal confidence of the superiority of a specific psychotherapy treatment. This factor has been linked with larger treatment effects in favor of the preferred treatment. However, various studies have displayed contradictory results between EA and the pattern of treatment effects.

AIMS

Using a systematic approach followed by meta-analysis, we aimed to evaluate the impact of an allegiance effect on the results of psychotherapeutic studies.

METHOD

We considered the meta-analyses of randomized controlled trials (RCTs) of different types of psychotherapies in the Cochrane Database of Systematic Reviews. Eligible articles included meta-analyses of RCTs with at least one study showing evidence of EA (i.e., allegiant study). Effect sizes in allegiant RCTs were compared with non-allegiant using random and fixed models and a summary relative odds ratio (ROR) were calculated. Heterogeneity was quantified with the I (2) metric.

RESULTS

A total of 30 meta-analyses including 240 RCTs were analyzed. The summary ROR was 1.31 [(95 % confidence interval (CI: 1.03-1.66) P = 0.30, I (2) = 53 %] indicating larger effects when allegiance exists. The impact of allegiance did not differ significantly (P > 0.05) when we compared psychiatric versus medical outcomes. Allegiance effect was significant for all forms of psychotherapy except for cognitive behavioral therapy. Moreover, the impact of allegiance was significant only when the treatment integrity of delivered psychotherapy was not assessed. Allegiance effect was even stronger where the experimenter was also both the developer of the preferred treatment and supervised or trained the therapists. No significant differences were found between allegiant and non-allegiant studies in terms of overall quality of studies.

CONCLUSIONS

Experimenter's allegiance influences the effect sizes of psychotherapy RCTs and can be considered non-financial conflict of interest introducing a form of optimism bias, especially since blinding is problematic in this kind of research. A clear reporting of EA in every single study should be given an opportunity to investigators of minimizing its overestimation effects.

A learning collaborative supporting the implementation of an evidence-informed program, the "4Rs and 2Ss for children with conduct difficulties and their families"

In this qualitative study the authors examine factors associated with the successful implementation and plans for continued use of an evidence-informed intervention, the 4Rs and 2Ss Program for Strengthening Families, in a sample of 29 New York State, Office of Mental Health licensed child mental health clinics. A learning collaborative (LC) approach was used as a vehicle for supporting training and implementation of the program. The PRISM theoretical framework ( Feldstein & Glasgow, 2008 ) was used to guide the data analysis. Data were analyzed using a multi-phase iterative process, identifying influences on implementation at multiple levels: the program (intervention), the external environment, implementation and sustainability infrastructure, and recipient characteristics. Clinics that were more proactive evidenced staff with advanced organizational skills were able to take advantage of the trainings and supports offered by the LC and fared better in their ability to adopt the intervention. The ability to adapt the intervention to the specific constraints of the clinics was a strong influence on continued use following the end of the LC. These preliminary results suggest that the supports provided by the LC are useful in consolidating information about the process of implementing evidence-informed interventions in community mental health settings. The impact of these supports is also based on their interactions with specific clinic contextual factors.

Developing a systematic evaluation approach for training programs within a train-the-trainer model for youth cognitive behavior therapy

The purpose of this small pilot study was three-fold: (a) to begin development of a coding scheme for supervisor and therapist skill acquisition, (b) to preliminarily investigate a pilot train-the-trainer paradigm for skill development, and (c) to evaluate self-reported versus observed indicators of skill mastery in that pilot program. Participants included four supervisor-therapist dyads (N = 8) working with public mental health sector youth. Master trainers taught cognitive-behavioral therapy techniques to supervisors, who in turn trained therapists on these techniques. Supervisor and therapist skill acquisition and supervisor use of teaching strategies were repeatedly assessed through coding of scripted role-plays with a multiple-baseline across participants and behaviors design. The coding system, the Practice Element Train the Trainer - Supervisor/Therapist Versions of the Therapy Process Observational Coding System for Child Psychotherapy, was developed and evaluated though the course of the investigation. The coding scheme demonstrated excellent reliability (ICCs [1,2] = 0.81-0.91) across 168 video recordings. As calculated through within-subject effect sizes, supervisor and therapist participants, respectively, evidenced skill improvements related to teaching and performing therapy techniques. Self-reported indicators of skill mastery were inflated in comparison to observed skill mastery. Findings lend initial support for further developing an evaluative approach for a train-the-trainer effort focused on disseminating evidence-based practices.

Engineering youth service system infrastructure: Hawaii's continued efforts at large-scale implementation through knowledge management strategies

Hawaii's Child and Adolescent Mental Health Division provides a unique illustration of a youth public mental health system with a long and successful history of large-scale quality improvement initiatives. Many advances are linked to flexibly organizing and applying knowledge gained from the scientific literature and move beyond installing a limited number of brand-named treatment approaches that might be directly relevant only to a small handful of system youth. This article takes a knowledge-to-action perspective and outlines five knowledge management strategies currently under way in Hawaii. Each strategy represents one component of a larger coordinated effort at engineering a service system focused on delivering both brand-named treatment approaches and complimentary strategies informed by the evidence base. The five knowledge management examples are (a) a set of modular-based professional training activities for currently practicing therapists, (b) an outreach initiative for supporting youth evidence-based practices training at Hawaii's mental health-related professional programs, (c) an effort to increase consumer knowledge of and demand for youth evidence-based practices, (d) a practice and progress agency performance feedback system, and (e) a sampling of system-level research studies focused on understanding treatment as usual. We end by outlining a small set of lessons learned and a longer term vision for embedding these efforts into the system's infrastructure.

The Technology of Measurement Feedback Systems

Usual care in the community is far from optimal. Sufficient evidence exists that dropout rates are significant, treatment is effective for only a small proportion of clients, and that the translation of evidence-based treatments to the real world is problematic. Technology has been shown to be helpful in health care in improving the effectiveness of treatment. A relatively new technology being used in mental health is measurement feedback systems (MFSs). MFSs are particularly applicable to couple and family psychology (CFP) because of its ability to provide information on the multiple perspectives involved in treatment. The Contextualized Feedback Systemstm (CFS®), developed at Vanderbilt University is used as an example of what can be accomplished with an MFS. The advantages and limitations of this technology are described as well as the anticipated reimbursement requirements that mental health services will need.

Assessing influences on intervention implementation: revision of the usage rating profile-intervention

Although treatment acceptability was originally proposed as a critical factor in determining the likelihood that a treatment will be used with integrity, more contemporary findings suggest that whether something is likely to be adopted into routine practice is dependent on the complex interplay among a number of different factors. The Usage Rating Profile-Intervention (URP-I; Chafouleas, Briesch, Riley-Tillman, & McCoach, 2009) was recently developed to assess these additional factors, conceptualized as potentially contributing to the quality of intervention use and maintenance over time. The purpose of the current study was to improve upon the URP-I by expanding and strengthening each of the original four subscales. Participants included 1005 elementary teachers who completed the instrument in response to a vignette depicting a common behavior intervention. Results of exploratory and confirmatory factor analyses, as well as reliability analyses, supported a measure containing 29 items and yielding 6 subscales: Acceptability, Understanding, Feasibility, Family-School Collaboration, System Climate, and System Support. Collectively, these items provide information about potential facilitators and barriers to usage that exist at the level of the individual, intervention, and environment. Information gleaned from the instrument is therefore likely to aid consultants in both the planning and evaluation of intervention efforts.

Therapists' attitudes toward evidence-based practices and implementation of parent-child interaction therapy

Child abuse and neglect affects many families each year, but evidence-based parent training programs can be instrumental in reducing maltreatment. Parent-Child Interaction Therapy, a parent training program developed for treatment of disruptive child behavior, has demonstrated effectiveness with families at risk of or exposed to child maltreatment. However, methods for disseminating this evidence-based intervention in community settings are not well understood. This study examined the association between community-based therapists' attitudes toward evidence-based practices (EBPs) and their participation in an implementation research project in which they received two forms of consultation. Results showed that therapists' self-reported unwillingness to diverge from EBPs was positively associated with their use of phone consultation and satisfaction with consultation. The degree to which therapists found EBPs appealing was positively associated with satisfaction as well. Open therapist attitudes toward EBPs were associated with greater attendance for online consultation. The next step in this line of research is to examine how therapists' attitudes toward EBPs can be improved, if changing attitudes affects therapist acquisition of treatment skills, and if such improvements enhance implementation efforts.

Routine outcome monitoring and feedback on physical or mental health status: evidence and theory

OBJECTIVES

Routine Outcome Monitoring (ROM) is an important quality tool for measuring outcome of treatment in health care. The objective of this article is to summarize the evidence base that supports the provision of feedback on ROM results to (mental) health care professionals and patients. Also, some relevant theoretical aspects are considered.

METHODS

Literature study (Pubmed, Medline, PsychINFO, Embase Psychiatry, 1975-2009) concerning randomized controlled trials (RTC's) of ROM and feedback on physical or mental health status of patients of all ages. Main search terms were routine outcome monitoring/measurement, feedback, health status measurement, patient reported outcome measures.

RESULTS

Included were 52 RCT's concerning ROM and feedback with adult or older patients: of these seven RCT's were exclusively focused on physical health and 45 RCT's (also) on the mental health of the patient, although not always in a mental health care setting or as primary outcome measure. There appears to be a positive impact of ROM on diagnosis and monitoring of treatment, and on communication between patient and therapist. Other results were less clear. There were no published RCT's on this topic with children or adolescents.

CONCLUSIONS

ROM appears especially effective for the monitoring of patients who are not doing well in therapy. Further research into this topic and the clinical-and cost-effectiveness of ROM is recommended, especially in mental health care for both adults and children. Also, more theory-driven research is needed with relevant conceptualizations such as Feedback Intervention Theory, Therapeutic Assessment.