At the Intersection of Health Information Technology and Decision Support: Measurement Feedback Systems... and Beyond

Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes

PURPOSE

Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions.

METHOD

Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events.

ANALYSIS

We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures).

RESULTS

For chart notes, practices had moderate interrater reliability (M k = 0.50, M ICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes.

CONCLUSION

Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

Future Directions in Youth and Family Treatment Engagement: Finishing the Bridge Between Science and Service

The field has spent more than 50 years investing in the quality of youth mental healthcare, with intervention science yielding roughly 1,300 efficacious treatments. In the latter half of this period, concurrent efforts in implementation science have developed effective methods for supporting front-line service organizations and therapists to begin to bridge the science to service gap. However, many youths and families still do not benefit fully from these strategic investments due to low treatment engagement: nearly half of youths in need of services pursue them, and among those who do, roughly another half terminate prematurely. The negative impact of low engagement is substantial, and is disproportionally and inequitably so for many. We contend that to build a robust and "finished" bridge connecting science and service, the field must go beyond its two historical foci of designing interventions and preparing therapists to deliver them, to include an intentional focus on the youths and families who participate in these interventions and who work with those therapists. In this paper, we highlight the significance of treatment engagement in youth mental healthcare and discuss the current state of the literature related to four priorities: conceptualization, theory, measurement, and interventions. Next, we offer an example from our own program of research as one illustration for advancing these priorities. Finally, we propose recommendations to act on these priorities.

Expect the Unexpected: A Qualitative Study of the Ripple Effects of Children's Mental Health Services Implementation Efforts

Background

Strategies to implement evidence-based interventions (EBIs) in children's mental health services have complex direct and indirect causal impacts on multiple outcomes. Ripple effects are outcomes caused by EBI implementation efforts that are unplanned, unanticipated, and/or more salient to stakeholders other than researchers and implementers. The purpose of the current paper is to provide a compilation of possible ripple effects associated with EBI implementation strategies in children's mental health services, to be used for implementation planning, research, and quality improvement.

Methods

Participants were identified via expert nomination and snowball sampling. Online surveys were completed by 81 participants, each representing one of five roles: providers of mental health services to children or youth, researchers, policy makers, caregivers, and youth. A partially directed conventional content analysis with consensus decision making was used to code ripple effects.

Results

Four hundred and four unique responses were coded into 66 ripple effects and 14 categories. Categories include general knowledge, skills, attitudes, and confidence about using EBIs; general job-related ripple effects; EBI treatment adherence, fidelity, and alignment; gaming the system; equity and stigma; shifting roles, role clarity, and task shifting; economic costs and benefits; EBI treatment availability, access, participation, attendance, barriers, and facilitators; clinical process and treatment quality; client engagement, therapeutic alliance, and client satisfaction; clinical organization structure, relationships in the organization, process, and functioning; youth client and caregiver outcomes; and use of EBI strategies and insights in one's own life.

Conclusions

This research advances the field by providing children's mental health implementers, researchers, funders, policy makers, and consumers with a menu of potential ripple effects. It can be a practical tool to ensure compliance with guidance from Quality Improvement/Quality Assurance, Complexity Science, and Diffusion of Innovation Theory. Future phases will match potential ripple effects with salient children's mental health implementation strategies for each participant role.

Factors affecting the implementation of an outcome measurement feedback system in youth mental health settings

Objective: Measurement feedback systems provide clinicians with regular snapshots of a client's mental health status, which can be used in treatment planning and client feedback. There are numerous barriers to clinicians using outcome measures routinely. This study aimed to investigate factors affecting the use of a measurement feedback system across youth mental health settings. Methods: The participants were 210 clinicians from headspace youth mental health services across Australia. They were surveyed on predictors and use of MyLifeTracker, a routine outcome measure. This was explored through three processes: looking at MyLifeTracker before session, using MyLifeTracker in treatment planning, and providing feedback of MyLifeTracker scores to clients. Results: Clinicians were more likely to look at MyLifeTracker before session, less likely to use it in treatment planning, and least likely to provide MyLifeTracker scores to clients. Each measurement feedback system process had a distinct group of predictors. Perceptions of MyLifeTracker's practicality was the only significant predictor of all three processes. Conclusion: Practically, organisations and supervisors can increase the use of measurement feedback systems through targeted supports.

Introducing a Measurement Feedback System for Youth Mental Health: Predictors and Impact of Implementation in a Community Agency

Measurement feedback systems (MFSs) that routinely collect and report client progress to mental health therapists have demonstrated beneficial impact on outcomes in numerous studies, with evidence that there is a dose-response relationship related to the implementation of the MFS. The current study examined the impact of MFS implementation (Implementation Index) on youth symptom outcomes separately by caregiver and youth self-report. Additionally, we tested the extent to which Implementation Index rates varied by individual therapists and clients, and whether therapist and client characteristics predicted MFS implementation. Methods: Administrative data (client charts, youth- and caregiver-reported Youth Outcome Questionnaires) for 229 youth (52.83% Latinx, 42.79% girls, M age = 10.33) treated during a 1-year period at a community mental health organization in Central Texas were analyzed using multi-level modeling. Caregiver-reported symptoms decreased faster for those with a higher MFS Implementation Index. Between-group differences among therapists accounted for a significant proportion of variance in the Implementation Index for caregiver report, whereas client differences accounted for most of the variance in the Implementation Index for youth self-report. Therapist trainee status predicted a significant increase in the Implementation Index for caregiver-report data. Youth symptom improvement as reported by caregivers varied with the extent of MFS implementation fidelity, and MFS implementation fidelity was higher for clients treated by trainees relative to staff therapists for caregiver report of symptoms.

Measurement training and feedback system for implementation of evidence-based treatment for adolescent externalizing problems: protocol for a randomized trial of pragmatic clinician training

BACKGROUND

Innovations in clinical training and support that enhance fidelity to evidence-based treatment (EBT) for adolescent behavior problems are sorely needed. This study will develop an online training system to address this gap: Measurement Training and Feedback System for Implementation (MTFS-I). Using procedures intended to be practical and sustainable, MTFS-I is designed to increase two aspects of therapist behavior that are fundamental to boosting EBT fidelity: therapist self-monitoring of EBT delivery, and therapist utilization of core techniques of EBTs in treatment sessions. This version of MTFS-I focuses on two empirically supported treatment approaches for adolescent conduct and substance use problems: family therapy and cognitive behavioral therapy (CBT).

METHODS/DESIGN

MTFS-I expands on conventional measurement feedback systems for client outcomes by adding training in observational coding to promote EBT self-monitoring and focusing on implementation of EBT treatment techniques. It has two primary components. (1) The training component, delivered weekly in two connected parts, involves self-monitored learning modules containing brief clinical descriptions of core EBT techniques and mock session coding exercises based on 5-8 min video segments that illustrate delivery of core techniques. (2) The feedback component summarizes aggregated therapist-reported data on EBT techniques used with their active caseloads. MTFS-I is hosted online and requires approximately 20 min per week to complete for each treatment approach. This randomized trial will first collect data on existing delivery of family therapy and CBT techniques for youth in outpatient behavioral health sites (Baseline phase). It will then randomize site clinicians to two study conditions (Implementation phase): Training Only versus Training + Feedback + Consultation. Therapists will choose whether to train in family therapy, CBT, or both. Study aims will compare clinician performance across study phase and between study conditions on MTFS-I uptake, reliability and accuracy in EBT self-monitoring, and utilization of EBT techniques in treatment sessions (based on observer coding of audiotapes).

DISCUSSION

Study contributions to implementation science and considerations of MTFS-I sustainability are discussed.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT03722654. Registered on 29 October 2018.

Measurement Training and Feedback System for Implementation of family-based services for adolescent substance use: protocol for a cluster randomized trial of two implementation strategies

BACKGROUND

This article describes a study protocol for testing the Measurement Training and Feedback System for Implementation (MTFS-I) and comparing two implementation strategies for MTFS-I delivery. MTFS-I is a web-based treatment quality improvement system designed to increase the delivery of evidence-based interventions for behavioral health problems in routine care settings. This version of MTFS-I focuses on family-based services (FBS) for adolescent substance use. FBS, comprising both family participation in treatment and family therapy technique use, have achieved the strongest evidence base for adolescent substance use and are a prime candidate for upgrading treatment quality in outpatient care. For FBS to fulfill their potential for widespread dissemination, FBS implementation must be bolstered by effective quality procedures that support sustainable delivery in usual care.

METHODS/DESIGN

Adapted from measurement feedback systems for client outcomes, MTFS-I contains three synergistic components: (a) weekly reporter training modules to instruct therapists in reliable post-session self-reporting on FBS utilization; (b) weekly mock session videos of FBS interventions (5-8 min) for supportive training in, and practice coding of, high-quality FBS; and (c) monthly feedback reports to therapists and supervisors displaying aggregated data on therapist-reported FBS use. MTFS-I is hosted online and requires approximately 20 min per week to complete. The study will experimentally compare two well-established implementation strategies designed to foster ongoing MTFS-I usage: Core Training, consisting of two 3-h training sessions focused on FBS site mapping, selecting FBS improvement goals, and sustaining MTFS-I, followed by routine remote technical assistance; and Core + Facilitation, which boosts Core Training sessions with collaborative phone-based clinical consultation and on-site facilitation meetings for 1 year to promote FBS goal achievement. The study design is a cluster randomized trial testing Core Training versus Core + Facilitation in ten substance use treatment clinics. Study aims will compare conditions on MTFS-I uptake, FBS delivery (based on therapist-report and observational data), and 1-year client outcomes.

DISCUSSION

Study contributions to implementation science and considerations of MTFS-I sustainability are discussed.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03342872 . Registered 10 November 2017.

The Heimdall Framework for Supporting Characterisation of Learning Health Systems

BACKGROUND

Learning Health Systems (LHS) can focus population medicine and Evidence Based Practice; smart technology delivering the next generation of improved healthcare described as Precision Medicine, and yet researchers in the LHS domain presently lack the ability to recognise their relevant works as falling within this domain.

OBJECTIVE

To review LHS literature and develop a framework describing the domain that can be used as a tool to analyse the literature and support researchers to identify health informatics investigations as falling with the domain of LHS.

METHOD

A scoping review is used to identify literature on which analysis was performed. This resolved the ontology and framework. The ontology was applied to quantify the distribution of classifications of LHS solutions. The framework was used to analyse and characterise the various works within the body of LHS literature.

RESULTS

The ontology and framework developed was shown to be easily applicable to the literature, consistently describing and representing the goals, intentions and solutions of each LHS investigation in the literature. More proposed or potential solutions are described in the literature than implemented LHS. This suggests immaturity in the domain and points to the existence of barriers preventing LHS realisation.

CONCLUSION

The lack of an ontology and framework may have been one of the causes for the failure to describe research works as falling within the LHS domain. Using our ontology and framework, LHS research works could be easily classified, demonstrating the comprehensiveness of our approach in contrast to earlier efforts.

Digital technology and clinical decision making in depression treatment: Current findings and future opportunities

Clinical decision making encompasses a broad set of processes that contribute to the effectiveness of depression treatments. There is emerging interest in using digital technologies to support effective and efficient clinical decision making. In this paper, we provide "snapshots" of research and current directions on ways that digital technologies can support clinical decision making in depression treatment. Practical facets of clinical decision making are reviewed, then research, design, and implementation opportunities where technology can potentially enhance clinical decision making are outlined. Discussions of these opportunities are organized around three established movements designed to enhance clinical decision making for depression treatment, including measurement-based care, integrated care, and personalized medicine. Research, design, and implementation efforts may support clinical decision making for depression by (1) improving tools to incorporate depression symptom data into existing electronic health record systems, (2) enhancing measurement of treatment fidelity and treatment processes, (3) harnessing smartphone and biosensor data to inform clinical decision making, (4) enhancing tools that support communication and care coordination between patients and providers and within provider teams, and (5) leveraging treatment and outcome data from electronic health record systems to support personalized depression treatment. The current climate of rapid changes in both healthcare and digital technologies facilitates an urgent need for research, design, and implementation of digital technologies that explicitly support clinical decision making. Ensuring that such tools are efficient, effective, and usable in frontline treatment settings will be essential for their success and will require engagement of stakeholders from multiple domains.

The need for data-informed clinical supervision in substance use disorder treatment

Effective clinical supervision is necessary for high-quality care in community-based substance use disorder treatment settings, yet little is known about current supervision practices. Some evidence suggests that supervisors and counselors differ in their experiences of clinical supervision; however, the impact of this misalignment on supervision quality is unclear. Clinical information monitoring systems may support supervision in substance use disorder treatment, but the potential use of these tools must first be explored. First, the current study examines the extent to which misaligned supervisor-counselor perceptions impact supervision satisfaction and emphasis on evidence-based treatments. This study also reports on formative work to develop a supervision-based clinical dashboard, an electronic information monitoring system and data visualization tool providing real-time clinical information to engage supervisors and counselors in a coordinated and data-informed manner, help align supervisor-counselor perceptions about supervision, and improve supervision effectiveness. Clinical supervisors and frontline counselors (N = 165) from five Midwestern agencies providing substance abuse services completed an online survey using Research Electronic Data Capture software, yielding a 75% response rate. Valid quantitative measures of supervision effectiveness were administered, along with qualitative perceptions of a supervision-based clinical dashboard. Through within-dyad analyses, misalignment between supervisor and counselor perceptions of supervision practices was negatively associated with satisfaction of supervision and reported frequency of discussing several important clinical supervision topics, including evidence-based treatments and client rapport. Participants indicated the most useful clinical dashboard functions and reported important benefits and challenges to using the proposed tool. Clinical supervision tends to be largely an informal and unstructured process in substance abuse treatment, which may compromise the quality of care. Clinical dashboards may be a well-targeted approach to facilitate data-informed clinical supervision in community-based treatment agencies.

Capabilities and Characteristics of Digital Measurement Feedback Systems: Results from a Comprehensive Review

Measurement feedback systems (MFS) are a class of health information technology (HIT) that function as an implementation support strategy for integrating measurement based care or routine outcome monitoring into clinical practice. Although many MFS have been developed, little is known about their functions. This paper reports findings from an application of health information technology-academic and commercial evaluation (HIT-ACE), a systematic and consolidated evaluation method, to MFS designed for use in behavioral healthcare settings. Forty-nine MFS were identified and subjected to systematic characteristic and capability coding. Results are presented with respect to the representation of characteristics and capabilities across MFS.

Collaborative Care in Schools: Enhancing Integration and Impact in Youth Mental Health

Collaborative Care is an innovative approach to integrated mental health service delivery that focuses on reducing access barriers, improving service quality, and lowering healthcare expenditures. A large body of evidence supports the effectiveness of Collaborative Care models with adults and, increasingly, for youth. Although existing studies examining these models for youth have focused exclusively on primary care, the education sector is also an appropriate analog for the accessibility that primary care offers to adults. Collaborative Care aligns closely with the practical realities of the education sector and may represent a strategy to achieve some of the objectives of increasingly popular multi-tiered systems of supports frameworks. Unfortunately, no resources exist to guide the application of Collaborative Care models in schools. Based on the existing evidence for Collaborative Care models, the current paper (1) provides a rationale for the adaptation of Collaborative Care models to improve mental health service accessibility and effectiveness in the education sector; (2) presents a preliminary Collaborative Care model for use in schools; and (3) describes avenues for research surrounding school-based Collaborative Care, including the currently funded Accessible, Collaborative Care for Effective School-based Services (ACCESS) project.