"It sounds like...": A natural language processing approach to detecting counselor reflections in motivational interviewing

Machine Learning-Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls

OBJECTIVE

Counselor assessment of suicide risk is one key component of crisis counseling, and standards require risk assessment in every crisis counseling conversation. Efforts to increase risk assessment frequency are limited by quality improvement tools that rely on human evaluation of conversations, which is labor intensive, slow, and impossible to scale. Advances in machine learning (ML) have made possible the development of tools that can automatically and immediately detect the presence of risk assessment in crisis counseling conversations.

METHODS

To train models, a coding team labeled every statement in 476 crisis counseling calls (193,257 statements) for a core element of risk assessment. The authors then fine-tuned a transformer-based ML model with the labeled data, utilizing separate training, validation, and test data sets.

RESULTS

Generally, the evaluated ML model was highly consistent with human raters. For detecting any risk assessment, ML model agreement with human ratings was 98% of human interrater agreement. Across specific labels, average F1 (the harmonic mean of precision and recall) was 0.86 at the call level and 0.66 at the statement level and often varied as a result of a low base rate for some risk labels.

CONCLUSIONS

ML models can reliably detect the presence of suicide risk assessment in crisis counseling conversations, presenting an opportunity to scale quality improvement efforts.

Scoping review on natural language processing applications in counselling and psychotherapy

Recent years have witnessed some rapid and tremendous progress in natural language processing (NLP) techniques that are used to analyse text data. This study endeavours to offer an up-to-date review of NLP applications by examining their use in counselling and psychotherapy from 1990 to 2021. The purpose of this scoping review is to identify trends, advancements, challenges and limitations of these applications. Among the 41 papers included in this review, 4 primary study purposes were identified: (1) developing automated coding; (2) predicting outcomes; (3) monitoring counselling sessions; and (4) investigating language patterns. Our findings showed a growing trend in the number of papers utilizing advanced machine learning methods, particularly neural networks. Unfortunately, only a third of the articles addressed the issues of bias and generalizability. Our findings provided a timely systematic update, shedding light on concerns related to bias, generalizability and validity in the context of NLP applications in counselling and psychotherapy.

The impact of alcohol on affiliative verbal behavior: A systematic review and meta-analysis

BACKGROUND

Language is a fundamental aspect of human social behavior that is linked to many rewarding social experiences, such as social bonding. Potential effects of alcohol on affiliative language may therefore be an essential feature of alcohol reward and may elucidate pathways through which alcohol is linked to social facilitation. Examinations of alcohol's impact on language content, however, are sparse. Accordingly, this investigation represents the first systematic review and meta-analysis of alcohol's effects on affiliative language. We test the hypothesis that alcohol increases affiliative verbal approach behaviors and discuss future research directions.

METHODS

PsycInfo and Web of Science were systematically searched in March 2023 according to our preregistered plan. Eligible studies included social alcohol administration experiments in which affiliative verbal language was assessed. We present a random-effects meta-analysis that examines the effect of alcohol compared to control on measures of affiliative verbal behavior.

RESULTS

Our search identified 16 distinct investigations (comprising 961 participants) that examined the effect of alcohol on affiliative verbal behavior. Studies varied greatly in methods and measures. Meta-analytic results demonstrated that alcohol is modestly associated with increases in affiliative verbal behavior (Hedges' g = 0.164, 95% CI [0.027, 0.301], p = 0.019). Study quality was rated using an adapted version of the Quality Assessment Tool for Quantitative Studies and did not significantly moderate alcohol's effects.

CONCLUSIONS

This study provides preliminary evidence that alcohol can increase affiliative verbal behaviors. This effect may be an important feature of alcohol reward. Given heterogeneity in study features, low study quality ratings, and limited reporting of effect size data, results simultaneously highlight the promise of this research area and the need for more work. Advances in language processing methodologies that could allow future work to systematically expand upon this finding are discussed.

A scoping review of empathy recognition in text using natural language processing

OBJECTIVE

To provide a scoping review of studies on empathy recognition in text using natural language processing (NLP) that can inform an approach to identifying physician empathic communication over patient portal messages.

MATERIALS AND METHODS

We searched 6 databases to identify relevant studies published through May 1, 2023. The study selection was conducted through a title screening, an abstract review, and a full-text review. Our process followed the PRISMA-ScR guidelines.

RESULTS

Of the 2446 publications identified from our searches, 39 studies were selected for the final review, which summarized: (1) definitions and context of empathy, (2) data sources and tested models, and (3) model performance. Definitions of empathy varied in their specificity to the context and setting of the study. The most common settings in which empathy was studied were reactions to news stories, health-related social media forums, and counseling sessions. We also observed an expected shift in methods used that coincided with the introduction of transformer-based models.

DISCUSSION

Aspects of the current approaches taken across various domains may be translatable to communication over a patient portal. However, the specific barriers to identifying empathic communication in this context are unclear. While modern NLP methods appear to be able to handle empathy-related tasks, challenges remain in precisely defining and measuring empathy in text.

CONCLUSION

Existing work that has attempted to measure empathy in text using NLP provides a useful basis for future studies of patient-physician asynchronous communication, but consideration for the conceptualization of empathy is needed.

Improving the quality of counseling and clinical supervision in opioid treatment programs: how can technology help?

BACKGROUND

The opioid epidemic has resulted in expanded substance use treatment services and strained the clinical workforce serving people with opioid use disorder. Focusing on evidence-based counseling practices like motivational interviewing may be of interest to counselors and their supervisors, but time-intensive adherence tasks like recording and feedback are aspirational in busy community-based opioid treatment programs. The need to improve and systematize clinical training and supervision might be addressed by the growing field of machine learning and natural language-based technology, which can promote counseling skill via self- and supervisor-monitoring of counseling session recordings.

METHODS

Counselors in an opioid treatment program were provided with an opportunity to use an artificial intelligence based, HIPAA compliant recording and supervision platform (Lyssn.io) to record counseling sessions. We then conducted four focus groups-two with counselors and two with supervisors-to understand the integration of technology with practice and supervision. Questions centered on the acceptability of the clinical supervision software and its potential in an OTP setting; we conducted a thematic coding of the responses.

RESULTS

The clinical supervision software was experienced by counselors and clinical supervisors as beneficial to counselor training, professional development, and clinical supervision. Focus group participants reported that the clinical supervision software could help counselors learn and improve motivational interviewing skills. Counselors said that using the technology highlights the value of counseling encounters (versus paperwork). Clinical supervisors noted that the clinical supervision software could help meet national clinical supervision guidelines and local requirements. Counselors and clinical supervisors alike talked about some of the potential challenges of requiring session recording.

CONCLUSIONS

Implementing evidence-based counseling practices can help the population served in OTPs; another benefit of focusing on clinical skills is to emphasize and hold up counselors' roles as worthy. Machine learning technology can have a positive impact on clinical practices among counselors and clinical supervisors in opioid treatment programs, settings whose clinical workforce continues to be challenged by the opioid epidemic. Using technology to focus on clinical skill building may enhance counselors' and clinical supervisors' overall experiences in their places of work.

Mental Health Counseling From Conversational Content With Transformer-Based Machine Learning

Importance

Use of asynchronous text-based counseling is rapidly growing as an easy-to-access approach to behavioral health care. Similar to in-person treatment, it is challenging to reliably assess as measures of process and content do not scale.

Objective

To use machine learning to evaluate clinical content and client-reported outcomes in a large sample of text-based counseling episodes of care.

Design, Setting, and Participants

In this quality improvement study, participants received text-based counseling between 2014 and 2019; data analysis was conducted from September 22, 2022, to November 28, 2023. The deidentified content of messages was retained as a part of ongoing quality assurance. Treatment was asynchronous text-based counseling via an online and mobile therapy app (Talkspace). Therapists were licensed to provide mental health treatment and were either independent contractors or employees of the product company. Participants were self-referred via online sign-up and received services via their insurance or self-pay and were assigned a diagnosis from their health care professional.

Exposure

All clients received counseling services from a licensed mental health clinician.

Main Outcomes and Measures

The primary outcomes were client engagement in counseling (number of weeks), treatment satisfaction, and changes in client symptoms, measured via the 8-item version of Patient Health Questionnaire (PHQ-8). A previously trained, transformer-based, deep learning model automatically categorized messages into types of therapist interventions and summaries of clinical content.

Results

The total sample included 166 644 clients treated by 4973 therapists (20 600 274 messages). Participating clients were predominantly female (75.23%), aged 26 to 35 years (55.4%), single (37.88%), earned a bachelor's degree (59.13%), and were White (61.8%). There was substantial variability in intervention use and treatment content across therapists. A series of mixed-effects regressions indicated that collectively, interventions and clinical content were associated with key outcomes: engagement (multiple R = 0.43), satisfaction (multiple R = 0.46), and change in PHQ-8 score (multiple R = 0.13).

Conclusions and Relevance

This quality improvement study found associations between therapist interventions, clinical content, and client-reported outcomes. Consistent with traditional forms of counseling, higher amounts of supportive counseling were associated with improved outcomes. These findings suggest that machine learning-based evaluations of content may increase the scale and specificity of psychotherapy research.

Natural language processing for mental health interventions: a systematic review and research framework

Neuropsychiatric disorders pose a high societal cost, but their treatment is hindered by lack of objective outcomes and fidelity metrics. AI technologies and specifically Natural Language Processing (NLP) have emerged as tools to study mental health interventions (MHI) at the level of their constituent conversations. However, NLP's potential to address clinical and research challenges remains unclear. We therefore conducted a pre-registered systematic review of NLP-MHI studies using PRISMA guidelines (osf.io/s52jh) to evaluate their models, clinical applications, and to identify biases and gaps. Candidate studies (n = 19,756), including peer-reviewed AI conference manuscripts, were collected up to January 2023 through PubMed, PsycINFO, Scopus, Google Scholar, and ArXiv. A total of 102 articles were included to investigate their computational characteristics (NLP algorithms, audio features, machine learning pipelines, outcome metrics), clinical characteristics (clinical ground truths, study samples, clinical focus), and limitations. Results indicate a rapid growth of NLP MHI studies since 2019, characterized by increased sample sizes and use of large language models. Digital health platforms were the largest providers of MHI data. Ground truth for supervised learning models was based on clinician ratings (n = 31), patient self-report (n = 29) and annotations by raters (n = 26). Text-based features contributed more to model accuracy than audio markers. Patients' clinical presentation (n = 34), response to intervention (n = 11), intervention monitoring (n = 20), providers' characteristics (n = 12), relational dynamics (n = 14), and data preparation (n = 4) were commonly investigated clinical categories. Limitations of reviewed studies included lack of linguistic diversity, limited reproducibility, and population bias. A research framework is developed and validated (NLPxMHI) to assist computational and clinical researchers in addressing the remaining gaps in applying NLP to MHI, with the goal of improving clinical utility, data access, and fairness.

Measuring the active elements of cognitive-behavioral therapies

Understanding how and for whom cognitive-behavioral therapies work is central to the development and improvement of mental health interventions. Suboptimal quantification of the active elements of cognitive-behavioral therapies has hampered progress in elucidating mechanisms of change. To advance process research on cognitive-behavioral therapies, we describe a theoretical measurement framework that focuses on the delivery, receipt, and application of the active elements of these interventions. We then provide recommendations for measuring the active elements of cognitive-behavioral therapies aligned with this framework. Finally, to support measurement harmonization and improve study comparability, we propose the development of a publicly available repository of assessment tools: the Active Elements of Cognitive-Behavioral Therapies Measurement Kit.

You never know what you are going to get: Large-scale assessment of therapists' supportive counseling skill use

Supportive counseling skills like empathy and active listening are critical ingredients of all psychotherapies, but most research relies on client or therapist reports of the treatment process. This study utilized machine-learning models trained to evaluate counseling skills to evaluate supportive skill use in 3,917 session recordings. We analyzed overall skill use and variation in practice patterns using a series of mixed effects models. On average, therapists scored moderately high on observer-rated empathy (i.e., 3.8 out of 5), 3.3% of the therapists' utterances in a session were open questions, and 12.9% of their utterances were reflections. However, there were substantial differences in skill use across therapists as well as across clients within-therapist caseloads. These findings highlight the substantial variability in the process of counseling that clients may experience when they access psychotherapy. We discuss findings in the context of both the need for therapists to be responsive and flexible with their clients, but also potential costs related to the lack of a more uniform experience of care. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol

BACKGROUND

Each year, millions of Americans receive evidence-based psychotherapies (EBPs) like cognitive behavioral therapy (CBT) for the treatment of mental and behavioral health problems. Yet, at present, there is no scalable method for evaluating the quality of psychotherapy services, leaving EBP quality and effectiveness largely unmeasured and unknown. Project AFFECT will develop and evaluate an AI-based software system to automatically estimate CBT fidelity from a recording of a CBT session. Project AFFECT is an NIMH-funded research partnership between the Penn Collaborative for CBT and Implementation Science and Lyssn.io, Inc. ("Lyssn") a start-up developing AI-based technologies that are objective, scalable, and cost efficient, to support training, supervision, and quality assurance of EBPs. Lyssn provides HIPAA-compliant, cloud-based software for secure recording, sharing, and reviewing of therapy sessions, which includes AI-generated metrics for CBT. The proposed tool will build from and be integrated into this core platform.

METHODS

Phase I will work from an existing software prototype to develop a LyssnCBT user interface geared to the needs of community mental health (CMH) agencies. Core activities include a user-centered design focus group and interviews with community mental health therapists, supervisors, and administrators to inform the design and development of LyssnCBT. LyssnCBT will be evaluated for usability and implementation readiness in a final stage of Phase I. Phase II will conduct a stepped-wedge, hybrid implementation-effectiveness randomized trial (N = 1,875 clients) to evaluate the effectiveness of LyssnCBT to improve therapist CBT skills and client outcomes and reduce client drop-out. Analyses will also examine the hypothesized mechanism of action underlying LyssnCBT.

DISCUSSION

Successful execution will provide automated, scalable CBT fidelity feedback for the first time ever, supporting high-quality training, supervision, and quality assurance, and providing a core technology foundation that could support the quality delivery of a range of EBPs in the future.

TRIAL REGISTRATION

ClinicalTrials.gov; NCT05340738 ; approved 4/21/2022.

Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study

Regular assessment of the effectiveness of behavioral interventions is a potent tool for improving their relevance to patients. However, poor provider and patient adherence characterize most measurement-based care tools. Therefore, a new approach for measuring intervention effects and communicating them to providers in a seamless manner is warranted. This paper provides a brief overview of the available research evidence on novel ways to measure the effects of behavioral treatments, integrating both objective and subjective data. We highlight the importance of analyzing therapeutic conversations through natural language processing. We then suggest a conceptual framework for capitalizing on data captured through directly collected and nondisruptive methodologies to describe the client's characteristics and needs and inform clinical decision-making. We then apply this context in exploring a new tool to integrate the content of therapeutic conversations and patients' self-reports. We present a case study of how both subjective and objective measures of treatment effects were implemented in cognitive-behavioral treatment for depression and anxiety and then utilized in treatment planning, delivery, and termination. In this tool, called Eleos, the patient completes standardized measures of depression and anxiety. The content of the treatment sessions was evaluated using nondisruptive, independent measures of conversation content, fidelity to the treatment model, and the back-and-forth of client-therapist dialogue. Innovative applications of advances in digital health are needed to disseminate empirically supported interventions and measure them in a noncumbersome way. Eleos appears to be a feasible, sustainable, and effective way to assess behavioral health care.

Characterizing DSM-5 and ICD-11 personality disorder features in psychiatric inpatients at scale using electronic health records

BACKGROUND

Investigation of personality traits and pathology in large, generalizable clinical cohorts has been hindered by inconsistent assessment and failure to consider a range of personality disorders (PDs) simultaneously.

METHODS

We applied natural language processing (NLP) of electronic health record notes to characterize a psychiatric inpatient cohort. A set of terms reflecting personality trait domains were derived, expanded, and then refined based on expert consensus. Latent Dirichlet allocation was used to score notes to estimate the extent to which any given note reflected PD topics. Regression models were used to examine the relationship of these estimates with sociodemographic features and length of stay.

RESULTS

Among 3623 patients with 4702 admissions, being male, non-white, having a low burden of medical comorbidity, being admitted through the emergency department, and having public insurance were independently associated with greater levels of disinhibition, detachment, and psychoticism. Being female, white, and having private insurance were independently associated with greater levels of negative affectivity. The presence of disinhibition, psychoticism, and negative affectivity were each significantly associated with a longer stay, while detachment was associated with a shorter stay.

CONCLUSIONS

Personality features can be systematically and scalably measured using NLP in the inpatient setting, and some of these features associate with length of stay. Developing treatment strategies for patients scoring high in certain personality dimensions may facilitate more efficient, targeted interventions, and may help reduce the impact of personality features on mental health service utilization.

A scoping review of machine learning in psychotherapy research

Machine learning (ML) offers robust statistical and probabilistic techniques that can help to make sense of large amounts of data. This scoping review paper aims to broadly explore the nature of research activity using ML in the context of psychological talk therapies, highlighting the scope of current methods and considerations for clinical practice and directions for future research. Using a systematic search methodology, fifty-one studies were identified. A narrative synthesis indicates two types of studies, those who developed and tested an ML model (k=44), and those who reported on the feasibility of a particular treatment tool that uses an ML algorithm (k=7). Most model development studies used supervised learning techniques to classify or predict labeled treatment process or outcome data, whereas others used unsupervised techniques to identify clusters in the unlabeled patient or treatment data. Overall, the current applications of ML in psychotherapy research demonstrated a range of possible benefits for indications of treatment process, adherence, therapist skills and treatment response prediction, as well as ways to accelerate research through automated behavioral or linguistic process coding. Given the novelty and potential of this research field, these proof-of-concept studies are encouraging, however, do not necessarily translate to improved clinical practice (yet).

Redesigning Implementation Measurement for Monitoring and Quality Improvement in Community Delivery Settings

The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the "New Beginnings Program" serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.

Story Arcs in Serious Illness: Natural Language Processing features of Palliative Care Conversations

OBJECTIVE

Serious illness conversations are complex clinical narratives that remain poorly understood. Natural Language Processing (NLP) offers new approaches for identifying hidden patterns within the lexicon of stories that may reveal insights about the taxonomy of serious illness conversations.

METHODS

We analyzed verbatim transcripts from 354 consultations involving 231 patients and 45 palliative care clinicians from the Palliative Care Communication Research Initiative. We stratified each conversation into deciles of "narrative time" based on word counts. We used standard NLP analyses to examine the frequency and distribution of words and phrases indicating temporal reference, illness terminology, sentiment and modal verbs (indicating possibility/desirability).

RESULTS

Temporal references shifted steadily from talking about the past to talking about the future over deciles of narrative time. Conversations progressed incrementally from "sadder" to "happier" lexicon; reduction in illness terminology accounted substantially for this pattern. We observed the following sequence in peak frequency over narrative time: symptom terms, treatment terms, prognosis terms and modal verbs indicating possibility.

CONCLUSIONS

NLP methods can identify narrative arcs in serious illness conversations.

PRACTICE IMPLICATIONS

Fully automating NLP methods will allow for efficient, large scale and real time measurement of serious illness conversations for research, education and system re-design.

Identifying Effective Motivational Interviewing Communication Sequences Using Automated Pattern Analysis

Motivational Interviewing (MI) is an evidence-based strategy for communicating with patients about behavior change. Although there is strong empirical evidence linking "MI-consistent" counselor behaviors and patient motivational statements (i.e., "change talk"), the specific counselor communication behaviors effective for eliciting patient change talk vary by treatment context and, thus, are a subject of ongoing research. An integral part of this research is the sequential analysis of pre-coded MI transcripts. In this paper, we evaluate the empirical effectiveness of the Hidden Markov Model, a probabilistic generative model for sequence data, for modeling sequences of behavior codes and closed frequent pattern mining, a method to identify frequently occurring sequential patterns of behavior codes in MI communication sequences to inform MI practice. We conducted experiments with 1,360 communication sequences from 37 transcribed audio recordings of weight loss counseling sessions with African-American adolescents with obesity and their caregivers. Transcripts had been previously annotated with patient-counselor behavior codes using a specialized codebook. Empirical results indicate that Hidden Markov Model and closed frequent pattern mining techniques can identify counselor communication strategies that are effective at eliciting patients' motivational statements to guide clinical practice.

Identifying Therapist and Client Personae for Therapeutic Alliance Estimation

Psychotherapy, from a narrative perspective, is the process in which a client relates an on-going life-story to a therapist. In each session, a client will recount events from their life, some of which stand out as more significant than others. These significant stories can ultimately shape one's identity. In this work we study these narratives in the context of therapeutic alliance-a self-reported measure on the perception of a shared bond between client and therapist. We propose that alliance can be predicted from the interactions between certain types of clients with types of therapists. To validate this method, we obtained 1235 transcribed sessions with client-reported alliance to train an unsupervised approach to discover groups of therapists and clients based on common types of narrative characters, or personae. We measure the strength of the relation between personae and alliance in two experiments. Our results show that (1) alliance can be explained by the interactions between the discovered character types, and (2) models trained on therapist and client personae achieve significant performance gains compared to competitive supervised baselines. Finally, exploratory analysis reveals important character traits that lead to an improved perception of alliance.

Development and Evaluation of ClientBot: Patient-Like Conversational Agent to Train Basic Counseling Skills

Background

Training therapists is both expensive and time-consuming. Degree–based training can require tens of thousands of dollars and hundreds of hours of expert instruction. Counseling skills practice often involves role-plays, standardized patients, or practice with real clients. Performance–based feedback is critical for skill development and expertise, but trainee therapists often receive minimal and subjective feedback, which is distal to their skill practice.

Objective

In this study, we developed and evaluated a patient-like neural conversational agent, which provides real-time feedback to trainees via chat–based interaction.

Methods

The text–based conversational agent was trained on an archive of 2354 psychotherapy transcripts and provided specific feedback on the use of basic interviewing and counseling skills (ie, open questions and reflections—summary statements of what a client has said). A total of 151 nontherapists were randomized to either (1) immediate feedback on their use of open questions and reflections during practice session with ClientBot or (2) initial education and encouragement on the skills.

Results

Participants in the ClientBot condition used 91% (21.4/11.2) more reflections during practice with feedback (P<.001) and 76% (14.1/8) more reflections after feedback was removed (P<.001) relative to the control group. The treatment group used more open questions during training but not after feedback was removed, suggesting that certain skills may not improve with performance–based feedback. Finally, after feedback was removed, the ClientBot group used 31% (32.5/24.7) more listening skills overall (P<.001).

Conclusions

This proof-of-concept study demonstrates that practice and feedback can improve trainee use of basic counseling skills.

Design feasibility of an automated, machine-learning based feedback system for motivational interviewing

Direct observation of psychotherapy and providing performance-based feedback is the gold-standard approach for training psychotherapists. At present, this requires experts and training human coding teams, which is slow, expensive, and labor intensive. Machine learning and speech signal processing technologies provide a way to scale up feedback in psychotherapy. We evaluated an initial proof of concept automated feedback system that generates motivational interviewing quality metrics and provides easy access to other session data (e.g., transcripts). The system automatically provides a report of session-level metrics (e.g., therapist empathy) and therapist behavior codes at the talk-turn level (e.g., reflections). We assessed usability, therapist satisfaction, perceived accuracy, and intentions to adopt. A sample of 21 novice (n = 10) or experienced (n = 11) therapists each completed a 10-min session with a standardized patient. The system received the audio from the session as input and then automatically generated feedback that therapists accessed via a web portal. All participants found the system easy to use and were satisfied with their feedback, 83% found the feedback consistent with their own perceptions of their clinical performance, and 90% reported they were likely to use the feedback in their practice. We discuss the implications of applying new technologies to evaluation of psychotherapy. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Developing Machine Learning Models for Behavioral Coding

OBJECTIVE

The goal of this research is to develop a machine learning supervised classification model to automatically code clinical encounter transcripts using a behavioral code scheme.

METHODS

We first evaluated the efficacy of eight state-of-the-art machine learning classification models to recognize patient-provider communication behaviors operationalized by the motivational interviewing framework. Data were collected during the course of a single weight loss intervention session with 37 African American adolescents and their caregivers. We then tested the transferability of the model to a novel treatment context, 80 patient-provider interactions during routine human immunodeficiency virus (HIV) clinic visits.

RESULTS

Of the eight models tested, the support vector machine model demonstrated the best performance, achieving a .680 F1-score (a function of model precision and recall) in adolescent and .639 in caregiver sessions. Adding semantic and contextual features improved accuracy with 75.1% of utterances in adolescent and 73.8% in caregiver sessions correctly coded. With no modification, the model correctly classified 72.0% of patient-provider utterances in HIV clinical encounters with reliability comparable to human coders (k = .639).

CONCLUSIONS

The development of a validated approach for automatic behavioral coding offers an efficient alternative to traditional, resource-intensive methods with the potential to dramatically accelerate the pace of outcomes-oriented behavioral research. The knowledge gained from computer-driven behavioral research can inform clinical practice by providing clinicians with empirically supported communication strategies to tailor their conversations with patients. Lastly, automatic behavioral coding is a critical first step toward fully automated eHealth/mHealth (electronic/mobile Health) behavioral interventions.

"It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation

We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.

Everyday couples' communication research: Overcoming methodological barriers with technology

Relationship behaviors contribute to compromised health or resilience. Everyday communication between intimate partners represents the vast majority of their interactions. When intimate partners take on new roles as patients and caregivers, everyday communication takes on a new and important role in managing both the transition and the adaptation to the change in health status. However, everyday communication and its relation to health has been little studied, likely due to barriers in collecting and processing this kind of data. The goal of this paper is to describe deterrents to capturing naturalistic, day-in-the-life communication data and share how technological advances have helped surmount them. We provide examples from a current study and describe how we anticipate technology will further change research capabilities.

Predicting couple therapy outcomes based on speech acoustic features

Automated assessment and prediction of marital outcome in couples therapy is a challenging task but promises to be a potentially useful tool for clinical psychologists. Computational approaches for inferring therapy outcomes using observable behavioral information obtained from conversations between spouses offer objective means for understanding relationship dynamics. In this work, we explore whether the acoustics of the spoken interactions of clinically distressed spouses provide information towards assessment of therapy outcomes. The therapy outcome prediction task in this work includes detecting whether there was a relationship improvement or not (posed as a binary classification) as well as discerning varying levels of improvement or decline in the relationship status (posed as a multiclass recognition task). We use each interlocutor's acoustic speech signal characteristics such as vocal intonation and intensity, both independently and in relation to one another, as cues for predicting the therapy outcome. We also compare prediction performance with one obtained via standardized behavioral codes characterizing the relationship dynamics provided by human experts as features for automated classification. Our experiments, using data from a longitudinal clinical study of couples in distressed relations, showed that predictions of relationship outcomes obtained directly from vocal acoustics are comparable or superior to those obtained using human-rated behavioral codes as prediction features. In addition, combining direct signal-derived features with manually coded behavioral features improved the prediction performance in most cases, indicating the complementarity of relevant information captured by humans and machine algorithms. Additionally, considering the vocal properties of the interlocutors in relation to one another, rather than in isolation, showed to be important for improving the automatic prediction. This finding supports the notion that behavioral outcome, like many other behavioral aspects, is closely related to the dynamics and mutual influence of the interlocutors during their interaction and their resulting behavioral patterns.

Implementing the teen marijuana check-up in schools-a study protocol

BACKGROUND

Substance misuse is now encountered in settings beyond addiction specialty care, with schools a point-of-contact for student access to behavioral health services. Marijuana is a leading impetus for adolescent treatment admissions despite declining risk perception, for which the Teen Marijuana Check-Up (TMCU)-a tailored adaptation of motivational enhancement therapy-offers an efficacious service option. To bridge the knowledge gap concerning effective and affordable technical assistance strategies for implementing empirically supported services, the described trial will test such a strategy to facilitate school-based TMCU implementation.

METHODS

A type II effectiveness/implementation hybrid trial will test a novel strategy for a TMCU purveyor to provide technical assistance on an 'as-needed' basis when triggered by a fidelity drift alarm bell, compared to resource-intensive 'gold-standard' technical assistance procedures of prior efficacy trials. Trial procedures adhere to the EPIS framework as follows: (1) initial mixed-method exploration of the involved school contexts and identification of TMCU interventionist candidates in elicitation interviews; (2) interventionist preparation via a formally evaluated training process involving a two-day workshop and sequence of three training cases; (3) post-training implementation for 24 months for which trained interventionists are randomized to 'as-needed' or 'gold-standard' technical assistance and self-referring students randomized (in 2:1 ratio) to TMCU or waitlist/control; and (4) examination of TMCU sustainment via interventionist completion of biannual outcome assessments, cost analyses, and exit interviews. Hypothesized effects include non-differential influence of the competing technical assistance methods on both TMCU fidelity and intervention effectiveness, with lesser school costs for the 'as-needed' than 'gold-standard' technical assistance and greater reduction in the frequency of marijuana use expected among TMCU-exposed students relative to those assigned to waitlist/control.

DISCUSSION

This trial-occurring in Washington state as legislative, fiscal, and sociocultural forces converge to heighten exposure of American adolescents to marijuana-related harms-is set to advance understanding of best implementation practices for this and other efficacious, school-based interventions through examination of a data-driven technical assistance method. If shown to be clinically useful and affordable, the concept of a fidelity drift alarm could be readily translated to other empirically supported services and in other health settings.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03111667 registered 7 April 2017.

Designing Contestability: Interaction Design, Machine Learning, and Mental Health

We describe the design of an automated assessment and training tool for psychotherapists to illustrate challenges with creating interactive machine learning (ML) systems, particularly in contexts where human life, livelihood, and wellbeing are at stake. We explore how existing theories of interaction design and machine learning apply to the psychotherapy context, and identify "contestability" as a new principle for designing systems that evaluate human behavior. Finally, we offer several strategies for making ML systems more accountable to human actors.

A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing

Motivational interviewing (MI) is an efficacious treatment for substance use disorders and other problem behaviors. Studies on MI fidelity and mechanisms of change typically use human raters to code therapy sessions, which requires considerable time, training, and financial costs. Natural language processing techniques have recently been utilized for coding MI sessions using machine learning techniques, rather than human coders, and preliminary results have suggested these methods hold promise. The current study extends this previous work by introducing two natural language processing models for automatically coding MI sessions via computer. The two models differ in the way they semantically represent session content, utilizing either 1) simple discrete sentence features (DSF model) and 2) more complex recursive neural networks (RNN model). Utterance- and session-level predictions from these models were compared to ratings provided by human coders using a large sample of MI sessions (N=341 sessions; 78,977 clinician and client talk turns) from 6 MI studies. Results show that the DSF model generally had slightly better performance compared to the RNN model. The DSF model had "good" or higher utterance-level agreement with human coders (Cohen's kappa>0.60) for open and closed questions, affirm, giving information, and follow/neutral (all therapist codes); considerably higher agreement was obtained for session-level indices, and many estimates were competitive with human-to-human agreement. However, there was poor agreement for client change talk, client sustain talk, and therapist MI-inconsistent behaviors. Natural language processing methods provide accurate representations of human derived behavioral codes and could offer substantial improvements to the efficiency and scale in which MI mechanisms of change research and fidelity monitoring are conducted.