Machine learning vs addiction therapists: A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication

Lack of evidence for predictive utility from resting state fMRI data for individual exposure-based cognitive behavioral therapy outcomes: A machine learning study in two large multi-site samples in anxiety disorders

Data-based predictions of individual Cognitive Behavioral Therapy (CBT) treatment response are a fundamental step towards precision medicine. Past studies demonstrated only moderate prediction accuracy (i.e. ability to discriminate between responders and non-responders of a given treatment) when using clinical routine data such as demographic and questionnaire data, while neuroimaging data achieved superior prediction accuracy. However, these studies may be considerably biased due to very limited sample sizes and bias-prone methodology. Adequately powered and cross-validated samples are a prerequisite to evaluate predictive performance and to identify the most promising predictors. We therefore analyzed resting state functional magnet resonance imaging (rs-fMRI) data from two large clinical trials to test whether functional neuroimaging data continues to provide good prediction accuracy in much larger samples. Data came from two distinct German multicenter studies on exposure-based CBT for anxiety disorders, the Protect-AD and SpiderVR studies. We separately and independently preprocessed baseline rs-fMRI data from n = 220 patients (Protect-AD) and n = 190 patients (SpiderVR) and extracted a variety of features, including ROI-to-ROI and edge-functional connectivity, sliding-windows, and graph measures. Including these features in sophisticated machine learning pipelines, we found that predictions of individual outcomes never significantly differed from chance level, even when conducting a range of exploratory post-hoc analyses. Moreover, resting state data never provided prediction accuracy beyond the sociodemographic and clinical data. The analyses were independent of each other in terms of selecting methods to process resting state data for prediction input as well as in the used parameters of the machine learning pipelines, corroborating the external validity of the results. These similar findings in two independent studies, analyzed separately, urge caution regarding the interpretation of promising prediction results based on neuroimaging data from small samples and emphasizes that some of the prediction accuracies from previous studies may result from overestimation due to homogeneous data and weak cross-validation schemes. The promise of resting-state neuroimaging data to play an important role in the prediction of CBT treatment outcomes in patients with anxiety disorders remains yet to be delivered.

Predicting recurrent chat contact in a psychological intervention for the youth using natural language processing

Chat-based counseling hotlines emerged as a promising low-threshold intervention for youth mental health. However, despite the resulting availability of large text corpora, little work has investigated Natural Language Processing (NLP) applications within this setting. Therefore, this preregistered approach (OSF: XA4PN) utilizes a sample of approximately 19,000 children and young adults that received a chat consultation from a 24/7 crisis service in Germany. Around 800,000 messages were used to predict whether chatters would contact the service again, as this would allow the provision of or redirection to additional treatment. We trained an XGBoost Classifier on the words of the anonymized conversations, using repeated cross-validation and bayesian optimization for hyperparameter search. The best model was able to achieve an AUROC score of 0.68 (p < 0.01) on the previously unseen 3942 newest consultations. A shapely-based explainability approach revealed that words indicating younger age or female gender and terms related to self-harm and suicidal thoughts were associated with a higher chance of recontacting. We conclude that NLP-based predictions of recurrent contact are a promising path toward personalized care at chat hotlines.

Dataset size versus homogeneity: A machine learning study on pooling intervention data in e-mental health dropout predictions

Objective

This study proposes a way of increasing dataset sizes for machine learning tasks in Internet-based Cognitive Behavioral Therapy through pooling interventions. To this end, it (1) examines similarities in user behavior and symptom data among online interventions for patients with depression, social anxiety, and panic disorder and (2) explores whether these similarities suffice to allow for pooling the data together, resulting in more training data when prediction intervention dropout.

Methods

A total of 6418 routine care patients from the Internet Psychiatry in Stockholm are analyzed using (1) clustering and (2) dropout prediction models. For the latter, prediction models trained on each individual intervention's data are compared to those trained on all three interventions pooled into one dataset. To investigate if results vary with dataset size, the prediction is repeated using small and medium dataset sizes.

Results

The clustering analysis identified three distinct groups that are almost equally spread across interventions and are instead characterized by different activity levels. In eight out of nine settings investigated, pooling the data improves prediction results compared to models trained on a single intervention dataset. It is further confirmed that models trained on small datasets are more likely to overestimate prediction results.

Conclusion

The study reveals similar patterns of patients with depression, social anxiety, and panic disorder regarding online activity and intervention dropout. As such, this work offers pooling different interventions' data as a possible approach to counter the problem of small dataset sizes in psychological research.

Who responds to a multi-component treatment for cannabis use disorder? Using multivariable and machine learning models to classify treatment responders and non-responders

BACKGROUND AND AIMS

Treatments for cannabis use disorder (CUD) have limited efficacy and little is known about who responds to existing treatments. Accurately predicting who will respond to treatment can improve clinical decision-making by allowing clinicians to offer the most appropriate level and type of care. This study aimed to determine whether multivariable/machine learning models can be used to classify CUD treatment responders versus non-responders.

METHODS

This secondary analysis used data from a National Drug Abuse Treatment Clinical Trials Network multi-site outpatient clinical trial in the United States. Adults with CUD (n = 302) received 12 weeks of contingency management, brief cessation counseling and were randomized to receive additionally either (1) N-Acetylcysteine or (2) placebo. Multivariable/machine learning models were used to classify treatment responders (i.e. two consecutive negative urine cannabinoid tests or a 50% reduction in days of use) versus non-responders using baseline demographic, medical, psychiatric and substance use information.

RESULTS

Prediction performance for various machine learning and regression prediction models yielded area under the curves (AUCs) >0.70 for four models (0.72-0.77), with support vector machine models having the highest overall accuracy (73%; 95% CI = 68-78%) and AUC (0.77; 95% CI = 0.72, 0.83). Fourteen variables were retained in at least three of four top models, including demographic (ethnicity, education), medical (diastolic/systolic blood pressure, overall health, neurological diagnosis), psychiatric (depressive symptoms, generalized anxiety disorder, antisocial personality disorder) and substance use (tobacco smoker, baseline cannabinoid level, amphetamine use, age of experimentation with other substances, cannabis withdrawal intensity) characteristics.

CONCLUSIONS

Multivariable/machine learning models can improve on chance prediction of treatment response to outpatient cannabis use disorder treatment, although further improvements in prediction performance are likely necessary for decisions about clinical care.

Prediction of Chinese clients' satisfaction with psychotherapy by machine learning

Background

Effective psychotherapy should satisfy the client, but that satisfaction depends on many factors. We do not fully understand the factors that affect client satisfaction with psychotherapy and how these factors synergistically affect a client's psychotherapy experience.

Aims

This study aims to use machine learning to predict Chinese clients' satisfaction with psychotherapy and analyze potential outcome contributors.

Methods

In this cross-sectional investigation, a self-compiled online questionnaire was delivered through the WeChat app. The information of 791 participants who had received psychotherapy was used in the study. A series of features, for example, the participants' demographic features and psychotherapy-related features, were chosen to distinguish between participants satisfied and dissatisfied with the psychotherapy they received. With our dataset, we trained seven supervised machine-learning-based algorithms to implement prediction models.

Results

Among the 791 participants, 619 (78.3%) reported being satisfied with the psychotherapy sessions that they received. The occupation of the clients, the location of psychotherapy, and the form of access to psychotherapy are the three most recognizable features that determined whether clients are satisfied with psychotherapy. The machine-learning model based on the CatBoost achieved the highest prediction performance in classifying satisfied and psychotherapy clients with an F1 score of 0.758.

Conclusion

This study clarified the factors related to clients' satisfaction with psychotherapy, and the machine-learning-based classifier accurately distinguished clients who were satisfied or unsatisfied with psychotherapy. These results will help provide better psychotherapy strategies for specific clients, so they may achieve better therapeutic outcomes.

A Machine Learning Approach for Predicting Non-Suicidal Self-Injury in Young Adults

Artificial intelligence techniques were explored to assess the ability to anticipate self-harming behaviour in the mental health context using a database collected by an app previously designed to record the emotional states and activities of a group of subjects exhibiting self-harm. Specifically, the Leave-One-Subject-Out technique was used to train classification trees with a maximum of five splits. The results show an accuracy of 84.78%, a sensitivity of 64.64% and a specificity of 85.53%. In addition, positive and negative predictive values were also obtained, with results of 14.48% and 98.47%, respectively. These results are in line with those reported in previous work using a multilevel mixed-effect regression analysis. The combination of apps and AI techniques is a powerful way to improve the tools to accompany and support the care and treatment of patients with this type of behaviour. These studies also guide the improvement of apps on the user side, simplifying and collecting more meaningful data, and on the therapist side, progressing in pathology treatments. Traditional therapy involves observing and reconstructing what had happened before episodes once they have occurred. This new generation of tools will make it possible to monitor the pathology more closely and to act preventively.

Using machine learning to predict heavy drinking during outpatient alcohol treatment

BACKGROUND

Accurate clinical prediction supports the effective treatment of alcohol use disorder (AUD) and other psychiatric disorders. Traditional statistical techniques have identified patient characteristics associated with treatment outcomes. However, less work has focused on systematically leveraging these associations to create optimal predictive models. The current study demonstrates how machine learning can be used to predict clinical outcomes in people completing outpatient AUD treatment.

METHOD

We used data from the COMBINE multisite clinical trial (n = 1383) to develop and test predictive models. We identified three priority prediction targets, including (1) heavy drinking during the first month of treatment, (2) heavy drinking during the last month of treatment, and (3) heavy drinking between weekly/bi-weekly sessions. Models were generated using the random forest algorithm. We used "leave sites out" partitioning to externally validate the models in trial sites that were not included in the model training. Stratified model development was used to test for sex differences in the relative importance of predictive features.

RESULTS

Models predicting heavy alcohol use during the first and last months of treatment showed internal cross-validation area under the curve (AUC) scores ranging from 0.67 to 0.74. AUC was comparable in the external validation using data from held-out sites (AUC range = 0.69 to 0.72). The model predicting between-session heavy drinking showed strong classification accuracy in internal cross-validation (AUC = 0.89) and external test samples (AUC range = 0.80 to 0.87). Stratified analyses showed substantial sex differences in optimal feature sets.

CONCLUSION

Machine learning techniques can predict alcohol treatment outcomes using routinely collected clinical data. This technique has the potential to greatly improve clinical prediction accuracy without requiring expensive or invasive assessment methods. More research is needed to understand how best to deploy these models.

Predicting therapy outcome in a digital mental health intervention for depression and anxiety: A machine learning approach

Objective

Predicting the outcomes of individual participants for treatment interventions appears central to making mental healthcare more tailored and effective. However, little work has been done to investigate the performance of machine learning-based predictions within digital mental health interventions. Therefore, this study evaluates the performance of machine learning in predicting treatment response in a digital mental health intervention designed for treating depression and anxiety.

Methods

Several algorithms were trained based on the data of 970 participants to predict a significant reduction in depression and anxiety symptoms using clinical and sociodemographic variables. As a random forest classifier performed best over cross-validation, it was used to predict the outcomes of 279 new participants.

Results

The random forest achieved an accuracy of 0.71 for the test set (base rate: 0.67, area under curve (AUC): 0.60, p = 0.001, balanced accuracy: 0.60). Additionally, predicted non-responders showed less average reduction of their Patient Health Questionnaire-9 (PHQ-9) (-2.7, p = 0.004) and General Anxiety Disorder Screener-7 values (-3.7, p < 0.001) compared to responders. Besides pre-treatment Patient Health Questionnaire-9 and General Anxiety Disorder Screener-7 values, the self-reported motivation, type of referral into the programme (self vs. healthcare provider) as well as Work Productivity and Activity Impairment Questionnaire items contributed most to the predictions.

Conclusions

This study provides evidence that social-demographic and clinical variables can be used for machine learning to predict therapy outcomes within the context of a therapist-supported digital mental health intervention. Despite the overall moderate performance, this appears promising as these predictions can potentially improve the outcomes of non-responders by monitoring their progress or by offering alternative or additional treatment.

Predicting Success of a Digital Self-Help Intervention for Alcohol and Substance Use With Machine Learning

Background

Digital self-help interventions for reducing the use of alcohol tobacco and other drugs (ATOD) have generally shown positive but small effects in controlling substance use and improving the quality of life of participants. Nonetheless, low adherence rates remain a major drawback of these digital interventions, with mixed results in (prolonged) participation and outcome. To prevent non-adherence, we developed models to predict success in the early stages of an ATOD digital self-help intervention and explore the predictors associated with participant's goal achievement.

Methods

We included previous and current participants from a widely used, evidence-based ATOD intervention from the Netherlands (Jellinek Digital Self-help). Participants were considered successful if they completed all intervention modules and reached their substance use goals (i.e., stop/reduce). Early dropout was defined as finishing only the first module. During model development, participants were split per substance (alcohol, tobacco, cannabis) and features were computed based on the log data of the first 3 days of intervention participation. Machine learning models were trained, validated and tested using a nested k-fold cross-validation strategy.

Results

From the 32,398 participants enrolled in the study, 80% of participants did not complete the first module of the intervention and were excluded from further analysis. From the remaining participants, the percentage of success for each substance was 30% for alcohol, 22% for cannabis and 24% for tobacco. The area under the Receiver Operating Characteristic curve was the highest for the Random Forest model trained on data from the alcohol and tobacco programs (0.71 95%CI 0.69-0.73) and (0.71 95%CI 0.67-0.76), respectively, followed by cannabis (0.67 95%CI 0.59-0.75). Quitting substance use instead of moderation as an intervention goal, initial daily consumption, no substance use on the weekends as a target goal and intervention engagement were strong predictors of success.

Discussion

Using log data from the first 3 days of intervention use, machine learning models showed positive results in identifying successful participants. Our results suggest the models were especially able to identify participants at risk of early dropout. Multiple variables were found to have high predictive value, which can be used to further improve the intervention.

Präzisionspsychiatrie und der Beitrag von Brain Imaging und anderen Biomarkern

'Precision Psychiatry' as the psychiatric variant of 'Precision Medicine' aims to provide high-level diagnosis and treatment based on robust biomarkers and tailored to the individual clinical, neurobiological, and genetic constitution of the patient. The specific peculiarity of psychiatry, in which disease entities are normatively defined based on clinical experience and are also significantly influenced by contemporary history, society and philosophy, has so far made the search for valid and reliable psychobiological connections difficult. Nevertheless, considerable progress has now been made in all areas of psychiatric research, made possible above all by the critical review and renewal of previous concepts of disease and psychopathology, the increased orientation towards neurobiology and genetics, and in particular the use of machine learning methods. Notably, modern machine learning methods make it possible to integrate high-dimensional and multimodal data sets and generate models which provide new psychobiological insights and offer the possibility of individualized, biomarker-driven single-subject prediction of diagnosis, therapy response and prognosis. The aim of the present review is therefore to introduce the concept of 'Precision Psychiatry' to the interested reader, to concisely present modern, machine learning methods required for this, and to clearly present the current state and future of biomarker-based 'precision psychiatry'.

Predicting alcohol dependence treatment outcomes: a prospective comparative study of clinical psychologists versus 'trained' machine learning models

BACKGROUND AND AIMS

Clinical staff are typically poor at predicting alcohol dependence treatment outcomes. Machine learning (ML) offers the potential to model complex clinical data more effectively. This study tested the predictive accuracy of ML algorithms demonstrated to be effective in predicting alcohol dependence outcomes, compared with clinical judgement and traditional linear regression.

DESIGN

Prospective study. ML models were trained on 1016 previously treated patients (training-set) who attended a hospital-based alcohol and drug clinic. ML models (n = 27), clinical psychologists (n = 10) and a 'traditional' logistic regression model (n = 1) predicted treatment outcome during the initial treatment session of an alcohol dependence programme.

SETTING

A 12-week cognitive behavioural therapy (CBT)-based abstinence programme for alcohol dependence in a hospital-based alcohol and drug clinic in Australia.

PARTICIPANTS

Prospective predictions were made for 220 new patients (test-set; 70.91% male, mean age = 35.78 years, standard deviation = 9.19). Sixty-nine (31.36%) patients successfully completed treatment.

MEASUREMENTS

Treatment success was the primary outcome variable. The cross-validated training-set accuracy of ML models was used to determine optimal parameters for selecting models for prospective prediction. Accuracy, sensitivity, specificity, area under the receiver operator curve (AUC), Brier score and calibration curves were calculated and compared across predictions.

FINDINGS

The mean aggregate accuracy of the ML models (63.06%) was higher than the mean accuracy of psychologist predictions (56.36%). The most accurate ML model achieved 70% accuracy, as did logistic regression. Both were more accurate than psychologists (P < 0.05) and had superior calibration. The high specificity for the selected ML (79%) and logistic regression (90%) meant they were significantly (P < 0.001) more effective than psychologists (50%) at correctly identifying patients whose treatment was unsuccessful. For ML and logistic regression, high specificity came at the expense of sensitivity (26 and 31%, respectively), resulting in poor prediction of successful patients.

CONCLUSIONS

Machine learning models and logistic regression appear to be more accurate than psychologists at predicting treatment outcomes in an abstinence programme for alcohol dependence, but sensitivity is low.

Using machine learning-based analysis for behavioral differentiation between anxiety and depression

Anxiety and depression are distinct—albeit overlapping—psychiatric diseases, currently diagnosed by self-reported-symptoms. This research presents a new diagnostic methodology, which tests rigorously for differences in cognitive biases among subclinical anxious and depressed individuals. 125 participants were divided into four groups based on the levels of their anxiety and depression symptoms. A comprehensive behavioral test battery detected and quantified various cognitive–emotional biases. Advanced machine-learning tools, developed for this study, analyzed these results. These tools detect unique patterns that characterize anxiety versus depression to predict group membership. The prediction model for differentiating between symptomatic participants (i.e., high symptoms of depression, anxiety, or both) compared to the non-symptomatic control group revealed a 71.44% prediction accuracy for the former (sensitivity) and 70.78% for the latter (specificity). 68.07% and 74.18% prediction accuracy was obtained for a two-group model with high depression/anxiety, respectively. The analysis also disclosed which specific behavioral measures contributed to the prediction, pointing to key cognitive mechanisms in anxiety versus depression. These results lay the ground for improved diagnostic instruments and more effective and focused individually-based treatment.

A scoping review of machine learning in psychotherapy research

Machine learning (ML) offers robust statistical and probabilistic techniques that can help to make sense of large amounts of data. This scoping review paper aims to broadly explore the nature of research activity using ML in the context of psychological talk therapies, highlighting the scope of current methods and considerations for clinical practice and directions for future research. Using a systematic search methodology, fifty-one studies were identified. A narrative synthesis indicates two types of studies, those who developed and tested an ML model (k=44), and those who reported on the feasibility of a particular treatment tool that uses an ML algorithm (k=7). Most model development studies used supervised learning techniques to classify or predict labeled treatment process or outcome data, whereas others used unsupervised techniques to identify clusters in the unlabeled patient or treatment data. Overall, the current applications of ML in psychotherapy research demonstrated a range of possible benefits for indications of treatment process, adherence, therapist skills and treatment response prediction, as well as ways to accelerate research through automated behavioral or linguistic process coding. Given the novelty and potential of this research field, these proof-of-concept studies are encouraging, however, do not necessarily translate to improved clinical practice (yet).

Machine-learning approaches to substance-abuse research: emerging trends and their implications

PURPOSE OF REVIEW

To provide an accessible overview of some of the most recent trends in the application of machine learning to the field of substance use disorders and their implications for future research and practice.

RECENT FINDINGS

Machine-learning (ML) techniques have recently been applied to substance use disorder (SUD) data for multiple predictive applications including detecting current abuse, assessing future risk and predicting treatment success. These models cover a wide range of machine-learning techniques and data types including physiological measures, longitudinal surveys, treatment outcomes, national surveys, medical records and social media.

SUMMARY

The application of machine-learning models to substance use disorder data shows significant promise, with some use cases and data types showing high predictive accuracy, particularly for models of physiological and behavioral measures for predicting current substance use, portending potential clinical diagnostic applications; however, these results are uneven, with some models performing poorly or at chance, a limitation likely reflecting insufficient data and/or weak validation methods. The field will likely benefit from larger and more multimodal datasets, greater standardization of data recording and rigorous testing protocols as well as greater use of modern deep neural network models applied to multimodal unstructured datasets.