Predicting therapy outcome in a digital mental health intervention for depression and anxiety: A machine learning approach

Predicting anxiety treatment outcome in community mental health services using linked health administrative data

Anxiety disorders is ranked as the most common class of mental illness disorders globally, affecting hundreds of millions of people and significantly impacting daily life. Developing reliable predictive models for anxiety treatment outcomes holds immense potential to help guide the development of personalised care, optimise resource allocation and improve patient outcomes. This research investigates whether community mental health treatment for anxiety disorder is associated with reliable changes in Kessler psychological distress scale (K10) scores and whether pre-treatment K10 scores and past health service interactions can accurately predict reliable change (improvement). The K10 assessment was administered to 46,938 public patients in a community setting within the Western Australia dataset in 2005-2022; of whom 3794 in 4067 episodes of care were reassessed at least twice for anxiety disorders, obsessive-compulsive disorder, or reaction to severe stress and adjustment disorders (ICD-10 codes F40-F43). Reliable change on the K10 was calculated and used with the post-treatment score as the outcome variables. Machine learning models were developed using features from a large health service administrative linked dataset that includes the pre-treatment K10 assessment as well as community mental health episodes of care, emergency department presentations, and inpatient admissions for prediction. The classification model achieved an area under the receiver operating characteristic curve of 0.76 as well as an F1 score, precision and recall of 0.69, and the regression model achieved an R2 of 0.37 with mean absolute error of 5.58 on the test dataset. While the prediction models achieved moderate performance, they also underscore the necessity for regular patient monitoring and the collection of more clinically relevant and contextual patient data to further improve prediction of treatment outcomes.

Predicting the Effectiveness of a Mindfulness Virtual Community Intervention for University Students: Machine Learning Model

BACKGROUND

Students' mental health crisis was recognized before the COVID-19 pandemic. Mindfulness virtual community (MVC), an 8-week web-based mindfulness and cognitive behavioral therapy program, has proven to be an effective web-based program to reduce symptoms of depression, anxiety, and stress. Predicting the success of MVC before a student enrolls in the program is essential to advise students accordingly.

OBJECTIVE

The objectives of this study were to investigate (1) whether we can predict MVC's effectiveness using sociodemographic and self-reported features and (2) whether exposure to mindfulness videos is highly predictive of the intervention's success.

METHODS

Machine learning models were developed to predict MVC's effectiveness, defined as success in reducing symptoms of depression, anxiety, and stress as measured using the Patient Health Questionnaire-9 (PHQ-9), the Beck Anxiety Inventory (BAI), and the Perceived Stress Scale (PSS), to at least the minimal clinically important difference. A data set representing a sample of undergraduate students (N=209) who took the MVC intervention between fall 2017 and fall 2018 was used for this secondary analysis. Random forest was used to measure the features' importance.

RESULTS

Gradient boosting achieved the best performance both in terms of area under the curve (AUC) and accuracy for predicting PHQ-9 (AUC=0.85 and accuracy=0.83) and PSS (AUC=1 and accuracy=1), and random forest had the best performance for predicting BAI (AUC=0.93 and accuracy=0.93). Exposure to online mindfulness videos was the most important predictor for the intervention's effectiveness for PHQ-9, BAI, and PSS, followed by the number of working hours per week.

CONCLUSIONS

The performance of the models to predict MVC intervention effectiveness for depression, anxiety, and stress is high. These models might be helpful for professionals to advise students early enough on taking the intervention or choosing other alternatives. The students' exposure to online mindfulness videos is the most important predictor for the effectiveness of the MVC intervention.

TRIAL REGISTRATION

ISRCTN Registry ISRCTN12249616; https://www.isrctn.com/ISRCTN12249616.

Which client with generalized anxiety disorder benefits from a mindfulness ecological momentary intervention versus a self-monitoring app? Developing a multivariable machine learning predictive model

Precision medicine methods (machine learning; ML) can identify which clients with generalized anxiety disorder (GAD) benefit from mindfulness ecological momentary intervention (MEMI) vs. self-monitoring app (SM). We used randomized controlled trial data of MEMI vs. SM for GAD (N = 110) and tested three ML models to predict one-month follow-up reliable improvement in GAD severity, perseverative cognitions (PC), trait mindfulness (TM), and executive function (EF). Eleven baseline predictors were tested regarding differential reliable change from MEMI vs. SM (age, sex, race, EF errors, inhibitory dyscontrol, set-shifting deficits, verbal fluency, working memory, GAD severity, TM, PC). The final top five prescriptive predictor models of all outcomes performed well (AUC = .752 .886). The following variables predicted better outcome from MEMI vs. SM: Higher GAD severity predicted more GAD improvement but less EF improvement. Elevated PC, inhibitory dyscontrol, and verbal dysfluency predicted better improvement in most outcomes. Greater set-shifting and TM predicted stronger improvements in GAD symptoms and TM. Older age predicted more alleviation of GAD and PC symptoms. Women exhibited more enhancements in trait mindfulness and EF than men. White individuals benefitted more than non-White. PC, TM, EF, and sociodemographic data might help predictive models optimize intervention selection for GAD.

Leveraging Machine Learning to Develop Digital Engagement Phenotypes of Users in a Digital Diabetes Prevention Program: Evaluation Study

BACKGROUND

Digital diabetes prevention programs (dDPPs) are effective "digital prescriptions" but have high attrition rates and program noncompletion. To address this, we developed a personalized automatic messaging system (PAMS) that leverages SMS text messaging and data integration into clinical workflows to increase dDPP engagement via enhanced patient-provider communication. Preliminary data showed positive results. However, further investigation is needed to determine how to optimize the tailoring of support technology such as PAMS based on a user's preferences to boost their dDPP engagement.

OBJECTIVE

This study evaluates leveraging machine learning (ML) to develop digital engagement phenotypes of dDPP users and assess ML's accuracy in predicting engagement with dDPP activities. This research will be used in a PAMS optimization process to improve PAMS personalization by incorporating engagement prediction and digital phenotyping. This study aims (1) to prove the feasibility of using dDPP user-collected data to build an ML model that predicts engagement and contributes to identifying digital engagement phenotypes, (2) to describe methods for developing ML models with dDPP data sets and present preliminary results, and (3) to present preliminary data on user profiling based on ML model outputs.

METHODS

Using the gradient-boosted forest model, we predicted engagement in 4 dDPP individual activities (physical activity, lessons, social activity, and weigh-ins) and general activity (engagement in any activity) based on previous short- and long-term activity in the app. The area under the receiver operating characteristic curve, the area under the precision-recall curve, and the Brier score metrics determined the performance of the model. Shapley values reflected the feature importance of the models and determined what variables informed user profiling through latent profile analysis.

RESULTS

We developed 2 models using weekly and daily DPP data sets (328,821 and 704,242 records, respectively), which yielded predictive accuracies above 90%. Although both models were highly accurate, the daily model better fitted our research plan because it predicted daily changes in individual activities, which was crucial for creating the "digital phenotypes." To better understand the variables contributing to the model predictor, we calculated the Shapley values for both models to identify the features with the highest contribution to model fit; engagement with any activity in the dDPP in the last 7 days had the most predictive power. We profiled users with latent profile analysis after 2 weeks of engagement (Bayesian information criterion=-3222.46) with the dDPP and identified 6 profiles of users, including those with high engagement, minimal engagement, and attrition.

CONCLUSIONS

Preliminary results demonstrate that applying ML methods with predicting power is an acceptable mechanism to tailor and optimize messaging interventions to support patient engagement and adherence to digital prescriptions. The results enable future optimization of our existing messaging platform and expansion of this methodology to other clinical domains.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04773834; https://www.clinicaltrials.gov/ct2/show/NCT04773834.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/26750.

Predictive Machine Learning Models for Assessing Lebanese University Students' Depression, Anxiety, and Stress During COVID-19

University students are experiencing a mental health crisis. COVID-19 has exacerbated this situation. We have surveyed students in 2 universities in Lebanon to gauge their mental health challenges. We have constructed a machine learning (ML) approach to predict symptoms of depression, anxiety, and stress based on demographics and self-rated health measures. Our approach involved developing 8 ML predictive models, including Logistic Regression (LR), multi-layer perceptron (MLP) neural network, support vector machine (SVM), random forest (RF) and XGBoost, AdaBoost, Naïve Bayes (NB), and K-Nearest neighbors (KNN). Following their construction, we compared their respective performances. Our evaluation shows that RF (AUC = 78.27%), NB (AUC = 76.37%), and AdaBoost (AUC = 72.96%) have provided the highest-performing AUC scores for depression, anxiety, and stress, respectively. Self-rated health is found to be the top feature in predicting depression, while age was the top feature in predicting anxiety and stress, followed by self-rated health. Future work will focus on using data augmentation approaches and extending to multi-class anxiety predictions.

Continuous Assessment of Function and Disability via Mobile Sensing: Real-World Data-Driven Feasibility Study

BACKGROUND

Functional limitations are associated with poor clinical outcomes, higher mortality, and disability rates, especially in older adults. Continuous assessment of patients' functionality is important for clinical practice; however, traditional questionnaire-based assessment methods are very time-consuming and infrequently used. Mobile sensing offers a great range of sources that can assess function and disability daily.

OBJECTIVE

This work aims to prove the feasibility of an interpretable machine learning pipeline for predicting function and disability based on the World Health Organization Disability Assessment Schedule (WHODAS) 2.0 outcomes of clinical outpatients, using passively collected digital biomarkers.

METHODS

One-month-long behavioral time-series data consisting of physical and digital activity descriptor variables were summarized using statistical measures (minimum, maximum, mean, median, SD, and IQR), creating 64 features that were used for prediction. We then applied a sequential feature selection to each WHODAS 2.0 domain (cognition, mobility, self-care, getting along, life activities, and participation) in order to find the most descriptive features for each domain. Finally, we predicted the WHODAS 2.0 functional domain scores using linear regression using the best feature subsets. We reported the mean absolute errors and the mean absolute percentage errors over 4 folds as goodness-of-fit statistics to evaluate the model and allow for between-domain performance comparison.

RESULTS

Our machine learning-based models for predicting patients' WHODAS functionality scores per domain achieved an average (across the 6 domains) mean absolute percentage error of 19.5%, varying between 14.86% (self-care domain) and 27.21% (life activities domain). We found that 5-19 features were sufficient for each domain, and the most relevant being the distance traveled, time spent at home, time spent walking, exercise time, and vehicle time.

CONCLUSIONS

Our findings show the feasibility of using machine learning-based methods to assess functional health solely from passively sensed mobile data. The feature selection step provides a set of interpretable features for each domain, ensuring better explainability to the models' decisions-an important aspect in clinical practice.

Predicting Individual Response to a Web-Based Positive Psychology Intervention: A Machine Learning Approach

Positive psychology interventions (PPIs) are effective at increasing happiness and decreasing depressive symptoms. PPIs are often administered as self-guided web-based interventions, but not all persons benefit from web-based interventions. Therefore, it is important to identify whether someone is likely to benefit from web-based PPIs, in order to triage persons who may not benefit from other interventions. In the current study, we used machine learning to predict individual response to a web-based PPI, in order to investigate baseline prognostic indicators of likelihood of response (N = 120). Our models demonstrated moderate correlations (happiness: r Test = 0.30 ± 0.09; depressive symptoms: r Test = 0.39 ± 0.06), indicating that baseline features can predict changes in happiness and depressive symptoms at a 6-month follow-up. Thus, machine learning can be used to predict outcome changes from a web-based PPI and has important clinical implications for matching individuals to PPIs based on their individual characteristics.

Personalization strategies in digital mental health interventions: a systematic review and conceptual framework for depressive symptoms

Introduction

Personalization is a much-discussed approach to improve adherence and outcomes for Digital Mental Health interventions (DMHIs). Yet, major questions remain open, such as (1) what personalization is, (2) how prevalent it is in practice, and (3) what benefits it truly has.

Methods

We address this gap by performing a systematic literature review identifying all empirical studies on DMHIs targeting depressive symptoms in adults from 2015 to September 2022. The search in Pubmed, SCOPUS and Psycinfo led to the inclusion of 138 articles, describing 94 distinct DMHIs provided to an overall sample of approximately 24,300 individuals.

Results

Our investigation results in the conceptualization of personalization as purposefully designed variation between individuals in an intervention's therapeutic elements or its structure. We propose to further differentiate personalization by what is personalized (i.e., intervention content, content order, level of guidance or communication) and the underlying mechanism [i.e., user choice, provider choice, decision rules, and machine-learning (ML) based approaches]. Applying this concept, we identified personalization in 66% of the interventions for depressive symptoms, with personalized intervention content (32% of interventions) and communication with the user (30%) being particularly popular. Personalization via decision rules (48%) and user choice (36%) were the most used mechanisms, while the utilization of ML was rare (3%). Two-thirds of personalized interventions only tailored one dimension of the intervention.

Discussion

We conclude that future interventions could provide even more personalized experiences and especially benefit from using ML models. Finally, empirical evidence for personalization was scarce and inconclusive, making further evidence for the benefits of personalization highly needed.

Systematic Review Registration

Identifier: CRD42022357408.

Prediction of Tinnitus Treatment Outcomes Based on EEG Sensors and TFI Score Using Deep Learning

Tinnitus is a hearing disorder that is characterized by the perception of sounds in the absence of an external source. Currently, there is no pharmaceutical cure for tinnitus, however, multiple therapies and interventions have been developed that improve or control associated distress and anxiety. We propose a new Artificial Intelligence (AI) algorithm as a digital prognostic health system that models electroencephalographic (EEG) data in order to predict patients' responses to tinnitus therapies. The EEG data was collected from patients prior to treatment and 3-months following a sound-based therapy. Feature selection techniques were utilised to identify predictive EEG variables with the best accuracy. The patients' EEG features from both the frequency and functional connectivity domains were entered as inputs that carry knowledge extracted from EEG into AI algorithms for training and predicting therapy outcomes. The AI models differentiated the patients' outcomes into either therapy responder or non-responder, as defined by their Tinnitus Functional Index (TFI) scores, with accuracies ranging from 98%-100%. Our findings demonstrate the potential use of AI, including deep learning, for predicting therapy outcomes in tinnitus. The research suggests an optimal configuration of the EEG sensors that are involved in measuring brain functional changes in response to tinnitus treatments. It identified which EEG electrodes are the most informative sensors and how the EEG frequency and functional connectivity can better classify patients into the responder and non-responder groups. This has potential for real-time monitoring of patient therapy outcomes at home.

Elite female footballers in England: an exploration of mental ill-health and help-seeking intentions

OBJECTIVES

Mental health research in sport is almost entirely focused on elite male athletes. However, recent research suggests that elite female athletes are at higher risk for mental ill-health when compared to their male counterparts. Given the recent growth of women's football in England and lack of research surrounding mental health in this population, this study sought to explore the prevalence of, and factors associated with depression, anxiety, and eating disorder symptoms in females competing in the top two tiers of English football.

METHODS

An anonymous online questionnaire pack, which measured personal and player characteristics and included the Patient Health Questionnaire (PHQ-9), Generalised Anxiety Disorder scale (GAD-7), Brief Eating Disorder Questionnaire (BEDA-Q), and General Help-Seeking Questionnaire (GHSQ), was completed between November 2020 and March 2021 by elite female footballers competing in the Women's Super League (WSL) and Women's Championship.

RESULTS

A total of 115 players completed the questionnaire (63 from the WSL; 52 from the Women's Championship). 36% displayed eating disorder symptoms (BEDAQ) 11% displayed moderate to severe anxiety symptoms (GAD-7), and 11% displayed moderate to severe depression symptoms (PHQ-9). Significant associations emerged between starting status, want for psychological support, student-athlete status, help-seeking intentions score, and mental ill-health symptoms.

CONCLUSION

In summary, elite female footballers in England reported significant mental health symptoms, particularly disordered eating symptoms. Further research should explore the experiences of mental ill-health in this population, focusing on the factors that were important in this study.

Advancing Prediction of Risk of Intraoperative Massive Blood Transfusion in Liver Transplantation With Machine Learning Models. A Multicenter Retrospective Study

Background

Liver transplantation surgery is often accompanied by massive blood loss and massive transfusion (MT), while MT can cause many serious complications related to high mortality. Therefore, there is an urgent need for a model that can predict the demand for MT to reduce the waste of blood resources and improve the prognosis of patients.

Objective

To develop a model for predicting intraoperative massive blood transfusion in liver transplantation surgery based on machine learning algorithms.

Methods

A total of 1,239 patients who underwent liver transplantation surgery in three large grade lll-A general hospitals of China from March 2014 to November 2021 were included and analyzed. A total of 1193 cases were randomly divided into the training set (70%) and test set (30%), and 46 cases were prospectively collected as a validation set. The outcome of this study was an intraoperative massive blood transfusion. A total of 27 candidate risk factors were collected, and recursive feature elimination (RFE) was used to select key features based on the Categorical Boosting (CatBoost) model. A total of ten machine learning models were built, among which the three best performing models and the traditional logistic regression (LR) method were prospectively verified in the validation set. The Area Under the Receiver Operating Characteristic Curve (AUROC) was used for model performance evaluation. The Shapley additive explanation value was applied to explain the complex ensemble learning models.

Results

Fifteen key variables were screened out, including age, weight, hemoglobin, platelets, white blood cells count, activated partial thromboplastin time, prothrombin time, thrombin time, direct bilirubin, aspartate aminotransferase, total protein, albumin, globulin, creatinine, urea. Among all algorithms, the predictive performance of the CatBoost model (AUROC: 0.810) was the best. In the prospective validation cohort, LR performed far less well than other algorithms.

Conclusion

A prediction model for massive blood transfusion in liver transplantation surgery was successfully established based on the CatBoost algorithm, and a certain degree of generalization verification is carried out in the validation set. The model may be superior to the traditional LR model and other algorithms, and it can more accurately predict the risk of massive blood transfusions and guide clinical decision-making.