An exploratory application of machine learning methods to optimize prediction of responsiveness to digital interventions for eating disorder symptoms

Review of machine learning solutions for eating disorders

BACKGROUND

Eating Disorders (EDs) are one of the most complex psychiatric disorders, with significant impairment of psychological and physical health, and psychosocial functioning, and are associated with low rates of early detection, low recovery and high relapse rates. This underscores the need for better diagnostic and treatment methods.

OBJECTIVE

This narrative review explores current Machine Learning (ML) and Artificial Intelligence (AI) applications in the domain of EDs, with a specific emphasis on clinical management in treatment settings. The primary objective are to (i) decrease the knowledge gap between ED researchers and AI-practitioners, by presenting the current state-of-the-art AI applications (including models for causality) in different ED use-cases; (ii) identify limitations of these existing AI interventions and how to address them.

RESULTS

AI/ML methods have been applied in different ED use-cases, including ED risk factor identification and incidence prediction (including the analysis of social media content in the general population), diagnosis, monitoring patients and treatment response and prognosis in clinical populations. A comparative analysis of AI-techniques deployed in these use-cases have been performed, considering factors such as complexity, flexibility, functionality, explainability and adaptability to healthcare constraints.

CONCLUSION

Multiple restrictions have been identified in the existing methods in ML and Causality in terms of achieving actionable healthcare for ED, like lack of good quality and quantity of data for models to train on, while requiring models to be flexible, high-performing, yet being explainable and producing counterfactual explanations, for ensuring the fairness and trustworthiness of its decisions. We conclude that to overcome these limitations and for future AI research and application in clinical management of ED, (i) careful considerations are required with regards to AI-model selection, and (ii) joint efforts from ED researcher and patient community are essential in building better quality and quantity of dedicated ED datasets and secure AI-solution framework.

Predictors, mediators, and moderators of response to digital interventions for eating disorders: A systematic review

OBJECTIVE

Digital interventions show promise as an effective prevention or self-management option for eating disorders (EDs). However, it remains unclear how, for whom, and through what mechanisms they work in this population, as a synthesis of outcome predictors, moderators, and mediators is lacking. This systematic review synthesized empirical research investigating predictors, mediators, and moderators of response to digital interventions for EDs.

METHOD

Six databases were searched (PROSPERO CRD42022295565) for studies that assessed predictors, moderators, or mediators of response (i.e., uptake, drop-out, engagement, and symptom level change) to a digital prevention or treatment program for EDs. Variables were grouped into several overarching categories (demographic, symptom severity, psychological, etc.) and were synthesized qualitatively across samples without a formally diagnosed ED (typically prevention-focused) and samples with a formally diagnosed ED (typically treatment-focused).

RESULTS

Eighty-six studies were included. For studies recruiting samples without a formal diagnosis (n = 70 studies), most predictors explored were statistically unrelated to outcome, although participant age, baseline symptom severity, confidence to change, motivation, and program engagement showed preliminary evidence of prognostic potential. No robust moderators or mediators were identified. Few studies recruiting samples with a formal diagnosis emerged (n = 16), of which no reliable predictors, moderators, or mediators were identified.

DISCUSSION

It remains unclear how, for whom, and under what circumstances digital programs targeting EDs work. We offer several recommendations for future research with the aim of advancing understanding of client characteristics and intervention elements that signal success from this intervention modality.

PUBLIC SIGNIFICANCE

Digital interventions have shown potential as an effective, scalable, and accessible intervention option for EDs. However, responsiveness varies, so advancing understanding of predictors, mediators, and moderators of outcome to digital interventions for EDs is needed. Such knowledge is important for enabling safe and efficient treatment matching, and for informing future development of effective digital interventions.

Exploration of the individual and combined effects of predictors of engagement, dropout, and change from digital interventions for recurrent binge eating

OBJECTIVE

Our ability to predict responsiveness to digital interventions for eating disorders has thus far been poor, potentially for three reasons: (1) there has been a narrow set of predictors explored; (2) prediction has mostly focused on symptom change, ignoring other aspects of the user journey (uptake, early engagement); and (3) there is an excessive focus on the unique effects of predictors rather than the combined contributions of a predictor set. We evaluated the univariate and multivariate effects of outcome predictors in the context of a randomized trial (n = 398) of digitally delivered interventions for recurrent binge eating.

METHOD

Thirty baseline variables were selected as predictors, ranging from specific symptoms, to key protective factors, to technological acceptance, and to online treatment attitudes. Outcomes included uptake, early engagement, and remission. Univariate (d) and multivariate (D) standardized mean differences were calculated to estimate the individual and combined effects of predictors, respectively.

RESULTS

At the univariate level, few predictors produced an effect size larger than what is considered small (d > .20) across outcomes. However, our multivariate approach enhanced prediction (Ds = .65 to 1.12), producing accuracy rates greater than chance (63%-71% accuracy). Less than half of the chosen variables proved to be useful in contributing to predictions in multivariate models.

CONCLUSION

Findings suggest that accuracy in outcome prediction from digitally delivered interventions may be better driven by the aggregation of many small effects rather than one or several largely influential predictors. Replication with different data streams (sensor, neuroimaging) would be useful.

PUBLIC SIGNIFICANCE

Our ability to predict who will and will not benefit from digital interventions for eating disorders has been poor. We highlight the viability of a multivariate approach to outcome prediction, whereby prediction may be better driven by the aggregation of many small effects rather than one or a few influential predictors.

Dataset size versus homogeneity: A machine learning study on pooling intervention data in e-mental health dropout predictions

Objective

This study proposes a way of increasing dataset sizes for machine learning tasks in Internet-based Cognitive Behavioral Therapy through pooling interventions. To this end, it (1) examines similarities in user behavior and symptom data among online interventions for patients with depression, social anxiety, and panic disorder and (2) explores whether these similarities suffice to allow for pooling the data together, resulting in more training data when prediction intervention dropout.

Methods

A total of 6418 routine care patients from the Internet Psychiatry in Stockholm are analyzed using (1) clustering and (2) dropout prediction models. For the latter, prediction models trained on each individual intervention's data are compared to those trained on all three interventions pooled into one dataset. To investigate if results vary with dataset size, the prediction is repeated using small and medium dataset sizes.

Results

The clustering analysis identified three distinct groups that are almost equally spread across interventions and are instead characterized by different activity levels. In eight out of nine settings investigated, pooling the data improves prediction results compared to models trained on a single intervention dataset. It is further confirmed that models trained on small datasets are more likely to overestimate prediction results.

Conclusion

The study reveals similar patterns of patients with depression, social anxiety, and panic disorder regarding online activity and intervention dropout. As such, this work offers pooling different interventions' data as a possible approach to counter the problem of small dataset sizes in psychological research.

Can a Single Variable Predict Early Dropout From Digital Health Interventions? Comparison of Predictive Models From Two Large Randomized Trials

BACKGROUND

A single generalizable metric that accurately predicts early dropout from digital health interventions has the potential to readily inform intervention targets and treatment augmentations that could boost retention and intervention outcomes. We recently identified a type of early dropout from digital health interventions for smoking cessation, specifically, users who logged in during the first week of the intervention and had little to no activity thereafter. These users also had a substantially lower smoking cessation rate with our iCanQuit smoking cessation app compared with users who used the app for longer periods.

OBJECTIVE

This study aimed to explore whether log-in count data, using standard statistical methods, can precisely predict whether an individual will become an iCanQuit early dropout while validating the approach using other statistical methods and randomized trial data from 3 other digital interventions for smoking cessation (combined randomized N=4529).

METHODS

Standard logistic regression models were used to predict early dropouts for individuals receiving the iCanQuit smoking cessation intervention app, the National Cancer Institute QuitGuide smoking cessation intervention app, the WebQuit.org smoking cessation intervention website, and the Smokefree.gov smoking cessation intervention website. The main predictors were the number of times a participant logged in per day during the first 7 days following randomization. The area under the curve (AUC) assessed the performance of the logistic regression models, which were compared with decision trees, support vector machine, and neural network models. We also examined whether 13 baseline variables that included a variety of demographics (eg, race and ethnicity, gender, and age) and smoking characteristics (eg, use of e-cigarettes and confidence in being smoke free) might improve this prediction.

RESULTS

The AUC for each logistic regression model using only the first 7 days of log-in count variables was 0.94 (95% CI 0.90-0.97) for iCanQuit, 0.88 (95% CI 0.83-0.93) for QuitGuide, 0.85 (95% CI 0.80-0.88) for WebQuit.org, and 0.60 (95% CI 0.54-0.66) for Smokefree.gov. Replacing logistic regression models with more complex decision trees, support vector machines, or neural network models did not significantly increase the AUC, nor did including additional baseline variables as predictors. The sensitivity and specificity were generally good, and they were excellent for iCanQuit (ie, 0.91 and 0.85, respectively, at the 0.5 classification threshold).

CONCLUSIONS

Logistic regression models using only the first 7 days of log-in count data were generally good at predicting early dropouts. These models performed well when using simple, automated, and readily available log-in count data, whereas including self-reported baseline variables did not improve the prediction. The results will inform the early identification of people at risk of early dropout from digital health interventions with the goal of intervening further by providing them with augmented treatments to increase their retention and, ultimately, their intervention outcomes.