Can personalized treatment prediction improve the outcomes, compared with the group average approach, in a randomized trial? Developing and validating a multivariable prediction model in a pragmatic megatrial of acute treatment for major depression

Machine learning for antidepressant treatment selection in depression

Finding the right antidepressant for the individual patient with major depressive disorder can be a difficult endeavor and is mostly based on trial-and-error. Machine learning (ML) is a promising tool to personalize antidepressant prescription. In this review, we summarize the current evidence of ML in the selection of antidepressants and conclude that its value for clinical practice is still limited. Apart from the current focus on effectiveness, several other factors should be taken into account to make ML-based prediction models useful for clinical application.

Towards Outcome-Driven Patient Subgroups: A Machine Learning Analysis Across Six Depression Treatment Studies

BACKGROUND

Major depressive disorder (MDD) is a heterogeneous condition; multiple underlying neurobiological and behavioral substrates are associated with treatment response variability. Understanding the sources of this variability and predicting outcomes has been elusive. Machine learning (ML) shows promise in predicting treatment response in MDD, but its application is limited by challenges to the clinical interpretability of ML models, and clinicians often lack confidence in model results. In order to improve the interpretability of ML models in clinical practice, our goal was to demonstrate the derivation of treatment-relevant patient profiles comprised of clinical and demographic information using a novel ML approach.

METHODS

We analyzed data from six clinical trials of pharmacological treatment for depression (total n = 5438) using the Differential Prototypes Neural Network (DPNN), a ML model that derives patient prototypes which can be used to derive treatment-relevant patient clusters while learning to generate probabilities for differential treatment response. A model classifying remission and outputting individual remission probabilities for five first-line monotherapies and three combination treatments was trained using clinical and demographic data. Prototypes were evaluated for interpretability by assessing differences in feature distributions (e.g. age, sex, symptom severity) and treatment-specific outcomes.

RESULTS

A 3-prototype model achieved an area under the receiver operating curve of 0.66 and an expected absolute improvement in remission rate for those receiving the best predicted treatment of 6.5% (relative improvement of 15.6%) compared to the population remission rate. We identified three treatment-relevant patient clusters. Cluster A patients tended to be younger, to have increased levels of fatigue, and more severe symptoms. Cluster B patients tended to be older, female, have less severe symptoms, and the highest remission rates. Cluster C patients had more severe symptoms, lower remission rates, more psychomotor agitation, more intense suicidal ideation, and more somatic genital symptoms.

CONCLUSION

It is possible to produce novel treatment-relevant patient profiles using ML models; doing so may improve interpretability of ML models and the quality of precision medicine treatments for MDD.

Development and validation of a prediction model for rehospitalization among people with schizophrenia discharged from acute inpatient care

Objective

Relapses and rehospitalization prevent the recovery of individuals with schizophrenia or related psychoses. We aimed to build a model to predict the risk of rehospitalization among people with schizophrenia or related psychoses, including those with multiple episodes.

Methods

This retrospective cohort study included individuals aged 18 years or older, with schizophrenia or related psychoses, and discharged between January 2014 and December 2018 from one of three Japanese psychiatric hospital acute inpatient care ward. We collected nine predictors at the time of recruitment, followed up with the participants for 12 months, and observed whether psychotic relapse had occurred. Next, we applied the Cox regression model and used an elastic net to avoid overfitting. Then, we examined discrimination using bootstrapping, Steyerberg's method, and "leave-one-hospital-out" cross-validation. We also constructed a bias-corrected calibration plot.

Results

Data from a total of 805 individuals were analyzed. The significant predictors were the number of previous hospitalizations (HR 1.42, 95% CI 1.22-1.64) and the current length of stay in days (HR 1.31, 95% CI 1.04-1.64). In model development for relapse, Harrell's c-index was 0.59 (95% CI 0.55-0.63). The internal and internal-external validation for rehospitalization showed Harrell's c-index to be 0.64 (95% CI 0.59-0.69) and 0.66 (95% CI 0.57-0.74), respectively. The calibration plot was found to be adequate.

Conclusion

The model showed moderate discrimination of readmission after discharge. Carefully defining a research question by seeking needs among the population with chronic schizophrenia with multiple episodes may be key to building a useful model.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Methodological and Quality Flaws in the Use of Artificial Intelligence in Mental Health Research: Systematic Review

BACKGROUND

Artificial intelligence (AI) is giving rise to a revolution in medicine and health care. Mental health conditions are highly prevalent in many countries, and the COVID-19 pandemic has increased the risk of further erosion of the mental well-being in the population. Therefore, it is relevant to assess the current status of the application of AI toward mental health research to inform about trends, gaps, opportunities, and challenges.

OBJECTIVE

This study aims to perform a systematic overview of AI applications in mental health in terms of methodologies, data, outcomes, performance, and quality.

METHODS

A systematic search in PubMed, Scopus, IEEE Xplore, and Cochrane databases was conducted to collect records of use cases of AI for mental health disorder studies from January 2016 to November 2021. Records were screened for eligibility if they were a practical implementation of AI in clinical trials involving mental health conditions. Records of AI study cases were evaluated and categorized by the International Classification of Diseases 11th Revision (ICD-11). Data related to trial settings, collection methodology, features, outcomes, and model development and evaluation were extracted following the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) guideline. Further, evaluation of risk of bias is provided.

RESULTS

A total of 429 nonduplicated records were retrieved from the databases and 129 were included for a full assessment-18 of which were manually added. The distribution of AI applications in mental health was found unbalanced between ICD-11 mental health categories. Predominant categories were Depressive disorders (n=70) and Schizophrenia or other primary psychotic disorders (n=26). Most interventions were based on randomized controlled trials (n=62), followed by prospective cohorts (n=24) among observational studies. AI was typically applied to evaluate quality of treatments (n=44) or stratify patients into subgroups and clusters (n=31). Models usually applied a combination of questionnaires and scales to assess symptom severity using electronic health records (n=49) as well as medical images (n=33). Quality assessment revealed important flaws in the process of AI application and data preprocessing pipelines. One-third of the studies (n=56) did not report any preprocessing or data preparation. One-fifth of the models were developed by comparing several methods (n=35) without assessing their suitability in advance and a small proportion reported external validation (n=21). Only 1 paper reported a second assessment of a previous AI model. Risk of bias and transparent reporting yielded low scores due to a poor reporting of the strategy for adjusting hyperparameters, coefficients, and the explainability of the models. International collaboration was anecdotal (n=17) and data and developed models mostly remained private (n=126).

CONCLUSIONS

These significant shortcomings, alongside the lack of information to ensure reproducibility and transparency, are indicative of the challenges that AI in mental health needs to face before contributing to a solid base for knowledge generation and for being a support tool in mental health management.

An individualized treatment rule to optimize probability of remission by continuation, switching, or combining antidepressant medications after failing a first-line antidepressant in a two-stage randomized trial

BACKGROUND

There is growing interest in using composite individualized treatment rules (ITRs) to guide depression treatment selection, but best approaches for doing this are not widely known. We develop an ITR for depression remission based on secondary analysis of a recently published trial for second-line antidepression medication selection using a cutting-edge ensemble machine learning method.

METHODS

Data come from the SUN(^_^)D trial, an open-label, assessor blinded pragmatic trial of previously-untreated patients with major depressive disorder from 48 clinics in Japan. Initial clinic-level randomization assigned patients to 50 or 100 mg/day sertraline. We focus on the 1549 patients who failed to remit within 3 weeks and were then rerandomized at the individual-level to continuation with sertraline, switching to mirtazapine, or combining mirtazapine with sertraline. The outcome was remission 9 weeks post-baseline. Predictors included socio-demographics, clinical characteristics, baseline symptoms, changes in symptoms between baseline and week 3, and week 3 side effects.

RESULTS

Optimized treatment was associated with significantly increased cross-validated week 9 remission rates in both samples [5.3% (2.4%), p = 0.016 50 mg/day sample; 5.1% (2.7%), p = 0.031 100 mg/day sample] compared to randomization (30.1-30.8%). Optimization was also associated with significantly increased remission in both samples compared to continuation [24.7% in both: 11.2% (3.8%), p = 0.002 50 mg/day sample; 11.7% (3.9%), p = 0.001 100 mg/day sample]. Non-significant gains were found for optimization compared to switching or combining.

CONCLUSIONS

An ITR can be developed to improve second-line antidepressant selection, but replication in a larger study with more comprehensive baseline predictors might produce stronger and more stable results.

Treatment success of internet-based vestibular rehabilitation in general practice: development and internal validation of a prediction model

OBJECTIVES

To develop and internally validate prediction models to assess treatment success of both stand-alone and blended online vestibular rehabilitation (VR) in patients with chronic vestibular syndrome.

DESIGN

Secondary analysis of a randomised controlled trial.

SETTING

59 general practices in The Netherlands.

PARTICIPANTS

202 adults, aged 50 years and older with a chronic vestibular syndrome who received either stand-alone VR (98) or blended VR (104). Stand-alone VR consisted of a 6-week, internet-based intervention with weekly online sessions and daily exercises. In blended VR, the same intervention was supplemented with physiotherapy support.

MAIN OUTCOME MEASURES

Successful treatment was defined as: clinically relevant improvement of (1) vestibular symptoms (≥3 points improvement Vertigo Symptom Scale-Short Form); (2) vestibular-related disability (>11 points improvement Dizziness Handicap Inventory); and (3) both vestibular symptoms and vestibular-related disability. We assessed performance of the predictive models by applying calibration plots, Hosmer-Lemeshow statistics, area under the receiver operating characteristic curves (AUC) and applied internal validation.

RESULTS

Improvement of vestibular symptoms, vestibular-related disability or both was seen in 121, 81 and 64 participants, respectively. We generated predictive models for each outcome, resulting in different predictors in the final models. Calibration for all models was adequate with non-significant Hosmer-Lemeshow statistics, but the discriminative ability of the final predictive models was poor (AUC 0.54 to 0.61). None of the identified models are therefore suitable for use in daily general practice to predict treatment success of online VR.

CONCLUSION

It is difficult to predict treatment success of internet-based VR and it remains unclear who should be treated with stand-alone VR or blended VR. Because we were unable to develop a useful prediction model, the decision to offer stand-alone or blended VR should for now be based on availability, cost effectiveness and patient preference.

TRIAL REGISTRATION NUMBER

The Netherlands Trial Register NTR5712.