An individualized treatment rule to optimize probability of remission by continuation, switching, or combining antidepressant medications after failing a first-line antidepressant in a two-stage randomized trial

Machine learning approaches to evaluate heterogeneous treatment effects in randomized controlled trials: a scoping review

BACKGROUND AND OBJECTIVES

Estimating heterogeneous treatment effects (HTEs) in randomized controlled trials (RCTs) has received substantial attention recently. This has led to the development of several statistical and machine learning (ML) algorithms to assess HTEs through identifying individualized treatment effects. However, a comprehensive review of these algorithms is lacking. We thus aimed to catalog and outline currently available statistical and ML methods for identifying HTEs via effect modeling using clinical RCT data and summarize how they have been applied in practice.

STUDY DESIGN AND SETTING

We performed a scoping review using prespecified search terms in MEDLINE and Embase, aiming to identify studies that assessed HTEs using advanced statistical and ML methods in RCT data published from 2010 to 2022.

RESULTS

Among a total of 32 studies identified in the review, 17 studies applied existing algorithms to RCT data, and 15 extended existing algorithms or proposed new algorithms. Applied algorithms included penalized regression, causal forest, Bayesian causal forest, and other metalearner frameworks. Of these methods, causal forest was the most frequently used (7 studies) followed by Bayesian causal forest (4 studies). Most applications were in cardiology (6 studies), followed by psychiatry (4 studies). We provide example R codes in simulated data to illustrate how to implement these algorithms.

CONCLUSION

This review identified and outlined various algorithms currently used to identify HTEs and individualized treatment effects in RCT data. Given the increasing availability of new algorithms, analysts should carefully select them after examining model performance and considering how the models will be used in practice.

Towards Outcome-Driven Patient Subgroups: A Machine Learning Analysis Across Six Depression Treatment Studies

BACKGROUND

Major depressive disorder (MDD) is a heterogeneous condition; multiple underlying neurobiological and behavioral substrates are associated with treatment response variability. Understanding the sources of this variability and predicting outcomes has been elusive. Machine learning (ML) shows promise in predicting treatment response in MDD, but its application is limited by challenges to the clinical interpretability of ML models, and clinicians often lack confidence in model results. In order to improve the interpretability of ML models in clinical practice, our goal was to demonstrate the derivation of treatment-relevant patient profiles comprised of clinical and demographic information using a novel ML approach.

METHODS

We analyzed data from six clinical trials of pharmacological treatment for depression (total n = 5438) using the Differential Prototypes Neural Network (DPNN), a ML model that derives patient prototypes which can be used to derive treatment-relevant patient clusters while learning to generate probabilities for differential treatment response. A model classifying remission and outputting individual remission probabilities for five first-line monotherapies and three combination treatments was trained using clinical and demographic data. Prototypes were evaluated for interpretability by assessing differences in feature distributions (e.g. age, sex, symptom severity) and treatment-specific outcomes.

RESULTS

A 3-prototype model achieved an area under the receiver operating curve of 0.66 and an expected absolute improvement in remission rate for those receiving the best predicted treatment of 6.5% (relative improvement of 15.6%) compared to the population remission rate. We identified three treatment-relevant patient clusters. Cluster A patients tended to be younger, to have increased levels of fatigue, and more severe symptoms. Cluster B patients tended to be older, female, have less severe symptoms, and the highest remission rates. Cluster C patients had more severe symptoms, lower remission rates, more psychomotor agitation, more intense suicidal ideation, and more somatic genital symptoms.

CONCLUSION

It is possible to produce novel treatment-relevant patient profiles using ML models; doing so may improve interpretability of ML models and the quality of precision medicine treatments for MDD.

A practical risk calculator for suicidal behavior among transitioning U.S. Army soldiers: results from the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS)

BACKGROUND

Risk of suicide-related behaviors is elevated among military personnel transitioning to civilian life. An earlier report showed that high-risk U.S. Army soldiers could be identified shortly before this transition with a machine learning model that included predictors from administrative systems, self-report surveys, and geospatial data. Based on this result, a Veterans Affairs and Army initiative was launched to evaluate a suicide-prevention intervention for high-risk transitioning soldiers. To make targeting practical, though, a streamlined model and risk calculator were needed that used only a short series of self-report survey questions.

METHODS

We revised the original model in a sample of n = 8335 observations from the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS) who participated in one of three Army STARRS 2011-2014 baseline surveys while in service and in one or more subsequent panel surveys (LS1: 2016-2018, LS2: 2018-2019) after leaving service. We trained ensemble machine learning models with constrained numbers of item-level survey predictors in a 70% training sample. The outcome was self-reported post-transition suicide attempts (SA). The models were validated in the 30% test sample.

RESULTS

Twelve-month post-transition SA prevalence was 1.0% (s.e. = 0.1). The best constrained model, with only 17 predictors, had a test sample ROC-AUC of 0.85 (s.e. = 0.03). The 10-30% of respondents with the highest predicted risk included 44.9-92.5% of 12-month SAs.

CONCLUSIONS

An accurate SA risk calculator based on a short self-report survey can target transitioning soldiers shortly before leaving service for intervention to prevent post-transition SA.

Methodological and Quality Flaws in the Use of Artificial Intelligence in Mental Health Research: Systematic Review

BACKGROUND

Artificial intelligence (AI) is giving rise to a revolution in medicine and health care. Mental health conditions are highly prevalent in many countries, and the COVID-19 pandemic has increased the risk of further erosion of the mental well-being in the population. Therefore, it is relevant to assess the current status of the application of AI toward mental health research to inform about trends, gaps, opportunities, and challenges.

OBJECTIVE

This study aims to perform a systematic overview of AI applications in mental health in terms of methodologies, data, outcomes, performance, and quality.

METHODS

A systematic search in PubMed, Scopus, IEEE Xplore, and Cochrane databases was conducted to collect records of use cases of AI for mental health disorder studies from January 2016 to November 2021. Records were screened for eligibility if they were a practical implementation of AI in clinical trials involving mental health conditions. Records of AI study cases were evaluated and categorized by the International Classification of Diseases 11th Revision (ICD-11). Data related to trial settings, collection methodology, features, outcomes, and model development and evaluation were extracted following the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) guideline. Further, evaluation of risk of bias is provided.

RESULTS

A total of 429 nonduplicated records were retrieved from the databases and 129 were included for a full assessment-18 of which were manually added. The distribution of AI applications in mental health was found unbalanced between ICD-11 mental health categories. Predominant categories were Depressive disorders (n=70) and Schizophrenia or other primary psychotic disorders (n=26). Most interventions were based on randomized controlled trials (n=62), followed by prospective cohorts (n=24) among observational studies. AI was typically applied to evaluate quality of treatments (n=44) or stratify patients into subgroups and clusters (n=31). Models usually applied a combination of questionnaires and scales to assess symptom severity using electronic health records (n=49) as well as medical images (n=33). Quality assessment revealed important flaws in the process of AI application and data preprocessing pipelines. One-third of the studies (n=56) did not report any preprocessing or data preparation. One-fifth of the models were developed by comparing several methods (n=35) without assessing their suitability in advance and a small proportion reported external validation (n=21). Only 1 paper reported a second assessment of a previous AI model. Risk of bias and transparent reporting yielded low scores due to a poor reporting of the strategy for adjusting hyperparameters, coefficients, and the explainability of the models. International collaboration was anecdotal (n=17) and data and developed models mostly remained private (n=126).

CONCLUSIONS

These significant shortcomings, alongside the lack of information to ensure reproducibility and transparency, are indicative of the challenges that AI in mental health needs to face before contributing to a solid base for knowledge generation and for being a support tool in mental health management.

Development and Validation of a Model to Predict Posttraumatic Stress Disorder and Major Depression After a Motor Vehicle Collision

IMPORTANCE

A substantial proportion of the 40 million people in the US who present to emergency departments (EDs) each year after traumatic events develop posttraumatic stress disorder (PTSD) or major depressive episode (MDE). Accurately identifying patients at high risk in the ED would facilitate the targeting of preventive interventions.

OBJECTIVES

To develop and validate a prediction tool based on ED reports after a motor vehicle collision to predict PTSD or MDE 3 months later.

DESIGN, SETTING, AND PARTICIPANTS

The Advancing Understanding of Recovery After Trauma (AURORA) study is a longitudinal study that examined adverse posttraumatic neuropsychiatric sequalae among patients who presented to 28 US urban EDs in the immediate aftermath of a traumatic experience. Enrollment began on September 25, 2017. The 1003 patients considered in this diagnostic/prognostic report completed 3-month assessments by January 31, 2020. Each patient received a baseline ED assessment along with follow-up self-report surveys 2 weeks, 8 weeks, and 3 months later. An ensemble machine learning method was used to predict 3-month PTSD or MDE from baseline information. Data analysis was performed from November 1, 2020, to May 31, 2021.

MAIN OUTCOMES AND MEASURES

The PTSD Checklist for DSM-5 was used to assess PTSD and the Patient Reported Outcomes Measurement Information System Depression Short-Form 8b to assess MDE.

RESULTS

A total of 1003 patients (median [interquartile range] age, 34.5 [24-43] years; 715 [weighted 67.9%] female; 100 [weighted 10.7%] Hispanic, 537 [weighted 52.7%] non-Hispanic Black, 324 [weighted 32.2%] non-Hispanic White, and 42 [weighted 4.4%] of non-Hispanic other race or ethnicity were included in this study. A total of 274 patients (weighted 26.6%) met criteria for 3-month PTSD or MDE. An ensemble machine learning model restricted to 30 predictors estimated in a training sample (patients from the Northeast or Midwest) had good prediction accuracy (mean [SE] area under the curve [AUC], 0.815 [0.031]) and calibration (mean [SE] integrated calibration index, 0.040 [0.002]; mean [SE] expected calibration error, 0.039 [0.002]) in an independent test sample (patients from the South). Patients in the top 30% of predicted risk accounted for 65% of all 3-month PTSD or MDE, with a mean (SE) positive predictive value of 58.2% (6.4%) among these patients at high risk. The model had good consistency across regions of the country in terms of both AUC (mean [SE], 0.789 [0.025] using the Northeast as the test sample and 0.809 [0.023] using the Midwest as the test sample) and calibration (mean [SE] integrated calibration index, 0.048 [0.003] using the Northeast as the test sample and 0.024 [0.001] using the Midwest as the test sample; mean [SE] expected calibration error, 0.034 [0.003] using the Northeast as the test sample and 0.025 [0.001] using the Midwest as the test sample). The most important predictors in terms of Shapley Additive Explanations values were symptoms of anxiety sensitivity and depressive disposition, psychological distress in the 30 days before motor vehicle collision, and peritraumatic psychosomatic symptoms.

CONCLUSIONS AND RELEVANCE

The results of this study suggest that a short set of questions feasible to administer in an ED can predict 3-month PTSD or MDE with good AUC, calibration, and geographic consistency. Patients at high risk can be identified in the ED for targeting if cost-effective preventive interventions are developed.