Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges

The challenges of using machine learning models in psychiatric research and clinical practice

To understand the complex nature of heterogeneous psychiatric disorders, scientists and clinicians are required to employ a wide range of clinical, endophenotypic, neuroimaging, genomic, and environmental data to understand the biological mechanisms of psychiatric illness before this knowledge is applied into clinical setting. Machine learning (ML) is an automated process that can detect patterns from large multidimensional datasets and can supersede conventional statistical methods as it can detect both linear and non-linear relationships. Due to this advantage, ML has potential to enhance our understanding, improve diagnosis, prognosis and treatment of psychiatric disorders. The current review provides an in-depth examination of, and offers practical guidance for, the challenges encountered in the application of ML models in psychiatric research and clinical practice. These challenges include the curse of dimensionality, data quality, the 'black box' problem, hyperparameter tuning, external validation, class imbalance, and data representativeness. These challenges are particularly critical in the context of psychiatry as it is expected that researchers will encounter them during the stages of ML model development and deployment. We detail practical solutions and best practices to effectively mitigate the outlined challenges. These recommendations have the potential to improve reliability and interpretability of ML models in psychiatry.

Use of Machine Learning Algorithms Based on Text, Audio, and Video Data in the Prediction of Anxiety and Posttraumatic Stress in General and Clinical Populations: A Systematic Review

Research in machine learning (ML) algorithms using natural behavior (i.e., text, audio, and video data) suggests that these techniques could contribute to personalization in psychology and psychiatry. However, a systematic review of the current state of the art is missing. Moreover, individual studies often target ML experts who may overlook potential clinical implications of their findings. In a narrative accessible to mental health professionals, we present a systematic review conducted in 5 psychology and 2 computer science databases. We included 128 studies that assessed the predictive power of ML algorithms using text, audio, and/or video data in the prediction of anxiety and posttraumatic stress disorder. Most studies (n = 87) were aimed at predicting anxiety, while the remainder (n = 41) focused on posttraumatic stress disorder. They were mostly published since 2019 in computer science journals and tested algorithms using text (n = 72) as opposed to audio or video. Studies focused mainly on general populations (n = 92) and less on laboratory experiments (n = 23) or clinical populations (n = 13). Methodological quality varied, as did reported metrics of the predictive power, hampering comparison across studies. Two-thirds of studies, which focused on both disorders, reported acceptable to very good predictive power (including high-quality studies only). The results of 33 studies were uninterpretable, mainly due to missing information. Research into ML algorithms using natural behavior is in its infancy but shows potential to contribute to diagnostics of mental disorders, such as anxiety and posttraumatic stress disorder, in the future if standardization of methods, reporting of results, and research in clinical populations are improved.

The Dynamic Interplay Between Puberty and Structural Brain Development as a Predictor of Mental Health Difficulties in Adolescence: A Systematic Review

Puberty is a time of intense reorganization of brain structure and a high-risk period for the onset of mental health problems, with variations in pubertal timing and tempo intensifying this risk. We conducted 2 systematic reviews of articles published up to February 1, 2024, focusing on 1) the role of brain structure in the relationship between puberty and mental health, and 2) precision psychiatry research evaluating the utility of puberty in making individualized predictions of mental health outcomes in young people. The first review provides inconsistent evidence about whether and how pubertal and psychopathological processes may interact in relation to brain development. While most studies found an association between early puberty and mental health difficulties in adolescents, evidence on whether brain structure mediates this relationship is mixed. The pituitary gland was found to be associated with mental health status during this time, possibly through its central role in regulating puberty and its function in the hypothalamic-pituitary-gonadal and hypothalamic-pituitary-adrenal axes. In the second review, the design of studies that have explored puberty in predictive models did not allow for a quantification of its predictive power. However, when puberty was evaluated through physically observable characteristics rather than hormonal measures, it was more commonly identified as a predictor of depression, anxiety, and suicidality in adolescence. Social processes may be more relevant than biological ones to the link between puberty and mental health problems and represent an important target for educational strategies.

Diagnostic accuracy of large language models in psychiatry

INTRODUCTION

Medical decision-making is crucial for effective treatment, especially in psychiatry where diagnosis often relies on subjective patient reports and a lack of high-specificity symptoms. Artificial intelligence (AI), particularly Large Language Models (LLMs) like GPT, has emerged as a promising tool to enhance diagnostic accuracy in psychiatry. This comparative study explores the diagnostic capabilities of several AI models, including Aya, GPT-3.5, GPT-4, GPT-3.5 clinical assistant (CA), Nemotron, and Nemotron CA, using clinical cases from the DSM-5.

METHODS

We curated 20 clinical cases from the DSM-5 Clinical Cases book, covering a wide range of psychiatric diagnoses. Four advanced AI models (GPT-3.5 Turbo, GPT-4, Aya, Nemotron) were tested using prompts to elicit detailed diagnoses and reasoning. The models' performances were evaluated based on accuracy and quality of reasoning, with additional analysis using the Retrieval Augmented Generation (RAG) methodology for models accessing the DSM-5 text.

RESULTS

The AI models showed varied diagnostic accuracy, with GPT-3.5 and GPT-4 performing notably better than Aya and Nemotron in terms of both accuracy and reasoning quality. While models struggled with specific disorders such as cyclothymic and disruptive mood dysregulation disorders, others excelled, particularly in diagnosing psychotic and bipolar disorders. Statistical analysis highlighted significant differences in accuracy and reasoning, emphasizing the superiority of the GPT models.

DISCUSSION

The application of AI in psychiatry offers potential improvements in diagnostic accuracy. The superior performance of the GPT models can be attributed to their advanced natural language processing capabilities and extensive training on diverse text data, enabling more effective interpretation of psychiatric language. However, models like Aya and Nemotron showed limitations in reasoning, indicating a need for further refinement in their training and application.

CONCLUSION

AI holds significant promise for enhancing psychiatric diagnostics, with certain models demonstrating high potential in interpreting complex clinical descriptions accurately. Future research should focus on expanding the dataset and integrating multimodal data to further enhance the diagnostic capabilities of AI in psychiatry.

Risk prediction model for in-stent restenosis following PCI: a systematic review

Introduction

The morbidity and mortality rates of coronary heart disease are significant, with PCI being the primary treatment. The high incidence of ISR following PCI poses a challenge to its effectiveness. Currently, there are numerous studies on ISR risk prediction models after PCI, but the quality varies and there is still a lack of systematic evaluation and analysis.

Methods

To systematically retrieve and evaluate the risk prediction models for ISR after PCI. A comprehensive search was conducted across 9 databases from inception to March 1, 2024. The screening of literature and extraction of data were independently carried out by two investigators, utilizing the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS). Additionally, the risk of bias and applicability were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

Results

A total of 17 studies with 29 models were included, with a sample size of 175-10,004 cases, and the incidence of outcome events was 5.79%-58.86%. The area under the receiver operating characteristic curve was 0.530-0.953. The top 5 predictors with high frequency were diabetes, number of diseased vessels, age, LDL-C and stent diameter. Bias risk assessment into the research of the risk of higher bias the applicability of the four study better.

Discussion

The overall risk of bias in the current ISR risk prediction model post-PCI is deemed high. Moving forward, it is imperative to enhance study design and specify the reporting process, optimize and validate the model, and enhance its performance.

Development and initial evaluation of a clinical prediction model for risk of treatment resistance in first-episode psychosis: Schizophrenia Prediction of Resistance to Treatment (SPIRIT)

BACKGROUND

A clinical tool to estimate the risk of treatment-resistant schizophrenia (TRS) in people with first-episode psychosis (FEP) would inform early detection of TRS and overcome the delay of up to 5 years in starting TRS medication.

AIMS

To develop and evaluate a model that could predict the risk of TRS in routine clinical practice.

METHOD

We used data from two UK-based FEP cohorts (GAP and AESOP-10) to develop and internally validate a prognostic model that supports identification of patients at high-risk of TRS soon after FEP diagnosis. Using sociodemographic and clinical predictors, a model for predicting risk of TRS was developed based on penalised logistic regression, with missing data handled using multiple imputation. Internal validation was undertaken via bootstrapping, obtaining optimism-adjusted estimates of the model's performance. Interviews and focus groups with clinicians were conducted to establish clinically relevant risk thresholds and understand the acceptability and perceived utility of the model.

RESULTS

We included seven factors in the prediction model that are predominantly assessed in clinical practice in patients with FEP. The model predicted treatment resistance among the 1081 patients with reasonable accuracy; the model's C-statistic was 0.727 (95% CI 0.723-0.732) prior to shrinkage and 0.687 after adjustment for optimism. Calibration was good (expected/observed ratio: 0.999; calibration-in-the-large: 0.000584) after adjustment for optimism.

CONCLUSIONS

We developed and internally validated a prediction model with reasonably good predictive metrics. Clinicians, patients and carers were involved in the development process. External validation of the tool is needed followed by co-design methodology to support implementation in early intervention services.

Developing an individualized treatment rule for Veterans with major depressive disorder using electronic health records

Efforts to develop an individualized treatment rule (ITR) to optimize major depressive disorder (MDD) treatment with antidepressant medication (ADM), psychotherapy, or combined ADM-psychotherapy have been hampered by small samples, small predictor sets, and suboptimal analysis methods. Analyses of large administrative databases designed to approximate experiments followed iteratively by pragmatic trials hold promise for resolving these problems. The current report presents a proof-of-concept study using electronic health records (EHR) of n = 43,470 outpatients beginning MDD treatment in Veterans Health Administration Primary Care Mental Health Integration (PC-MHI) clinics, which offer access not only to ADMs but also psychotherapy and combined ADM-psychotherapy. EHR and geospatial databases were used to generate an extensive baseline predictor set (5,865 variables). The outcome was a composite measure of at least one serious negative event (suicide attempt, psychiatric emergency department visit, psychiatric hospitalization, suicide death) over the next 12 months. Best-practices methods were used to adjust for nonrandom treatment assignment and to estimate a preliminary ITR in a 70% training sample and to evaluate the ITR in the 30% test sample. Statistically significant aggregate variation was found in overall probability of the outcome related to baseline predictors (AU-ROC = 0.68, S.E. = 0.01), with test sample outcome prevalence of 32.6% among the 5% of patients having highest predicted risk compared to 7.1% in the remainder of the test sample. The ITR found that psychotherapy-only was the optimal treatment for 56.0% of patients (roughly 20% lower risk of the outcome than if receiving one of the other treatments) and that treatment type was unrelated to outcome risk among other patients. Change in aggregate treatment costs of implementing this ITR would be negligible, as 16.1% fewer patients would be prescribed ADMs and 2.9% more would receive psychotherapy. A pragmatic trial would be needed to confirm the accuracy of the ITR.

Individual treatment selection for patients with post-traumatic stress disorder: External validation of a personalised advantage index

OBJECTIVE

To test the predictive accuracy and generalisability of a personalised advantage index (PAI) model designed to support treatment selection for Post-Traumatic Stress Disorder (PTSD).

METHOD

A PAI model developed by Deisenhofer et al. (2018) was used to predict treatment outcomes in a statistically independent dataset including archival records for N = 152 patients with PSTD who accessed either trauma-focussed cognitive behavioural therapy or eye movement desensitisation and reprocessing in routine care. Outcomes were compared between patients who received their PAI-indicated optimal treatment versus those who received their suboptimal treatment.

RESULTS

The model did not yield treatment specific predictions and patients who had received their PAI-indicated optimal treatment did not have better treatment outcomes in this external validation sample.

CONCLUSION

This PAI model did not generalise to an external validation sample.

Precision psychiatry: predicting predictability

Precision psychiatry is an emerging field that aims to provide individualized approaches to mental health care. An important strategy to achieve this precision is to reduce uncertainty about prognosis and treatment response. Multivariate analysis and machine learning are used to create outcome prediction models based on clinical data such as demographics, symptom assessments, genetic information, and brain imaging. While much emphasis has been placed on technical innovation, the complex and varied nature of mental health presents significant challenges to the successful implementation of these models. From this perspective, I review ten challenges in the field of precision psychiatry, including the need for studies on real-world populations and realistic clinical outcome definitions, and consideration of treatment-related factors such as placebo effects and non-adherence to prescriptions. Fairness, prospective validation in comparison to current practice and implementation studies of prediction models are other key issues that are currently understudied. A shift is proposed from retrospective studies based on linear and static concepts of disease towards prospective research that considers the importance of contextual factors and the dynamic and complex nature of mental health.

Prospective Follow-Up of Adolescents With and at Risk for Depression: Protocol and Methods of the Identifying Depression Early in Adolescence Risk Stratified Cohort Longitudinal Assessments

Objective

To present the protocol and methods for the prospective longitudinal assessments-including clinical and digital phenotyping approaches-of the Identifying Depression Early in Adolescence Risk Stratified Cohort (IDEA-RiSCo) study, which comprises Brazilian adolescents stratified at baseline by risk of developing depression or presence of depression.

Method

Of 7,720 screened adolescents aged 14 to 16 years, we recruited 150 participants (75 boys, 75 girls) based on a composite risk score: 50 with low risk for developing depression (LR), 50 with high risk for developing depression (HR), and 50 with an active untreated major depressive episode (MDD). Three annual follow-up assessments were conducted, involving clinical measures (parent- and adolescent-reported questionnaires and psychiatrist assessments), active and passive data sensing via smartphones, and neurobiological measures (neuroimaging and biological material samples). Retention rates were 96% (Wave 1), 94% (Wave 2), and 88% (Wave 3), with no significant differences by sex or group (p > .05). Participants highlighted their familiarity with the research team and assessment process as a motivator for sustained engagement.

Discussion

This protocol relied on novel aspects, such as the use of a WhatsApp bot, which is particularly pertinent for low- to-middle-income countries, and the collection of information from diverse sources in a longitudinal design, encompassing clinical data, self-reports, parental reports, Global Positioning System (GPS) data, and ecological momentary assessments. The study engaged adolescents over an extensive period and demonstrated the feasibility of conducting a prospective follow-up study with a risk-enriched cohort of adolescents in a middle-income country, integrating mobile technology with traditional methodologies to enhance longitudinal data collection.

Optimising the use of electronic medical records for large scale research in psychiatry

The explosion and abundance of digital data could facilitate large-scale research for psychiatry and mental health. Research using so-called "real world data"-such as electronic medical/health records-can be resource-efficient, facilitate rapid hypothesis generation and testing, complement existing evidence (e.g. from trials and evidence-synthesis) and may enable a route to translate evidence into clinically effective, outcomes-driven care for patient populations that may be under-represented. However, the interpretation and processing of real-world data sources is complex because the clinically important 'signal' is often contained in both structured and unstructured (narrative or "free-text") data. Techniques for extracting meaningful information (signal) from unstructured text exist and have advanced the re-use of routinely collected clinical data, but these techniques require cautious evaluation. In this paper, we survey the opportunities, risks and progress made in the use of electronic medical record (real-world) data for psychiatric research.

Predicting post-treatment symptom severity for adults receiving psychological therapy in routine care for generalised anxiety disorder: a machine learning approach

Approximately half of generalised anxiety disorder (GAD) patients do not recover from first-line treatments, and no validated prediction models exist to inform individuals or clinicians of potential treatment benefits. This study aimed to develop and validate an accurate and explainable prediction model of post-treatment GAD symptom severity. Data from adults receiving treatment for GAD in eight Improving Access to Psychological Therapies (IAPT) services (n=15,859) were separated into training, validation and holdout datasets. Thirteen machine learning algorithms were compared using 10-fold cross-validation, against two simple clinically relevant comparison models. The best-performing model was tested on the holdout dataset and model-specific explainability measures identified the most important predictors. A Bayesian Additive Regression Trees model out-performed all comparison models (MSE=16.54 [95 % CI=15.58; 17.51]; MAE=3.19; R²=0.33, including a single predictor linear regression model: MSE=20.70 [95 % CI=19.58; 21.82]; MAE=3.94; R²=0.14). The five most important predictors were: PHQ-9 anhedonia, GAD-7 annoyance/irritability, restlessness and fear items, then the referral-assessment waiting time. The best-performing model accurately predicted post-treatment GAD symptom severity using only pre-treatment data, outperforming comparison models that approximated clinical judgement and remaining within the GAD-7 error of measurement and minimal clinically important differences. This model could inform treatment decision-making and provide desired information to clinicians and patients receiving treatment for GAD.

On the Road to Individualizing Pharmacotherapy for Adolescents and Adults with Schizophrenia - Results from an Expert Consensus Following the Delphi Method

Introduction

Schizophrenia is a severe mental illness that usually begins in late adolescence or early adulthood. Current pharmacological treatments, while acceptably effective for many patients, are rarely clinically tailored or individualized. The lack of sufficient etiopathological knowledge of the disease, together with overall comparable effect sizes for efficacy between available antipsychotics and the absence of clinically actionable biomarkers, has hindered the advance of individualized medicine in the treatment of schizophrenia. Nevertheless, some degree of stratification based on clinical markers could guide treatment choices and help clinicians move toward individualized psychiatry. To this end, a panel of experts met to formally discuss the current approach to individualized treatment in schizophrenia and to define how treatment individualization could help improve clinical outcomes.

Methods

A task force of seven experts iteratively developed, evaluated, and refined questionnaire items, which were then evaluated using the Delphi method. Descriptive statistics were used to summarize and rank expert responses. Expert discussion, informed by the results of a scoping review on personalizing the pharmacologic treatment of adults and adolescents with schizophrenia, ultimately generated recommendations to guide individualized pharmacologic treatment in this population.

Results

There was substantial agreement among the expert group members, resulting in the following recommendations: 1) individualization of treatment requires consideration of the patient's diagnosis, clinical presentation, comorbidities, previous treatment response, drug tolerability, adherence patterns, and social factors; 2) patient preferences should be considered in a shared decision-making approach; 3) identified barriers to personalized care that need to be overcome include the lack of actionable biomarkers and mechanistic similarities between available treatments, but digital tools should be increasingly used to enhance individualized treatment.

Conclusion

Individualized care can help provide effective, tailored treatments based on an individual's clinical characteristics, disease trajectory, family and social environment, and goals and preferences.

Predictive modeling of response to repetitive transcranial magnetic stimulation in treatment-resistant depression

Identifying predictors of treatment response to repetitive transcranial magnetic stimulation (rTMS) remain elusive in treatment-resistant depression (TRD). Leveraging electronic medical records (EMR), this retrospective cohort study applied supervised machine learning (ML) to sociodemographic, clinical, and treatment-related data to predict depressive symptom response (>50% reduction on PHQ-9) and remission (PHQ-9 < 5) following rTMS in 232 patients with TRD (mean age: 54.5, 63.4% women) treated at the University of California, San Diego Interventional Psychiatry Program between 2017 and 2023. ML models were internally validated using nested cross-validation and Shapley values were calculated to quantify contributions of each feature to response prediction. The best-fit models proved reasonably accurate at discriminating treatment responders (Area under the curve (AUC): 0.689 [0.638, 0.740], p < 0.01) and remitters (AUC 0.745 [0.692, 0.797], p < 0.01), though only the response model was well-calibrated. Both models were associated with significant net benefits, indicating their potential utility for clinical decision-making. Shapley values revealed that patients with comorbid anxiety, obesity, concurrent psychiatric medication use, and more chronic TRD were less likely to respond or remit following rTMS. Patients with trauma and former tobacco users were more likely to respond. Furthermore, delivery of intermittent theta burst stimulation and more rTMS sessions were associated with superior outcomes. These findings highlight the potential of ML-guided techniques to guide clinical decision-making for rTMS treatment in patients with TRD to optimize therapeutic outcomes.

Individualized prediction models in ADHD: a systematic review and meta-regression

There have been increasing efforts to develop prediction models supporting personalised detection, prediction, or treatment of ADHD. We overviewed the current status of prediction science in ADHD by: (1) systematically reviewing and appraising available prediction models; (2) quantitatively assessing factors impacting the performance of published models. We did a PRISMA/CHARMS/TRIPOD-compliant systematic review (PROSPERO: CRD42023387502), searching, until 20/12/2023, studies reporting internally and/or externally validated diagnostic/prognostic/treatment-response prediction models in ADHD. Using meta-regressions, we explored the impact of factors affecting the area under the curve (AUC) of the models. We assessed the study risk of bias with the Prediction Model Risk of Bias Assessment Tool (PROBAST). From 7764 identified records, 100 prediction models were included (88% diagnostic, 5% prognostic, and 7% treatment-response). Of these, 96% and 7% were internally and externally validated, respectively. None was implemented in clinical practice. Only 8% of the models were deemed at low risk of bias; 67% were considered at high risk of bias. Clinical, neuroimaging, and cognitive predictors were used in 35%, 31%, and 27% of the studies, respectively. The performance of ADHD prediction models was increased in those models including, compared to those models not including, clinical predictors (β = 6.54, p = 0.007). Type of validation, age range, type of model, number of predictors, study quality, and other type of predictors did not alter the AUC. Several prediction models have been developed to support the diagnosis of ADHD. However, efforts to predict outcomes or treatment response have been limited, and none of the available models is ready for implementation into clinical practice. The use of clinical predictors, which may be combined with other type of predictors, seems to improve the performance of the models. A new generation of research should address these gaps by conducting high quality, replicable, and externally validated models, followed by implementation research.

Five-year follow-up of the iBerry Study: screening in early adolescence to identify those at risk of psychopathology in emerging adulthood

The iBerry Study, a Dutch population-based high-risk cohort (n = 1022) examines the transition from subclinical symptoms to psychiatric disorders in adolescents. Here, we present the first follow-up measurement, approximately 3 years after baseline assessment and 5 years after the screening based on self-reported emotional and behavioral problems (SDQ-Y). We give an update on the data collection, details on the (non)response, and the results on psychopathology outcomes. The first follow-up (2019-2022) had a response rate of 79% (n = 807). Our results at baseline (mean age 15.0 years) have shown the effectiveness of using the SDQ-Y to select a cohort oversampled for the risk of psychopathology. At first follow-up (mean age 18.1 years), the previously administered SDQ-Y remains predictive for selecting adolescents at risk. At follow-up, 47% of the high-risk adolescents showed significant mental health problems based on self- and parent reports and 46% of the high-risk adolescents met the criteria for multiple DSM-5 diagnoses. Compared to low-risk adolescents, high-risk adolescents had a sevenfold higher odds of significant emotional and behavioral problems at follow-up. Comprehensive assessment on psychopathology, substance abuse, psychotic symptoms, suicidality, nonsuicidal self-injury, addiction to social media and/or video gaming, and delinquency, as well as social development, and the utilization of healthcare and social services were conducted. This wave, as well as the ones to follow, track these adolescents into their young adulthood to identify risk factors, elucidate causal mechanisms, and discern pathways leading to both common and severe mental disorders. Results from the iBerry Study will provide leads for preventive interventions.

Predicting treatment outcome based on resting-state functional connectivity in internalizing mental disorders: A systematic review and meta-analysis

Predicting treatment outcome in internalizing mental disorders prior to treatment initiation is pivotal for precision mental healthcare. In this regard, resting-state functional connectivity (rs-FC) and machine learning have often shown promising prediction accuracies. This systematic review and meta-analysis evaluates these studies, considering their risk of bias through the Prediction Model Study Risk of Bias Assessment Tool (PROBAST). We examined the predictive performance of features derived from rs-FC, identified features with the highest predictive value, and assessed the employed machine learning pipelines. We searched the electronic databases Scopus, PubMed and PsycINFO on the 12th of December 2022, which resulted in 13 included studies. The mean balanced accuracy for predicting treatment outcome was 77% (95% CI: [72%- 83%]). rs-FC of the dorsolateral prefrontal cortex had high predictive value in most studies. However, a high risk of bias was identified in all studies, compromising interpretability. Methodological recommendations are provided based on a comprehensive exploration of the studies' machine learning pipelines, and potential fruitful developments are discussed.

Proteomic Biomarkers for the Prediction of Transition to Psychosis in Individuals at Clinical High Risk: A Multi-cohort Model Development Study

Psychosis risk prediction is one of the leading challenges in psychiatry. Previous investigations have suggested that plasma proteomic data may be useful in accurately predicting transition to psychosis in individuals at clinical high risk (CHR). We hypothesized that an a priori-specified proteomic prediction model would have strong predictive accuracy for psychosis risk and aimed to replicate longitudinal associations between plasma proteins and transition to psychosis. This study used plasma samples from participants in 3 CHR cohorts: the North American Prodrome Longitudinal Studies 2 and 3, and the NEURAPRO randomized control trial (total n = 754). Plasma proteomic data were quantified using mass spectrometry. The primary outcome was transition to psychosis over the study follow-up period. Logistic regression models were internally validated, and optimism-corrected performance metrics derived with a bootstrap procedure. In the overall sample of CHR participants (age: 18.5, SD: 3.9; 51.9% male), 20.4% (n = 154) developed psychosis within 4.4 years. The a priori-specified model showed poor risk-prediction accuracy for the development of psychosis (C-statistic: 0.51 [95% CI: 0.50, 0.59], calibration slope: 0.45). At a group level, Complement C8B, C4B, C5, and leucine-rich α-2 glycoprotein 1 (LRG1) were associated with transition to psychosis but did not surpass correction for multiple comparisons. This study did not confirm the findings from a previous proteomic prediction model of transition from CHR to psychosis. Certain complement proteins may be weakly associated with transition at a group level. Previous findings, derived from small samples, should be interpreted with caution.

Machine learning models predict the emergence of depression in Argentinean college students during periods of COVID-19 quarantine

Introduction

The COVID-19 pandemic has exacerbated mental health challenges, particularly depression among college students. Detecting at-risk students early is crucial but remains challenging, particularly in developing countries. Utilizing data-driven predictive models presents a viable solution to address this pressing need.

Aims

1) To develop and compare machine learning (ML) models for predicting depression in Argentinean students during the pandemic. 2) To assess the performance of classification and regression models using appropriate metrics. 3) To identify key features driving depression prediction.

Methods

A longitudinal dataset (N = 1492 college students) captured T1 and T2 measurements during the Argentinean COVID-19 quarantine. ML models, including linear logistic regression classifiers/ridge regression (LogReg/RR), random forest classifiers/regressors, and support vector machines/regressors (SVM/SVR), are employed. Assessed features encompass depression and anxiety scores (at T1), mental disorder/suicidal behavior history, quarantine sub-period information, sex, and age. For classification, models' performance on test data is evaluated using Area Under the Precision-Recall Curve (AUPRC), Area Under the Receiver Operating Characteristic curve, Balanced Accuracy, F1 score, and Brier loss. For regression, R-squared (R2), Mean Absolute Error, and Mean Squared Error are assessed. Univariate analyses are conducted to assess the predictive strength of each individual feature with respect to the target variable. The performance of multi- vs univariate models is compared using the mean AUPRC score for classifiers and the R2 score for regressors.

Results

The highest performance is achieved by SVM and LogReg (e.g., AUPRC: 0.76, 95% CI: 0.69, 0.81) and SVR and RR models (e.g., R2 for SVR and RR: 0.56, 95% CI: 0.45, 0.64 and 0.45, 0.63, respectively). Univariate models, particularly LogReg and SVM using depression (AUPRC: 0.72, 95% CI: 0.64, 0.79) or anxiety scores (AUPRC: 0.71, 95% CI: 0.64, 0.78) and RR using depression scores (R2: 0.48, 95% CI: 0.39, 0.57) exhibit performance levels close to those of the multivariate models, which include all features.

Discussion

These findings highlight the relevance of pre-existing depression and anxiety conditions in predicting depression during quarantine, underscoring their comorbidity. ML models, particularly SVM/SVR and LogReg/RR, demonstrate potential in the timely detection of at-risk students. However, further studies are needed before clinical implementation.

A Systematic Evaluation of Machine Learning-Based Biomarkers for Major Depressive Disorder

Importance

Biological psychiatry aims to understand mental disorders in terms of altered neurobiological pathways. However, for one of the most prevalent and disabling mental disorders, major depressive disorder (MDD), no informative biomarkers have been identified.

Objective

To evaluate whether machine learning (ML) can identify a multivariate biomarker for MDD.

Design, Setting, and Participants

This study used data from the Marburg-Münster Affective Disorders Cohort Study, a case-control clinical neuroimaging study. Patients with acute or lifetime MDD and healthy controls aged 18 to 65 years were recruited from primary care and the general population in Münster and Marburg, Germany, from September 11, 2014, to September 26, 2018. The Münster Neuroimaging Cohort (MNC) was used as an independent partial replication sample. Data were analyzed from April 2022 to June 2023.

Exposure

Patients with MDD and healthy controls.

Main Outcome and Measure

Diagnostic classification accuracy was quantified on an individual level using an extensive ML-based multivariate approach across a comprehensive range of neuroimaging modalities, including structural and functional magnetic resonance imaging and diffusion tensor imaging as well as a polygenic risk score for depression.

Results

Of 1801 included participants, 1162 (64.5%) were female, and the mean (SD) age was 36.1 (13.1) years. There were a total of 856 patients with MDD (47.5%) and 945 healthy controls (52.5%). The MNC replication sample included 1198 individuals (362 with MDD [30.1%] and 836 healthy controls [69.9%]). Training and testing a total of 4 million ML models, mean (SD) accuracies for diagnostic classification ranged between 48.1% (3.6%) and 62.0% (4.8%). Integrating neuroimaging modalities and stratifying individuals based on age, sex, treatment, or remission status does not enhance model performance. Findings were replicated within study sites and also observed in structural magnetic resonance imaging within MNC. Under simulated conditions of perfect reliability, performance did not significantly improve. Analyzing model errors suggests that symptom severity could be a potential focus for identifying MDD subgroups.

Conclusion and Relevance

Despite the improved predictive capability of multivariate compared with univariate neuroimaging markers, no informative individual-level MDD biomarker-even under extensive ML optimization in a large sample of diagnosed patients-could be identified.

Validation of a youth suicide risk calculator in an adult sample with bipolar disorder

BACKGROUND

Bipolar disorder (BD) conveys the highest risk of suicide of all mental disorders. We sought to externally validate a risk calculator (RC) of suicide attempts developed in youth with BD from the Course and Outcome of Bipolar Youth (COBY) study in an adult sample.

METHODS

A prospective cohort of adults with BD from the National Institute of Mental Health Collaborative Depression Study (CDS; N = 427; mean (+/- SD) age at intake (36 +/- 13 years)) was secondarily analyzed to validate the COBY RC for one-year risk of suicide attempts/deaths. Nine of the ten predictor variables from the COBY RC were available in the CDS and used: age, age of mood disorder onset, first and second (partial) degree family history of suicide, history of psychotic symptoms, substance use disorder, prior suicide attempt, socioeconomic status, and non-suicidal self-injury (prospectively, incompletely at baseline).

RESULTS

Over a mean (SD) follow-up of 19 (10) years, 29 % of the CDS sample attempted suicide. The RC predicted suicide attempts/deaths over one-year follow-up with an area under the receiver operating characteristic curve (AUC) of 0.78 (95 % CI 0.75-0.80). The RC performed slightly better in those with a younger age of mood disorder onset.

LIMITATIONS

Clinical samples may limit generalizability; the RC does not assess more acute suicide risk.

CONCLUSIONS

One-year risk of suicide attempts/deaths can be predicted with acceptable accuracy in youth and adults with BD, comparable to commonly used RCs to predict cardiovascular risk. This RC may help identify higher-risk individuals with BD for personalized treatment and research. https://cobysuicideattemptsrc.shinyapps.io/Shiny.

Risk prediction models for postoperative delirium in elderly patients with fragility hip fracture: A systematic review and critical appraisal

BACKGROUND

Elderly patients with fragility hip fracture continue to experience a high rate of postoperative delirium (POD), which has a significant negative impact on prognosis and imposes a huge economic burden. A number of risk prediction models have been developed to detect POD early. However, the risk of bias and clinical applicability of the models remain unclear. The aim of this study was to systematically evaluate risk prediction models for POD.

METHODS

CNKI, WanFang DATA, Vip Database, SinoMed, PubMed, Web of Science, Embase, and the Cochrane Library were searched for studies published by July 2023. The literature was screened independently by two investigators. The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) and the Prediction Model Risk of Bias Assessment Tool (PROBAST) were respectively used for data extraction, risk of bias, and applicability assessment.

RESULTS

A total of 16 studies on the construction of POD risk prediction models were included. The area under the ROC curve of the models ranges from 0.670 to 0.957. The most common predictors of POD included age, history of dementia, history of delirium, ASA classification, preoperative waiting time, and preoperative albumin level. All models had a high risk of bias, mainly due to inadequate sample size, inappropriate handling of missing data, a lack of model performance evaluation, and overfitting of the models.

CONCLUSIONS

Overall, risk prediction models for POD in fragility hip fracture patients are still in the development stage. The majority of POD prediction models have substantial bias risks, attributable primarily to poor reporting of analysis and evaluation of model performance. In future research, it is recommended to conduct validation of the models or develop localized prediction models with demonstrated high performance, with the aim of benefiting POD screening.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42023449153.

Brain Functional Connectivity and Anatomical Features as Predictors of Cognitive Behavioral Therapy Outcome for Anxiety in Youths

Background

Because pediatric anxiety disorders precede the onset of many other problems, successful prediction of response to the first-line treatment, cognitive-behavioral therapy (CBT), could have major impact. However, existing clinical models are weakly predictive. The current study evaluates whether structural and resting-state functional magnetic resonance imaging can predict post-CBT anxiety symptoms.

Methods

Two datasets were studied: (A) one consisted of n=54 subjects with an anxiety diagnosis, who received 12 weeks of CBT, and (B) one consisted of n=15 subjects treated for 8 weeks. Connectome Predictive Modeling (CPM) was used to predict treatment response, as assessed with the PARS; additionally we investigated models using anatomical features, instead of functional connectivity. The main analysis included network edges positively correlated with treatment outcome, and age, sex, and baseline anxiety severity as predictors. Results from alternative models and analyses also are presented. Model assessments utilized 1000 bootstraps, resulting in a 95% CI for R2, r and mean absolute error (MAE).

Outcomes

The main model showed a mean absolute error of approximately 3.5 (95%CI: [3.1-3.8]) points a R2 of 0.08 [-0.14 - 0.26] and r of 0.38 [0.24 - 0.511]. When testing this model in the left-out sample (B) the results were similar, with a MAE of 3.4 [2.8 - 4.7], R2-0.65 [-2.29 - 0.16] and r of 0.4 [0.24 - 0.54]. The anatomical metrics showed a similar pattern, where models rendered overall low R2.

Interpretation

The analysis showed that models based on earlier promising results failed to predict clinical outcomes. Despite the small sample size, the current study does not support extensive use of CPM to predict outcome in pediatric anxiety.

The Translational Future of Stress Neurobiology and Psychosis Vulnerability: A Review of the Evidence

Psychosocial stress is a well-established risk factor for psychosis, yet the neurobiological mechanisms underlying this relationship have yet to be fully elucidated. Much of the research in this field has investigated hypothalamic-pituitary-adrenal (HPA) axis function and immuno-inflammatory processes among individuals with established psychotic disorders. However, as such studies are limited in their ability to provide knowledge that can be used to develop preventative interventions, it is important to shift the focus to individuals with increased vulnerability for psychosis (i.e., high-risk groups). In the present article, we provide an overview of the current methods for identifying individuals at high-risk for psychosis and review the psychosocial stressors that have been most consistently associated with psychosis risk. We then describe a network of interacting physiological systems that are hypothesised to mediate the relationship between psychosocial stress and the manifestation of psychotic illness and critically review evidence that abnormalities within these systems characterise highrisk populations. We found that studies of high-risk groups have yielded highly variable findings, likely due to (i) the heterogeneity both within and across high-risk samples, (ii) the diversity of psychosocial stressors implicated in psychosis, and (iii) that most studies examine single markers of isolated neurobiological systems. We propose that to move the field forward, we require well-designed, largescale translational studies that integrate multi-domain, putative stress-related biomarkers to determine their prognostic value in high-risk samples. We advocate that such investigations are highly warranted, given that psychosocial stress is undoubtedly a relevant risk factor for psychotic disorders.

Electronic health records and stratified psychiatry: bridge to precision treatment?

The use of a stratified psychiatry approach that combines electronic health records (EHR) data with machine learning (ML) is one potentially fruitful path toward rapidly improving precision treatment in clinical practice. This strategy, however, requires confronting pervasive methodological flaws as well as deficiencies in transparency and reporting in the current conduct of ML-based studies for treatment prediction. EHR data shares many of the same data quality issues as other types of data used in ML prediction, plus some unique challenges. To fully leverage EHR data's power for patient stratification, increased attention to data quality and collection of patient-reported outcome data is needed.

[Ketamine and suicidal behavior: Contribution of animal models of aggression-impulsivity to understanding its mechanism of action]

More than two-thirds of suicides occur during a major depressive episode. Acting out prevention measures and therapeutic options to manage the suicidal crisis are limited. The impulsive-aggressive dimensions are vulnerability factors associated with suicide in patients suffering from a characterized depressive episode: this can be a dimension involved in animals. Impulsive and aggressive rodent models can help analyze, at least in part, the neurobiology of suicide and the beneficial effects of treatments. Ketamine, a glutamatergic antagonist, by rapidly improving the symptoms of depressive episodes, would help reduce suicidal thoughts in the short term. Animal models share with humans impulsive and aggressive endophenotypes modulated by the serotonergic system (5-HTB receptor, MAO-A enzyme), neuroinflammation or the hypothalamic-pituitary-adrenal axis and stress. Significant effects of ketamine on these endophenotypes remain to be demonstrated.

Prognostic models predicting transition to psychotic disorder using blood-based biomarkers: a systematic review and critical appraisal

Accumulating evidence suggests individuals with psychotic disorder show abnormalities in metabolic and inflammatory processes. Recently, several studies have employed blood-based predictors in models predicting transition to psychotic disorder in risk-enriched populations. A systematic review of the performance and methodology of prognostic models using blood-based biomarkers in the prediction of psychotic disorder from risk-enriched populations is warranted. Databases (PubMed, EMBASE and PsycINFO) were searched for eligible texts from 1998 to 15/05/2023, which detailed model development or validation studies. The checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was used to guide data extraction from eligible texts and the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias and applicability of the studies. A narrative synthesis of the included studies was performed. Seventeen eligible studies were identified: 16 eligible model development studies and one eligible model validation study. A wide range of biomarkers were assessed, including nucleic acids, proteins, metabolites, and lipids. The range of C-index (area under the curve) estimates reported for the models was 0.67-1.00. No studies assessed model calibration. According to PROBAST criteria, all studies were at high risk of bias in the analysis domain. While a wide range of potentially predictive biomarkers were identified in the included studies, most studies did not account for overfitting in model performance estimates, no studies assessed calibration, and all models were at high risk of bias according to PROBAST criteria. External validation of the models is needed to provide more accurate estimates of their performance. Future studies which follow the latest available methodological and reporting guidelines and adopt strategies to accommodate required sample sizes for model development or validation will clarify the value of including blood-based biomarkers in models predicting psychosis.

Predicting functional impairment in euthymic patients with mood disorder: A 5-year follow-up

Major Depressive Disorder and Bipolar Disorder are psychiatric disorders associated with psychosocial impairment. Despite clinical improvement, functional complaints usually remain, mainly impairing occupational and cognitive performance. The aim of this study was to use machine learning techniques to predict functional impairment in patients with mood disorders. For that, analyzes were performed using a population-based cohort study. Participants diagnosed with a mood disorder at baseline and reassessed were considered (n = 282). Random forest (RF) with previous recursive feature selection and LASSO algorithms were applied to a training set with imputed data by bagged trees resulting in two main models. Following recursive feature selection, 25 variables were retained. The RF model had the best performance compared to LASSO. The most important variables in predicting functional impairment were sexual abuse, severity of depressive, anxiety, and somatic symptoms, physical neglect, emotional abuse, and physical abuse. The model demonstrated acceptable performance to predict functional impairment. However, our sample is composed of young participants and the model may not generalize to older individuals with mood disorders. More studies are needed in this direction. The presented calculator has clinical, sociodemographic, and environmental data, demonstrating that it is possible to use such information to predict functional performance.

Stress Hormone Dynamics Are Coupled to Brain Serotonin 4 Receptor Availability in Unmedicated Patients With Major Depressive Disorder: A NeuroPharm Study

BACKGROUND

A prominent finding in major depressive disorder (MDD) is distorted stress hormone dynamics, which is regulated by serotonergic brain signaling. An interesting feature of the cerebral serotonin system is the serotonin 4 receptor (5-HT4R), which is lower in depressed relative to healthy individuals and also has been highlighted as a promising novel antidepressant target. Here, we test the novel hypothesis that brain 5-HT4R availability in untreated patients with MDD is correlated with cortisol dynamics, indexed by the cortisol awakening response (CAR). Further, we evaluate if CAR changes with antidepressant treatment, including a selective serotonin reuptake inhibitor, and if pretreatment CAR can predict treatment outcome.

METHODS

Sixty-six patients (76% women) with a moderate to severe depressive episode underwent positron emission tomography imaging with [11C]SB207145 for quantification of brain 5-HT4R binding using BPND as outcome. Serial home sampling of saliva in the first hour from awakening was performed to assess CAR before and after 8 weeks of antidepressant treatment. Treatment outcome was measured by change in Hamilton Depression Rating Scale 6 items.

RESULTS

In the unmedicated depressed state, prefrontal and anterior cingulate cortices 5-HT4R binding was positively associated with CAR. CAR remained unaltered after 8 weeks of antidepressant treatment, and pretreatment CAR did not significantly predict treatment outcome.

CONCLUSIONS

Our findings highlight a link between serotonergic disturbances in MDD and cortisol dynamics, which likely is involved in disease and treatment mechanisms. Further, our data support 5-HT4R agonism as a promising precision target in patients with MDD and disturbed stress hormone dynamics.

Evidence of questionable research practices in clinical prediction models

BACKGROUND

Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.

METHODS

We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.

RESULTS

The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.

CONCLUSIONS

The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

An umbrella review of candidate predictors of response, remission, recovery, and relapse across mental disorders

We aimed to identify diagnosis-specific/transdiagnostic/transoutcome multivariable candidate predictors (MCPs) of key outcomes in mental disorders. We conducted an umbrella review (protocol  link ), searching MEDLINE/Embase (19/07/2022), including systematic reviews of studies reporting on MCPs of response, remission, recovery, or relapse, in DSM/ICD-defined mental disorders. From published predictors, we filtered MCPs, validating MCP criteria. AMSTAR2/PROBAST measured quality/risk of bias of systematic reviews/individual studies. We included 117 systematic reviews, 403 studies, 299,888 individuals with mental disorders, testing 796 prediction models. Only 4.3%/1.2% of the systematic reviews/individual studies were at low risk of bias. The most frequently targeted outcome was remission (36.9%), the least frequent was recovery (2.5%). Studies mainly focused on depressive (39.4%), substance-use (17.9%), and schizophrenia-spectrum (11.9%) disorders. We identified numerous MCPs within disorders for response, remission and relapse, but none for recovery. Transdiagnostic MCPs of remission included lower disease-specific symptoms (disorders = 5), female sex/higher education (disorders = 3), and quality of life/functioning (disorders = 2). Transdiagnostic MCPs of relapse included higher disease-specific symptoms (disorders = 5), higher depressive symptoms (disorders = 3), and younger age/higher anxiety symptoms/global illness severity/ number of previous episodes/negative life events (disorders = 2). Finally, positive trans-outcome MCPs for depression included less negative life events/depressive symptoms (response, remission, less relapse), female sex (response, remission) and better functioning (response, less relapse); for schizophrenia, less positive symptoms/higher depressive symptoms (remission, less relapse); for substance use disorder, marital status/higher education (remission, less relapse). Male sex, younger age, more clinical symptoms and comorbid mental/physical symptoms/disorders were poor prognostic factors, while positive factors included social contacts and employment, absent negative life events, higher education, early access/intervention, lower disease-specific and comorbid mental and physical symptoms/conditions, across mental disorders. Current data limitations include high risk of bias of studies and extraction of single predictors from multivariable models. Identified MCPs can inform future development, validation or refinement of prediction models of key outcomes in mental disorders.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

BACKGROUND

Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome.

METHODS

We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size.

RESULTS

A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84).

CONCLUSIONS

Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Predictive models for predicting the risk of maternal postpartum depression: A systematic review and evaluation

OBJECTIVES

Clinical prediction models have been widely used to screen and diagnose postpartum depression (PPD). This study systematically reviews and evaluates the risk of bias and the applicability of PPD prediction models.

METHODS

A systematic search was performed in eight databases from inception to June 1, 2022. The literature was independently screened, and data were extracted by two investigators using the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS). The risk of bias and applicability was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

After the screening, 12 studies of PPD risk prediction models were included, with the area under the ROC curve of the models ranging from 0.611 to 0.937. The most-reported predictors of PPD included several aspects, including prenatal mood disorders, endocrine and hormonal influences, psychosocial aspects, the influence of family factors, and somatic illness factors. The applicability of all studies was good. However, there was some bias, mainly due to inadequate outcome events, missing data not appropriately handled, lack of model performance assessment, and overfitting of the models.

CONCLUSIONS

This systematic review and evaluation indicate that most present PPD prediction models have a high risk of bias during development and validation. Despite some models' predictive solid performance, the models' clinical practice rate is low. Therefore, future research should develop predictive models with excellent performance in all aspects and clinical applicability to better inform maternal medical decisions.

Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

BACKGROUND

The development of machine learning models for aiding in the diagnosis of mental disorder is recognized as a significant breakthrough in the field of psychiatry. However, clinical practice of such models remains a challenge, with poor generalizability being a major limitation.

METHODS

Here, we conducted a pre-registered meta-research assessment on neuroimaging-based models in the psychiatric literature, quantitatively examining global and regional sampling issues over recent decades, from a view that has been relatively underexplored. A total of 476 studies (n = 118,137) were included in the current assessment. Based on these findings, we built a comprehensive 5-star rating system to quantitatively evaluate the quality of existing machine learning models for psychiatric diagnoses.

RESULTS

A global sampling inequality in these models was revealed quantitatively (sampling Gini coefficient (G) = 0.81, p < .01), varying across different countries (regions) (e.g., China, G = 0.47; the USA, G = 0.58; Germany, G = 0.78; the UK, G = 0.87). Furthermore, the severity of this sampling inequality was significantly predicted by national economic levels (β =  - 2.75, p < .001, R²_adj = 0.40; r =  - .84, 95% CI: - .41 to - .97), and was plausibly predictable for model performance, with higher sampling inequality for reporting higher classification accuracy. Further analyses showed that lack of independent testing (84.24% of models, 95% CI: 81.0-87.5%), improper cross-validation (51.68% of models, 95% CI: 47.2-56.2%), and poor technical transparency (87.8% of models, 95% CI: 84.9-90.8%)/availability (80.88% of models, 95% CI: 77.3-84.4%) are prevailing in current diagnostic classifiers despite improvements over time. Relating to these observations, model performances were found decreased in studies with independent cross-country sampling validations (all p < .001, BF₁₀ > 15). In light of this, we proposed a purpose-built quantitative assessment checklist, which demonstrated that the overall ratings of these models increased by publication year but were negatively associated with model performance.

CONCLUSIONS

Together, improving sampling economic equality and hence the quality of machine learning models may be a crucial facet to plausibly translating neuroimaging-based diagnostic classifiers into clinical practice.

Candidate biomarkers in psychiatric disorders: state of the field

The field of psychiatry is hampered by a lack of robust, reliable and valid biomarkers that can aid in objectively diagnosing patients and providing individualized treatment recommendations. Here we review and critically evaluate the evidence for the most promising biomarkers in the psychiatric neuroscience literature for autism spectrum disorder, schizophrenia, anxiety disorders and post-traumatic stress disorder, major depression and bipolar disorder, and substance use disorders. Candidate biomarkers reviewed include various neuroimaging, genetic, molecular and peripheral assays, for the purposes of determining susceptibility or presence of illness, and predicting treatment response or safety. This review highlights a critical gap in the biomarker validation process. An enormous societal investment over the past 50 years has identified numerous candidate biomarkers. However, to date, the overwhelming majority of these measures have not been proven sufficiently reliable, valid and useful to be adopted clinically. It is time to consider whether strategic investments might break this impasse, focusing on a limited number of promising candidates to advance through a process of definitive testing for a specific indication. Some promising candidates for definitive testing include the N170 signal, an event-related brain potential measured using electroencephalography, for subgroup identification within autism spectrum disorder; striatal resting-state functional magnetic resonance imaging (fMRI) measures, such as the striatal connectivity index (SCI) and the functional striatal abnormalities (FSA) index, for prediction of treatment response in schizophrenia; error-related negativity (ERN), an electrophysiological index, for prediction of first onset of generalized anxiety disorder, and resting-state and structural brain connectomic measures for prediction of treatment response in social anxiety disorder. Alternate forms of classification may be useful for conceptualizing and testing potential biomarkers. Collaborative efforts allowing the inclusion of biosystems beyond genetics and neuroimaging are needed, and online remote acquisition of selected measures in a naturalistic setting using mobile health tools may significantly advance the field. Setting specific benchmarks for well-defined target application, along with development of appropriate funding and partnership mechanisms, would also be crucial. Finally, it should never be forgotten that, for a biomarker to be actionable, it will need to be clinically predictive at the individual level and viable in clinical settings.

Identifying the neurodevelopmental and psychiatric signatures of genomic disorders associated with intellectual disability: a machine learning approach

BACKGROUND

Genomic conditions can be associated with developmental delay, intellectual disability, autism spectrum disorder, and physical and mental health symptoms. They are individually rare and highly variable in presentation, which limits the use of standard clinical guidelines for diagnosis and treatment. A simple screening tool to identify young people with genomic conditions associated with neurodevelopmental disorders (ND-GCs) who could benefit from further support would be of considerable value. We used machine learning approaches to address this question.

METHOD

A total of 493 individuals were included: 389 with a ND-GC, mean age = 9.01, 66% male) and 104 siblings without known genomic conditions (controls, mean age = 10.23, 53% male). Primary carers completed assessments of behavioural, neurodevelopmental and psychiatric symptoms and physical health and development. Machine learning techniques (penalised logistic regression, random forests, support vector machines and artificial neural networks) were used to develop classifiers of ND-GC status and identified limited sets of variables that gave the best classification performance. Exploratory graph analysis was used to understand associations within the final variable set.

RESULTS

All machine learning methods identified variable sets giving high classification accuracy (AUROC between 0.883 and 0.915). We identified a subset of 30 variables best discriminating between individuals with ND-GCs and controls which formed 5 dimensions: conduct, separation anxiety, situational anxiety, communication and motor development.

LIMITATIONS

This study used cross-sectional data from a cohort study which was imbalanced with respect to ND-GC status. Our model requires validation in independent datasets and with longitudinal follow-up data for validation before clinical application.

CONCLUSIONS

In this study, we developed models that identified a compact set of psychiatric and physical health measures that differentiate individuals with a ND-GC from controls and highlight higher-order structure within these measures. This work is a step towards developing a screening instrument to identify young people with ND-GCs who might benefit from further specialist assessment.

Pharmacological Mechanism of Ketamine in Suicidal Behavior Based on Animal Models of Aggressiveness and Impulsivity: A Narrative Review

Around 700,000 people die from suicide each year in the world. Approximately 90% of suicides have a history of mental illness, and more than two-thirds occur during a major depressive episode. Specific therapeutic options to manage the suicidal crisis are limited and measures to prevent acting out also remain limited. Drugs shown to reduce the risk of suicide (antidepressants, lithium, or clozapine) necessitate a long delay of onset. To date, no treatment is indicated for the treatment of suicidality. Ketamine, a glutamate NMDA receptor antagonist, is a fast-acting antidepressant with significant effects on suicidal ideation in the short term, while its effects on suicidal acts still need to be demonstrated. In the present article, we reviewed the literature on preclinical studies in order to identify the potential anti-suicidal pharmacological targets of ketamine. Impulsive-aggressive traits are one of the vulnerability factors common to suicide in patients with unipolar and bipolar depression. Preclinical studies in rodent models with impulsivity, aggressiveness, and anhedonia may help to analyze, at least in part, suicide neurobiology, as well as the beneficial effects of ketamine/esketamine on reducing suicidal ideations and preventing suicidal acts. The present review focuses on disruptions in the serotonergic system (5-HT_B receptor, MAO-A enzyme), neuroinflammation, and/or the HPA axis in rodent models with an impulsive/aggressive phenotype, because these traits are critical risk factors for suicide in humans. Ketamine can modulate these endophenotypes of suicide in human as well as in animal models. The main pharmacological properties of ketamine are then summarized. Finally, numerous questions arose regarding the mechanisms by which ketamine may prevent an impulsive-aggressive phenotype in rodents and suicidal ideations in humans. Animal models of anxiety/depression are important tools to better understand the pathophysiology of depressed patients, and in helping develop novel and fast antidepressant drugs with anti-suicidal properties and clinical utility.

Electronic health record-based prediction models for in-hospital adverse drug event diagnosis or prognosis: a systematic review

OBJECTIVE

We conducted a systematic review to characterize and critically appraise developed prediction models based on structured electronic health record (EHR) data for adverse drug event (ADE) diagnosis and prognosis in adult hospitalized patients.

MATERIALS AND METHODS

We searched the Embase and Medline databases (from January 1, 1999, to July 4, 2022) for articles utilizing structured EHR data to develop ADE prediction models for adult inpatients. For our systematic evidence synthesis and critical appraisal, we applied the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS).

RESULTS

Twenty-five articles were included. Studies often did not report crucial information such as patient characteristics or the method for handling missing data. In addition, studies frequently applied inappropriate methods, such as univariable screening for predictor selection. Furthermore, the majority of the studies utilized ADE labels that only described an adverse symptom while not assessing causality or utilizing a causal model. None of the models were externally validated.

CONCLUSIONS

Several challenges should be addressed before the models can be widely implemented, including the adherence to reporting standards and the adoption of best practice methods for model development and validation. In addition, we propose a reorientation of the ADE prediction modeling domain to include causality as a fundamental challenge that needs to be addressed in future studies, either through acquiring ADE labels via formal causality assessments or the usage of adverse event labels in combination with causal prediction modeling.

Predicting suicide risk in 137,112 people with severe mental illness in Finland: external validation of the Oxford Mental Illness and Suicide tool (OxMIS)

Oxford Mental Illness and Suicide tool (OxMIS) is a standardised, scalable, and transparent instrument for suicide risk assessment in people with severe mental illness (SMI) based on 17 sociodemographic, criminal history, familial, and clinical risk factors. However, alongside most prediction models in psychiatry, external validations are currently lacking. We utilised a Finnish population sample of all persons diagnosed by mental health services with SMI (schizophrenia-spectrum and bipolar disorders) between 1996 and 2017 (n = 137,112). To evaluate the performance of OxMIS, we initially calculated the predicted 12-month suicide risk for each individual by weighting risk factors by effect sizes reported in the original OxMIS prediction model and converted to a probability. This probability was then used to assess the discrimination and calibration of the OxMIS model in this external sample. Within a year of assessment, 1.1% of people with SMI (n = 1475) had died by suicide. The overall discrimination of the tool was good, with an area under the curve of 0.70 (95% confidence interval: 0.69-0.71). The model initially overestimated suicide risks in those with elevated predicted risks of >5% over 12 months (Harrell's E_max = 0.114), which applied to 1.3% (n = 1780) of the cohort. However, when we used a 5% maximum predicted suicide risk threshold as is recommended clinically, the calibration was excellent (ICI = 0.002; E_max = 0.005). Validating clinical prediction tools using routinely collected data can address research gaps in prediction psychiatry and is a necessary step to translating such models into clinical practice.

Validation of the Collaborative Outcomes study on Health and Functioning during Infection Times (COH-FIT) questionnaire for adults

BACKGROUND

The Collaborative Outcome study on Health and Functioning during Infection Times (COH-FIT; www.coh-fit.com) is an anonymous and global online survey measuring health and functioning during the COVID-19 pandemic. The aim of this study was to test concurrently the validity of COH-FIT items and the internal validity of the co-primary outcome, a composite psychopathology "P-score".

METHODS

The COH-FIT survey has been translated into 30 languages (two blind forward-translations, consensus, one independent English back-translation, final harmonization). To measure mental health, 1-4 items ("COH-FIT items") were extracted from validated questionnaires (e.g. Patient Health Questionnaire 9). COH-FIT items measured anxiety, depressive, post-traumatic, obsessive-compulsive, bipolar and psychotic symptoms, as well as stress, sleep and concentration. COH-FIT Items which correlated r ≥ 0.5 with validated companion questionnaires, were initially retained. A P-score factor structure was then identified from these items using exploratory factor analysis (EFA) and confirmatory factor analyses (CFA) on data split into training and validation sets. Consistency of results across languages, gender and age was assessed.

RESULTS

From >150,000 adult responses by May 6th, 2022, a subset of 22,456 completed both COH-FIT items and validated questionnaires. Concurrent validity was consistently demonstrated across different languages for COH-FIT items. CFA confirmed EFA results of five first-order factors (anxiety, depression, post-traumatic, psychotic, psychophysiologic symptoms) and revealed a single second-order factor P-score, with high internal reliability (ω = 0.95). Factor structure was consistent across age and sex.

CONCLUSIONS

COH-FIT is a valid instrument to globally measure mental health during infection times. The P-score is a valid measure of multidimensional mental health.

Deep phenotyping towards precision psychiatry of first-episode depression - the Brain Drugs-Depression cohort

BACKGROUND

Major Depressive Disorder (MDD) is a heterogenous brain disorder, with potentially multiple psychosocial and biological disease mechanisms. This is also a plausible explanation for why patients do not respond equally well to treatment with first- or second-line antidepressants, i.e., one-third to one-half of patients do not remit in response to first- or second-line treatment. To map MDD heterogeneity and markers of treatment response to enable a precision medicine approach, we will acquire several possible predictive markers across several domains, e.g., psychosocial, biochemical, and neuroimaging.

METHODS

All patients are examined before receiving a standardised treatment package for adults aged 18-65 with first-episode depression in six public outpatient clinics in the Capital Region of Denmark. From this population, we will recruit a cohort of 800 patients for whom we will acquire clinical, cognitive, psychometric, and biological data. A subgroup (subcohort I, n = 600) will additionally provide neuroimaging data, i.e., Magnetic Resonance Imaging, and Electroencephalogram, and a subgroup of patients from subcohort I unmedicated at inclusion (subcohort II, n = 60) will also undergo a brain Positron Emission Tomography with the [¹¹C]-UCB-J tracer binding to the presynaptic glycoprotein-SV2A. Subcohort allocation is based on eligibility and willingness to participate. The treatment package typically lasts six months. Depression severity is assessed with the Quick Inventory of Depressive Symptomatology (QIDS) at baseline, and 6, 12 and 18 months after treatment initiation. The primary outcome is remission (QIDS ≤ 5) and clinical improvement (≥ 50% reduction in QIDS) after 6 months. Secondary endpoints include remission at 12 and 18 months and %-change in QIDS, 10-item Symptom Checklist, 5-item WHO Well-Being Index, and modified Disability Scale from baseline through follow-up. We also assess psychotherapy and medication side-effects. We will use machine learning to determine a combination of characteristics that best predict treatment outcomes and statistical models to investigate the association between individual measures and clinical outcomes. We will assess associations between patient characteristics, treatment choices, and clinical outcomes using path analysis, enabling us to estimate the effect of treatment choices and timing on the clinical outcome.

DISCUSSION

The BrainDrugs-Depression study is a real-world deep-phenotyping clinical cohort study of first-episode MDD patients.

TRIAL REGISTRATION

Registered at clinicaltrials.gov November 15th, 2022 (NCT05616559).

Predicting childhood and adolescent attention-deficit/hyperactivity disorder onset: a nationwide deep learning approach

Attention-deficit/hyperactivity disorder (ADHD) is a heterogeneous disorder with a high degree of psychiatric and physical comorbidity, which complicates its diagnosis in childhood and adolescence. We analyzed registry data from 238,696 persons born and living in Sweden between 1995 and 1999. Several machine learning techniques were used to assess the ability of registry data to inform the diagnosis of ADHD in childhood and adolescence: logistic regression, random Forest, gradient boosting, XGBoost, penalized logistic regression, deep neural network (DNN), and ensemble models. The best fitting model was the DNN, achieving an area under the receiver operating characteristic curve of 0.75, 95% CI (0.74-0.76) and balanced accuracy of 0.69. At the 0.45 probability threshold, sensitivity was 71.66% and specificity was 65.0%. There was an overall agreement in the feature importance among all models (τ > .5). The top 5 features contributing to classification were having a parent with criminal convictions, male sex, having a relative with ADHD, number of academic subjects failed, and speech/learning disabilities. A DNN model predicting childhood and adolescent ADHD trained exclusively on Swedish register data achieved good discrimination. If replicated and validated in an external sample, and proven to be cost-effective, this model could be used to alert clinicians to individuals who ought to be screened for ADHD and to aid clinicians' decision-making with the goal of decreasing misdiagnoses. Further research is needed to validate results in different populations and to incorporate new predictors.

Evaluation of Risk of Bias in Neuroimaging-Based Artificial Intelligence Models for Psychiatric Diagnosis: A Systematic Review

IMPORTANCE

Neuroimaging-based artificial intelligence (AI) diagnostic models have proliferated in psychiatry. However, their clinical applicability and reporting quality (ie, feasibility) for clinical practice have not been systematically evaluated.

OBJECTIVE

To systematically assess the risk of bias (ROB) and reporting quality of neuroimaging-based AI models for psychiatric diagnosis.

EVIDENCE REVIEW

PubMed was searched for peer-reviewed, full-length articles published between January 1, 1990, and March 16, 2022. Studies aimed at developing or validating neuroimaging-based AI models for clinical diagnosis of psychiatric disorders were included. Reference lists were further searched for suitable original studies. Data extraction followed the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. A closed-loop cross-sequential design was used for quality control. The PROBAST (Prediction Model Risk of Bias Assessment Tool) and modified CLEAR (Checklist for Evaluation of Image-Based Artificial Intelligence Reports) benchmarks were used to systematically evaluate ROB and reporting quality.

FINDINGS

A total of 517 studies presenting 555 AI models were included and evaluated. Of these models, 461 (83.1%; 95% CI, 80.0%-86.2%) were rated as having a high overall ROB based on the PROBAST. The ROB was particular high in the analysis domain, including inadequate sample size (398 of 555 models [71.7%; 95% CI, 68.0%-75.6%]), poor model performance examination (with 100% of models lacking calibration examination), and lack of handling data complexity (550 of 555 models [99.1%; 95% CI, 98.3%-99.9%]). None of the AI models was perceived to be applicable to clinical practices. Overall reporting completeness (ie, number of reported items/number of total items) for the AI models was 61.2% (95% CI, 60.6%-61.8%), and the completeness was poorest for the technical assessment domain with 39.9% (95% CI, 38.8%-41.1%).

CONCLUSIONS AND RELEVANCE

This systematic review found that the clinical applicability and feasibility of neuroimaging-based AI models for psychiatric diagnosis were challenged by a high ROB and poor reporting quality. Particularly in the analysis domain, ROB in AI diagnostic models should be addressed before clinical application.

Deep Learning for Cross-Diagnostic Prediction of Mental Disorder Diagnosis and Prognosis Using Danish Nationwide Register and Genetic Data

Importance

Diagnoses and treatment of mental disorders are hampered by the current lack of objective markers needed to provide a more precise diagnosis and treatment strategy.

Objective

To develop deep learning models to predict mental disorder diagnosis and severity spanning multiple diagnoses using nationwide register data, family and patient-specific diagnostic history, birth-related measurement, and genetics.

Design, Setting, and Participants

This study was conducted from May 1, 1981, to December 31, 2016. For the analysis, which used a Danish population-based case-cohort sample of individuals born between 1981 and 2005, genotype data and matched longitudinal health register data were taken from the longitudinal Danish population-based Integrative Psychiatric Research Consortium 2012 case-cohort study. Included were individuals with mental disorders (attention-deficit/hyperactivity disorder [ADHD]), autism spectrum disorder (ASD), major depressive disorder (MDD), bipolar disorder (BD), schizophrenia spectrum disorders (SCZ), and population controls. Data were analyzed from February 1, 2021, to January 24, 2022.

Exposure

At least 1 hospital contact with diagnosis of ADHD, ASD, MDD, BD, or SCZ.

Main Outcomes and Measures

The predictability of (1) mental disorder diagnosis and (2) severity trajectories (measured by future outpatient hospital contacts, admissions, and suicide attempts) were investigated using both a cross-diagnostic and single-disorder setup. Predictive power was measured by AUC, accuracy, and Matthews correlation coefficient (MCC), including an estimate of feature importance.

Results

A total of 63 535 individuals (mean [SD] age, 23 [7] years; 34 944 male [55%]; 28 591 female [45%]) were included in the model. Based on data prior to diagnosis, the specific diagnosis was predicted in a multidiagnostic prediction model including the background population with an overall area under the curve (AUC) of 0.81 and MCC of 0.28, whereas the single-disorder models gave AUCs/MCCs of 0.84/0.54 for SCZ, 0.79/0.41 for BD, 0.77/0.39 for ASD, 0.74/0.38, for ADHD, and 0.74/0.38 for MDD. The most important data sets for multidiagnostic prediction were previous mental disorders and age (11%-23% reduction in prediction accuracy when removed) followed by family diagnoses, birth-related measurements, and genetic data (3%-5% reduction in prediction accuracy when removed). Furthermore, when predicting subsequent disease trajectories of the disorder, the most severe cases were the most easily predictable, with an AUC of 0.72.

Conclusions and Relevance

Results of this diagnostic study suggest the possibility of combining genetics and registry data to predict both mental disorder diagnosis and disorder progression in a clinically relevant, cross-diagnostic setting prior to clinical assessment.

Multi-omics data integration methods and their applications in psychiatric disorders

To study mental illness and health, in the past researchers have often broken down their complexity into individual subsystems (e.g., genomics, transcriptomics, proteomics, clinical data) and explored the components independently. Technological advancements and decreasing costs of high throughput sequencing has led to an unprecedented increase in data generation. Furthermore, over the years it has become increasingly clear that these subsystems do not act in isolation but instead interact with each other to drive mental illness and health. Consequently, individual subsystems are now analysed jointly to promote a holistic understanding of the underlying biological complexity of health and disease. Complementing the increasing data availability, current research is geared towards developing novel methods that can efficiently combine the information rich multi-omics data to discover biologically meaningful biomarkers for diagnosis, treatment, and prognosis. However, clinical translation of the research is still challenging. In this review, we summarise conventional and state-of-the-art statistical and machine learning approaches for discovery of biomarker, diagnosis, as well as outcome and treatment response prediction through integrating multi-omics and clinical data. In addition, we describe the role of biological model systems and in silico multi-omics model designs in clinical translation of psychiatric research from bench to bedside. Finally, we discuss the current challenges and explore the application of multi-omics integration in future psychiatric research. The review provides a structured overview and latest updates in the field of multi-omics in psychiatry.