From models to tools: clinical translation of machine learning studies in psychosis

Development and initial evaluation of a clinical prediction model for risk of treatment resistance in first-episode psychosis: Schizophrenia Prediction of Resistance to Treatment (SPIRIT)

BACKGROUND

A clinical tool to estimate the risk of treatment-resistant schizophrenia (TRS) in people with first-episode psychosis (FEP) would inform early detection of TRS and overcome the delay of up to 5 years in starting TRS medication.

AIMS

To develop and evaluate a model that could predict the risk of TRS in routine clinical practice.

METHOD

We used data from two UK-based FEP cohorts (GAP and AESOP-10) to develop and internally validate a prognostic model that supports identification of patients at high-risk of TRS soon after FEP diagnosis. Using sociodemographic and clinical predictors, a model for predicting risk of TRS was developed based on penalised logistic regression, with missing data handled using multiple imputation. Internal validation was undertaken via bootstrapping, obtaining optimism-adjusted estimates of the model's performance. Interviews and focus groups with clinicians were conducted to establish clinically relevant risk thresholds and understand the acceptability and perceived utility of the model.

RESULTS

We included seven factors in the prediction model that are predominantly assessed in clinical practice in patients with FEP. The model predicted treatment resistance among the 1081 patients with reasonable accuracy; the model's C-statistic was 0.727 (95% CI 0.723-0.732) prior to shrinkage and 0.687 after adjustment for optimism. Calibration was good (expected/observed ratio: 0.999; calibration-in-the-large: 0.000584) after adjustment for optimism.

CONCLUSIONS

We developed and internally validated a prediction model with reasonably good predictive metrics. Clinicians, patients and carers were involved in the development process. External validation of the tool is needed followed by co-design methodology to support implementation in early intervention services.

Multivariate brain-behaviour associations in psychiatric disorders

Mapping brain-behaviour associations is paramount to understand and treat psychiatric disorders. Standard approaches involve investigating the association between one brain and one behavioural variable (univariate) or multiple variables against one brain/behaviour feature ('single' multivariate). Recently, large multimodal datasets have propelled a new wave of studies that leverage on 'doubly' multivariate approaches capable of parsing the multifaceted nature of both brain and behaviour simultaneously. Within this movement, canonical correlation analysis (CCA) and partial least squares (PLS) emerge as the most popular techniques. Both seek to capture shared information between brain and behaviour in the form of latent variables. We provide an overview of these methods, review the literature in psychiatric disorders, and discuss the main challenges from a predictive modelling perspective. We identified 39 studies across four diagnostic groups: attention deficit and hyperactive disorder (ADHD, k = 4, N = 569), autism spectrum disorders (ASD, k = 6, N = 1731), major depressive disorder (MDD, k = 5, N = 938), psychosis spectrum disorders (PSD, k = 13, N = 1150) and one transdiagnostic group (TD, k = 11, N = 5731). Most studies (67%) used CCA and focused on the association between either brain morphology, resting-state functional connectivity or fractional anisotropy against symptoms and/or cognition. There were three main findings. First, most diagnoses shared a link between clinical/cognitive symptoms and two brain measures, namely frontal morphology/brain activity and white matter association fibres (tracts between cortical areas in the same hemisphere). Second, typically less investigated behavioural variables in multivariate models such as physical health (e.g., BMI, drug use) and clinical history (e.g., childhood trauma) were identified as important features. Finally, most studies were at risk of bias due to low sample size/feature ratio and/or in-sample testing only. We highlight the importance of carefully mitigating these sources of bias with an exemplar application of CCA.

Assisting schizophrenia diagnosis using clinical electroencephalography and interpretable graph neural networks: a real-world and cross-site study

Schizophrenia (SCZ) is a chronic and serious mental disorder with a high mortality rate. At present, there is a lack of objective, cost-effective and widely disseminated diagnosis tools to address this mental health crisis globally. Clinical electroencephalogram (EEG) is a noninvasive technique to measure brain activity with high temporal resolution, and accumulating evidence demonstrates that clinical EEG is capable of capturing abnormal SCZ neuropathology. Although EEG-based automated diagnostic tools have obtained impressive performance on individual datasets, the transportability of potential EEG biomarkers in cross-site real-world application is still an open question. To address the challenges of small sample sizes and population heterogeneity, we develop an advanced interpretable deep learning model using multimodal clinical EEG features and demographic information as inputs to graph neural networks, and further propose different transfer learning strategies to adapt to different clinical scenarios. Taking the disease discrimination of health control (HC) and SCZ with 1030 participants as a use case, our model is trained on a small clinical dataset (N = 188, Chinese) and enhanced using a large-scale public dataset (N = 508, American) of adult participants. Cross-site validation from an independent dataset of adult participants (N = 157, Chinese) produced stable performance, with AUCs of 0.793-0.852 and accuracies of 0.786-0.858 for different SCZ prevalence, respectively. In addition, cross-site validation from another dataset of adolescent boys (N = 84, Russian) yielded an AUC of 0.702 and an accuracy of 0.690. Moreover, feature visualization further revealed that the ranking of feature importance varied significantly among different datasets, and that EEG theta and alpha band power appeared to be the most significant and translational biomarkers of SCZ pathology. Overall, our promising results demonstrate the feasibility of SCZ discrimination using EEG biomarkers in multiple clinical settings.

Translation of AI into oncology clinical practice

Artificial intelligence (AI) is a transformative technology that is capturing popular imagination and can revolutionize biomedicine. AI and machine learning (ML) algorithms have the potential to break through existing barriers in oncology research and practice such as automating workflow processes, personalizing care, and reducing healthcare disparities. Emerging applications of AI/ML in the literature include screening and early detection of cancer, disease diagnosis, response prediction, prognosis, and accelerated drug discovery. Despite this excitement, only few AI/ML models have been properly validated and fewer have become regulated products for routine clinical use. In this review, we highlight the main challenges impeding AI/ML clinical translation. We present different clinical use cases from the domains of radiology, radiation oncology, immunotherapy, and drug discovery in oncology. We dissect the unique challenges and opportunities associated with each of these cases. Finally, we summarize the general requirements for successful AI/ML implementation in the clinic, highlighting specific examples and points of emphasis including the importance of multidisciplinary collaboration of stakeholders, role of domain experts in AI augmentation, transparency of AI/ML models, and the establishment of a comprehensive quality assurance program to mitigate risks of training bias and data drifts, all culminating toward safer and more beneficial AI/ML applications in oncology labs and clinics.

External validation of binary machine learning models for pain intensity perception classification from EEG in healthy individuals

Discrimination of pain intensity using machine learning (ML) and electroencephalography (EEG) has significant potential for clinical applications, especially in scenarios where self-report is unsuitable. However, existing research is limited due to a lack of external validation (assessing performance using novel data). We aimed for the first external validation study for pain intensity classification with EEG. Pneumatic pressure stimuli were delivered to the fingernail bed at high and low pain intensities during two independent EEG experiments with healthy participants. Study one (n = 25) was utilised for training and cross-validation. Study two (n = 15) was used for external validation one (identical stimulation parameters to study one) and external validation two (new stimulation parameters). Time-frequency features of peri-stimulus EEG were computed on a single-trial basis for all electrodes. ML training and analysis were performed on a subset of features, identified through feature selection, which were distributed across scalp electrodes and included frontal, central, and parietal regions. Results demonstrated that ML models outperformed chance. The Random Forest (RF) achieved the greatest accuracies of 73.18, 68.32 and 60.42% for cross-validation, external validation one and two, respectively. Importantly, this research is the first to externally validate ML and EEG for the classification of intensity during experimental pain, demonstrating promising performance which generalises to novel samples and paradigms. These findings offer the most rigorous estimates of ML's clinical potential for pain classification.

The potential of precision psychiatry: what is in reach?

Progress in developing personalised care for mental disorders is supported by numerous proof-of-concept machine learning studies in the area of risk assessment, diagnostics and precision prescribing. Most of these studies primarily use clinical data, but models might benefit from additional neuroimaging, blood and genetic data to improve accuracy. Combined, multimodal models might offer potential for stratification of patients for treatment. Clinical implementation of machine learning is impeded by a lack of wider generalisability, with efforts primarily focused on psychosis and dementia. Studies across all diagnostic groups should work to test the robustness of machine learning models, which is an essential first step to clinical implementation, and then move to prospective clinical validation. Models need to exceed clinicians' heuristics to be useful, and safe, in routine decision-making. Engagement of clinicians, researchers and patients in digitalisation and 'big data' approaches are vital to allow the generation and accessibility of large, longitudinal, prospective data needed for precision psychiatry to be applied into real-world psychiatric care.

Machine learning for brain age prediction: Introduction to methods and clinical applications

The rise of machine learning has unlocked new ways of analysing structural neuroimaging data, including brain age prediction. In this state-of-the-art review, we provide an introduction to the methods and potential clinical applications of brain age prediction. Studies on brain age typically involve the creation of a regression machine learning model of age-related neuroanatomical changes in healthy people. This model is then applied to new subjects to predict their brain age. The difference between predicted brain age and chronological age in a given individual is known as ‘brain-age gap’. This value is thought to reflect neuroanatomical abnormalities and may be a marker of overall brain health. It may aid early detection of brain-based disorders and support differential diagnosis, prognosis, and treatment choices. These applications could lead to more timely and more targeted interventions in age-related disorders.

Translational Application of a Neuro-Scientific Multi-Modal Approach Into Forensic Psychiatric Evaluation: Why and How?

A prominent body of literature indicates that insanity evaluations, which are intended to provide influential expert reports for judges to reach a decision "beyond any reasonable doubt," suffer from a low inter-rater reliability. This paper reviews the limitations of the classical approach to insanity evaluation and the criticisms to the introduction of neuro-scientific approach in court. Here, we explain why in our opinion these criticisms, that seriously hamper the translational implementation of neuroscience into the forensic setting, do not survive scientific scrutiny. Moreover, we discuss how the neuro-scientific multimodal approach may improve the inter-rater reliability in insanity evaluation. Critically, neuroscience does not aim to introduce a brain-based concept of insanity. Indeed, criteria for responsibility and insanity are and should remain clinical. Rather, following the falsificationist approach and the convergence of evidence principle, the neuro-scientific multimodal approach is being proposed as a way to improve reliability of insanity evaluation and to mitigate the influence of cognitive biases on the formulation of insanity opinions, with the final aim to reduce errors and controversies.

A multidisciplinary approach to insanity assessment as a way to reduce cognitive biases

Insanity assessment requires the evaluation of the psychopathological condition that underlies the mens rea. Psychopathological evaluation may be quite challenging due to (i) absence of biomarkers; (ii) low inter-rater reliability; (iii) presence of cognitive bias. This intrinsic low reliability of forensic psychiatric diagnosis does impact on insanity assessment, leading to arbitrary and unjust legal outcomes for the examinee. Thus, strategies to improve the reliability of insanity evaluation are strongly needed. A multidisciplinary approach has been proposed as a way to enrich clinical diagnosis with reliable and biologically founded data, thus minimizing subjectivity, reducing controversies and increasing inter-subject concordance in insanity assessment. By discussing a real case, here we show how the convergence of multiple indices can produce evidence that cannot be denied without introducing logical fallacies. Applying this approach, the forensic discussion will move from the presence/absence of psychopathology to the impact of psychopathology on insanity. This article illustrates how a multidisciplinary evaluation, which integrates neuroscientific methods with the classical insanity assessment, may lead to a more accurate approach in insanity evaluation. Critically, this approach will minimize the impact of cognitive bias on insanity opinion and thus result in an improvement of the whole criminal justice process.

Computational Psychiatry Research Map (CPSYMAP): A New Database for Visualizing Research Papers

The field of computational psychiatry is growing in prominence along with recent advances in computational neuroscience, machine learning, and the cumulative scientific understanding of psychiatric disorders. Computational approaches based on cutting-edge technologies and high-dimensional data are expected to provide an understanding of psychiatric disorders with integrating the notions of psychology and neuroscience, and to contribute to clinical practices. However, the multidisciplinary nature of this field seems to limit the development of computational psychiatry studies. Computational psychiatry combines knowledge from neuroscience, psychiatry, and computation; thus, there is an emerging need for a platform to integrate and coordinate these perspectives. In this study, we developed a new database for visualizing research papers as a two-dimensional "map" called the Computational Psychiatry Research Map (CPSYMAP). This map shows the distribution of papers along neuroscientific, psychiatric, and computational dimensions to enable anyone to find niche research and deepen their understanding ofthe field.