On the generalizability of resting-state fMRI machine learning classifiers

Advancing brain network models to reconcile functional neuroimaging and clinical research

Functional magnetic resonance imaging (fMRI) captures information on brain function beyond the anatomical alterations that are traditionally visually examined by neuroradiologists. However, the fMRI signals are complex in addition to being noisy, so fMRI still faces limitations for clinical applications. Here we review methods that have been proposed as potential solutions so far, namely statistical, biophysical and decoding models, with their strengths and weaknesses. We especially evaluate the ability of these models to directly predict clinical variables from their parameters (predictability) and to extract clinically relevant information regarding biological mechanisms and relevant features for classification and prediction (interpretability). We then provide guidelines for useful applications and pitfalls of such fMRI-based models in a clinical research context, looking beyond the current state of the art. In particular, we argue that the clinical relevance of fMRI calls for a new generation of models for fMRI data, which combine the strengths of both biophysical and decoding models. This leads to reliable and biologically meaningful model parameters, which thus fulfills the need for simultaneous interpretability and predictability. In our view, this synergy is fundamental for the discovery of new pharmacological and interventional targets, as well as the use of models as biomarkers in neurology and psychiatry.

Supervised machine learning for diagnostic classification from large-scale neuroimaging datasets

There are growing concerns about the generalizability of machine learning classifiers in neuroimaging. In order to evaluate this aspect across relatively large heterogeneous populations, we investigated four disorders: Autism spectrum disorder (N = 988), Attention deficit hyperactivity disorder (N = 930), Post-traumatic stress disorder (N = 87) and Alzheimer's disease (N = 132). We applied 18 different machine learning classifiers (based on diverse principles) wherein the training/validation and the hold-out test data belonged to samples with the same diagnosis but differing in either the age range or the acquisition site. Our results indicate that overfitting can be a huge problem in heterogeneous datasets, especially with fewer samples, leading to inflated measures of accuracy that fail to generalize well to the general clinical population. Further, different classifiers tended to perform well on different datasets. In order to address this, we propose a consensus-classifier by combining the predictive power of all 18 classifiers. The consensus-classifier was less sensitive to unmatched training/validation and holdout test data. Finally, we combined feature importance scores obtained from all classifiers to infer the discriminative ability of connectivity features. The functional connectivity patterns thus identified were robust to the classification algorithm used, age and acquisition site differences, and had diagnostic predictive ability in addition to univariate statistically significant group differences between the groups. A MATLAB toolbox called Machine Learning in NeuroImaging (MALINI), which implements all the 18 different classifiers along with the consensus classifier is available from Lanka et al. (2019) The toolbox can also be found at the following URL: https://github.com/pradlanka/malini .

Multisite generalizability of schizophrenia diagnosis classification based on functional brain connectivity

Our objective was to assess the generalizability, across sites and cognitive contexts, of schizophrenia classification based on functional brain connectivity. We tested different training-test scenarios combining fMRI data from 191 schizophrenia patients and 191 matched healthy controls obtained at 6 scanning sites and under different task conditions. Diagnosis classification accuracy generalized well to a novel site and cognitive context provided data from multiple sites were used for classifier training. By contrast, lower classification accuracy was achieved when data from a single distinct site was used for training. These findings indicate that it is beneficial to use multisite data to train fMRI-based classifiers intended for large-scale use in the clinical realm.

On wakefulness fluctuations as a source of BOLD functional connectivity dynamics

Human brain dynamics and functional connectivity fluctuate over a range of temporal scales in coordination with internal states and environmental demands. However, the neurobiological significance and consequences of functional connectivity dynamics during rest have not yet been established. We show that the coarse-grained clustering of whole-brain dynamic connectivity measured with magnetic resonance imaging reveals discrete patterns (dynamic connectivity states) associated with wakefulness and sleep. We validate this using EEG in healthy subjects and patients with narcolepsy and by matching our results with previous findings in a large collaborative database. We also show that drowsiness may account for previous reports of metastable connectivity states associated with different levels of functional integration. This implies that future studies of transient functional connectivity must independently monitor wakefulness. We conclude that a possible neurobiological significance of dynamic connectivity states, computed at a sufficiently coarse temporal scale, is that of fluctuations in wakefulness.

Feature selection of fMRI data based on normalized mutual information and fisher discriminant ratio

Pattern classification has been increasingly used in functional magnetic resonance imaging (fMRI) data analysis. However, the classification performance is restricted by the high dimensional property and noises of the fMRI data. In this paper, a new feature selection method (named as "NMI-F") was proposed by sequentially combining the normalized mutual information (NMI) and fisher discriminant ratio. In NMI-F, the normalized mutual information was firstly used to evaluate the relationships between features, and fisher discriminant ratio was then applied to calculate the importance of each feature involved. Two fMRI datasets (task-related and resting state) were used to test the proposed method. It was found that classification base on the NMI-F method could differentiate the brain cognitive and disease states effectively, and the proposed NMI-F method was prior to the other related methods. The current results also have implications to the future studies.