Representation learning of resting state fMRI with variational autoencoder

Brain fingerprinting and cognitive behavior predicting using functional connectome of high inter-subject variability

The functional connectivity (FC) graph of the brain has been widely recognized as a ``fingerprint'' that can be used to identify individuals from a group of subjects. Research has indicated that individual identification accuracy can be improved by eliminating the impact of shared information among individuals. However, current research extracts not only shared information of inter-subject but also individual-specific information from FC graphs, resulting in incomplete separation of shared information and fingerprint information among individuals, leading to lower individual identification accuracy across all functional magnetic resonance imaging (fMRI) states session pairs and poor cognitive behavior prediction performance. In this paper, we propose a method to enhance inter-subject variability combining conditional variational autoencoder (CVAE) network and sparse dictionary learning (SDL) module. By embedding fMRI state information in the encoding and decoding processes, the CVAE network can better capture and represent the common features among individuals and enhance inter-subject variability by residual. Our experimental results on Human Connectome Project (HCP) data show that the refined connectomes obtained by using CVAE with SDL can accurately distinguish an individual from the remaining participants. The success accuracies reached 99.7 % and 99.6 % in the session pair rest1-rest2 and reverse rest2-rest1, respectively. In the identification experiment involving task-task combinations carried out on the same day, the identification accuracies ranged from 94.2 % to 98.8 %. Furthermore, we showed the Frontoparietal and Default networks make the most significant contributions to individual identification and the edges that significantly contribute to individual identification are found within and between the Frontoparietal and Default networks. Additionally, high-level cognitive behaviors can also be better predicted with the obtained refined connectomes, suggesting that higher fingerprinting can be useful for resulting in higher behavioral associations. In summary, our proposed framework provides a promising approach to use functional connectivity networks for studying cognition and behavior, promoting a deeper understanding of brain functions.

Explaining deep learning-based representations of resting state functional connectivity data: focusing on interpreting nonlinear patterns in autism spectrum disorder

Background

Resting state Functional Magnetic Resonance Imaging fMRI (rs-fMRI) has been used extensively to study brain function in psychiatric disorders, yielding insights into brain organization. However, the high dimensionality of the rs-fMRI data presents significant challenges for data analysis. Variational autoencoders (VAEs), a type of neural network, have been instrumental in extracting low-dimensional latent representations of resting state functional connectivity (rsFC) patterns, thereby addressing the complex nonlinear structure of rs-fMRI data. Despite these advances, interpreting these latent representations remains a challenge. This paper aims to address this gap by developing explainable VAE models and testing their utility using rs-fMRI data in autism spectrum disorder (ASD).

Methods

One-thousand one hundred and fifty participants (601 healthy controls [HC] and 549 patients with ASD) were included in the analysis. RsFC correlation matrices were extracted from the preprocessed rs-fMRI data using the Power atlas, which includes 264 regions of interest (ROIs). Then VAEs were trained in an unsupervised manner. Lastly, we introduce our latent contribution scores to explain the relationship between estimated representations and the original rs-fMRI brain measures.

Results

We quantified the latent contribution scores for both the ASD and HC groups at the network level. We found that both ASD and HC groups share the top network connectivitives contributing to all estimated latent components. For example, latent 0 was driven by rsFC within ventral attention network (VAN) in both the ASD and HC. However, we found significant differences in the latent contribution scores between the ASD and HC groups within the VAN for latent 0 and the sensory/somatomotor network for latent 2.

Conclusion

This study introduced latent contribution scores to interpret nonlinear patterns identified by VAEs. These scores effectively capture changes in each observed rsFC feature as the estimated latent representation changes, enabling an explainable deep learning model that better understands the underlying neural mechanisms of ASD.

A Demographic-Conditioned Variational Autoencoder for fMRI Distribution Sampling and Removal of Confounds

Objective

fMRI and derived measures such as functional connectivity (FC) have been used to predict brain age, general fluid intelligence, psychiatric disease status, and preclinical neurodegenerative disease. However, it is not always clear that all demographic confounds, such as age, sex, and race, have been removed from fMRI data. Additionally, many fMRI datasets are restricted to authorized researchers, making dissemination of these valuable data sources challenging.

Methods

We create a variational autoencoder (VAE)-based model, DemoVAE, to decorrelate fMRI features from demographics and generate high-quality synthetic fMRI data based on user-supplied demographics. We train and validate our model using two large, widely used datasets, the Philadelphia Neurodevelopmental Cohort (PNC) and Bipolar and Schizophrenia Network for Intermediate Phenotypes (BSNIP).

Results

We find that DemoVAE recapitulates group differences in fMRI data while capturing the full breadth of individual variations. Significantly, we also find that most clinical and computerized battery fields that are correlated with fMRI data are not correlated with DemoVAE latents. An exception are several fields related to schizophrenia medication and symptom severity.

Conclusion

Our model generates fMRI data that captures the full distribution of FC better than traditional VAE or GAN models. We also find that most prediction using fMRI data is dependent on correlation with, and prediction of, demographics.

Significance

Our DemoVAE model allows for generation of high quality synthetic data conditioned on subject demographics as well as the removal of the confounding effects of demographics. We identify that FC-based prediction tasks are highly influenced by demographic confounds.

Using a deep generation network reveals neuroanatomical specificity in hemispheres

Asymmetry is an important property of brain organization, but its nature is still poorly understood. Capturing the neuroanatomical components specific to each hemisphere facilitates the understanding of the establishment of brain asymmetry. Since deep generative networks (DGNs) have powerful inference and recovery capabilities, we use one hemisphere to predict the opposite hemisphere by training the DGNs, which automatically fit the built-in dependencies between the left and right hemispheres. After training, the reconstructed images approximate the homologous components in the hemisphere. We use the difference between the actual and reconstructed hemispheres to measure hemisphere-specific components due to asymmetric expression of environmental and genetic factors. The results show that our model is biologically plausible and that our proposed metric of hemispheric specialization is reliable, representing a wide range of individual variation. Together, this work provides promising tools for exploring brain asymmetry and new insights into self-supervised DGNs for representing the brain.

Conditional Variational Autoencoder for Functional Connectivity Analysis of Autism Spectrum Disorder Functional Magnetic Resonance Imaging Data: A Comparative Study

Generative models, such as Variational Autoencoders (VAEs), are increasingly employed for atypical pattern detection in brain imaging. During training, these models learn to capture the underlying patterns within "normal" brain images and generate new samples from those patterns. Neurodivergent states can be observed by measuring the dissimilarity between the generated/reconstructed images and the input images. This paper leverages VAEs to conduct Functional Connectivity (FC) analysis from functional Magnetic Resonance Imaging (fMRI) scans of individuals with Autism Spectrum Disorder (ASD), aiming to uncover atypical interconnectivity between brain regions. In the first part of our study, we compare multiple VAE architectures-Conditional VAE, Recurrent VAE, and a hybrid of CNN parallel with RNN VAE-aiming to establish the effectiveness of VAEs in application FC analysis. Given the nature of the disorder, ASD exhibits a higher prevalence among males than females. Therefore, in the second part of this paper, we investigate if introducing phenotypic data could improve the performance of VAEs and, consequently, FC analysis. We compare our results with the findings from previous studies in the literature. The results showed that CNN-based VAE architecture is more effective for this application than the other models.

Functional brain network identification and fMRI augmentation using a VAE-GAN framework

Recently, deep learning models have achieved superior performance for mapping functional brain networks from functional magnetic resonance imaging (fMRI) data compared with traditional methods. However, due to the lack of sufficient data and the high dimensionality of brain volume, deep learning models of fMRI tend to suffer from overfitting. In addition, existing methods rarely studied fMRI data augmentation and its application. To address these issues, we developed a VAE-GAN framework that combined a VAE (variational auto-encoder) with a GAN (generative adversarial net) for functional brain network identification and fMRI augmentation. As a generative model, the VAE-GAN models the distribution of fMRI so that it enables the extraction of more generalized features, and thus relieve the overfitting issue. The VAE-GAN is easier to train on fMRI than a standard GAN since it uses latent variables from VAE to generate fake data rather than relying on random noise that is used in a GAN, and it can generate higher quality of fake data than VAE since the discriminator can promote the training of the generator. In other words, the VAE-GAN inherits the advantages of VAE and GAN and avoids their limitations in modeling of fMRI data. Extensive experiments on task fMRI datasets from HCP have proved the effectiveness and superiority of the proposed VAE-GAN framework for identifying both temporal features and functional brain networks compared with existing models, and the quality of fake data is higher than those from VAE and GAN. The results on resting state fMRI of Attention Deficit Hyperactivity Disorder (ADHD)-200 dataset further demonstrated that the fake data generated by the VAE-GAN can help improve the performance of brain network modeling and ADHD classification.

Explaining Deep Learning-Based Representations of Resting State Functional Connectivity Data: Focusing on Interpreting Nonlinear Patterns in Autism Spectrum Disorder

Background

Resting state Functional Magnetic Resonance Imaging fMRI (rs-fMRI) has been used to study brain function in psychiatric disorders, yielding insight into brain organization. However, the high dimensionality of the rs-fMRI data presents challenges, and requires dimensionality reduction before applying machine learning techniques. Neural networks, specifically variational autoencoders (VAEs), have been instrumental in extracting low-dimensional latent representations of resting state functional connectivity patterns, addressing the complex nonlinear structure of rs-fMRI. However, interpreting those latent representations remains a challenge. This paper aims to address this gap by creating explainable VAE models and testing their utility using rs-fMRI data in autism spectrum disorder (ASD).

Methods

One-thousand one hundred and fifty participants (601 HC and 549 patients with ASD) were included in the analysis. We extracted functional connectivity correlation matrices from the preprocessed rs-fMRI data using Power atlas with 264 ROIs. Then VAEs were trained in an unsupervised fashion. Lastly, we introduce our latent contribution scores to explain the relationship between estimated representations and the original rs-fMRI brain measures.

Results

We quantified the latent contribution scores for the ASD and control groups at the network level. We found that both ASD and control groups share the top network connectivity that contribute to all estimated latent components. For example, latent 0 was driven by resting state functional connectivity patterns (rsFC) within ventral attention network in both the ASD and control. However, significant differences in the latent contribution scores between the ASD and control groups were discovered within the ventral attention network in latent 0 and the sensory/somatomotor network in latent 2.

Conclusion

This study introduced latent contribution scores to interpret nonlinear patterns identified by VAEs. These scores effectively capture changes in each observed rsFC features as estimated latent representation changes, enabling an explainable deep learning model to better understand the underlying neural mechanism of ASD.

Manifold learning for fMRI time-varying functional connectivity

Whole-brain functional connectivity (FC) measured with functional MRI (fMRI) evolves over time in meaningful ways at temporal scales going from years (e.g., development) to seconds [e.g., within-scan time-varying FC (tvFC)]. Yet, our ability to explore tvFC is severely constrained by its large dimensionality (several thousands). To overcome this difficulty, researchers often seek to generate low dimensional representations (e.g., 2D and 3D scatter plots) hoping those will retain important aspects of the data (e.g., relationships to behavior and disease progression). Limited prior empirical work suggests that manifold learning techniques (MLTs)-namely those seeking to infer a low dimensional non-linear surface (i.e., the manifold) where most of the data lies-are good candidates for accomplishing this task. Here we explore this possibility in detail. First, we discuss why one should expect tvFC data to lie on a low dimensional manifold. Second, we estimate what is the intrinsic dimension (ID; i.e., minimum number of latent dimensions) of tvFC data manifolds. Third, we describe the inner workings of three state-of-the-art MLTs: Laplacian Eigenmaps (LEs), T-distributed Stochastic Neighbor Embedding (T-SNE), and Uniform Manifold Approximation and Projection (UMAP). For each method, we empirically evaluate its ability to generate neuro-biologically meaningful representations of tvFC data, as well as their robustness against hyper-parameter selection. Our results show that tvFC data has an ID that ranges between 4 and 26, and that ID varies significantly between rest and task states. We also show how all three methods can effectively capture subject identity and task being performed: UMAP and T-SNE can capture these two levels of detail concurrently, but LE could only capture one at a time. We observed substantial variability in embedding quality across MLTs, and within-MLT as a function of hyper-parameter selection. To help alleviate this issue, we provide heuristics that can inform future studies. Finally, we also demonstrate the importance of feature normalization when combining data across subjects and the role that temporal autocorrelation plays in the application of MLTs to tvFC data. Overall, we conclude that while MLTs can be useful to generate summary views of labeled tvFC data, their application to unlabeled data such as resting-state remains challenging.

Toward a more informative representation of the fetal-neonatal brain connectome using variational autoencoder

Recent advances in functional magnetic resonance imaging (fMRI) have helped elucidate previously inaccessible trajectories of early-life prenatal and neonatal brain development. To date, the interpretation of fetal-neonatal fMRI data has relied on linear analytic models, akin to adult neuroimaging data. However, unlike the adult brain, the fetal and newborn brain develops extraordinarily rapidly, far outpacing any other brain development period across the life span. Consequently, conventional linear computational models may not adequately capture these accelerated and complex neurodevelopmental trajectories during this critical period of brain development along the prenatal-neonatal continuum. To obtain a nuanced understanding of fetal-neonatal brain development, including nonlinear growth, for the first time, we developed quantitative, systems-wide representations of brain activity in a large sample (>500) of fetuses, preterm, and full-term neonates using an unsupervised deep generative model called variational autoencoder (VAE), a model previously shown to be superior to linear models in representing complex resting-state data in healthy adults. Here, we demonstrated that nonlinear brain features, that is, latent variables, derived with the VAE pretrained on rsfMRI of human adults, carried important individual neural signatures, leading to improved representation of prenatal-neonatal brain maturational patterns and more accurate and stable age prediction in the neonate cohort compared to linear models. Using the VAE decoder, we also revealed distinct functional brain networks spanning the sensory and default mode networks. Using the VAE, we are able to reliably capture and quantify complex, nonlinear fetal-neonatal functional neural connectivity. This will lay the critical foundation for detailed mapping of healthy and aberrant functional brain signatures that have their origins in fetal life.

Unsupervised representation learning of spontaneous MEG data with Nonlinear ICA

Resting-state magnetoencephalography (MEG) data show complex but structured spatiotemporal patterns. However, the neurophysiological basis of these signal patterns is not fully known and the underlying signal sources are mixed in MEG measurements. Here, we developed a method based on the nonlinear independent component analysis (ICA), a generative model trainable with unsupervised learning, to learn representations from resting-state MEG data. After being trained with a large dataset from the Cam-CAN repository, the model has learned to represent and generate patterns of spontaneous cortical activity using latent nonlinear components, which reflects principal cortical patterns with specific spectral modes. When applied to the downstream classification task of audio-visual MEG, the nonlinear ICA model achieves competitive performance with deep neural networks despite limited access to labels. We further validate the generalizability of the model across different datasets by applying it to an independent neurofeedback dataset for decoding the subject's attentional states, providing a real-time feature extraction and decoding mindfulness and thought-inducing tasks with an accuracy of around 70% at the individual level, which is much higher than obtained by linear ICA or other baseline methods. Our results demonstrate that nonlinear ICA is a valuable addition to existing tools, particularly suited for unsupervised representation learning of spontaneous MEG activity which can then be applied to specific goals or tasks when labelled data are scarce.

Computing personalized brain functional networks from fMRI using self-supervised deep learning

A novel self-supervised deep learning (DL) method is developed to compute personalized brain functional networks (FNs) for characterizing brain functional neuroanatomy based on functional MRI (fMRI). Specifically, a DL model of convolutional neural networks with an encoder-decoder architecture is developed to compute personalized FNs directly from fMRI data. The DL model is trained to optimize functional homogeneity of personalized FNs without utilizing any external supervision in an end-to-end fashion. We demonstrate that a DL model trained on fMRI scans from the Human Connectome Project can identify personalized FNs and generalizes well across four different datasets. We further demonstrate that the identified personalized FNs are informative for predicting individual differences in behavior, brain development, and schizophrenia status. Taken together, the self-supervised DL allows for rapid, generalizable computation of personalized FNs.

Manifold Learning for fMRI time-varying FC

Whole-brain functional connectivity ( FC ) measured with functional MRI (fMRI) evolve over time in meaningful ways at temporal scales going from years (e.g., development) to seconds (e.g., within-scan time-varying FC ( tvFC )). Yet, our ability to explore tvFC is severely constrained by its large dimensionality (several thousands). To overcome this difficulty, researchers seek to generate low dimensional representations (e.g., 2D and 3D scatter plots) expected to retain its most informative aspects (e.g., relationships to behavior, disease progression). Limited prior empirical work suggests that manifold learning techniques ( MLTs )-namely those seeking to infer a low dimensional non-linear surface (i.e., the manifold) where most of the data lies-are good candidates for accomplishing this task. Here we explore this possibility in detail. First, we discuss why one should expect tv FC data to lie on a low dimensional manifold. Second, we estimate what is the intrinsic dimension (i.e., minimum number of latent dimensions; ID ) of tvFC data manifolds. Third, we describe the inner workings of three state-of-the-art MLTs : Laplacian Eigenmaps ( LE ), T-distributed Stochastic Neighbor Embedding ( T-SNE ), and Uniform Manifold Approximation and Projection ( UMAP ). For each method, we empirically evaluate its ability to generate neuro-biologically meaningful representations of tvFC data, as well as their robustness against hyper-parameter selection. Our results show that tvFC data has an ID that ranges between 4 and 26, and that ID varies significantly between rest and task states. We also show how all three methods can effectively capture subject identity and task being performed: UMAP and T-SNE can capture these two levels of detail concurrently, but L E could only capture one at a time. We observed substantial variability in embedding quality across MLTs , and within- MLT as a function of hyper-parameter selection. To help alleviate this issue, we provide heuristics that can inform future studies. Finally, we also demonstrate the importance of feature normalization when combining data across subjects and the role that temporal autocorrelation plays in the application of MLTs to tvFC data. Overall, we conclude that while MLTs can be useful to generate summary views of labeled tvFC data, their application to unlabeled data such as resting-state remains challenging.

A transformer model for learning spatiotemporal contextual representation in fMRI data

Representation learning is a core component in data-driven modeling of various complex phenomena. Learning a contextually informative representation can especially benefit the analysis of fMRI data because of the complexities and dynamic dependencies present in such datasets. In this work, we propose a framework based on transformer models to learn an embedding of the fMRI data by taking the spatiotemporal contextual information in the data into account. This approach takes the multivariate BOLD time series of the regions of the brain as well as their functional connectivity network simultaneously as the input to create a set of meaningful features that can in turn be used in various downstream tasks such as classification, feature extraction, and statistical analysis. The proposed spatiotemporal framework uses the attention mechanism as well as the graph convolution neural network to jointly inject the contextual information regarding the dynamics in time series data and their connectivity into the representation. We demonstrate the benefits of this framework by applying it to two resting-state fMRI datasets, and provide further discussion on various aspects and advantages of it over a number of other commonly adopted architectures.

Brain Decoding Using fMRI Images for Multiple Subjects through Deep Learning

Substantial information related to human cerebral conditions can be decoded through various noninvasive evaluating techniques like fMRI. Exploration of the neuronal activity of the human brain can divulge the thoughts of a person like what the subject is perceiving, thinking, or visualizing. Furthermore, deep learning techniques can be used to decode the multifaceted patterns of the brain in response to external stimuli. Existing techniques are capable of exploring and classifying the thoughts of the human subject acquired by the fMRI imaging data. fMRI images are the volumetric imaging scans which are highly dimensional as well as require a lot of time for training when fed as an input in the deep learning network. However, the hassle for more efficient learning of highly dimensional high-level features in less training time and accurate interpretation of the brain voxels with less misclassification error is needed. In this research, we propose an improved CNN technique where features will be functionally aligned. The optimal features will be selected after dimensionality reduction. The highly dimensional feature vector will be transformed into low dimensional space for dimensionality reduction through autoadjusted weights and combination of best activation functions. Furthermore, we solve the problem of increased training time by using Swish activation function, making it denser and increasing efficiency of the model in less training time. Finally, the experimental results are evaluated and compared with other classifiers which demonstrated the supremacy of the proposed model in terms of accuracy.