4D Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)

A survey of brain functional network extraction methods using fMRI data

Functional network (FN) analyses play a pivotal role in uncovering insights into brain function and understanding the pathophysiology of various brain disorders. This paper focuses on classical and advanced methods for deriving brain FNs from functional magnetic resonance imaging (fMRI) data. We systematically review their foundational principles, advantages, shortcomings, and interrelations, encompassing both static and dynamic FN extraction approaches. In the context of static FN extraction, we present hypothesis-driven methods such as region of interest (ROI)-based approaches as well as data-driven methods including matrix decomposition, clustering, and deep learning. For dynamic FN extraction, both window-based and windowless methods are surveyed with respect to the estimation of time-varying FN and the subsequent computation of FN states. We also discuss the scope of application of the various methods and avenues for future improvements.

Modeling default mode network patterns via a universal spatio-temporal brain attention skip network

Designing a comprehensive four-dimensional resting-state functional magnetic resonance imaging (4D Rs-fMRI) based default mode network (DMN) modeling methodology to reveal the spatio-temporal patterns of individual DMN, is crucial for understanding the cognitive mechanisms of the brain and the pathogenesis of psychiatric disorders. However, there are still two limitations of existing approaches for DMN modeling. The approaches either (1) simply split the spatio-temporal components and ignore the overall character of the spatio-temporal patterns or (2) are biased in the process of feature extraction for DMN modeling, and their spatio-temporal accuracy is thus not warranted. To this end, we propose a novel Spatio-Temporal Brain Attention Skip Network (STBAS-Net) to model the personalized spatio-temporal patterns of the DMN. STBAS-Net consists of spatial and temporal components, where the multi-head attention skip connection block in the spatial component achieves detailed feature extraction and enhancement in the shallow stage. Under the guidance of spatial information, we technically fuse multiple spatio-temporal information in the temporal component, which dexterously exploits the overall spatio-temporal features and achieves mutual constraints of spatio-temporal patterns to characterize the spatio-temporal patterns of the DMN. We verify the proposed STBAS-Net on a publicly released 4D Rs-fMRI dataset and an EMCI dataset. The experimental results show that compared with existing advanced methods, the proposed network can more accurately model the personalized spatio-temporal patterns of the human brain DMN and successfully identify abnormal spatio-temporal patterns in EMCI patients. This study provides a potential tool for revealing the spatio-temporal patterns of the human brain DMN and is expected to provide an effective methodological framework for future exploration of abnormal brain spatio-temporal patterns and modeling of other functional brain networks.

Computing personalized brain functional networks from fMRI using self-supervised deep learning

A novel self-supervised deep learning (DL) method is developed to compute personalized brain functional networks (FNs) for characterizing brain functional neuroanatomy based on functional MRI (fMRI). Specifically, a DL model of convolutional neural networks with an encoder-decoder architecture is developed to compute personalized FNs directly from fMRI data. The DL model is trained to optimize functional homogeneity of personalized FNs without utilizing any external supervision in an end-to-end fashion. We demonstrate that a DL model trained on fMRI scans from the Human Connectome Project can identify personalized FNs and generalizes well across four different datasets. We further demonstrate that the identified personalized FNs are informative for predicting individual differences in behavior, brain development, and schizophrenia status. Taken together, the self-supervised DL allows for rapid, generalizable computation of personalized FNs.

Machine Learning for Brain MRI Data Harmonisation: A Systematic Review

BACKGROUND

Magnetic Resonance Imaging (MRI) data collected from multiple centres can be heterogeneous due to factors such as the scanner used and the site location. To reduce this heterogeneity, the data needs to be harmonised. In recent years, machine learning (ML) has been used to solve different types of problems related to MRI data, showing great promise.

OBJECTIVE

This study explores how well various ML algorithms perform in harmonising MRI data, both implicitly and explicitly, by summarising the findings in relevant peer-reviewed articles. Furthermore, it provides guidelines for the use of current methods and identifies potential future research directions.

METHOD

This review covers articles published through PubMed, Web of Science, and IEEE databases through June 2022. Data from studies were analysed based on the criteria of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Quality assessment questions were derived to assess the quality of the included publications.

RESULTS

a total of 41 articles published between 2015 and 2022 were identified and analysed. In the review, MRI data has been found to be harmonised either in an implicit (n = 21) or an explicit (n = 20) way. Three MRI modalities were identified: structural MRI (n = 28), diffusion MRI (n = 7) and functional MRI (n = 6).

CONCLUSION

Various ML techniques have been employed to harmonise different types of MRI data. There is currently a lack of consistent evaluation methods and metrics used across studies, and it is recommended that the issue be addressed in future studies. Harmonisation of MRI data using ML shows promises in improving performance for ML downstream tasks, while caution should be exercised when using ML-harmonised data for direct interpretation.

Unsupervised Feature Selection via Graph Regularized Nonnegative CP Decomposition

Unsupervised feature selection has attracted remarkable attention recently. With the development of data acquisition technology, multi-dimensional tensor data has been appeared in enormous real-world applications. However, most existing unsupervised feature selection methods are non-tensor-based which results the vectorization of tensor data as a preprocessing step. This seemingly ordinary operation has led to an unnecessary loss of the multi-dimensional structural information and eventually restricted the quality of the selected features. To overcome the limitation, in this paper, we propose a novel unsupervised feature selection model: Nonnegative tensor CP (CANDECOMP/PARAFAC) decomposition based unsupervised feature selection, CPUFS for short. In specific, we devise new tensor-oriented linear classifier and feature selection matrix for CPUFS. In addition, CPUFS simultaneously conducts graph regularized nonnegative CP decomposition and newly-designed tensor-oriented pseudo label regression and feature selection to fully preserve the multi-dimensional data structure. To solve the CPUFS model, we propose an efficient iterative optimization algorithm with theoretically guaranteed convergence, whose computational complexity scales linearly in the number of features. A variation of the CPUFS model by incorporating nonnegativity into the linear classifier, namely CPUFSnn, is also proposed and studied. Experimental results on ten real-world benchmark datasets demonstrate the effectiveness of both CPUFS and CPUFSnn over the state-of-the-arts.

Characterizing functional brain networks via Spatio-Temporal Attention 4D Convolutional Neural Networks (STA-4DCNNs)

Characterizing individualized spatio-temporal patterns of functional brain networks (FBNs) via functional magnetic resonance imaging (fMRI) provides a foundation for understanding complex brain function. Although previous studies have achieved promising performances based on either shallow or deep learning models, there is still much space to improve the accuracy of spatio-temporal pattern characterization of FBNs by optimally integrating the four-dimensional (4D) features of fMRI. In this study, we introduce a novel Spatio-Temporal Attention 4D Convolutional Neural Network (STA-4DCNN) model to characterize individualized spatio-temporal patterns of FBNs. Particularly, STA-4DCNN is composed of two subnetworks, in which the first Spatial Attention 4D CNN (SA-4DCNN) models the spatio-temporal features of 4D fMRI data and then characterizes the spatial pattern of FBNs, and the second Temporal Guided Attention Network (T-GANet) further characterizes the temporal pattern of FBNs under the guidance of the spatial pattern together with 4D fMRI data. We evaluate the proposed STA-4DCNN on seven different task fMRI and one resting state fMRI datasets from the publicly released Human Connectome Project. The experimental results demonstrate that STA-4DCNN has superior ability and generalizability in characterizing individualized spatio-temporal patterns of FBNs when compared to other state-of-the-art models. We further apply STA-4DCNN on another independent ABIDE I resting state fMRI dataset including both autism spectrum disorder (ASD) and typical developing (TD) subjects, and successfully identify abnormal spatio-temporal patterns of FBNs in ASD compared to TD. In general, STA-4DCNN provides a powerful tool for FBN characterization and for clinical applications on brain disease characterization at the individual level.

An explainable deep learning framework for characterizing and interpreting human brain states

Deep learning approaches have been widely adopted in the medical image analysis field. However, a most of existing deep learning approaches focus on achieving promising performances such as classification, detection, and segmentation, and much less effort is devoted to the explanation of the designed models. Similarly, in the brain imaging field, many deep learning approaches have been designed and applied to characterize and predict human brain states. However, these models lack interpretation. In response, we propose a novel domain knowledge informed self-attention graph pooling-based (SAGPool) graph convolutional neural network to study human brain states. Specifically, the dense individualized and common connectivity-based cortical landmarks system (DICCCOL, structural brain connectivity profiles) and holistic atlases of functional networks and interactions system (HAFNI, functional brain connectivity profiles) are integrated with the SAGPool model to better characterize and interpret the brain states. Extensive experiments are designed and carried out on the large-scale human connectome project (HCP) Q1 and S1200 dataset. Promising brain state classification performances are observed (e.g., an average of 93.7% for seven-task classification and 100% for binary classification). In addition, the importance of the brain regions, which contributes most to the accurate classification, is successfully quantified and visualized. A thorough neuroscientific interpretation suggests that these extracted brain regions and their importance calculated from self-attention graph pooling layer offer substantial explainability.

Modern views of machine learning for precision psychiatry

In light of the National Institute of Mental Health (NIMH)'s Research Domain Criteria (RDoC), the advent of functional neuroimaging, novel technologies and methods provide new opportunities to develop precise and personalized prognosis and diagnosis of mental disorders. Machine learning (ML) and artificial intelligence (AI) technologies are playing an increasingly critical role in the new era of precision psychiatry. Combining ML/AI with neuromodulation technologies can potentially provide explainable solutions in clinical practice and effective therapeutic treatment. Advanced wearable and mobile technologies also call for the new role of ML/AI for digital phenotyping in mobile mental health. In this review, we provide a comprehensive review of ML methodologies and applications by combining neuroimaging, neuromodulation, and advanced mobile technologies in psychiatry practice. We further review the role of ML in molecular phenotyping and cross-species biomarker identification in precision psychiatry. We also discuss explainable AI (XAI) and neuromodulation in a closed human-in-the-loop manner and highlight the ML potential in multi-media information extraction and multi-modal data fusion. Finally, we discuss conceptual and practical challenges in precision psychiatry and highlight ML opportunities in future research.

Decoding Task-Based fMRI Data with Graph Neural Networks, Considering Individual Differences

Task fMRI provides an opportunity to analyze the working mechanisms of the human brain during specific experimental paradigms. Deep learning models have increasingly been applied for decoding and encoding purposes study to representations in task fMRI data. More recently, graph neural networks, or neural networks models designed to leverage the properties of graph representations, have recently shown promise in task fMRI decoding studies. Here, we propose an end-to-end graph convolutional network (GCN) framework with three convolutional layers to classify task fMRI data from the Human Connectome Project dataset. We compared the predictive performance of our GCN model across four of the most widely used node embedding algorithms—NetMF, RandNE, Node2Vec, and Walklets—to automatically extract the structural properties of the nodes in the functional graph. The empirical results indicated that our GCN framework accurately predicted individual differences (0.978 and 0.976) with the NetMF and RandNE embedding methods, respectively. Furthermore, to assess the effects of individual differences, we tested the classification performance of the model on sub-datasets divided according to gender and fluid intelligence. Experimental results indicated significant differences in the classification predictions of gender, but not high/low fluid intelligence fMRI data. Our experiments yielded promising results and demonstrated the superior ability of our GCN in modeling task fMRI data.

Modeling spatio-temporal patterns of holistic functional brain networks via multi-head guided attention graph neural networks (Multi-Head GAGNNs)

Mounting evidence has demonstrated that complex brain function processes are realized by the interaction of holistic functional brain networks which are spatially distributed across specific brain regions in a temporally dynamic fashion. Therefore, modeling spatio-temporal patterns of holistic functional brain networks plays an important role in understanding brain function. Compared to traditional modeling methods such as principal component analysis, independent component analysis, and sparse coding, superior performance has been achieved by recent deep learning methodologies. However, there are still two limitations of existing deep learning approaches for functional brain network modeling. They either (1) merely modeled a single targeted network and ignored holistic ones at one time, or (2) underutilized both spatial and temporal features of fMRI during network modeling, and the spatial/temporal accuracy was thus not warranted. To address these limitations, we proposed a novel Multi-Head Guided Attention Graph Neural Network (Multi-Head GAGNN) to simultaneously model both spatial and temporal patterns of holistic functional brain networks. Specifically, a spatial Multi-Head Attention Graph U-Net was first adopted to model the spatial patterns of multiple brain networks, and a temporal Multi-Head Guided Attention Network was then introduced to model the corresponding temporal patterns under the guidance of modeled spatial patterns. Based on seven task fMRI datasets from the public Human Connectome Project and resting state fMRI datasets from the public Autism Brain Imaging Data Exchange I of 1448 subjects, the proposed Multi-Head GAGNN showed superior ability and generalizability in modeling both spatial and temporal patterns of holistic functional brain networks in individual brains compared to other state-of-the-art (SOTA) models. Furthermore, the modeled spatio-temporal patterns of functional brain networks via the proposed Multi-Head GAGNN can better predict the individual cognitive behavioral measures compared to the other SOTA models. This study provided a novel and powerful tool for brain function modeling as well as for understanding the brain-cognitive behavior associations.

Hierarchical Spatio-Temporal Modeling of Naturalistic Functional Magnetic Resonance Imaging Signals via Two-Stage Deep Belief Network With Neural Architecture Search

Naturalistic functional magnetic resonance imaging (NfMRI) has become an effective tool to study brain functional activities in real-life context, which reduces the anxiety or boredom due to difficult or repetitive tasks and avoids the problem of unreliable collection of brain activity caused by the subjects’ microsleeps during resting state. Recent studies have made efforts on characterizing the brain’s hierarchical organizations from fMRI data by various deep learning models. However, most of those models have ignored the properties of group-wise consistency and inter-subject difference in brain function under naturalistic paradigm. Another critical issue is how to determine the optimal neural architecture of deep learning models, as manual design of neural architecture is time-consuming and less reliable. To tackle these problems, we proposed a two-stage deep belief network (DBN) with neural architecture search (NAS) combined framework (two-stage NAS-DBN) to model both the group-consistent and individual-specific naturalistic functional brain networks (FBNs), which reflected the hierarchical organization of brain function and the nature of brain functional activities under naturalistic paradigm. Moreover, the test-retest reliability and spatial overlap rate of the FBNs identified by our model reveal better performance than that of widely used traditional methods. In general, our model provides a promising method for characterizing hierarchical spatiotemporal features under the natural paradigm.

Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition

Using deep neural networks (DNNs) to explore spatial patterns and temporal dynamics of human brain activities has been an important yet challenging problem because the artificial neural networks are hard to be designed manually. There have been several promising deep learning methods, e.g., deep belief network (DBN), convolutional neural network (CNN), and deep sparse recurrent auto-encoder (DSRAE), that can decompose neuroscientific and meaningful spatiotemporal patterns from 4D functional Magnetic Resonance Imaging (fMRI) data. However, those previous studies still depend on hand-crafted neural network architectures and hyperparameters, which are not optimal in various senses. In this paper, we employ the evolutionary algorithms (EA) to optimize the deep neural architecture of DSRAE by minimizing the expected loss of initialized models, named eNAS-DSRAE (evolutionary Neural Architecture Search on Deep Sparse Recurrent Auto-Encoder). Also, validation experiments are designed and performed on the publicly available human connectome project (HCP) 900 datasets, and the results achieved by the optimized eNAS-DSRAE suggested that our framework can successfully identify the spatiotemporal features and perform better than the hand-crafted neural network models. To our best knowledge, the proposed eNAS-DSRAE is not only among the earliest NAS models that can extract connectome-scale meaningful spatiotemporal brain networks from 4D fMRI data, but also is an effective framework to optimize the RNN-based models.

Simultaneous spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder

Exploring the spatial patterns and temporal dynamics of human brain activity has been of great interest, in the quest to better understand connectome-scale brain networks. Though modeling spatial and temporal patterns of functional brain networks have been researched for a long time, the development of a unified and simultaneous spatial-temporal model has yet to be realized. For instance, although some deep learning methods have been proposed recently in order to model functional brain networks, most of them can only represent either spatial or temporal perspective of functional Magnetic Resonance Imaging (fMRI) data and rarely model both domains simultaneously. Due to the recent success in applying sequential auto-encoders for brain decoding, in this paper, we propose a deep sparse recurrent auto-encoder (DSRAE) to be applied unsupervised to learn spatial patterns and temporal fluctuations of brain networks at the same time. The proposed DSRAE was evaluated and validated based on three tasks of the publicly available Human Connectome Project (HCP) fMRI dataset, resulting with promising evidence. To the best of our knowledge, the proposed DSRAE is among the early efforts in developing unified models that can extract connectome-scale spatial-temporal networks from 4D fMRI data simultaneously.