Constructing fine-granularity functional brain network atlases via deep convolutional autoencoder

fMRI-based spatio-temporal parcellations of the human brain

PURPOSE OF REVIEW

Human brain parcellation based on functional magnetic resonance imaging (fMRI) plays an essential role in neuroscience research. By segmenting vast and intricate fMRI data into functionally similar units, researchers can better decipher the brain's structure in both healthy and diseased states. This article reviews current methodologies and ideas in this field, while also outlining the obstacles and directions for future research.

RECENT FINDINGS

Traditional brain parcellation techniques, which often rely on cytoarchitectonic criteria, overlook the functional and temporal information accessible through fMRI. The adoption of machine learning techniques, notably deep learning, offers the potential to harness both spatial and temporal information for more nuanced brain segmentation. However, the search for a one-size-fits-all solution to brain segmentation is impractical, with the choice between group-level or individual-level models and the intended downstream analysis influencing the optimal parcellation strategy. Additionally, evaluating these models is complicated by our incomplete understanding of brain function and the absence of a definitive "ground truth".

SUMMARY

While recent methodological advancements have significantly enhanced our grasp of the brain's spatial and temporal dynamics, challenges persist in advancing fMRI-based spatio-temporal representations. Future efforts will likely focus on refining model evaluation and selection as well as developing methods that offer clear interpretability for clinical usage, thereby facilitating further breakthroughs in our comprehension of the brain.

Neuroimaging-based classification of PTSD using data-driven computational approaches: A multisite big data study from the ENIGMA-PGC PTSD consortium

BACKGROUND

Recent advances in data-driven computational approaches have been helpful in devising tools to objectively diagnose psychiatric disorders. However, current machine learning studies limited to small homogeneous samples, different methodologies, and different imaging collection protocols, limit the ability to directly compare and generalize their results. Here we aimed to classify individuals with PTSD versus controls and assess the generalizability using a large heterogeneous brain datasets from the ENIGMA-PGC PTSD Working group.

METHODS

We analyzed brain MRI data from 3,477 structural-MRI; 2,495 resting state-fMRI; and 1,952 diffusion-MRI. First, we identified the brain features that best distinguish individuals with PTSD from controls using traditional machine learning methods. Second, we assessed the utility of the denoising variational autoencoder (DVAE) and evaluated its classification performance. Third, we assessed the generalizability and reproducibility of both models using leave-one-site-out cross-validation procedure for each modality.

RESULTS

We found lower performance in classifying PTSD vs. controls with data from over 20 sites (60 % test AUC for s-MRI, 59 % for rs-fMRI and 56 % for d-MRI), as compared to other studies run on single-site data. The performance increased when classifying PTSD from HC without trauma history in each modality (75 % AUC). The classification performance remained intact when applying the DVAE framework, which reduced the number of features. Finally, we found that the DVAE framework achieved better generalization to unseen datasets compared with the traditional machine learning frameworks, albeit performance was slightly above chance.

CONCLUSION

These results have the potential to provide a baseline classification performance for PTSD when using large scale neuroimaging datasets. Our findings show that the control group used can heavily affect classification performance. The DVAE framework provided better generalizability for the multi-site data. This may be more significant in clinical practice since the neuroimaging-based diagnostic DVAE classification models are much less site-specific, rendering them more generalizable.

ReMiND: Recovery of Missing Neuroimaging using Diffusion Models with Application to Alzheimer's Disease

Objective

Missing data is a significant challenge in medical research. In longitudinal studies of Alzheimer's disease (AD) where structural magnetic resonance imaging (MRI) is collected from individuals at multiple time points, participants may miss a study visit or drop out. Additionally, technical issues such as participant motion in the scanner may result in unusable imaging data at designated visits. Such missing data may hinder the development of high-quality imaging-based biomarkers. Furthermore, when imaging data are unavailable in clinical practice, patients may not benefit from effective application of biomarkers for disease diagnosis and monitoring.

Methods

To address the problem of missing MRI data in studies of AD, we introduced a novel 3D diffusion model specifically designed for imputing missing structural MRI (Recovery of Missing Neuroimaging using Diffusion models (ReMiND)). The model generates a whole-brain image conditional on a single structural MRI observed at a past visit or conditional on one past and one future observed structural MRI relative to the missing observation.

Results

Experimental results show that our method can generate high-quality individual 3D structural MRI with high similarity to ground truth, observed images. Additionally, images generated using ReMiND exhibit relatively lower error rates and more accurately estimated rates of atrophy over time in important anatomical brain regions compared with two alternative imputation approaches: forward filling and image generation using variational autoencoders.

Conclusion

Our 3D diffusion model can impute missing structural MRI data at a single designated visit and outperforms alternative methods for imputing whole-brain images that are missing from longitudinal trajectories.

Characterizing functional brain networks via Spatio-Temporal Attention 4D Convolutional Neural Networks (STA-4DCNNs)

Characterizing individualized spatio-temporal patterns of functional brain networks (FBNs) via functional magnetic resonance imaging (fMRI) provides a foundation for understanding complex brain function. Although previous studies have achieved promising performances based on either shallow or deep learning models, there is still much space to improve the accuracy of spatio-temporal pattern characterization of FBNs by optimally integrating the four-dimensional (4D) features of fMRI. In this study, we introduce a novel Spatio-Temporal Attention 4D Convolutional Neural Network (STA-4DCNN) model to characterize individualized spatio-temporal patterns of FBNs. Particularly, STA-4DCNN is composed of two subnetworks, in which the first Spatial Attention 4D CNN (SA-4DCNN) models the spatio-temporal features of 4D fMRI data and then characterizes the spatial pattern of FBNs, and the second Temporal Guided Attention Network (T-GANet) further characterizes the temporal pattern of FBNs under the guidance of the spatial pattern together with 4D fMRI data. We evaluate the proposed STA-4DCNN on seven different task fMRI and one resting state fMRI datasets from the publicly released Human Connectome Project. The experimental results demonstrate that STA-4DCNN has superior ability and generalizability in characterizing individualized spatio-temporal patterns of FBNs when compared to other state-of-the-art models. We further apply STA-4DCNN on another independent ABIDE I resting state fMRI dataset including both autism spectrum disorder (ASD) and typical developing (TD) subjects, and successfully identify abnormal spatio-temporal patterns of FBNs in ASD compared to TD. In general, STA-4DCNN provides a powerful tool for FBN characterization and for clinical applications on brain disease characterization at the individual level.

Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility

Probabilistic generative models are attractive for scientific modeling because their inferred parameters can be used to generate hypotheses and design experiments. This requires that the learned model provides an accurate representation of the input data and yields a latent space that effectively predicts outcomes relevant to the scientific question. Supervised Variational Autoencoders (SVAEs) have previously been used for this purpose, as a carefully designed decoder can be used as an interpretable generative model of the data, while the supervised objective ensures a predictive latent representation. Unfortunately, the supervised objective forces the encoder to learn a biased approximation to the generative posterior distribution, which renders the generative parameters unreliable when used in scientific models. This issue has remained undetected as reconstruction losses commonly used to evaluate model performance do not detect bias in the encoder. We address this previously-unreported issue by developing a second-order supervision framework (SOS-VAE) that updates the decoder parameters, rather than the encoder, to induce a predictive latent representation. This ensures that the encoder maintains a reliable posterior approximation and the decoder parameters can be effectively interpreted. We extend this technique to allow the user to trade-off the bias in the generative parameters for improved predictive performance, acting as an intermediate option between SVAEs and our new SOS-VAE. We also use this methodology to address missing data issues that often arise when combining recordings from multiple scientific experiments. We demonstrate the effectiveness of these developments using synthetic data and electrophysiological recordings with an emphasis on how our learned representations can be used to design scientific experiments.

Learning brain representation using recurrent Wasserstein generative adversarial net

BACKGROUND AND OBJECTIVE

To understand brain cognition and disorders, modeling the mapping between mind and brain has been of great interest to the neuroscience community. The key is the brain representation, including functional brain networks (FBN) and their corresponding temporal features. Recently, it has been proven that deep learning models have superb representation power on functional magnetic resonance imaging (fMRI) over traditional machine learning methods. However, due to the lack of high-quality data and labels, deep learning models tend to suffer from overfitting in the training process.

METHODS

In this work, we applied a recurrent Wasserstein generative adversarial net (RWGAN) to learn brain representation from volumetric fMRI data. Generative adversarial net (GAN) is widely used in natural image generation and is able to capture the distribution of the input data, which enables the extraction of generalized features from fMRI and thus relieves the overfitting issue. The recurrent layers in RWGAN are designed to better model the local temporal features of the fMRI time series. The discriminator of RWGAN works as a deep feature extractor. With LASSO regression, the RWGAN model can decompose the fMRI data into temporal features and spatial features (FBNs). Furthermore, the generator of RWGAN can generate high-quality new data for fMRI augmentation.

RESULTS

The experimental results on seven tasks from the HCP dataset showed that the RWGAN can learn meaningful and interpretable temporal features and FBNs, compared to HCP task designs and general linear model (GLM) derived networks. Besides, the results on different training datasets showed that the RWGAN performed better on small datasets than other deep learning models. Moreover, we used the generator of RWGAN to yield fake subjects. The result showed that the fake data can also be used to learn meaningful representation compared to those learned from real data.

CONCLUSIONS

To our best knowledge, this work is among the earliest attempts of applying generative deep learning for modeling fMRI data. The proposed RWGAN offers a novel methodology for learning brain representation from fMRI, and it can generate high-quality fake data for the potential use of fMRI data augmentation.

Multi-Source Domain Adaptation Techniques for Mitigating Batch Effects: A Comparative Study

The past decade has seen an increasing number of applications of deep learning (DL) techniques to biomedical fields, especially in neuroimaging-based analysis. Such DL-based methods are generally data-intensive and require a large number of training instances, which might be infeasible to acquire from a single acquisition site, especially for data, such as fMRI scans, due to the time and costs that they demand. We can attempt to address this issue by combining fMRI data from various sites, thereby creating a bigger heterogeneous dataset. Unfortunately, the inherent differences in the combined data, known as batch effects, often hamper learning a model. To mitigate this issue, techniques such as multi-source domain adaptation [Multi-source Domain Adversarial Networks (MSDA)] aim at learning an effective classification function that uses (learned) domain-invariant latent features. This article analyzes and compares the performance of various popular MSDA methods [MDAN, Domain AggRegation Networks (DARN), Multi-Domain Matching Networks (MDMN), and Moment Matching for MSDA (M3SDA)] at predicting different labels (illness, age, and sex) of images from two public rs-fMRI datasets: ABIDE 1and ADHD-200. It also evaluates the impact of various conditions such as class imbalance, the number of sites along with a comparison of the degree of adaptation of each of the methods, thereby presenting the effectiveness of MSDA models in neuroimaging-based applications.

4D Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)

Since the human brain functional mechanism has been enabled for investigation by the functional Magnetic Resonance Imaging (fMRI) technology, simultaneous modeling of both the spatial and temporal patterns of brain functional networks from 4D fMRI data has been a fundamental but still challenging research topic for neuroimaging and medical image analysis fields. Currently, general linear model (GLM), independent component analysis (ICA), sparse dictionary learning, and recently deep learning models, are major methods for fMRI data analysis in either spatial or temporal domains, but there are few joint spatial-temporal methods proposed, as far as we know. As a result, the 4D nature of fMRI data has not been effectively investigated due to this methodological gap. The recent success of deep learning applications for functional brain decoding and encoding greatly inspired us in this work to propose a novel framework called spatio-temporal convolutional neural network (ST-CNN) to extract both spatial and temporal characteristics from targeted networks jointly and automatically identify of functional networks. The identification of Default Mode Network (DMN) from fMRI data was used for evaluation of the proposed framework. Results show that only training the framework on one fMRI dataset is sufficiently generalizable to identify the DMN from different datasets of different cognitive tasks and resting state. Further investigation of the results shows that the joint-learning scheme can capture the intrinsic relationship between the spatial and temporal characteristics of DMN and thus it ensures the accurate identification of DMN from independent datasets. The ST-CNN model brings new tools and insights for fMRI analysis in cognitive and clinical neuroscience studies.

Evolution of Human Brain Atlases in Terms of Content, Applications, Functionality, and Availability

Human brain atlases have been evolving tremendously, propelled recently by brain big projects, and driven by sophisticated imaging techniques, advanced brain mapping methods, vast data, analytical strategies, and powerful computing. We overview here this evolution in four categories: content, applications, functionality, and availability, in contrast to other works limited mostly to content. Four atlas generations are distinguished: early cortical maps, print stereotactic atlases, early digital atlases, and advanced brain atlas platforms, and 5 avenues in electronic atlases spanning the last two generations. Content-wise, new electronic atlases are categorized into eight groups considering their scope, parcellation, modality, plurality, scale, ethnicity, abnormality, and a mixture of them. Atlas content developments in these groups are heading in 23 various directions. Application-wise, we overview atlases in neuroeducation, research, and clinics, including stereotactic and functional neurosurgery, neuroradiology, neurology, and stroke. Functionality-wise, tools and functionalities are addressed for atlas creation, navigation, individualization, enabling operations, and application-specific. Availability is discussed in media and platforms, ranging from mobile solutions to leading-edge supercomputers, with three accessibility levels. The major application-wise shift has been from research to clinical practice, particularly in stereotactic and functional neurosurgery, although clinical applications are still lagging behind the atlas content progress. Atlas functionality also has been relatively neglected until recently, as the management of brain data explosion requires powerful tools. We suggest that the future human brain atlas-related research and development activities shall be founded on and benefit from a standard framework containing the core virtual brain model cum the brain atlas platform general architecture.

fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Semi-supervised labelling of the femur in a whole-body post-mortem CT database using deep learning

A deep learning pipeline was developed and used to localize and classify a variety of implants in the femur contained in whole-body post-mortem computed tomography (PMCT) scans. The results provide a proof-of-principle approach for labelling content not described in medical/autopsy reports. The pipeline, which incorporated residual networks and an autoencoder, was trained and tested using n = 450 full-body PMCT scans. For the localization component, Dice scores of 0.99, 0.96, and 0.98 and mean absolute errors of 3.2, 7.1, and 4.2 mm were obtained in the axial, coronal, and sagittal views, respectively. A regression analysis found the orientation of the implant to the scanner axis and also the relative positioning of extremities to be statistically significant factors. For the classification component, test cases were properly labelled as nail (N⁺), hip replacement (H⁺), knee replacement (K⁺) or without-implant (I^-) with an accuracy >97%. The recall for I^- and H⁺ cases was 1.00, but fell to 0.82 and 0.65 for cases with K⁺ and N⁺. This semi-automatic approach provides a generalized structure for image-based labelling of features, without requiring time-consuming segmentation.

Identifying Brain Networks at Multiple Time Scales via Deep Recurrent Neural Network

For decades, task functional magnetic resonance imaging has been a powerful noninvasive tool to explore the organizational architecture of human brain function. Researchers have developed a variety of brain network analysis methods for task fMRI data, including the general linear model, independent component analysis, and sparse representation methods. However, these shallow models are limited in faithful reconstruction and modeling of the hierarchical and temporal structures of brain networks, as demonstrated in more and more studies. Recently, recurrent neural networks (RNNs) exhibit great ability of modeling hierarchical and temporal dependence features in the machine learning field, which might be suitable for task fMRI data modeling. To explore such possible advantages of RNNs for task fMRI data, we propose a novel framework of a deep recurrent neural network (DRNN) to model the functional brain networks from task fMRI data. Experimental results on the motor task fMRI data of Human Connectome Project 900 subjects release demonstrated that the proposed DRNN can not only faithfully reconstruct functional brain networks, but also identify more meaningful brain networks with multiple time scales which are overlooked by traditional shallow models. In general, this work provides an effective and powerful approach to identifying functional brain networks at multiple time scales from task fMRI data.

Functional Neuroimaging in the New Era of Big Data

The field of functional neuroimaging has substantially advanced as a big data science in the past decade, thanks to international collaborative projects and community efforts. Here we conducted a literature review on functional neuroimaging, with focus on three general challenges in big data tasks: data collection and sharing, data infrastructure construction, and data analysis methods. The review covers a wide range of literature types including perspectives, database descriptions, methodology developments, and technical details. We show how each of the challenges was proposed and addressed, and how these solutions formed the three core foundations for the functional neuroimaging as a big data science and helped to build the current data-rich and data-driven community. Furthermore, based on our review of recent literature on the upcoming challenges and opportunities toward future scientific discoveries, we envisioned that the functional neuroimaging community needs to advance from the current foundations to better data integration infrastructure, methodology development toward improved learning capability, and multi-discipline translational research framework for this new era of big data.

Automatic recognition of holistic functional brain networks using iteratively optimized convolutional neural networks (IO-CNN) with weak label initialization

fMRI data decomposition techniques have advanced significantly from shallow models such as Independent Component Analysis (ICA) and Sparse Coding and Dictionary Learning (SCDL) to deep learning models such Deep Belief Networks (DBN) and Convolutional Autoencoder (DCAE). However, interpretations of those decomposed networks are still open questions due to the lack of functional brain atlases, no correspondence across decomposed or reconstructed networks across different subjects, and significant individual variabilities. Recent studies showed that deep learning, especially deep convolutional neural networks (CNN), has extraordinary ability of accommodating spatial object patterns, e.g., our recent works using 3D CNN for fMRI-derived network classifications achieved high accuracy with a remarkable tolerance for mistakenly labelled training brain networks. However, the training data preparation is one of the biggest obstacles in these supervised deep learning models for functional brain network map recognitions, since manual labelling requires tedious and time-consuming labours which will sometimes even introduce label mistakes. Especially for mapping functional networks in large scale datasets such as hundreds of thousands of brain networks used in this paper, the manual labelling method will become almost infeasible. In response, in this work, we tackled both the network recognition and training data labelling tasks by proposing a new iteratively optimized deep learning CNN (IO-CNN) framework with an automatic weak label initialization, which enables the functional brain networks recognition task to a fully automatic large-scale classification procedure. Our extensive experiments based on ABIDE-II 1099 brains' fMRI data showed the great promise of our IO-CNN framework.

Discovering hierarchical common brain networks via multimodal deep belief network

Studying a common architecture reflecting both brain's structural and functional organizations across individuals and populations in a hierarchical way has been of significant interest in the brain mapping field. Recently, deep learning models exhibited ability in extracting meaningful hierarchical structures from brain imaging data, e.g., fMRI and DTI. However, deep learning models have been rarely used to explore the relation between brain structure and function yet. In this paper, we proposed a novel multimodal deep believe network (DBN) model to discover and quantitatively represent the hierarchical organizations of common and consistent brain networks from both fMRI and DTI data. A prominent characteristic of DBN is that it is capable of extracting meaningful features from complex neuroimaging data with a hierarchical manner. With our proposed DBN model, three hierarchical layers with hundreds of common and consistent brain networks across individual brains are successfully constructed through learning a large dimension of representative features from fMRI/DTI data.

Multi-Site Diagnostic Classification of Schizophrenia Using Discriminant Deep Learning with Functional Connectivity MRI

Background

A lack of a sufficiently large sample at single sites causes poor generalizability in automatic diagnosis classification of heterogeneous psychiatric disorders such as schizophrenia based on brain imaging scans. Advanced deep learning methods may be capable of learning subtle hidden patterns from high dimensional imaging data, overcome potential site-related variation, and achieve reproducible cross-site classification. However, deep learning-based cross-site transfer classification, despite less imaging site-specificity and more generalizability of diagnostic models, has not been investigated in schizophrenia.

Methods

A large multi-site functional MRI sample (n = 734, including 357 schizophrenic patients from seven imaging resources) was collected, and a deep discriminant autoencoder network, aimed at learning imaging site-shared functional connectivity features, was developed to discriminate schizophrenic individuals from healthy controls.

Findings

Accuracies of approximately 85·0% and 81·0% were obtained in multi-site pooling classification and leave-site-out transfer classification, respectively. The learned functional connectivity features revealed dysregulation of the cortical-striatal-cerebellar circuit in schizophrenia, and the most discriminating functional connections were primarily located within and across the default, salience, and control networks.

Interpretation

The findings imply that dysfunctional integration of the cortical-striatal-cerebellar circuit across the default, salience, and control networks may play an important role in the “disconnectivity” model underlying the pathophysiology of schizophrenia. The proposed discriminant deep learning method may be capable of learning reliable connectome patterns and help in understanding the pathophysiology and achieving accurate prediction of schizophrenia across multiple independent imaging sites.

•

A deep discriminant autoencoder network is developed for cross-site classification of schizophrenia.

•

The deep learning method can learn site-shared brain connectivity features and achieve accurate prediction of schizophrenia.

•

The learned features reveal dysfunctional integration of the cortical-striatal-cerebellar circuit in schizophrenia.