Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks

Task sub-type states decoding via group deep bidirectional recurrent neural network

Decoding brain states under different cognitive tasks from functional magnetic resonance imaging (fMRI) data has attracted great attention in the neuroimaging filed. However, the well-known temporal dependency in fMRI sequences has not been fully exploited in existing studies, due to the limited temporal-modeling capacity of the backbone machine learning algorithms and rigid training sample organization strategies upon which the brain decoding methods are built. To address these limitations, we propose a novel method for fine-grain brain state decoding, namely, group deep bidirectional recurrent neural network (Group-DBRNN) model. We first propose a training sample organization strategy that consists of a group-task sample generation module and a multiple-scale random fragment strategy (MRFS) module to collect training samples that contain rich task-relevant brain activity contrast (i.e., the comparison of neural activity patterns between different tasks) and maintain the temporal dependency. We then develop a novel decoding model by replacing the unidirectional RNNs that are widely used in existing brain state decoding studies with bidirectional stacked RNNs to better capture the temporal dependency, and by introducing a multi-task interaction layer (MTIL) module to effectively model the task-relevant brain activity contrast. Our experimental results on the Human Connectome Project task fMRI dataset (7 tasks consisting of 23 task sub-type states) show that the proposed model achieves an average decoding accuracy of 94.7% over the 23 fine-grain sub-type states. Meanwhile, our extensive interpretations of the intermediate features learned in the proposed model via visualizations and quantitative assessments of their discriminability and inter-subject alignment evidence that the proposed model can effectively capture the temporal dependency and task-relevant contrast.

Discovering individual fingerprints in resting-state functional connectivity using deep neural networks

Non-negligible idiosyncrasy due to interindividual differences is an ongoing issue in resting-state functional MRI (rfMRI) analysis. We show that a deep neural network (DNN) can be employed for individual identification by learning important features from the time-varying functional connectivity (FC) of rfMRI in the Human Connectome Project. We employed the trained DNN to identify individuals from an independent dataset acquired at our institution. The results revealed that the DNN could successfully identify 300 individuals with an error rate of 2.9% using 15 s time-window and 870 individuals with an error rate of 6.7%. A trained DNN with nonlinear hidden layers led to the proposal of the "fingerprint of FC" (fpFC) as representative edges of individual FC. The fpFCs for individuals exhibited commonly important and individual-specific edges across time-window lengths (from 5 min to 15 s). Furthermore, the utility of our model for another group of subjects was validated, supporting the feasibility of our technique in the context of transfer learning. In conclusion, our study offers an insight into the discovery of the intrinsic mode of the human brain using whole-brain resting-state FC and DNNs.

A hybrid spatiotemporal deep belief network and sparse representation-based framework reveals multilevel core functional components in decoding multitask fMRI signals

Decoding human brain activity on various task-based functional brain imaging data is of great significance for uncovering the functioning mechanism of the human mind. Currently, most feature extraction model-based methods for brain state decoding are shallow machine learning models, which may struggle to capture complex and precise spatiotemporal patterns of brain activity from the highly noisy fMRI raw data. Moreover, although decoding models based on deep learning methods benefit from their multilayer structure that could extract spatiotemporal features at multiscale, the relatively large populations of fMRI datasets are indispensable, and the explainability of their results is elusive. To address the above problems, we proposed a computational framework based on hybrid spatiotemporal deep belief network and sparse representations to differentiate multitask fMRI (tfMRI) signals. Using a relatively small cohort of tfMRI data as a test bed, our framework can achieve an average classification accuracy of 97.86% and define the multilevel temporal and spatial patterns of multiple cognitive tasks. Intriguingly, our model can characterize the key components for differentiating the multitask fMRI signals. Overall, the proposed framework can identify the interpretable and discriminative fMRI composition patterns at multiple scales, offering an effective methodology for basic neuroscience and clinical research with relatively small cohorts.

Deep learning with explainability for characterizing age-related intrinsic differences in dynamic brain functional connectivity

Although many deep learning models-based medical applications are performance-driven, i.e., accuracy-oriented, their explainability is more critical. This is especially the case with neuroimaging, where we are often interested in identifying biomarkers underlying brain development or disorders. Herein we propose an explainable deep learning approach by elucidating the information transmission mechanism between two layers of a deep network with a joint feature selection strategy that considers several shallow-layer explainable machine learning models and sparse learning of the deep network. At the end, we apply and validate the proposed approach to the analysis of dynamic brain functional connectivity (FC) from fMRI in a brain development study. Our approach can identify the differences within and between functional brain networks over age during development. The results indicate that the brain network transits from undifferentiated structures to more specialized and organized ones, and the information processing ability becomes more efficient as age increases. In addition, we detect two developmental patterns in the brain network: the FCs in regions related to visual and sound processing and mental regulation become weakened, while those between regions corresponding to emotional processing and cognitive activities are enhanced.

Autoencoder and restricted Boltzmann machine for transfer learning in functional magnetic resonance imaging task classification

Deep neural networks (DNNs) have been adopted widely as classifiers for functional magnetic resonance imaging (fMRI) data, advancing beyond traditional machine learning models. Consequently, transfer learning of the pre-trained DNN becomes crucial to enhance DNN classification performance, specifically by alleviating an overfitting issue that occurs when a substantial number of DNN parameters are fitted to a relatively small number of fMRI samples. In this study, we first systematically compared the two most popularly used, unsupervised pretraining models for resting-state fMRI (rfMRI) volume data to pre-train the DNNs, namely autoencoder (AE) and restricted Boltzmann machine (RBM). The group in-brain mask used when training AE and RBM displayed a sizable overlap ratio with Yeo's seven functional brain networks (FNs). The parcellated FNs obtained from the RBM were fine-grained compared to those from the AE. The pre-trained AE and RBM served as the weight parameters of the first of the two hidden DNN layers, and the DNN fulfilled the task classifier role for fMRI (tfMRI) data in the Human Connectome Project (HCP). We tested two transfer learning schemes: (1) fixing and (2) fine-tuning the DNN's pre-trained AE or RBM weights. The DNN with transfer learning was compared to a baseline DNN, trained using random initial weights. Overall, DNN classification performance from the transfer learning proved superior when the pre-trained RBM weights were fixed and when the pre-trained AE weights were fine-tuned (average error rates: 14.8% for fixed RBM, 15.1% fine-tuned AE, and 15.5% for the baseline model) compared to the alternative scenarios of DNN transfer learning schemes. Moreover, the optimal transfer learning scheme between the fixed RBM and fine-tuned AE varied according to seven task conditions in the HCP. Nonetheless, the computational load reduced substantially for the fixed-weight-based transfer learning compared to the fine-tuning-based transfer learning (e.g., the number of weight parameters for the fixed-weight-based DNN model reduced to 1.9% compared with a baseline/fine-tuned DNN model). Our findings suggest that weight initialization at the DNN's first layer using RBM-based pre-trained weights provides the most promising approach when the whole-brain fMRI volume supports associated task classification. We believe that our proposed scheme could be applied to a variety of task conditions to improve their classification performance and to utilize computational resources efficiently using our AE/RBM-based pre-trained weights compared to random initial weights for DNN training.

An Invasive Ductal Carcinomas Breast Cancer Grade Classification Using an Ensemble of Convolutional Neural Networks

Invasive Ductal Carcinoma Breast Cancer (IDC-BC) is the most common type of cancer and its asymptomatic nature has led to an increased mortality rate globally. Advancements in artificial intelligence and machine learning have revolutionized the medical field with the development of AI-enabled computer-aided diagnosis (CAD) systems, which help in determining diseases at an early stage. CAD systems assist pathologists in their decision-making process to produce more reliable outcomes in order to treat patients well. In this work, the potential of pre-trained convolutional neural networks (CNNs) (i.e., EfficientNetV2L, ResNet152V2, DenseNet201), singly or as an ensemble, was thoroughly explored. The performances of these models were evaluated for IDC-BC grade classification using the DataBiox dataset. Data augmentation was used to avoid the issues of data scarcity and data imbalances. The performance of the best model was compared to three different balanced datasets of Databiox (i.e., 1200, 1400, and 1600 images) to determine the implications of this data augmentation. Furthermore, the effects of the number of epochs were analysed to ensure the coherency of the most optimal model. The experimental results analysis revealed that the proposed ensemble model outperformed the existing state-of-the-art techniques in relation to classifying the IDC-BC grades of the Databiox dataset. The proposed ensemble model of the CNNs achieved a 94% classification accuracy and attained a significant area under the ROC curves for grades 1, 2, and 3, i.e., 96%, 94%, and 96%, respectively.

Computing personalized brain functional networks from fMRI using self-supervised deep learning

A novel self-supervised deep learning (DL) method is developed to compute personalized brain functional networks (FNs) for characterizing brain functional neuroanatomy based on functional MRI (fMRI). Specifically, a DL model of convolutional neural networks with an encoder-decoder architecture is developed to compute personalized FNs directly from fMRI data. The DL model is trained to optimize functional homogeneity of personalized FNs without utilizing any external supervision in an end-to-end fashion. We demonstrate that a DL model trained on fMRI scans from the Human Connectome Project can identify personalized FNs and generalizes well across four different datasets. We further demonstrate that the identified personalized FNs are informative for predicting individual differences in behavior, brain development, and schizophrenia status. Taken together, the self-supervised DL allows for rapid, generalizable computation of personalized FNs.

Sparse non-convex regularization based explainable DBN in the analysis of brain abnormalities in schizophrenia

Deep belief networks have been widely used in medical image analysis. However, the high-dimensional but small-sample-size characteristic of medical image data makes the model prone to dimensional disaster and overfitting. Meanwhile, the traditional DBN is driven by performance and ignores the explainability which is important for medical image analysis. In this paper, a sparse non-convex based explainable deep belief network is proposed by combining DBN with non-convex sparsity learning. For sparsity, the non-convex regularization and Kullback-Leibler divergence penalty are embedded into DBN to obtain the sparse connection and sparse response representation of the network. It effectively reduces the complexity of the model and improves the generalization ability of the model. Considering explainability, the crucial features for decision-making are selected through the feature back-selection based on the row norm of each layer's weight after network training. We apply the model to schizophrenia data and demonstrate it achieves the best performance among several typical feature selection models. It reveals 28 functional connections highly correlated with schizophrenia, which provides an effective foundation for the treatment and prevention of schizophrenia and methodological assurance for similar brain disorders.

An explainable autoencoder with multi-paradigm fMRI fusion for identifying differences in dynamic functional connectivity during brain development

Multi-paradigm deep learning models show great potential for dynamic functional connectivity (dFC) analysis by integrating complementary information. However, many of them cannot use information from different paradigms effectively and have poor explainability, that is, the ability to identify significant features that contribute to decision making. In this paper, we propose a multi-paradigm fusion-based explainable deep sparse autoencoder (MF-EDSAE) to address these issues. Considering explainability, the MF-EDSAE is constructed based on a deep sparse autoencoder (DSAE). For integrating information effectively, the MF-EDASE contains the nonlinear fusion layer and multi-paradigm hypergraph regularization. We apply the model to the Philadelphia Neurodevelopmental Cohort and demonstrate it achieves better performance in detecting dynamic FC (dFC) that differ significantly during brain development than the single-paradigm DSAE. The experimental results show that children have more dispersive dFC patterns than adults. The function of the brain transits from undifferentiated systems to specialized networks during brain development. Meanwhile, adults have stronger connectivities between task-related functional networks for a given task than children. As the brain develops, the patterns of the global dFC change more quickly when stimulated by a task.

Deep learning in neuroimaging data analysis: Applications, challenges, and solutions

Methods for the analysis of neuroimaging data have advanced significantly since the beginning of neuroscience as a scientific discipline. Today, sophisticated statistical procedures allow us to examine complex multivariate patterns, however most of them are still constrained by assuming inherent linearity of neural processes. Here, we discuss a group of machine learning methods, called deep learning, which have drawn much attention in and outside the field of neuroscience in recent years and hold the potential to surpass the mentioned limitations. Firstly, we describe and explain the essential concepts in deep learning: the structure and the computational operations that allow deep models to learn. After that, we move to the most common applications of deep learning in neuroimaging data analysis: prediction of outcome, interpretation of internal representations, generation of synthetic data and segmentation. In the next section we present issues that deep learning poses, which concerns multidimensionality and multimodality of data, overfitting and computational cost, and propose possible solutions. Lastly, we discuss the current reach of DL usage in all the common applications in neuroimaging data analysis, where we consider the promise of multimodality, capability of processing raw data, and advanced visualization strategies. We identify research gaps, such as focusing on a limited number of criterion variables and the lack of a well-defined strategy for choosing architecture and hyperparameters. Furthermore, we talk about the possibility of conducting research with constructs that have been ignored so far or/and moving toward frameworks, such as RDoC, the potential of transfer learning and generation of synthetic data.

Decoding Task-Based fMRI Data with Graph Neural Networks, Considering Individual Differences

Task fMRI provides an opportunity to analyze the working mechanisms of the human brain during specific experimental paradigms. Deep learning models have increasingly been applied for decoding and encoding purposes study to representations in task fMRI data. More recently, graph neural networks, or neural networks models designed to leverage the properties of graph representations, have recently shown promise in task fMRI decoding studies. Here, we propose an end-to-end graph convolutional network (GCN) framework with three convolutional layers to classify task fMRI data from the Human Connectome Project dataset. We compared the predictive performance of our GCN model across four of the most widely used node embedding algorithms-NetMF, RandNE, Node2Vec, and Walklets-to automatically extract the structural properties of the nodes in the functional graph. The empirical results indicated that our GCN framework accurately predicted individual differences (0.978 and 0.976) with the NetMF and RandNE embedding methods, respectively. Furthermore, to assess the effects of individual differences, we tested the classification performance of the model on sub-datasets divided according to gender and fluid intelligence. Experimental results indicated significant differences in the classification predictions of gender, but not high/low fluid intelligence fMRI data. Our experiments yielded promising results and demonstrated the superior ability of our GCN in modeling task fMRI data.

Learning brain representation using recurrent Wasserstein generative adversarial net

BACKGROUND AND OBJECTIVE

To understand brain cognition and disorders, modeling the mapping between mind and brain has been of great interest to the neuroscience community. The key is the brain representation, including functional brain networks (FBN) and their corresponding temporal features. Recently, it has been proven that deep learning models have superb representation power on functional magnetic resonance imaging (fMRI) over traditional machine learning methods. However, due to the lack of high-quality data and labels, deep learning models tend to suffer from overfitting in the training process.

METHODS

In this work, we applied a recurrent Wasserstein generative adversarial net (RWGAN) to learn brain representation from volumetric fMRI data. Generative adversarial net (GAN) is widely used in natural image generation and is able to capture the distribution of the input data, which enables the extraction of generalized features from fMRI and thus relieves the overfitting issue. The recurrent layers in RWGAN are designed to better model the local temporal features of the fMRI time series. The discriminator of RWGAN works as a deep feature extractor. With LASSO regression, the RWGAN model can decompose the fMRI data into temporal features and spatial features (FBNs). Furthermore, the generator of RWGAN can generate high-quality new data for fMRI augmentation.

RESULTS

The experimental results on seven tasks from the HCP dataset showed that the RWGAN can learn meaningful and interpretable temporal features and FBNs, compared to HCP task designs and general linear model (GLM) derived networks. Besides, the results on different training datasets showed that the RWGAN performed better on small datasets than other deep learning models. Moreover, we used the generator of RWGAN to yield fake subjects. The result showed that the fake data can also be used to learn meaningful representation compared to those learned from real data.

CONCLUSIONS

To our best knowledge, this work is among the earliest attempts of applying generative deep learning for modeling fMRI data. The proposed RWGAN offers a novel methodology for learning brain representation from fMRI, and it can generate high-quality fake data for the potential use of fMRI data augmentation.

Wavelet-Based Fractal Analysis of rs-fMRI for Classification of Alzheimer's Disease

The resting-state functional magnetic resonance imaging (rs-fMRI) modality has gained widespread acceptance as a promising method for analyzing a variety of neurological and psychiatric diseases. It is established that resting-state neuroimaging data exhibit fractal behavior, manifested in the form of slow-decaying auto-correlation and power-law scaling of the power spectrum across low-frequency components. With this property, the rs-fMRI signal can be broken down into fractal and nonfractal components. The fractal nature originates from several sources, such as cardiac fluctuations, respiration and system noise, and carries no information on the brain’s neuronal activities. As a result, the conventional correlation of rs-fMRI signals may not accurately reflect the functional dynamic of spontaneous neuronal activities. This problem can be solved by using a better representation of neuronal activities provided by the connectivity of nonfractal components. In this work, the nonfractal connectivity of rs-fMRI is used to distinguish Alzheimer’s patients from healthy controls. The automated anatomical labeling (AAL) atlas is used to extract the blood-oxygenation-level-dependent time series signals from 116 brain regions, yielding a 116 × 116 nonfractal connectivity matrix. From this matrix, significant connections evaluated using the p-value are selected as an input to a classifier for the classification of Alzheimer’s vs. normal controls. The nonfractal-based approach provides a good representation of the brain’s neuronal activity. It outperformed the fractal and Pearson-based connectivity approaches by 16.4% and 17.2%, respectively. The classification algorithm developed based on the nonfractal connectivity feature and support vector machine classifier has shown an excellent performance, with an accuracy of 90.3% and 83.3% for the XHSLF dataset and ADNI dataset, respectively. For further validation of our proposed work, we combined the two datasets (XHSLF+ADNI) and still received an accuracy of 90.2%. The proposed work outperformed the recently published work by a margin of 8.18% and 11.2%, respectively.

Decoding task specific and task general functional architectures of the brain

Functional magnetic resonance imaging (fMRI) is used to capture complex and dynamic interactions between brain regions while performing tasks. Task related alterations in the brain have been classified as task specific and task general, depending on whether they are particular to a task or common across multiple tasks. Using recent attempts in interpreting deep learning models, we propose an approach to determine both task specific and task general architectures of the functional brain. We demonstrate our methods with a reference‐based decoder on deep learning classifiers trained on 12,500 rest and task fMRI samples from the Human Connectome Project (HCP). The decoded task general and task specific motor and language architectures were validated with findings from previous studies. We found that unlike intersubject variability that is characteristic of functional pathology of neurological diseases, a small set of connections are sufficient to delineate the rest and task states. The nodes and connections in the task general architecture could serve as potential disease biomarkers as alterations in task general brain modulations are known to be implicated in several neuropsychiatric disorders.

Robust Brain State Decoding using Bidirectional Long Short Term Memory Networks in functional MRI

Decoding brain states of the underlying cognitive processes via learning discriminative feature representations has recently gained a lot of interest in brain imaging studies. Particularly, there has been an impetus to encode the dynamics of brain functioning by analyzing temporal information available in the fMRI data. Long-short term memory (LSTM), a class of machine learning model possessing a "memory" component, to retain previously seen temporal information, is increasingly being observed to perform well in various applications with dynamic temporal behavior, including brain state decoding. Because of the dynamics and inherent latency in fMRI BOLD responses, future temporal context is crucial. However, it is neither encoded nor captured by the conventional LSTM model. This paper performs robust brain state decoding via information encapsulation from both the past and future instances of fMRI data via bi-directional LSTM. This allows for explicitly modeling the dynamics of BOLD response without any delay adjustment. To this end, we utilize a bidirectional LSTM, wherein, the input sequence is fed in normal time-order for one LSTM network, and in the reverse time-order, for another. The two hidden activations of forward and reverse directions in bi-LSTM are collated to build the "memory" of the model and are used to robustly predict the brain states at every time instance. Working memory data from the Human Connectome Project (HCP) is utilized for validation and was observed to perform 18% better than it's unidirectional counterpart in terms of accuracy in predicting the brain states.

Natural and Artificial Intelligence: A brief introduction to the interplay between AI and neuroscience research

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.

Design of Deep Learning Model for Task-Evoked fMRI Data Classification

Machine learning methods have been successfully applied to neuroimaging signals, one of which is to decode specific task states from functional magnetic resonance imaging (fMRI) data. In this paper, we propose a model that simultaneously utilizes characteristics of both spatial and temporal sequential information of fMRI data with deep neural networks to classify the fMRI task states. We designed a convolution network module and a recurrent network module to extract the spatial and temporal features of fMRI data, respectively. In particular, we also add the attention mechanism to the recurrent network module, which more effectively highlights the brain activation state at the moment of reaction. We evaluated the model using task-evoked fMRI data from the Human Connectome Project (HCP) dataset, the classification accuracy got 94.31%, and the experimental results have shown that the model can effectively distinguish the brain states under different task stimuli.

Deep learning methods for predicting brain abnormalities and compute human cognitive power using fMRI

In modern times, digital medical images play a significant progression in clinical diagnosis to treat the populace earlier to hoard their lives. Magnetic Resonance Imaging (MRI) is one of the most advanced medical imaging modalities that facilitate scanning various parts of the human body like the head, chest, abdomen, and pelvis and identify the diseases. Numerous studies on the same discipline have proposed different algorithms, techniques, and methods for analyzing medical digital images, especially MRI. Most of them have mainly focused on identifying and classifying the images as either normal or abnormal. Computing brainpower is essential to understand and handle various brain diseases efficiently in critical situations. This paper knuckles down to design and implement a computer-aided framework, enhancing the identification of humans’ cognitive power from their MRI. Images. The proposed framework converts the 3D DICOM images into 2D medical images, preprocessing, enhancement, learning, and extracting various image information to classify it as normal or abnormal and provide the brain’s cognitive power. This study widens the efficient use of machine learning methods, Voxel Residual Network (VRN), with multimodality fusion architecture to learn and analyze the image to classify and predict cognitive power. The experimental results denote that the proposed framework demonstrates better performance than the existing approaches.

Modeling and augmenting of fMRI data using deep recurrent variational auto-encoder

Objective. Recently, deep learning models have been successfully applied in functional magnetic resonance imaging (fMRI) modeling and associated applications. However, there still exist at least two challenges. Firstly, due to the lack of sufficient data, deep learning models tend to suffer from overfitting in the training process. Secondly, it is still challenging to model the temporal dynamics from fMRI, due to that the brain state is continuously changing over scan time. In addition, existing methods rarely studied and applied fMRI data augmentation.Approach. In this work, we construct a deep recurrent variational auto-encoder (DRVAE) that combined variational auto-encoder and recurrent neural network, aiming to address all of the above mentioned challenges. The encoder of DRVAE can extract more generalized temporal features from assumed Gaussian distribution of input data, and the decoder of DRVAE can generate new data to increase training samples and thus partially relieve the overfitting issue. The recurrent layers in DRVAE are designed to effectively model the temporal dynamics of functional brain activities. LASSO (least absolute shrinkage and selection operator) regression is applied on the temporal features and input fMRI data to estimate the corresponding spatial networks.Main results. Extensive experimental results on seven tasks from HCP dataset showed that the DRVAE and LASSO framework can learn meaningful temporal patterns and spatial networks from both real data and generated data. The results on group-wise data and single subject suggest that the brain activities may follow certain distribution. Moreover, we applied DRVAE on four resting state fMRI datasets from ADHD-200 for data augmentation, and the results showed that the classification performances on augmented datasets have been considerably improved.Significance. The proposed method can not only derive meaningful temporal features and spatial networks from fMRI, but also generate high-quality new data for fMRI data augmentation and associated applications.

Cigarette craving modulation is more feasible than resistance modulation for heavy cigarette smokers: empirical evidence from functional MRI data

BACKGROUND

Modulation of cigarette craving and neuronal activations from nicotine-dependent cigarette smokers using real-time functional MRI (rtfMRI)-based neurofeedback (rtfMRI-NF) has been previously reported.

OBJECTIVES

The aim of this study was to evaluate the efficacy of rtfMRI-NF training in reducing cigarette cravings using fMRI data acquired before and after training.

METHODS

Treatment-seeking male heavy cigarette smokers (N = 14) were enrolled and randomly assigned to two conditions related to rtfMRI-NF training aiming at resisting the urge to smoke. In one condition, subjects underwent conventional rtfMRI-NF training using neuronal activity as the neurofeedback signal (activity-based) within regions-of-interest (ROIs) implicated in cigarette craving. In another condition, subjects underwent rtfMRI-NF training with additional functional connectivity information included in the neurofeedback signal (functional connectivity-added). Before and after rtfMRI-NF training at each of two visits, participants underwent two fMRI runs with cigarette smoking stimuli and were asked to crave or resist the urge to smoke without neurofeedback. Cigarette craving-related or resistance-related regions were identified using a general linear model followed by paired t-tests and were evaluated using regression analysis on the basis of neuronal activation and subjective craving scores (CRSs).

RESULTS

Visual areas were mainly implicated in craving, whereas the superior frontal areas were associated with resistance. The degree of (a) CRS reduction and (b) the correlation between neuronal activation and CRSs were statistically significant (P < 0.05) in the functional connectivity-added neurofeedback group for craving-related ROIs.

CONCLUSION

Our study demonstrated the feasibility of altering cigarette craving in craving-related ROIs but not in resistance-related ROIs via rtfMRI-NF training.

Classifying Cognitive Normal and Early Mild Cognitive Impairment of Alzheimer’s Disease by Applying Restricted Boltzmann Machine to fMRI Data

Background:

Neuroimaging is an important tool in early detection of Alzheimer’s disease (AD), which is a serious neurodegenerative brain disease among the elderly subjects. Independent component analysis (ICA) is arguably one of the most widely used algorithm for the analysis of brain imaging data, which can be used to extract intrinsic networks of brain from functional magnetic resonance imaging (fMRI).

Method:

Witnessed by recent studies, a more flexible model known as restricted Boltzmann machine (RBM) can also be used to extract spatial maps and time courses of intrinsic networks from resting state fMRI, moreover, RBM shows superior temporal features than ICA. Here, we seek to employ RBM to improve the performance of classifying individuals. Experiments are performed on healthy controls and subjects at the early stage of AD, i.e., cognitive normal (CN) and early mild cognitive impairment participants (EMCI), and two types of data, i.e., structural magnetic resonance imaging (sMRI) and fMRI data.

Results:

(1) By separately employing ICA for sMRI and fMRI, the features extracted from fMRI improve classification accuracy by 7.5% for CN and EMCI; (2) instead of applying ICA to fMRI, using RBM further improves classification accuracy by 7.75% for CN and EMCI; (3) the lesions at the early stage of AD are more likely to occur in the regions around slices 4, 6, 10, 14, 19, 51 and 59 of the whole brain in the longitudinal direction.

Conclusion:

By using fMRI instead of sMRI and RBM instead of ICA, we can classify CN and EMCI more efficiently.

Diagnosis of Alzheimer's Disease Severity with fMRI Images Using Robust Multitask Feature Extraction Method and Convolutional Neural Network (CNN)

The automatic diagnosis of Alzheimer's disease plays an important role in human health, especially in its early stage. Because it is a neurodegenerative condition, Alzheimer's disease seems to have a long incubation period. Therefore, it is essential to analyze Alzheimer's symptoms at different stages. In this paper, the classification is done with several methods of machine learning consisting of K-nearest neighbor (KNN), support vector machine (SVM), decision tree (DT), linear discrimination analysis (LDA), and random forest (RF). Moreover, novel convolutional neural network (CNN) architecture is presented to diagnose Alzheimer's severity. The relationship between Alzheimer's patients' functional magnetic resonance imaging (fMRI) images and their scores on the MMSE is investigated to achieve the aim. The feature extraction is performed based on the robust multitask feature learning algorithm. The severity is also calculated based on the Mini-Mental State Examination score, including low, mild, moderate, and severe categories. Results show that the accuracy of the KNN, SVM, DT, LDA, RF, and presented CNN method is 77.5%, 85.8%, 91.7%, 79.5%, 85.1%, and 96.7%, respectively. Moreover, for the presented CNN architecture, the sensitivity of low, mild, moderate, and severe status of Alzheimer patients is 98.1%, 95.2%,89.0%, and 87.5%, respectively. Based on the findings, the presented CNN architecture classifier outperforms other methods and can diagnose the severity and stages of Alzheimer's disease with maximum accuracy.

Diagnosis of Alzheimer Disease Using 2D MRI Slices by Convolutional Neural Network

There are many kinds of brain abnormalities that cause changes in different parts of the brain. Alzheimer's disease is a chronic condition that degenerates the cells of the brain leading to memory asthenia. Cognitive mental troubles such as forgetfulness and confusion are one of the most important features of Alzheimer's patients. In the literature, several image processing techniques, as well as machine learning strategies, were introduced for the diagnosis of the disease. This study is aimed at recognizing the presence of Alzheimer's disease based on the magnetic resonance imaging of the brain. We adopted a deep learning methodology for the discrimination between Alzheimer's patients and healthy patients from 2D anatomical slices collected using magnetic resonance imaging. Most of the previous researches were based on the implementation of a 3D convolutional neural network, whereas we incorporated the usage of 2D slices as input to the convolutional neural network. The data set of this research was obtained from the OASIS website. We trained the convolutional neural network structure using the 2D slices to exhibit the deep network weightings that we named as the Alzheimer Network (AlzNet). The accuracy of our enhanced network was 99.30%. This work investigated the effects of many parameters on AlzNet, such as the number of layers, number of filters, and dropout rate. The results were interesting after using many performance metrics for evaluating the proposed AlzNet.

Deep Generative Analysis for Task-Based Functional MRI Experiments

While functional magnetic resonance imaging (fMRI) remains one of the most widespread and important methods in basic and clinical neuroscience, the data it produces-time series of brain volumes-continue to pose daunting analysis challenges. The current standard ("mass univariate") approach involves constructing a matrix of task regressors, fitting a separate general linear model at each volume pixel ("voxel"), computing test statistics for each model, and correcting for false positives post hoc using bootstrap or other resampling methods. Despite its simplicity, this approach has enjoyed great success over the last two decades due to: 1) its ability to produce effect maps highlighting brain regions whose activity significantly correlates with a given variable of interest; and 2) its modeling of experimental effects as separable and thus easily interpretable. However, this approach suffers from several well-known drawbacks, namely: inaccurate assumptions of linearity and noise Gaussianity; a limited ability to capture individual effects and variability; and difficulties in performing proper statistical testing secondary to independently fitting voxels. In this work, we adopt a different approach, modeling entire volumes directly in a manner that increases model flexibility while preserving interpretability. Specifically, we use a generalized additive model (GAM) in which the effects of each regressor remain separable, the product of a spatial map produced by a variational autoencoder and a (potentially nonlinear) gain modeled by a covariate-specific Gaussian Process. The result is a model that yields group-level effect maps comparable or superior to the ones obtained with standard fMRI analysis software while also producing single-subject effect maps capturing individual differences. This suggests that generative models with a decomposable structure might offer a more flexible alternative for the analysis of task-based fMRI data.

Deep learning approaches for neural decoding across architectures and recording modalities

Decoding behavior, perception or cognitive state directly from neural signals is critical for brain-computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

Convolution-GRU Based on Independent Component Analysis for fMRI Analysis with Small and Imbalanced Samples

Functional magnetic resonance imaging (fMRI) is a commonly used method of brain research. However, due to the complexity and particularity of the fMRI task, it is difficult to find enough subjects, resulting in a small and, often, imbalanced dataset. A dataset with small samples causes overfitting of the learning model, and the imbalance will make the model insensitive to the minority class, which has been a problem in classification. It is of great significance to classify fMRI data with small and imbalanced samples. In the present study, we propose a 3-step method on a small and imbalanced fMRI dataset from a word-scene memory task. The steps of the method are as follows: (1) An independent component analysis is performed to reduce the dimension of data; (2) The synthetic minority oversampling technique is used to generate new samples of the minority class to balance data; (3) A convolution-Gated Recurrent Unit (GRU) network is used to classify the independent component signals, indicating whether the subjects are performing episodic memory tasks. The accuracy of the proposed method is 72.2%, which improves the classification performance compared with traditional classifiers such as support vector machines (SVM), logistic regression (LGR), linear discriminant analysis (LDA) and k-nearest neighbor (KNN), and this study gives a biomarker for evaluating the reactivation of episodic memory.

Functional magnetic resonance imaging multivoxel pattern analysis reveals neuronal substrates for collaboration and competition with myopic and predictive strategic reasoning

Competition and collaboration are strategies that can be used to optimize the outcomes of social interactions. Research into the neuronal substrates underlying these aspects of social behavior has been limited due to the difficulty in distinguishing complex activation via univariate analysis. Therefore, we employed multivoxel pattern analysis of functional magnetic resonance imaging to reveal the neuronal activations underlying competitive and collaborative processes when the collaborator/opponent used myopic/predictive reasoning. Twenty‐four healthy subjects participated in 2 × 2 matrix‐based sequential‐move games. Searchlight‐based multivoxel patterns were used as input for a support vector machine using nested cross‐validation to distinguish game conditions, and identified voxels were validated via the regression of the behavioral data with bootstrapping. The left anterior insula (accuracy = 78.5%) was associated with competition, and middle frontal gyrus (75.1%) was associated with predictive reasoning. The inferior/superior parietal lobules (84.8%) and middle frontal gyrus (84.7%) were associated with competition, particularly in trials with a predictive opponent. The visual/motor areas were related to response time as a proxy for visual attention and task difficulty. Our results suggest that multivoxel patterns better represent the neuronal substrates underlying the social cognition of collaboration and competition intermixed with myopic and predictive reasoning than do univariate features.

A Two Consequent Multi-layers Deep Discriminative Approach for Classifying fMRI Images

Functional Magnetic Resonance Imaging (fMRI), for many decades acts as a potential aiding method for diagnosing medical problems. Several successful machine learning algorithms have been proposed in literature to extract valuable knowledge from fMRI. One of these algorithms is the convolutional neural network (CNN) that competent with high capabilities for learning optimal abstractions of fMRI. This is because the CNN learns features similarly to human brain where it preserves local structure and avoids distortion of the global feature space. Focusing on the achievements of using the CNN for the fMRI, and accordingly, the Deep Convolutional Auto-Encoder (DCAE) benefits from the data-driven approach with CNN’s optimal features to strengthen the fMRI classification. In this paper, a new two consequent multi-layers DCAE deep discriminative approach for classifying fMRI Images is proposed. The first DCAE is unsupervised sub-model that is composed of four CNN. It focuses on learning weights to utilize discriminative characteristics of the extracted features for robust reconstruction of fMRI with lower dimensional considering tiny details and refining by its deep multiple layers. Then the second DCAE is a supervised sub-model that focuses on training labels to reach an outperformed results. The proposed approach proved its effectiveness and improved literately reported results on a large brain disorder fMRI dataset.

fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Towards a brain-based predictome of mental illness

Neuroimaging-based approaches have been extensively applied to study mental illness in recent years and have deepened our understanding of both cognitively healthy and disordered brain structure and function. Recent advancements in machine learning techniques have shown promising outcomes for individualized prediction and characterization of patients with psychiatric disorders. Studies have utilized features from a variety of neuroimaging modalities, including structural, functional, and diffusion magnetic resonance imaging data, as well as jointly estimated features from multiple modalities, to assess patients with heterogeneous mental disorders, such as schizophrenia and autism. We use the term "predictome" to describe the use of multivariate brain network features from one or more neuroimaging modalities to predict mental illness. In the predictome, multiple brain network-based features (either from the same modality or multiple modalities) are incorporated into a predictive model to jointly estimate features that are unique to a disorder and predict subjects accordingly. To date, more than 650 studies have been published on subject-level prediction focusing on psychiatric disorders. We have surveyed about 250 studies including schizophrenia, major depression, bipolar disorder, autism spectrum disorder, attention-deficit hyperactivity disorder, obsessive-compulsive disorder, social anxiety disorder, posttraumatic stress disorder, and substance dependence. In this review, we present a comprehensive review of recent neuroimaging-based predictomic approaches, current trends, and common shortcomings and share our vision for future directions.

A heuristic information cluster search approach for precise functional brain mapping

Detection of the relevant brain regions for characterizing the distinction between cognitive conditions is one of the most sought after objectives in neuroimaging research. A popular approach for achieving this goal is the multivariate pattern analysis which is currently conducted through a number of approaches such as the popular searchlight procedure. This is due to several advantages such as being automatic and flexible with regards to size of the search region. However, these approaches suffer from a number of limitations which can lead to misidentification of truly informative regions which in turn results in imprecise information maps. These limitations mainly stem from several factors such as the fact that the information value of the search spheres are assigned to the voxel at the center of them (in case of searchlight), the requirement for manual tuning of parameters such as searchlight radius and shape, and high complexity and low interpretability in commonly used machine learning-based approaches. Other drawbacks include overlooking the structure and interactions within the regions, and the disadvantages of using certain regularization techniques in analysis of datasets with characteristics of common functional magnetic resonance imaging data. In this article, we propose a fully data-driven maximum relevance minimum redundancy search algorithm for detecting precise information value of the clusters within brain regions while alleviating the above-mentioned limitations. Moreover, in order to make the proposed method faster, we propose an efficient algorithmic implementation. We evaluate and compare the proposed algorithm with the searchlight procedure as well as least absolute shrinkage and selection operator regularization-based mapping approach using both real and synthetic datasets. The analysis results of the proposed approach demonstrate higher information detection precision and map specificity compared to the benchmark approaches.

Automated MRI-Based Deep Learning Model for Detection of Alzheimer’s Disease Process

In the context of neuro-pathological disorders, neuroimaging has been widely accepted as a clinical tool for diagnosing patients with Alzheimer’s disease (AD) and mild cognitive impairment (MCI). The advanced deep learning method, a novel brain imaging technique, was applied in this study to evaluate its contribution to improving the diagnostic accuracy of AD. Three-dimensional convolutional neural networks (3D-CNNs) were applied with magnetic resonance imaging (MRI) to execute binary and ternary disease classification models. The dataset from the Alzheimer’s disease neuroimaging initiative (ADNI) was used to compare the deep learning performances across 3D-CNN, 3D-CNN-support vector machine (SVM) and two-dimensional (2D)-CNN models. The outcomes of accuracy with ternary classification for 2D-CNN, 3D-CNN and 3D-CNN-SVM were [Formula: see text]%, [Formula: see text]% and [Formula: see text]% respectively. The 3D-CNN-SVM yielded a ternary classification accuracy of 93.71%, 96.82% and 96.73% for NC, MCI and AD diagnoses, respectively. Furthermore, 3D-CNN-SVM showed the best performance for binary classification. Our study indicated that ‘NC versus MCI’ showed accuracy, sensitivity and specificity of 98.90%, 98.90% and 98.80%; ‘NC versus AD’ showed accuracy, sensitivity and specificity of 99.10%, 99.80% and 98.40%; and ‘MCI versus AD’ showed accuracy, sensitivity and specificity of 89.40%, 86.70% and 84.00%, respectively. This study clearly demonstrates that 3D-CNN-SVM yields better performance with MRI compared to currently utilized deep learning methods. In addition, 3D-CNN-SVM proved to be efficient without having to manually perform any prior feature extraction and is totally independent of the variability of imaging protocols and scanners. This suggests that it can potentially be exploited by untrained operators and extended to virtual patient imaging data. Furthermore, owing to the safety, noninvasiveness and nonirradiative properties of the MRI modality, 3D-CNN-SMV may serve as an effective screening option for AD in the general population. This study holds value in distinguishing AD and MCI subjects from normal controls and to improve value-based care of patients in clinical practice.

Predicting reading ability from brain anatomy and function: From areas to connections

Reading is a complex task involving different brain areas. As a crystallized ability, reading is also known to have effects on brain structure and function development. However, there are still open questions about what are the elements of the reading networks and how structural and functional brain measures shape the reading ability. The present study used a data-driven approach to investigate whether reading-related brain structural measures of cortical thickness, myelination, sulcus depth and structural connectivity and functional connectivity from the whole brain can predict individual differences in reading skills. It used different brain measures and performance scores from the Oral Reading Recognition Test (ORRT) measuring reading ability from 998 participants. We revealed reading-related brain areas and connections, and evaluated how well area and connection measures predict reading performance. Interestingly, the combination of all brain measures obtained the best predictions. We further grouped reading-related areas into positive and negative networks, each with four different levels (Core Regions, Extended-Regions 1, 2, 3), representing different correlation levels with the reading scores, and the non-correlated Region irrelevant to reading ability. The Core Regions are composed of areas that are most strongly correlated with reading performance. Insular and frontal opercular cortex, lateral temporal cortex, and early auditory cortex occupy the positive Core Region, while inferior temporal and motor cortex occupy the negative Core Region. Aside from those areas, the present study also found more reading-related areas including visual and language-related areas. In addition, connections predicting reading scores are denser inside the reading-related networks than outside. Together, the present study reveals extended reading networks of the brain and provides an extended data-driven analytical framework to study interpretable brain-behavior relationships, which are transferable also to studying other abilities.

Decoding and mapping task states of the human brain via deep learning

Support vector machine (SVM)-based multivariate pattern analysis (MVPA) has delivered promising performance in decoding specific task states based on functional magnetic resonance imaging (fMRI) of the human brain. Conventionally, the SVM-MVPA requires careful feature selection/extraction according to expert knowledge. In this study, we propose a deep neural network (DNN) for directly decoding multiple brain task states from fMRI signals of the brain without any burden for feature handcrafts. We trained and tested the DNN classifier using task fMRI data from the Human Connectome Project's S1200 dataset (N = 1,034). In tests to verify its performance, the proposed classification method identified seven tasks with an average accuracy of 93.7%. We also showed the general applicability of the DNN for transfer learning to small datasets (N = 43), a situation encountered in typical neuroscience research. The proposed method achieved an average accuracy of 89.0 and 94.7% on a working memory task and a motor classification task, respectively, higher than the accuracy of 69.2 and 68.6% obtained by the SVM-MVPA. A network visualization analysis showed that the DNN automatically detected features from areas of the brain related to each task. Without incurring the burden of handcrafting the features, the proposed deep decoding method can classify brain task states highly accurately, and is a powerful tool for fMRI researchers.

Hierarchical Organization of Functional Brain Networks Revealed by Hybrid Spatiotemporal Deep Learning

Hierarchical organization of brain function has been an established concept in the neuroscience field for a long time, however, it has been rarely demonstrated how such hierarchical macroscale functional networks are actually organized in the human brain. In this study, to answer this question, we propose a novel methodology to provide an evidence of hierarchical organization of functional brain networks. This article introduces the hybrid spatiotemporal deep learning (HSDL), by jointly using deep belief networks (DBNs) and deep least absolute shrinkage and selection operator (LASSO) to reveal the temporal hierarchical features and spatial hierarchical maps of brain networks based on the Human Connectome Project 900 functional magnetic resonance imaging (fMRI) data sets. Briefly, the key idea of HSDL is to extract the weights between two adjacent layers of DBNs, which are then treated as the hierarchical dictionaries for deep LASSO to identify the corresponding hierarchical spatial maps. Our results demonstrate that both spatial and temporal aspects of dozens of functional networks exhibit multiscale properties that can be well characterized and interpreted based on existing computational tools and neuroscience knowledge. Our proposed novel hybrid deep model is used to provide the first insightful opportunity to reveal the potential hierarchical organization of time series and functional brain networks, using task-based fMRI signals of human brain.

LNTP-MDBN: Big Data Integrated Learning Framework for Heterogeneous Image Set Classification

BACKGROUND

With the explosive growth of global data, the term Big Data describes the enormous size of dataset through the detailed analysis. The big data analytics revealed the hidden patterns and secret correlations among the values. The major challenges in Big data analysis are due to increase of volume, variety, and velocity. The capturing of images with multi-directional views initiates the image set classification which is an attractive research study in the volumetricbased medical image processing.

METHODS

This paper proposes the Local N-ary Ternary Patterns (LNTP) and Modified Deep Belief Network (MDBN) to alleviate the dimensionality and robustness issues. Initially, the proposed LNTP-MDBN utilizes the filtering technique to identify and remove the dependent and independent noise from the images. Then, the application of smoothening and the normalization techniques on the filtered image improves the intensity of the images.

RESULTS

The LNTP-based feature extraction categorizes the heterogeneous images into different categories and extracts the features from each category. Based on the extracted features, the modified DBN classifies the normal and abnormal categories in the image set finally.

CONCLUSION

The comparative analysis of proposed LNTP-MDBN with the existing pattern extraction and DBN learning models regarding classification accuracy and runtime confirms the effectiveness in mining applications.

Functional Imaging of Visuospatial Attention in Complex and Naturalistic Conditions

One of the ultimate goals of cognitive neuroscience is to understand how the brain works in the real world. Functional imaging with naturalistic stimuli provides us with the opportunity to study the brain in situations similar to the everyday life. This includes the processing of complex stimuli that can trigger many types of signals related both to the physical characteristics of the external input and to the internal knowledge that we have about natural objects and environments. In this chapter, I will first outline different types of stimuli that have been used in naturalistic imaging studies. These include static pictures, short video clips, full-length movies, and virtual reality, each comprising specific advantages and disadvantages. Next, I will turn to the main issue of visual-spatial orienting in naturalistic conditions and its neural substrates. I will discuss different classes of internal signals, related to objects, scene structure, and long-term memory. All of these, together with external signals about stimulus salience, have been found to modulate the activity and the connectivity of the frontoparietal attention networks. I will conclude by pointing out some promising future directions for functional imaging with naturalistic stimuli. Despite this field of research is still in its early days, I consider that it will play a major role in bridging the gap between standard laboratory paradigms and mechanisms of brain functioning in the real world.

Analyzing Neuroimaging Data Through Recurrent Deep Learning Models

The application of deep learning (DL) models to neuroimaging data poses several challenges, due to the high dimensionality, low sample size, and complex temporo-spatial dependency structure of these data. Even further, DL models often act as black boxes, impeding insight into the association of cognitive state and brain activity. To approach these challenges, we introduce the DeepLight framework, which utilizes long short-term memory (LSTM) based DL models to analyze whole-brain functional Magnetic Resonance Imaging (fMRI) data. To decode a cognitive state (e.g., seeing the image of a house), DeepLight separates an fMRI volume into a sequence of axial brain slices, which is then sequentially processed by an LSTM. To maintain interpretability, DeepLight adapts the layer-wise relevance propagation (LRP) technique. Thereby, decomposing its decoding decision into the contributions of the single input voxels to this decision. Importantly, the decomposition is performed on the level of single fMRI volumes, enabling DeepLight to study the associations between cognitive state and brain activity on several levels of data granularity, from the level of the group down to the level of single time points. To demonstrate the versatility of DeepLight, we apply it to a large fMRI dataset of the Human Connectome Project. We show that DeepLight outperforms conventional approaches of uni- and multivariate fMRI analysis in decoding the cognitive states and in identifying the physiologically appropriate brain regions associated with these states. We further demonstrate DeepLight's ability to study the fine-grained temporo-spatial variability of brain activity over sequences of single fMRI samples.

Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks

Decoding brain functional states underlying cognitive processes from functional MRI (fMRI) data using multivariate pattern analysis (MVPA) techniques has achieved promising performance for characterizing brain activation patterns and providing neurofeedback signals. However, it remains challenging to decode subtly distinct brain states for individual fMRI data points due to varying temporal durations and dependency among different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in intrinsic functional network modeling and sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, subject-specific intrinsic functional networks (FNs) are computed from resting-state fMRI data and are used to characterize functional signals of task fMRI data with a compact representation for building brain decoding models, and LSTM RNNs are adopted to learn brain decoding mappings between functional profiles and brain states. Validation results on fMRI data from the HCP dataset have demonstrated that brain decoding models built on training data using the proposed method could learn discriminative latent feature representations and effectively distinguish subtly distinct working memory tasks of different subjects with significantly higher accuracy than conventional decoding models. Informative FNs of the brain decoding models identified as brain activation patterns of working memory tasks were largely consistent with the literature. The method also obtained promising decoding performance on motor and social cognition tasks. Our results suggest that LSTM RNNs in conjunction with FNs could build interpretable, highly accurate brain decoding models.

Deep neural networks in psychiatry

Machine and deep learning methods, today's core of artificial intelligence, have been applied with increasing success and impact in many commercial and research settings. They are powerful tools for large scale data analysis, prediction and classification, especially in very data-rich environments ("big data"), and have started to find their way into medical applications. Here we will first give an overview of machine learning methods, with a focus on deep and recurrent neural networks, their relation to statistics, and the core principles behind them. We will then discuss and review directions along which (deep) neural networks can be, or already have been, applied in the context of psychiatry, and will try to delineate their future potential in this area. We will also comment on an emerging area that so far has been much less well explored: by embedding semantically interpretable computational models of brain dynamics or behavior into a statistical machine learning context, insights into dysfunction beyond mere prediction and classification may be gained. Especially this marriage of computational models with statistical inference may offer insights into neural and behavioral mechanisms that could open completely novel avenues for psychiatric treatment.

Regression-based machine-learning approaches to predict task activation using resting-state fMRI

Resting‐state fMRI has shown the ability to predict task activation on an individual basis by using a general linear model (GLM) to map resting‐state network features to activation z‐scores. The question remains whether the relatively simplistic GLM is the best approach to accomplish this prediction. In this study, several regression‐based machine‐learning approaches were compared, including GLMs, feed‐forward neural networks, and random forest bootstrap aggregation (bagging). Resting‐state and task data from 350 Human Connectome Project subjects were analyzed. First, the effect of the number of training subjects on the prediction accuracy was evaluated. In addition, the prediction accuracy and Dice coefficient were compared across models. Prediction accuracy increased with the training number up to 200 subjects; however, an elbow in the prediction curve occurred around 30–40 training subjects. All models performed well with correlation matrices, which displayed correlation between actual and predicted task activation for all subjects, exhibiting a strong diagonal trend for all tasks. Overall, the neural network and random forest bagging techniques outperformed the GLM. These approaches, however, require additional computing power and processing time. These results show that, while the GLM performs well, resting‐state fMRI prediction of task activation could benefit from more complex machine learning approaches.

Deep Learning Intervention for Health Care Challenges: Some Biomedical Domain Considerations

The use of deep learning (DL) for the analysis and diagnosis of biomedical and health care problems has received unprecedented attention in the last decade. The technique has recorded a number of achievements for unearthing meaningful features and accomplishing tasks that were hitherto difficult to solve by other methods and human experts. Currently, biological and medical devices, treatment, and applications are capable of generating large volumes of data in the form of images, sounds, text, graphs, and signals creating the concept of big data. The innovation of DL is a developing trend in the wake of big data for data representation and analysis. DL is a type of machine learning algorithm that has deeper (or more) hidden layers of similar function cascaded into the network and has the capability to make meaning from medical big data. Current transformation drivers to achieve personalized health care delivery will be possible with the use of mobile health (mHealth). DL can provide the analysis for the deluge of data generated from mHealth apps. This paper reviews the fundamentals of DL methods and presents a general view of the trends in DL by capturing literature from PubMed and the Institute of Electrical and Electronics Engineers database publications that implement different variants of DL. We highlight the implementation of DL in health care, which we categorize into biological system, electronic health record, medical image, and physiological signals. In addition, we discuss some inherent challenges of DL affecting biomedical and health domain, as well as prospective research directions that focus on improving health management by promoting the application of physiological signals and modern internet technology.

Data-driven models in human neuroscience and neuroengineering

Discoveries in modern human neuroscience are increasingly driven by quantitative understanding of complex data. Data-intensive approaches to modeling have promise to dramatically advance our understanding of the brain and critically enable neuroengineering capabilities. In this review, we provide an accessible primer to modern modeling approaches and highlight recent data-driven discoveries in the domains of neuroimaging, single-neuron and neuronal population responses, and device neuroengineering. Further, we suggest that meaningful progress requires the community to tackle open challenges in the realms of model interpretability and generalizability, training pipelines of data-fluent human neuroscientists, and integrated consideration of data ethics.