1
|
Bougou V, Vanhoyland M, Bertrand A, Van Paesschen W, Op De Beeck H, Janssen P, Theys T. Neuronal tuning and population representations of shape and category in human visual cortex. Nat Commun 2024; 15:4608. [PMID: 38816391 PMCID: PMC11139926 DOI: 10.1038/s41467-024-49078-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 05/22/2024] [Indexed: 06/01/2024] Open
Abstract
Object recognition and categorization are essential cognitive processes which engage considerable neural resources in the human ventral visual stream. However, the tuning properties of human ventral stream neurons for object shape and category are virtually unknown. We performed large-scale recordings of spiking activity in human Lateral Occipital Complex in response to stimuli in which the shape dimension was dissociated from the category dimension. Consistent with studies in nonhuman primates, the neuronal representations were primarily shape-based, although we also observed category-like encoding for images of animals. Surprisingly, linear decoders could reliably classify stimulus category even in data sets that were entirely shape-based. In addition, many recording sites showed an interaction between shape and category tuning. These results represent a detailed study on shape and category coding at the neuronal level in the human ventral visual stream, furnishing essential evidence that reconciles human imaging and macaque single-cell studies.
Collapse
Affiliation(s)
- Vasiliki Bougou
- Research Group of Experimental Neurosurgery and Neuroanatomy, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium
- Laboratory for Neuro-and Psychophysiology, Research Group Neurophysiology, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium
| | - Michaël Vanhoyland
- Research Group of Experimental Neurosurgery and Neuroanatomy, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium
- Laboratory for Neuro-and Psychophysiology, Research Group Neurophysiology, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium
- Department of Neurosurgery, University Hospitals Leuven, Leuven, Belgium
| | | | - Wim Van Paesschen
- Department of Neurology, University Hospitals Leuven, Leuven, Belgium
- Laboratory for Epilepsy Research, KU Leuven, Leuven, Belgium
| | - Hans Op De Beeck
- Laboratory Biological Psychology, Department of Neurosciences, KU Leuven, Leuven, Belgium
| | - Peter Janssen
- Laboratory for Neuro-and Psychophysiology, Research Group Neurophysiology, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium.
| | - Tom Theys
- Research Group of Experimental Neurosurgery and Neuroanatomy, Department of Neurosciences, KU Leuven and the Leuven Brain Institute, Leuven, Belgium
- Department of Neurosurgery, University Hospitals Leuven, Leuven, Belgium
| |
Collapse
|
2
|
Dado T, Papale P, Lozano A, Le L, Wang F, van Gerven M, Roelfsema P, Güçlütürk Y, Güçlü U. Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain. PLoS Comput Biol 2024; 20:e1012058. [PMID: 38709818 PMCID: PMC11098503 DOI: 10.1371/journal.pcbi.1012058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 05/16/2024] [Accepted: 04/08/2024] [Indexed: 05/08/2024] Open
Abstract
A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.
Collapse
Affiliation(s)
- Thirza Dado
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Paolo Papale
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Antonio Lozano
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Lynn Le
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Feng Wang
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Marcel van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Pieter Roelfsema
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
- Laboratory of Visual Brain Therapy, Sorbonne University, Paris, France
- Department of Integrative Neurophysiology, VU Amsterdam, Amsterdam, Netherlands
- Department of Psychiatry, Amsterdam UMC, Amsterdam, Netherlands
| | - Yağmur Güçlütürk
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Umut Güçlü
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
3
|
Duman AN, Tatar AE. Topological data analysis for revealing dynamic brain reconfiguration in MEG data. PeerJ 2023; 11:e15721. [PMID: 37489123 PMCID: PMC10363343 DOI: 10.7717/peerj.15721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 06/16/2023] [Indexed: 07/26/2023] Open
Abstract
In recent years, the focus of the functional connectivity community has shifted from stationary approaches to the ones that include temporal dynamics. Especially, non-invasive electrophysiological data (magnetoencephalography/electroencephalography (MEG/EEG)) with high temporal resolution and good spatial coverage have made it possible to measure the fast alterations in the neural activity in the brain during ongoing cognition. In this article, we analyze dynamic brain reconfiguration using MEG images collected from subjects during the rest and the cognitive tasks. Our proposed topological data analysis method, called Mapper, produces biomarkers that differentiate cognitive tasks without prior spatial and temporal collapse of the data. The suggested method provides an interactive visualization of the rapid fluctuations in electrophysiological data during motor and cognitive tasks; hence, it has the potential to extract clinically relevant information at an individual level without temporal and spatial collapse.
Collapse
Affiliation(s)
- Ali Nabi Duman
- Department of Mathematics, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia
| | - Ahmet E. Tatar
- Center for Information Technology, University of Groningen, Groningen, Netherlands
| |
Collapse
|
4
|
Yan M, Zhang WH, Wang H, Wong KYM. Bimodular continuous attractor neural networks with static and moving stimuli. Phys Rev E 2023; 107:064302. [PMID: 37464697 DOI: 10.1103/physreve.107.064302] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2022] [Accepted: 05/08/2023] [Indexed: 07/20/2023]
Abstract
We investigated the dynamical behaviors of bimodular continuous attractor neural networks, each processing a modality of sensory input and interacting with each other. We found that when bumps coexist in both modules, the position of each bump is shifted towards the other input when the intermodular couplings are excitatory and is shifted away when inhibitory. When one intermodular coupling is excitatory while another is moderately inhibitory, temporally modulated population spikes can be generated. On further increase of the inhibitory coupling, momentary spikes will emerge. In the regime of bump coexistence, bump heights are primarily strengthened by excitatory intermodular couplings, but there is a lesser weakening effect due to a bump being displaced from the direct input. When bimodular networks serve as decoders of multisensory integration, we extend the Bayesian framework to show that excitatory and inhibitory couplings encode attractive and repulsive priors, respectively. At low disparity, the bump positions decode the posterior means in the Bayesian framework, whereas at high disparity, multiple steady states exist. In the regime of multiple steady states, the less stable state can be accessed if the input causing the more stable state arrives after a sufficiently long delay. When one input is moving, the bump in the corresponding module is pinned when the moving stimulus is weak, unpinned at intermediate stimulus strength, and tracks the input at strong stimulus strength, and the stimulus strengths for these transitions increase with the velocity of the moving stimulus. These results are important to understanding multisensory integration of static and dynamic stimuli.
Collapse
Affiliation(s)
- Min Yan
- Department of Physics, Hong Kong University of Science and Technology, Hong Kong SAR, People's Republic of China
| | - Wen-Hao Zhang
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, Texas 75390, USA
- O'Donnell Brain Institute, UT Southwestern Medical Center, Dallas, Texas 75390, USA
| | - He Wang
- Department of Physics, Hong Kong University of Science and Technology, Hong Kong SAR, People's Republic of China
- Hong Kong University of Science and Technology, Shenzhen Research Institute, Shenzhen 518057, China
| | - K Y Michael Wong
- Department of Physics, Hong Kong University of Science and Technology, Hong Kong SAR, People's Republic of China
| |
Collapse
|
5
|
Doerig A, Sommers RP, Seeliger K, Richards B, Ismael J, Lindsay GW, Kording KP, Konkle T, van Gerven MAJ, Kriegeskorte N, Kietzmann TC. The neuroconnectionist research programme. Nat Rev Neurosci 2023:10.1038/s41583-023-00705-w. [PMID: 37253949 DOI: 10.1038/s41583-023-00705-w] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/21/2023] [Indexed: 06/01/2023]
Abstract
Artificial neural networks (ANNs) inspired by biology are beginning to be widely used to model behavioural and neural data, an approach we call 'neuroconnectionism'. ANNs have been not only lauded as the current best models of information processing in the brain but also criticized for failing to account for basic cognitive functions. In this Perspective article, we propose that arguing about the successes and failures of a restricted set of current ANNs is the wrong approach to assess the promise of neuroconnectionism for brain science. Instead, we take inspiration from the philosophy of science, and in particular from Lakatos, who showed that the core of a scientific research programme is often not directly falsifiable but should be assessed by its capacity to generate novel insights. Following this view, we present neuroconnectionism as a general research programme centred around ANNs as a computational language for expressing falsifiable theories about brain computation. We describe the core of the programme, the underlying computational framework and its tools for testing specific neuroscientific hypotheses and deriving novel understanding. Taking a longitudinal view, we review past and present neuroconnectionist projects and their responses to challenges and argue that the research programme is highly progressive, generating new and otherwise unreachable insights into the workings of the brain.
Collapse
Affiliation(s)
- Adrien Doerig
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Rowan P Sommers
- Department of Neurobiology of Language, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Katja Seeliger
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Blake Richards
- Department of Neurology and Neurosurgery, McGill University, Montréal, QC, Canada
- School of Computer Science, McGill University, Montréal, QC, Canada
- Mila, Montréal, QC, Canada
- Montréal Neurological Institute, Montréal, QC, Canada
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
| | | | | | - Konrad P Kording
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
- Bioengineering, Neuroscience, University of Pennsylvania, Pennsylvania, PA, USA
| | | | | | | | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
| |
Collapse
|
6
|
He M, Hou X, Ge E, Wang Z, Kang Z, Qiang N, Zhang X, Ge B. Multi-head attention-based masked sequence model for mapping functional brain networks. Front Neurosci 2023; 17:1183145. [PMID: 37214388 PMCID: PMC10192686 DOI: 10.3389/fnins.2023.1183145] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 04/17/2023] [Indexed: 05/24/2023] Open
Abstract
The investigation of functional brain networks (FBNs) using task-based functional magnetic resonance imaging (tfMRI) has gained significant attention in the field of neuroimaging. Despite the availability of several methods for constructing FBNs, including traditional methods like GLM and deep learning methods such as spatiotemporal self-attention mechanism (STAAE), these methods have design and training limitations. Specifically, they do not consider the intrinsic characteristics of fMRI data, such as the possibility that the same signal value at different time points could represent different brain states and meanings. Furthermore, they overlook prior knowledge, such as task designs, during training. This study aims to overcome these limitations and develop a more efficient model by drawing inspiration from techniques in the field of natural language processing (NLP). The proposed model, called the Multi-head Attention-based Masked Sequence Model (MAMSM), uses a multi-headed attention mechanism and mask training approach to learn different states corresponding to the same voxel values. Additionally, it combines cosine similarity and task design curves to construct a novel loss function. The MAMSM was applied to seven task state datasets from the Human Connectome Project (HCP) tfMRI dataset. Experimental results showed that the features acquired by the MAMSM model exhibit a Pearson correlation coefficient with the task design curves above 0.95 on average. Moreover, the model can extract more meaningful networks beyond the known task-related brain networks. The experimental results demonstrated that MAMSM has great potential in advancing the understanding of functional brain networks.
Collapse
Affiliation(s)
- Mengshen He
- Key Laboratory of Modern Teaching Technology, Ministry of Education, Shaanxi Normal University, Xi’an, China
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Xiangyu Hou
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Enjie Ge
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Zhenwei Wang
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Zili Kang
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Ning Qiang
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| | - Xin Zhang
- Institute of Medical Research, Northwestern Polytechnical University, Xi’an, Shaanxi, China
| | - Bao Ge
- Key Laboratory of Modern Teaching Technology, Ministry of Education, Shaanxi Normal University, Xi’an, China
- School of Physics and Information Technology, Shaanxi Normal University, Xi’an, China
| |
Collapse
|
7
|
Taylor J, Xu Y. Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks. J Cogn Neurosci 2023; 35:816-840. [PMID: 36877074 DOI: 10.1162/jocn_a_01979] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/07/2023]
Abstract
Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.
Collapse
|
8
|
Barbulescu R, Mestre G, Oliveira AL, Silveira LM. Learning the dynamics of realistic models of C. elegans nervous system with recurrent neural networks. Sci Rep 2023; 13:467. [PMID: 36627317 PMCID: PMC9832137 DOI: 10.1038/s41598-022-25421-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Accepted: 11/29/2022] [Indexed: 01/12/2023] Open
Abstract
Given the inherent complexity of the human nervous system, insight into the dynamics of brain activity can be gained from studying smaller and simpler organisms. While some of the potential target organisms are simple enough that their behavioural and structural biology might be well-known and understood, others might still lead to computationally intractable models that require extensive resources to simulate. Since such organisms are frequently only acting as proxies to further our understanding of underlying phenomena or functionality, often one is not interested in the detailed evolution of every single neuron in the system. Instead, it is sufficient to observe the subset of neurons that capture the effect that the profound nonlinearities of the neuronal system have in response to different stimuli. In this paper, we consider the well-known nematode Caenorhabditis elegans and seek to investigate the possibility of generating lower complexity models that capture the system's dynamics with low error using only measured or simulated input-output information. Such models are often termed black-box models. We show how the nervous system of C. elegans can be modelled and simulated with data-driven models using different neural network architectures. Specifically, we target the use of state-of-the-art recurrent neural network architectures such as Long Short-Term Memory and Gated Recurrent Units and compare these architectures in terms of their properties and their accuracy (Root Mean Square Error), as well as the complexity of the resulting models. We show that Gated Recurrent Unit models with a hidden layer size of 4 are able to accurately reproduce the system response to very different stimuli. We furthermore explore the relative importance of their inputs as well as scalability to more scenarios.
Collapse
Affiliation(s)
| | - Gonçalo Mestre
- grid.14647.300000 0001 0279 8114INESC-ID Lisboa, Rua Alves Redol 9, Lisbon, 1000-029 Portugal ,grid.9983.b0000 0001 2181 4263IST Técnico Lisboa, Universidade de Lisboa, Av. Rovisco Pais 1, Lisbon, 1049-001 Portugal
| | - Arlindo L. Oliveira
- grid.14647.300000 0001 0279 8114INESC-ID Lisboa, Rua Alves Redol 9, Lisbon, 1000-029 Portugal ,grid.9983.b0000 0001 2181 4263IST Técnico Lisboa, Universidade de Lisboa, Av. Rovisco Pais 1, Lisbon, 1049-001 Portugal
| | - Luís Miguel Silveira
- grid.14647.300000 0001 0279 8114INESC-ID Lisboa, Rua Alves Redol 9, Lisbon, 1000-029 Portugal ,grid.9983.b0000 0001 2181 4263IST Técnico Lisboa, Universidade de Lisboa, Av. Rovisco Pais 1, Lisbon, 1049-001 Portugal
| |
Collapse
|
9
|
High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks. Brain Sci 2022; 12:brainsci12081101. [PMID: 36009164 PMCID: PMC9406060 DOI: 10.3390/brainsci12081101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 07/26/2022] [Accepted: 08/18/2022] [Indexed: 11/20/2022] Open
Abstract
Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance. The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model’s performance in high-level visual areas. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas.
Collapse
|
10
|
Qiang N, Dong Q, Liang H, Li J, Zhang S, Zhang C, Ge B, Sun Y, Gao J, Liu T, Yue H, Zhao S. Learning brain representation using recurrent Wasserstein generative adversarial net. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 223:106979. [PMID: 35792364 DOI: 10.1016/j.cmpb.2022.106979] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 04/26/2022] [Accepted: 06/26/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE To understand brain cognition and disorders, modeling the mapping between mind and brain has been of great interest to the neuroscience community. The key is the brain representation, including functional brain networks (FBN) and their corresponding temporal features. Recently, it has been proven that deep learning models have superb representation power on functional magnetic resonance imaging (fMRI) over traditional machine learning methods. However, due to the lack of high-quality data and labels, deep learning models tend to suffer from overfitting in the training process. METHODS In this work, we applied a recurrent Wasserstein generative adversarial net (RWGAN) to learn brain representation from volumetric fMRI data. Generative adversarial net (GAN) is widely used in natural image generation and is able to capture the distribution of the input data, which enables the extraction of generalized features from fMRI and thus relieves the overfitting issue. The recurrent layers in RWGAN are designed to better model the local temporal features of the fMRI time series. The discriminator of RWGAN works as a deep feature extractor. With LASSO regression, the RWGAN model can decompose the fMRI data into temporal features and spatial features (FBNs). Furthermore, the generator of RWGAN can generate high-quality new data for fMRI augmentation. RESULTS The experimental results on seven tasks from the HCP dataset showed that the RWGAN can learn meaningful and interpretable temporal features and FBNs, compared to HCP task designs and general linear model (GLM) derived networks. Besides, the results on different training datasets showed that the RWGAN performed better on small datasets than other deep learning models. Moreover, we used the generator of RWGAN to yield fake subjects. The result showed that the fake data can also be used to learn meaningful representation compared to those learned from real data. CONCLUSIONS To our best knowledge, this work is among the earliest attempts of applying generative deep learning for modeling fMRI data. The proposed RWGAN offers a novel methodology for learning brain representation from fMRI, and it can generate high-quality fake data for the potential use of fMRI data augmentation.
Collapse
Affiliation(s)
- Ning Qiang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China; Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Qinglin Dong
- Advanced Medical Computing and Analysis, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | - Hongtao Liang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Jin Li
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Shu Zhang
- Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Cheng Zhang
- School of Electronics Engineering and Computer Science, Peking University, Beijing, China
| | - Bao Ge
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China; Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Yifei Sun
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Jie Gao
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, Greece
| | - Huiji Yue
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China.
| | - Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi'an, China.
| |
Collapse
|
11
|
Armeni K, Güçlü U, van Gerven M, Schoffelen JM. A 10-hour within-participant magnetoencephalography narrative dataset to test models of language comprehension. Sci Data 2022; 9:278. [PMID: 35676293 PMCID: PMC9177538 DOI: 10.1038/s41597-022-01382-7] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Accepted: 05/10/2022] [Indexed: 11/13/2022] Open
Abstract
Recently, cognitive neuroscientists have increasingly studied the brain responses to narratives. At the same time, we are witnessing exciting developments in natural language processing where large-scale neural network models can be used to instantiate cognitive hypotheses in narrative processing. Yet, they learn from text alone and we lack ways of incorporating biological constraints during training. To mitigate this gap, we provide a narrative comprehension magnetoencephalography (MEG) data resource that can be used to train neural network models directly on brain data. We recorded from 3 participants, 10 separate recording hour-long sessions each, while they listened to audiobooks in English. After story listening, participants answered short questions about their experience. To minimize head movement, the participants wore MEG-compatible head casts, which immobilized their head position during recording. We report a basic evoked-response analysis showing that the responses accurately localize to primary auditory areas. The responses are robust and conserved across 10 sessions for every participant. We also provide usage notes and briefly outline possible future uses of the resource.
Collapse
Affiliation(s)
- Kristijan Armeni
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Umut Güçlü
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Marcel van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Jan-Mathijs Schoffelen
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands.
| |
Collapse
|
12
|
Zhao M, Yan W, Luo N, Zhi D, Fu Z, Du Y, Yu S, Jiang T, Calhoun VD, Sui J. An attention-based hybrid deep learning framework integrating brain connectivity and activity of resting-state functional MRI data. Med Image Anal 2022; 78:102413. [PMID: 35305447 PMCID: PMC9035078 DOI: 10.1016/j.media.2022.102413] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2021] [Revised: 10/27/2021] [Accepted: 03/01/2022] [Indexed: 12/30/2022]
Abstract
Functional magnetic resonance imaging (fMRI) as a promising tool to investigate psychotic disorders can be decomposed into useful imaging features such as time courses (TCs) of independent components (ICs) and functional network connectivity (FNC) calculated by TC cross-correlation. TCs reflect the temporal dynamics of brain activity and the FNC characterizes temporal coherence across intrinsic brain networks. Both features have been used as input to deep learning approaches with decent results. However, few studies have tried to leverage their complementary information to learn optimal representations at multiple facets. Motivated by this, we proposed a Hybrid Deep Learning Framework integrating brain Connectivity and Activity (HDLFCA) together by combining convolutional recurrent neural network (C-RNN) and deep neural network (DNN), aiming to improve classification accuracy and interpretability simultaneously. Specifically, C-RNNAM was proposed to extract temporal dynamic dependencies with an attention module (AM) to automatically learn discriminative knowledge from TC nodes, while DNN was applied to identify the most group-discriminative FNC patterns with layer-wise relevance propagation (LRP). Then, both prediction outputs were concatenated to build a new feature matrix, generating the final decision by logistic regression. The effectiveness of HDLFCA was validated on both multi-site schizophrenia (SZ, n ∼ 1100) and public autism datasets (ABIDE, n ∼ 1522) by outperforming 12 alternative models at 2.8-8.9% accuracy, including 8 models using either static FNC or TCs and 4 models using dynamic FNC. Appreciable classification accuracy was achieved for HC vs. SZ (85.3%) and HC vs. Autism (72.4%) respectively. More importantly, the most group-discriminative brain regions can be easily attributed and visualized, providing meaningful biological interpretability and highlighting the great potential of the proposed HDLFCA model in the identification of valid neuroimaging biomarkers.
Collapse
Affiliation(s)
- Min Zhao
- Brainnetome Center and National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Weizheng Yan
- Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA
| | - Na Luo
- Brainnetome Center and National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Dongmei Zhi
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China
| | - Zening Fu
- Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA
| | - Yuhui Du
- School of Computer and Information Technology, Shanxi University, Taiyuan, China
| | - Shan Yu
- Brainnetome Center and National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Tianzi Jiang
- Brainnetome Center and National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Vince D Calhoun
- Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA
| | - Jing Sui
- Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA; State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China.
| |
Collapse
|
13
|
Gaurav A, Song X, Manhas S, Gilra A, Vasilaki E, Roy P, De Souza MM. Reservoir Computing for Temporal Data Classification Using a Dynamic Solid Electrolyte ZnO Thin Film Transistor. FRONTIERS IN ELECTRONICS 2022. [DOI: 10.3389/felec.2022.869013] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
The processing of sequential and temporal data is essential to computer vision and speech recognition, two of the most common applications of artificial intelligence (AI). Reservoir computing (RC) is a branch of AI that offers a highly efficient framework for processing temporal inputs at a low training cost compared to conventional Recurrent Neural Networks (RNNs). However, despite extensive effort, two-terminal memristor-based reservoirs have, until now, been implemented to process sequential data by reading their conductance states only once, at the end of the entire sequence. This method reduces the dimensionality, related to the number of signals from the reservoir and thereby lowers the overall performance of reservoir systems. Higher dimensionality facilitates the separation of originally inseparable inputs by reading out from a larger set of spatiotemporal features of inputs. Moreover, memristor-based reservoirs either use multiple pulse rates, fast or slow read (immediately or with a delay introduced after the end of the sequence), or excitatory pulses to enhance the dimensionality of reservoir states. This adds to the complexity of the reservoir system and reduces power efficiency. In this paper, we demonstrate the first reservoir computing system based on a dynamic three terminal solid electrolyte ZnO/Ta2O5 Thin-film Transistor fabricated at less than 100°C. The inherent nonlinearity and dynamic memory of the device lead to a rich separation property of reservoir states that results in, to our knowledge, the highest accuracy of 94.44%, using electronic charge-based system, for the classification of hand-written digits. This improvement is attributed to an increase in the dimensionality of the reservoir by reading the reservoir states after each pulse rather than at the end of the sequence. The third terminal enables a read operation in the off state, that is when no pulse is applied at the gate terminal, via a small read pulse at the drain. This fundamentally allows multiple read operations without increasing energy consumption, which is not possible in the conventional two-terminal memristor counterpart. Further, we have also shown that devices do not saturate even after multiple write pulses which demonstrates the device’s ability to process longer sequences.
Collapse
|
14
|
|
15
|
Wasmann JWA, Lanting CP, Huinck WJ, Mylanus EAM, van der Laak JWM, Govaerts PJ, Swanepoel DW, Moore DR, Barbour DL. Computational Audiology: New Approaches to Advance Hearing Health Care in the Digital Age. Ear Hear 2021; 42:1499-1507. [PMID: 33675587 PMCID: PMC8417156 DOI: 10.1097/aud.0000000000001041] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
The global digital transformation enables computational audiology for advanced clinical applications that can reduce the global burden of hearing loss. In this article, we describe emerging hearing-related artificial intelligence applications and argue for their potential to improve access, precision, and efficiency of hearing health care services. Also, we raise awareness of risks that must be addressed to enable a safe digital transformation in audiology. We envision a future where computational audiology is implemented via interoperable systems using shared data and where health care providers adopt expanded roles within a network of distributed expertise. This effort should take place in a health care system where privacy, responsibility of each stakeholder, and patients' safety and autonomy are all guarded by design.
Collapse
Affiliation(s)
- Jan-Willem A Wasmann
- Department of Otorhinolaryngology, Donders Institute for Brain, Cognition and Behaviour, Radboud University Medical Center Nijmegen, the Netherlands
| | - Cris P Lanting
- Department of Otorhinolaryngology, Donders Institute for Brain, Cognition and Behaviour, Radboud University Medical Center Nijmegen, the Netherlands
| | - Wendy J Huinck
- Department of Otorhinolaryngology, Donders Institute for Brain, Cognition and Behaviour, Radboud University Medical Center Nijmegen, the Netherlands
| | - Emmanuel A M Mylanus
- Department of Otorhinolaryngology, Donders Institute for Brain, Cognition and Behaviour, Radboud University Medical Center Nijmegen, the Netherlands
| | - Jeroen W M van der Laak
- Department of Pathology, Radboud University Medical Center Nijmegen, the Netherlands
- Center for Medical Image Science and Visualization, Linköping University, Sweden
| | | | - De Wet Swanepoel
- Department of Speech-Language Pathology and Audiology, University of Pretoria, South Africa
| | - David R Moore
- Communication Sciences Research Center, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
- Department of Otolaryngology, University of Cincinnati, Cincinnati, Ohio, USA
- Manchester Centre for Audiology and Deafness, University of Manchester, Manchester, United Kingdom
| | - Dennis L Barbour
- Department of Biomedical Engineering. Washington University in St. Louis, St. Louis, Missouri, USA
| |
Collapse
|
16
|
Decoding neurobiological spike trains using recurrent neural networks: a case study with electrophysiological auditory cortex recordings. Neural Comput Appl 2021. [DOI: 10.1007/s00521-021-06589-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
AbstractRecent advancements in multielectrode methods and spike-sorting algorithms enable the in vivo recording of the activities of many neurons at a high temporal resolution. These datasets offer new opportunities in the investigation of the biological neural code, including the direct testing of specific coding hypotheses, but they also reveal the limitations of present decoder algorithms. Classical methods rely on a manual feature extraction step, resulting in a feature vector, like the firing rates of an ensemble of neurons. In this paper, we present a recurrent neural-network-based decoder and evaluate its performance on experimental and artificial datasets. The experimental datasets were obtained by recording the auditory cortical responses of rats exposed to sound stimuli, while the artificial datasets represent preset encoding schemes. The task of the decoder was to classify the action potential timeseries according to the corresponding sound stimuli. It is illustrated that, depending on the coding scheme, the performance of the recurrent-network-based decoder can exceed the performance of the classical methods. We also show how randomized copies of the training datasets can be used to reveal the role of candidate spike-train features. We conclude that artificial neural network decoders can be a useful alternative to classical population vector-based techniques in studies of the biological neural code.
Collapse
|
17
|
Li Q, Zhang W, Zhao L, Wu X, Liu T. Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition. IEEE Trans Biomed Eng 2021; 69:624-634. [PMID: 34357861 DOI: 10.1109/tbme.2021.3102466] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Using deep neural networks (DNNs) to explore spatial patterns and temporal dynamics of human brain activities has been an important yet challenging problem because the artificial neural networks are hard to be designed manually. There have been several promising deep learning methods, e.g., deep belief network (DBN), convolutional neural network (CNN), and deep sparse recurrent auto-encoder (DSRAE), that can decompose neuroscientific and meaningful spatiotemporal patterns from 4D functional Magnetic Resonance Imaging (fMRI) data. However, those previous studies still depend on hand-crafted neural network architectures and hyperparameters, which are not optimal in various senses. In this paper, we employ the evolutionary algorithms (EA) to optimize the deep neural architecture of DSRAE by minimizing the expected loss of initialized models, named eNAS-DSRAE (evolutionary Neural Architecture Search on Deep Sparse Recurrent Auto-Encoder). Also, validation experiments are designed and performed on the publicly available human connectome project (HCP) 900 datasets, and the results achieved by the optimized eNAS-DSRAE suggested that our framework can successfully identify the spatiotemporal features and perform better than the hand-crafted neural network models. To our best knowledge, the proposed eNAS-DSRAE is not only among the earliest NAS models that can extract connectome-scale meaningful spatiotemporal brain networks from 4D fMRI data, but also is an effective framework to optimize the RNN-based models.
Collapse
|
18
|
Zhang Z, Li G, Xu Y, Tang X. Application of Artificial Intelligence in the MRI Classification Task of Human Brain Neurological and Psychiatric Diseases: A Scoping Review. Diagnostics (Basel) 2021; 11:1402. [PMID: 34441336 PMCID: PMC8392727 DOI: 10.3390/diagnostics11081402] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2021] [Revised: 07/21/2021] [Accepted: 07/21/2021] [Indexed: 12/12/2022] Open
Abstract
Artificial intelligence (AI) for medical imaging is a technology with great potential. An in-depth understanding of the principles and applications of magnetic resonance imaging (MRI), machine learning (ML), and deep learning (DL) is fundamental for developing AI-based algorithms that can meet the requirements of clinical diagnosis and have excellent quality and efficiency. Moreover, a more comprehensive understanding of applications and opportunities would help to implement AI-based methods in an ethical and sustainable manner. This review first summarizes recent research advances in ML and DL techniques for classifying human brain magnetic resonance images. Then, the application of ML and DL methods to six typical neurological and psychiatric diseases is summarized, including Alzheimer's disease (AD), Parkinson's disease (PD), major depressive disorder (MDD), schizophrenia (SCZ), attention-deficit/hyperactivity disorder (ADHD), and autism spectrum disorder (ASD). Finally, the limitations of the existing research are discussed, and possible future research directions are proposed.
Collapse
Affiliation(s)
- Zhao Zhang
- 715-3 Teaching Building No.5, Department of Biomedical Engineering, School of Life Sciences, Beijing Institute of Technology, 5 South Zhongguancun Road, Haidian District, Beijing 100081, China; (Z.Z.); (G.L.)
| | - Guangfei Li
- 715-3 Teaching Building No.5, Department of Biomedical Engineering, School of Life Sciences, Beijing Institute of Technology, 5 South Zhongguancun Road, Haidian District, Beijing 100081, China; (Z.Z.); (G.L.)
| | - Yong Xu
- Department of Cardiology, Chinese PLA General Hospital, Beijing 100853, China;
| | - Xiaoying Tang
- 715-3 Teaching Building No.5, Department of Biomedical Engineering, School of Life Sciences, Beijing Institute of Technology, 5 South Zhongguancun Road, Haidian District, Beijing 100081, China; (Z.Z.); (G.L.)
| |
Collapse
|
19
|
Qiang N, Dong Q, Liang H, Ge B, Zhang S, Sun Y, Zhang C, Zhang W, Gao J, Liu T. Modeling and augmenting of fMRI data using deep recurrent variational auto-encoder. J Neural Eng 2021; 18. [PMID: 34229310 DOI: 10.1088/1741-2552/ac1179] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 07/06/2021] [Indexed: 11/11/2022]
Abstract
Objective. Recently, deep learning models have been successfully applied in functional magnetic resonance imaging (fMRI) modeling and associated applications. However, there still exist at least two challenges. Firstly, due to the lack of sufficient data, deep learning models tend to suffer from overfitting in the training process. Secondly, it is still challenging to model the temporal dynamics from fMRI, due to that the brain state is continuously changing over scan time. In addition, existing methods rarely studied and applied fMRI data augmentation.Approach. In this work, we construct a deep recurrent variational auto-encoder (DRVAE) that combined variational auto-encoder and recurrent neural network, aiming to address all of the above mentioned challenges. The encoder of DRVAE can extract more generalized temporal features from assumed Gaussian distribution of input data, and the decoder of DRVAE can generate new data to increase training samples and thus partially relieve the overfitting issue. The recurrent layers in DRVAE are designed to effectively model the temporal dynamics of functional brain activities. LASSO (least absolute shrinkage and selection operator) regression is applied on the temporal features and input fMRI data to estimate the corresponding spatial networks.Main results. Extensive experimental results on seven tasks from HCP dataset showed that the DRVAE and LASSO framework can learn meaningful temporal patterns and spatial networks from both real data and generated data. The results on group-wise data and single subject suggest that the brain activities may follow certain distribution. Moreover, we applied DRVAE on four resting state fMRI datasets from ADHD-200 for data augmentation, and the results showed that the classification performances on augmented datasets have been considerably improved.Significance. The proposed method can not only derive meaningful temporal features and spatial networks from fMRI, but also generate high-quality new data for fMRI data augmentation and associated applications.
Collapse
Affiliation(s)
- Ning Qiang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, People's Republic of China.,Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, People's Republic of China
| | - Qinglin Dong
- Advanced Medical Computing and Analysis, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, United States of America
| | - Hongtao Liang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, People's Republic of China
| | - Bao Ge
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, People's Republic of China.,Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, People's Republic of China
| | - Shu Zhang
- Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, People's Republic of China
| | - Yifei Sun
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, People's Republic of China
| | - Cheng Zhang
- School of Electronics Engineering and Computer Science, Peking University, Beijing, People's Republic of China
| | - Wei Zhang
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, CA, United States of America
| | - Jie Gao
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, People's Republic of China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, United States of America
| |
Collapse
|
20
|
Park YH, Shin SA, Kim S, Lee JM. Key Intrinsic Connectivity Networks for Individual Identification With Siamese Long Short-Term Memory. Front Neurosci 2021; 15:660187. [PMID: 34220422 PMCID: PMC8249867 DOI: 10.3389/fnins.2021.660187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Accepted: 05/11/2021] [Indexed: 11/27/2022] Open
Abstract
In functional magnetic resonance imaging (fMRI) analysis, many studies have been conducted on inter-subject variability as well as intra-subject reproducibility. These studies indicate that fMRI could have unique characteristics for individuals. In this study, we hypothesized that the dynamic information during 1 min of fMRI was unique and repetitive enough for each subject, so we applied long short-term memory (LSTM) using initial time points of dynamic resting-state fMRI for individual identification. Siamese network is used to obtain robust individual identification performance without additional learning on a new dataset. In particular, by adding a new structure called region of interest–wise average pooling (RAP), individual identification performance could be improved, and key intrinsic connectivity networks (ICNs) for individual identification were also identified. The average performance of individual identification was 97.88% using the test dataset in eightfold cross-validation analysis. Through the visualization of features learned by Siamese LSTM with RAP, ICNs spanning the parietal region were observed as the key ICNs in identifying individuals. These results suggest the key ICNs in fMRI could represent individual uniqueness.
Collapse
Affiliation(s)
- Yeong-Hun Park
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea
| | - Seong A Shin
- Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Seonggyu Kim
- Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Jong-Min Lee
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| |
Collapse
|
21
|
Seeliger K, Ambrogioni L, Güçlütürk Y, van den Bulk LM, Güçlü U, van Gerven MAJ. End-to-end neural system identification with neural information flow. PLoS Comput Biol 2021; 17:e1008558. [PMID: 33539366 PMCID: PMC7888598 DOI: 10.1371/journal.pcbi.1008558] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2019] [Revised: 02/17/2021] [Accepted: 11/24/2020] [Indexed: 11/19/2022] Open
Abstract
Neural information flow (NIF) provides a novel approach for system identification in neuroscience. It models the neural computations in multiple brain regions and can be trained end-to-end via stochastic gradient descent from noninvasive data. NIF models represent neural information processing via a network of coupled tensors, each encoding the representation of the sensory input contained in a brain region. The elements of these tensors can be interpreted as cortical columns whose activity encodes the presence of a specific feature in a spatiotemporal location. Each tensor is coupled to the measured data specific to a brain region via low-rank observation models that can be decomposed into the spatial, temporal and feature receptive fields of a localized neuronal population. Both these observation models and the convolutional weights defining the information processing within regions are learned end-to-end by predicting the neural signal during sensory stimulation. We trained a NIF model on the activity of early visual areas using a large-scale fMRI dataset recorded in a single participant. We show that we can recover plausible visual representations and population receptive fields that are consistent with empirical findings.
Collapse
Affiliation(s)
- K. Seeliger
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - L. Ambrogioni
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Y. Güçlütürk
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - L. M. van den Bulk
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - U. Güçlü
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - M. A. J. van Gerven
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| |
Collapse
|
22
|
Bae H, Kim SJ, Kim CE. Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks. Front Syst Neurosci 2021; 14:615129. [PMID: 33519390 PMCID: PMC7843526 DOI: 10.3389/fnsys.2020.615129] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2020] [Accepted: 12/14/2020] [Indexed: 12/26/2022] Open
Abstract
One of the central goals in systems neuroscience is to understand how information is encoded in the brain, and the standard approach is to identify the relation between a stimulus and a neural response. However, the feature of a stimulus is typically defined by the researcher's hypothesis, which may cause biases in the research conclusion. To demonstrate potential biases, we simulate four likely scenarios using deep neural networks trained on the image classification dataset CIFAR-10 and demonstrate the possibility of selecting suboptimal/irrelevant features or overestimating the network feature representation/noise correlation. Additionally, we present studies investigating neural coding principles in biological neural networks to which our points can be applied. This study aims to not only highlight the importance of careful assumptions and interpretations regarding the neural response to stimulus features but also suggest that the comparative study between deep and biological neural networks from the perspective of machine learning can be an effective strategy for understanding the coding principles of the brain.
Collapse
Affiliation(s)
- Hyojin Bae
- Department of Physiology, Gachon University College of Korean Medicine, Seongnam, South Korea
| | - Sang Jeong Kim
- Laboratory of Neurophysiology, Department of Physiology, Seoul National University College of Medicine, Seoul, South Korea
| | - Chang-Eop Kim
- Department of Physiology, Gachon University College of Korean Medicine, Seongnam, South Korea
| |
Collapse
|
23
|
Cui Y, Zhao S, Chen Y, Han J, Guo L, Xie L, Liu T. Modeling Brain Diverse and Complex Hemodynamic Response Patterns via Deep Recurrent Autoencoder. IEEE Trans Cogn Dev Syst 2020. [DOI: 10.1109/tcds.2019.2949195] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
24
|
Wang C, Yan H, Huang W, Li J, Yang J, Li R, Zhang L, Li L, Zhang J, Zuo Z, Chen H. ‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework. J Neural Eng 2020; 17:056013. [DOI: 10.1088/1741-2552/abb691] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
25
|
Sobczak F, He Y, Sejnowski TJ, Yu X. Predicting the fMRI Signal Fluctuation with Recurrent Neural Networks Trained on Vascular Network Dynamics. Cereb Cortex 2020; 31:826-844. [PMID: 32940658 PMCID: PMC7906791 DOI: 10.1093/cercor/bhaa260] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Revised: 07/19/2020] [Accepted: 08/12/2020] [Indexed: 02/06/2023] Open
Abstract
Resting-state functional MRI (rs-fMRI) studies have revealed specific low-frequency hemodynamic signal fluctuations (<0.1 Hz) in the brain, which could be related to neuronal oscillations through the neurovascular coupling mechanism. Given the vascular origin of the fMRI signal, it remains challenging to separate the neural correlates of global rs-fMRI signal fluctuations from other confounding sources. However, the slow-oscillation detected from individual vessels by single-vessel fMRI presents strong correlation to neural oscillations. Here, we use recurrent neural networks (RNNs) to predict the future temporal evolution of the rs-fMRI slow oscillation from both rodent and human brains. The RNNs trained with vessel-specific rs-fMRI signals encode the unique brain oscillatory dynamic feature, presenting more effective prediction than the conventional autoregressive model. This RNN-based predictive modeling of rs-fMRI datasets from the Human Connectome Project (HCP) reveals brain state-specific characteristics, demonstrating an inverse relationship between the global rs-fMRI signal fluctuation with the internal default-mode network (DMN) correlation. The RNN prediction method presents a unique data-driven encoding scheme to specify potential brain state differences based on the global fMRI signal fluctuation, but not solely dependent on the global variance.
Collapse
Affiliation(s)
- Filip Sobczak
- Translational Neuroimaging and Neural Control Group, High Field Magnetic Resonance Department, Max Planck Institute for Biological Cybernetics, 72076 Tuebingen, Germany.,Graduate Training Centre of Neuroscience, International Max Planck Research School, University of Tuebingen, 72074 Tuebingen, Germany
| | - Yi He
- Translational Neuroimaging and Neural Control Group, High Field Magnetic Resonance Department, Max Planck Institute for Biological Cybernetics, 72076 Tuebingen, Germany.,Danish Research Centre for Magnetic Resonance, 2650, Hvidovre, Denmark
| | - Terrence J Sejnowski
- Howard Hughes Medical Institute, Computational Neurobiology Laboratory, Salk Institute for Biological Studies, La Jolla, CA 92037, USA.,Division of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA
| | - Xin Yu
- Translational Neuroimaging and Neural Control Group, High Field Magnetic Resonance Department, Max Planck Institute for Biological Cybernetics, 72076 Tuebingen, Germany.,Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA 02129, USA
| |
Collapse
|
26
|
Deep Reinforcement Learning and Its Neuroscientific Implications. Neuron 2020; 107:603-616. [DOI: 10.1016/j.neuron.2020.06.014] [Citation(s) in RCA: 57] [Impact Index Per Article: 14.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2020] [Revised: 06/08/2020] [Accepted: 06/12/2020] [Indexed: 11/23/2022]
|
27
|
Berezutskaya J, Freudenburg ZV, Ambrogioni L, Güçlü U, van Gerven MAJ, Ramsey NF. Cortical network responses map onto data-driven features that capture visual semantics of movie fragments. Sci Rep 2020; 10:12077. [PMID: 32694561 PMCID: PMC7374611 DOI: 10.1038/s41598-020-68853-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2020] [Accepted: 06/30/2020] [Indexed: 11/08/2022] Open
Abstract
Research on how the human brain extracts meaning from sensory input relies in principle on methodological reductionism. In the present study, we adopt a more holistic approach by modeling the cortical responses to semantic information that was extracted from the visual stream of a feature film, employing artificial neural network models. Advances in both computer vision and natural language processing were utilized to extract the semantic representations from the film by combining perceptual and linguistic information. We tested whether these representations were useful in studying the human brain data. To this end, we collected electrocorticography responses to a short movie from 37 subjects and fitted their cortical patterns across multiple regions using the semantic components extracted from film frames. We found that individual semantic components reflected fundamental semantic distinctions in the visual input, such as presence or absence of people, human movement, landscape scenes, human faces, etc. Moreover, each semantic component mapped onto a distinct functional cortical network involving high-level cognitive regions in occipitotemporal, frontal and parietal cortices. The present work demonstrates the potential of the data-driven methods from information processing fields to explain patterns of cortical responses, and contributes to the overall discussion about the encoding of high-level perceptual information in the human brain.
Collapse
Affiliation(s)
- Julia Berezutskaya
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands.
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Montessorilaan 3, 6525 HR, Nijmegen, The Netherlands.
| | - Zachary V Freudenburg
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - Luca Ambrogioni
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Montessorilaan 3, 6525 HR, Nijmegen, The Netherlands
| | - Umut Güçlü
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Montessorilaan 3, 6525 HR, Nijmegen, The Netherlands
| | - Marcel A J van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Montessorilaan 3, 6525 HR, Nijmegen, The Netherlands
| | - Nick F Ramsey
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| |
Collapse
|
28
|
Huang W, Yan H, Wang C, Li J, Yang X, Li L, Zuo Z, Zhang J, Chen H. Long short-term memory-based neural decoding of object categories evoked by natural images. Hum Brain Mapp 2020; 41:4442-4453. [PMID: 32648632 PMCID: PMC7502843 DOI: 10.1002/hbm.25136] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2020] [Revised: 05/19/2020] [Accepted: 06/29/2020] [Indexed: 01/18/2023] Open
Abstract
Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding. Most previous studies used peak response signals to decode object categories. However, brain activities measured by functional magnetic resonance imaging are a dynamic process with time dependence, so peak signals cannot fully represent the whole process, which may affect the performance of decoding. Here, we propose a decoding model based on long short‐term memory (LSTM) network to decode five object categories from multitime response signals evoked by natural images. Experimental results show that the average decoding accuracy using the multitime (2–6 s) response signals is 0.540 from the five subjects, which is significantly higher than that using the peak ones (6 s; accuracy: 0.492; p < .05). In addition, from the perspective of different durations, methods and visual areas, the decoding performances of the five object categories are deeply and comprehensively explored. The analysis of different durations and decoding methods reveals that the LSTM‐based decoding model with sequence simulation ability can fit the time dependence of the multitime visual response signals to achieve higher decoding performance. The comparative analysis of different visual areas demonstrates that the higher visual cortex (VC) contains more semantic category information needed for visual perceptual decoding than lower VC.
Collapse
Affiliation(s)
- Wei Huang
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Hongmei Yan
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Chong Wang
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Jiyi Li
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Xiaoqing Yang
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Liang Li
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| | - Zhentao Zuo
- State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing, China
| | - Jiang Zhang
- Department of Medical Information Engineering, Sichuan University, Chengdu, China
| | - Huafu Chen
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, People's Republic of China
| |
Collapse
|
29
|
Dvornek NC, Ventola P, Duncan JS. ESTIMATING REPRODUCIBLE FUNCTIONAL NETWORKS ASSOCIATED WITH TASK DYNAMICS USING UNSUPERVISED LSTMS. PROCEEDINGS. IEEE INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING 2020; 2020. [PMID: 34422224 DOI: 10.1109/isbi45749.2020.9098377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
We propose a method for estimating more reproducible functional networks that are more strongly associated with dynamic task activity by using recurrent neural networks with long short term memory (LSTMs). The LSTM model is trained in an unsupervised manner to learn to generate the functional magnetic resonance imaging (fMRI) time-series data in regions of interest. The learned functional networks can then be used for further analysis, e.g., correlation analysis to determine functional networks that are strongly associated with an fMRI task paradigm. We test our approach and compare to other methods for decomposing functional networks from fMRI activity on 2 related but separate datasets that employ a biological motion perception task. We demonstrate that the functional networks learned by the LSTM model are more strongly associated with the task activity and dynamics compared to other approaches. Furthermore, the patterns of network association are more closely replicated across subjects within the same dataset as well as across datasets. More reproducible functional networks are essential for better characterizing the neural correlates of a target task.
Collapse
Affiliation(s)
- Nicha C Dvornek
- Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT.,Biomedical Engineering, Yale University, New Haven, CT
| | - Pamela Ventola
- Child Study Center, Yale School of Medicine, New Haven, CT
| | - James S Duncan
- Biomedical Engineering, Yale University, New Haven, CT.,Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT.,Electrical Engineering, Yale University, New Haven, CT
| |
Collapse
|
30
|
Kam TE, Zhang H, Jiao Z, Shen D. Deep Learning of Static and Dynamic Brain Functional Networks for Early MCI Detection. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:478-487. [PMID: 31329111 PMCID: PMC7122732 DOI: 10.1109/tmi.2019.2928790] [Citation(s) in RCA: 56] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2023]
Abstract
While convolutional neural network (CNN) has been demonstrating powerful ability to learn hierarchical spatial features from medical images, it is still difficult to apply it directly to resting-state functional MRI (rs-fMRI) and the derived brain functional networks (BFNs). We propose a novel CNN framework to simultaneously learn embedded features from BFNs for brain disease diagnosis. Since BFNs can be built by considering both static and dynamic functional connectivity (FC), we first decompose rs-fMRI into multiple static BFNs with modified independent component analysis. Then, the voxel-wise variability in dynamic FC is used to quantify BFN dynamics. A set of paired 3D images representing static/dynamic BFNs can be fed into 3D CNNs, from which we can hierarchically and simultaneously learn static/dynamic BFN features. As a result, the dynamic BFN features can complement static BFN features and, at the meantime, different BFNs can help each other toward a joint and better classification. We validate our method with a publicly accessible, large cohort of rs-fMRI dataset in early-stage mild cognitive impairment (eMCI) diagnosis, which is one of the most challenging problems to the clinicians. By comparing with a conventional method, our method shows significant diagnostic performance improvement by almost 10%. This result demonstrates the effectiveness of deep learning in preclinical Alzheimer's disease diagnosis, based on the complex and high-dimensional voxel-wise spatiotemporal patterns of the resting-state brain functional connectomics. The framework provides a new but intuitive way to fully exploit deeply embedded diagnostic features from rs-fMRI for a better-individualized diagnosis of various neurological diseases.
Collapse
|
31
|
Kim HC, Jang H, Lee JH. Test–retest reliability of spatial patterns from resting-state functional MRI using the restricted Boltzmann machine and hierarchically organized spatial patterns from the deep belief network. J Neurosci Methods 2020; 330:108451. [DOI: 10.1016/j.jneumeth.2019.108451] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2019] [Revised: 07/25/2019] [Accepted: 09/27/2019] [Indexed: 12/01/2022]
|
32
|
Li H, Fan Y. Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks. Neuroimage 2019; 202:116059. [PMID: 31362049 PMCID: PMC6819260 DOI: 10.1016/j.neuroimage.2019.116059] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2019] [Revised: 07/25/2019] [Accepted: 07/26/2019] [Indexed: 11/17/2022] Open
Abstract
Decoding brain functional states underlying cognitive processes from functional MRI (fMRI) data using multivariate pattern analysis (MVPA) techniques has achieved promising performance for characterizing brain activation patterns and providing neurofeedback signals. However, it remains challenging to decode subtly distinct brain states for individual fMRI data points due to varying temporal durations and dependency among different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in intrinsic functional network modeling and sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, subject-specific intrinsic functional networks (FNs) are computed from resting-state fMRI data and are used to characterize functional signals of task fMRI data with a compact representation for building brain decoding models, and LSTM RNNs are adopted to learn brain decoding mappings between functional profiles and brain states. Validation results on fMRI data from the HCP dataset have demonstrated that brain decoding models built on training data using the proposed method could learn discriminative latent feature representations and effectively distinguish subtly distinct working memory tasks of different subjects with significantly higher accuracy than conventional decoding models. Informative FNs of the brain decoding models identified as brain activation patterns of working memory tasks were largely consistent with the literature. The method also obtained promising decoding performance on motor and social cognition tasks. Our results suggest that LSTM RNNs in conjunction with FNs could build interpretable, highly accurate brain decoding models.
Collapse
Affiliation(s)
- Hongming Li
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Yong Fan
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA.
| |
Collapse
|
33
|
Cui Y, Zhao S, Wang H, Xie L, Chen Y, Han J, Guo L, Zhou F, Liu T. Identifying Brain Networks at Multiple Time Scales via Deep Recurrent Neural Network. IEEE J Biomed Health Inform 2019; 23:2515-2525. [PMID: 30475739 PMCID: PMC6914656 DOI: 10.1109/jbhi.2018.2882885] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
For decades, task functional magnetic resonance imaging has been a powerful noninvasive tool to explore the organizational architecture of human brain function. Researchers have developed a variety of brain network analysis methods for task fMRI data, including the general linear model, independent component analysis, and sparse representation methods. However, these shallow models are limited in faithful reconstruction and modeling of the hierarchical and temporal structures of brain networks, as demonstrated in more and more studies. Recently, recurrent neural networks (RNNs) exhibit great ability of modeling hierarchical and temporal dependence features in the machine learning field, which might be suitable for task fMRI data modeling. To explore such possible advantages of RNNs for task fMRI data, we propose a novel framework of a deep recurrent neural network (DRNN) to model the functional brain networks from task fMRI data. Experimental results on the motor task fMRI data of Human Connectome Project 900 subjects release demonstrated that the proposed DRNN can not only faithfully reconstruct functional brain networks, but also identify more meaningful brain networks with multiple time scales which are overlooked by traditional shallow models. In general, this work provides an effective and powerful approach to identifying functional brain networks at multiple time scales from task fMRI data.
Collapse
Affiliation(s)
- Yan Cui
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, 310027, China
| | - Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Han Wang
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, 310027, China
| | - Li Xie
- College of Biomedical Engineering & Instrument Science, and the State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou, 310027, China
| | - Yaowu Chen
- College of Biomedical Engineering & Instrument Science, and Zhejiang Provincial Key Laboratory for Network Multimedia Technologies, Zhejiang University, Hangzhou, 310027, China
| | - Junwei Han
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Lei Guo
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Fan Zhou
- College of Biomedical Engineering & Instrument Science, Zhejiang University, and the Zhejiang University Embedded System Engineering Research Center, Ministry of Education of China, Hangzhou, 310027, China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, 30602, USA
| |
Collapse
|
34
|
Dvornek NC, Li X, Zhuang J, Duncan JS. Jointly Discriminative and Generative Recurrent Neural Networks for Learning from fMRI. MACHINE LEARNING IN MEDICAL IMAGING. MLMI (WORKSHOP) 2019; 11861:382-390. [PMID: 32274470 DOI: 10.1007/978-3-030-32692-0_44] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
Recurrent neural networks (RNNs) were designed for dealing with time-series data and have recently been used for creating predictive models from functional magnetic resonance imaging (fMRI) data. However, gathering large fMRI datasets for learning is a difficult task. Furthermore, network interpretability is unclear. To address these issues, we utilize multitask learning and design a novel RNN-based model that learns to discriminate between classes while simultaneously learning to generate the fMRI time-series data. Employing the long short-term memory (LSTM) structure, we develop a discriminative model based on the hidden state and a generative model based on the cell state. The addition of the generative model constrains the network to learn functional communities represented by the LSTM nodes that are both consistent with the data generation as well as useful for the classification task. We apply our approach to the classification of subjects with autism vs. healthy controls using several datasets from the Autism Brain Imaging Data Exchange. Experiments show that our jointly discriminative and generative model improves classification learning while also producing robust and meaningful functional communities for better model understanding.
Collapse
Affiliation(s)
- Nicha C Dvornek
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Xiaoxiao Li
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Juntang Zhuang
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - James S Duncan
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
- Department of Electrical Engineering, Yale University, New Haven, CT, USA
- Department of Statistics and Data Science, Yale University, New Haven, CT, USA
| |
Collapse
|
35
|
Li X, Guo N, Li Q. Functional Neuroimaging in the New Era of Big Data. GENOMICS, PROTEOMICS & BIOINFORMATICS 2019; 17:393-401. [PMID: 31809864 PMCID: PMC6943787 DOI: 10.1016/j.gpb.2018.11.005] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/01/2018] [Revised: 09/17/2018] [Accepted: 12/25/2018] [Indexed: 12/15/2022]
Abstract
The field of functional neuroimaging has substantially advanced as a big data science in the past decade, thanks to international collaborative projects and community efforts. Here we conducted a literature review on functional neuroimaging, with focus on three general challenges in big data tasks: data collection and sharing, data infrastructure construction, and data analysis methods. The review covers a wide range of literature types including perspectives, database descriptions, methodology developments, and technical details. We show how each of the challenges was proposed and addressed, and how these solutions formed the three core foundations for the functional neuroimaging as a big data science and helped to build the current data-rich and data-driven community. Furthermore, based on our review of recent literature on the upcoming challenges and opportunities toward future scientific discoveries, we envisioned that the functional neuroimaging community needs to advance from the current foundations to better data integration infrastructure, methodology development toward improved learning capability, and multi-discipline translational research framework for this new era of big data.
Collapse
Affiliation(s)
- Xiang Li
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA
| | - Ning Guo
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA
| | - Quanzheng Li
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA.
| |
Collapse
|
36
|
Angrick M, Herff C, Mugler E, Tate MC, Slutzky MW, Krusienski DJ, Schultz T. Speech synthesis from ECoG using densely connected 3D convolutional neural networks. J Neural Eng 2019; 16:036019. [PMID: 30831567 PMCID: PMC6822609 DOI: 10.1088/1741-2552/ab0c59] [Citation(s) in RCA: 71] [Impact Index Per Article: 14.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
OBJECTIVE Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. APPROACH Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. MAIN RESULTS In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. SIGNIFICANCE To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.
Collapse
Affiliation(s)
- Miguel Angrick
- Cognitive Systems Lab, University of Bremen, Bremen, Germany
| | | | | | | | | | | | | |
Collapse
|
37
|
Wang L, Li K, Chen X, Hu XP. Application of Convolutional Recurrent Neural Network for Individual Recognition Based on Resting State fMRI Data. Front Neurosci 2019; 13:434. [PMID: 31118882 PMCID: PMC6504790 DOI: 10.3389/fnins.2019.00434] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2018] [Accepted: 04/15/2019] [Indexed: 11/28/2022] Open
Abstract
In most task and resting state fMRI studies, a group consensus is often sought, where individual variability is considered a nuisance. None the less, biological variability is an important factor that cannot be ignored and is gaining more attention in the field. One recent development is the individual identification based on static functional connectome. While the original work was based on the static connectome, subsequent efforts using recurrent neural networks (RNN) demonstrated that the inclusion of temporal features greatly improved identification accuracy. Given that convolutional RNN (ConvRNN) seamlessly integrates spatial and temporal features, the present work applied ConvRNN for individual identification with resting state fMRI data. Our result demonstrates ConvRNN achieving a higher identification accuracy than conventional RNN, likely due to better extraction of local features between neighboring ROIs. Furthermore, given that each convolutional output assembles in-place features, they provide a natural way for us to visualize the informative spatial pattern and temporal information, opening up a promising new avenue for analyzing fMRI data.
Collapse
Affiliation(s)
- Lebo Wang
- Department of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, United States
| | - Kaiming Li
- Department of Bioengineering, University of California, Riverside, Riverside, CA, United States
| | - Xu Chen
- Center for Advanced Neuroimaging, University of California, Riverside, Riverside, CA, United States
| | - Xiaoping P Hu
- Department of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, United States.,Department of Bioengineering, University of California, Riverside, Riverside, CA, United States.,Center for Advanced Neuroimaging, University of California, Riverside, Riverside, CA, United States
| |
Collapse
|
38
|
Wang H, Zhao S, Dong Q, Cui Y, Chen Y, Han J, Xie L, Liu T. Recognizing Brain States Using Deep Sparse Recurrent Neural Network. IEEE TRANSACTIONS ON MEDICAL IMAGING 2019; 38:1058-1068. [PMID: 30369441 PMCID: PMC6508593 DOI: 10.1109/tmi.2018.2877576] [Citation(s) in RCA: 27] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2023]
Abstract
Brain activity is a dynamic combination of different sensory responses and thus brain activity/state is continuously changing over time. However, the brain's dynamical functional states recognition at fast time-scales in task fMRI data have been rarely explored. In this paper, we propose a novel 5-layer deep sparse recurrent neural network (DSRNN) model to accurately recognize the brain states across the whole scan session. Specifically, the DSRNN model includes an input layer, one fully-connected layer, two recurrent layers, and a softmax output layer. The proposed framework has been tested on seven task fMRI data sets of Human Connectome Project. Extensive experiment results demonstrate that the proposed DSRNN model can accurately identify the brain's state in different task fMRI data sets and significantly outperforms other auto-correlation methods or non-temporal approaches in the dynamic brain state recognition accuracy. In general, the proposed DSRNN offers a new methodology for basic neuroscience and clinical research.
Collapse
|
39
|
Lage-Castellanos A, Valente G, Formisano E, De Martino F. Methods for computing the maximum performance of computational models of fMRI responses. PLoS Comput Biol 2019; 15:e1006397. [PMID: 30849071 PMCID: PMC6426260 DOI: 10.1371/journal.pcbi.1006397] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Revised: 03/20/2019] [Accepted: 01/17/2019] [Indexed: 11/19/2022] Open
Abstract
Computational neuroimaging methods aim to predict brain responses (measured e.g. with functional magnetic resonance imaging [fMRI]) on the basis of stimulus features obtained through computational models. The accuracy of such prediction is used as an indicator of how well the model describes the computations underlying the brain function that is being considered. However, the prediction accuracy is bounded by the proportion of the variance of the brain response which is related to the measurement noise and not to the stimuli (or cognitive functions). This bound to the performance of a computational model has been referred to as the noise ceiling. In previous fMRI applications two methods have been proposed to estimate the noise ceiling based on either a split-half procedure or Monte Carlo simulations. These methods make different assumptions over the nature of the effects underlying the data, and, importantly, their relation has not been clarified yet. Here, we derive an analytical form for the noise ceiling that does not require computationally expensive simulations or a splitting procedure that reduce the amount of data. The validity of this analytical definition is proved in simulations, we show that the analytical solution results in the same estimate of the noise ceiling as the Monte Carlo method. Considering different simulated noise structure, we evaluate different estimators of the variance of the responses and their impact on the estimation of the noise ceiling. We furthermore evaluate the interplay between regularization (often used to estimate model fits to the data when the number of computational features in the model is large) and model complexity on the performance with respect to the noise ceiling. Our results indicate that when considering the variance of the responses across runs, computing the noise ceiling analytically results in similar estimates as the split half estimator and approaches the true noise ceiling under a variety of simulated noise scenarios. Finally, the methods are tested on real fMRI data acquired at 7 Tesla. Encoding computational models in brain responses measured with fMRI allows testing the algorithmic representations carried out by the neural population within voxels. The accuracy of a model in predicting new responses is used as a measure of the brain validity of the computational model being tested, but the result of this analysis is determined not only by how precisely the model describes the responses but also by the quality of the data. In this article, we evaluate existing approaches to estimate the best possible accuracy that any computational model can achieve conditioned to the amount of measurement noise that is present in the experimental data (i.e. the noise ceiling). Additionally we introduce a close form estimation of the noise ceiling that does not require computationally or data expensive procedures. All the methods are compared using simulated and real fMRI data. We draw conclusions over the impact of regularization procedures and make practical recommendations on how to report the results of computational models in neuroimaging.
Collapse
Affiliation(s)
- Agustin Lage-Castellanos
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
- Department of NeuroInformatics, Cuban Center for Neuroscience, Cuba
- * E-mail:
| | - Giancarlo Valente
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Elia Formisano
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
- Maastricht Centre for Systems Biology (MaCSBio), Maastricht University, Maastricht, The Netherlands
| | - Federico De Martino
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
- Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, Minneapolis, United States of America
| |
Collapse
|
40
|
Guang H, Ji L. Proprioceptive Recognition with Artificial Neural Networks Based on Organizations of Spinocerebellar Tract and Cerebellum. Int J Neural Syst 2019; 29:1850056. [PMID: 30776987 DOI: 10.1142/s0129065718500569] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Muscle kinematics and kinetics are nonlinearly encoded by proprioceptors, and the changes in muscle length and velocity are integrated into Ia afferent. Besides, proprioceptive signals from multiple muscles are probably mixed in afferent pathways, which all lead to difficulties in proprioceptive recognition for the cerebellum. In this study, the artificial neural networks, whose organizations are biologically based on the spinocerebellar tract and cerebellum, are utilized to decode the proprioceptive signals. Consistent with the controversy of the proprioceptive division in the dorsal spinocerebellar tract, the spinocerebellar tract networks incorporated two distinct inferences, (1) the centralized networks, which mixed Ia, II, and Ib and processed them together; (2) the decentralized networks, which processed Ia, II, and Ib afferents separately. The cerebellar networks were based on the Marr-Albus model to recognize the kinematic states. The networks were trained by a specific movement, and the trained networks were subsequently required to predict kinematic states of six movements. The results demonstrated that the centralized networks, which were more consistent with the physiological findings in recent years, had better recognition accuracy than the decentralized networks, and the networks were still effective even when proprioceptive afferents from multiple muscles were integrated.
Collapse
Affiliation(s)
- Hui Guang
- 1Department of Mechanical Engineering, Tsinghua University, Beijing 100084, P. R. China
| | - Linhong Ji
- 1Department of Mechanical Engineering, Tsinghua University, Beijing 100084, P. R. China
| |
Collapse
|
41
|
Wang H, Xie K, Lian Z, Cui Y, Chen Y, Zhang J, Xie L, Tsien J, Liu T. Large-Scale Circuitry Interactions Upon Earthquake Experiences Revealed by Recurrent Neural Networks. IEEE Trans Neural Syst Rehabil Eng 2018; 26:2115-2125. [PMID: 30296236 PMCID: PMC6298947 DOI: 10.1109/tnsre.2018.2872919] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Brain dynamics has recently received increasing interest due to its significant importance in basic and clinical neurosciences. However, due to inherent difficulties in both data acquisition and data analysis methods, studies on large-scale brain dynamics of mouse with local field potential (LFP) recording are very rare. In this paper, we did a series of works on modeling large-scale mouse brain dynamic activities responding to fearful earthquake. Based on LFP recording data from 13 brain regions that are closely related to fear learning and memory and the effective Bayesian connectivity change point model, we divided the response time series into four stages: "Before," "Earthquake," "Recovery," and "After." We first reported the changes in power and theta-gamma coupling during stage transitions. Then, a recurrent neural network model was designed to model the functional dynamics in these thirteen brain regions and six frequency bands in response to the fear stimulus. Interestingly, our results showed that the functional brain connectivities in theta and gamma bands exhibited distinct response processes: in theta band, there is a separated-united-separated alternation in whole-brain connectivity and a low-high-low change in connectivity strength; however, gamma bands have a united-separated-united transition and a high-low-high alternation in connectivity pattern and strength. In general, our results offer a novel perspective in studying functional brain dynamics under fearful stimulus and reveal its relationship to the brain's structural connectivity substrates.
Collapse
Affiliation(s)
- Han Wang
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, China (
| | - Kun Xie
- The Brain Decoding Center, Banna Biomedical Research Institute, Yunnan Academy of Science and Technology, Yunnan, China; and Brain and Behavior Discovery Institute, Medical College of Georgia at Augusta University, Augusta, GA, USA ()
| | - Zhichao Lian
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China ()
| | - Yan Cui
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, China )
| | - Yaowu Chen
- Zhejiang Provincial Key Laboratory for Network Multimedia Technologies, Hangzhou, China; and Zhejiang University Embedded System Engineering Research Center, Ministry of Education of China, Hangzhou, China ()
| | - Jing Zhang
- Department of Math and Statistics, Georgia State University, Atlanta, GA ()
| | - Li Xie
- State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou, China ()
| | - Joe Tsien
- Brain and Behavior Discovery Institute, Medical College of Georgia, Augusta University, Augusta, GA, USA ()
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, 30602 USA (phone: (706) 542-3478; )
| |
Collapse
|
42
|
Hjelm RD, Damaraju E, Cho K, Laufs H, Plis SM, Calhoun VD. Spatio-Temporal Dynamics of Intrinsic Networks in Functional Magnetic Imaging Data Using Recurrent Neural Networks. Front Neurosci 2018; 12:600. [PMID: 30294250 PMCID: PMC6158311 DOI: 10.3389/fnins.2018.00600] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2018] [Accepted: 08/09/2018] [Indexed: 11/18/2022] Open
Abstract
We introduce a novel recurrent neural network (RNN) approach to account for temporal dynamics and dependencies in brain networks observed via functional magnetic resonance imaging (fMRI). Our approach directly parameterizes temporal dynamics through recurrent connections, which can be used to formulate blind source separation with a conditional (rather than marginal) independence assumption, which we call RNN-ICA. This formulation enables us to visualize the temporal dynamics of both first order (activity) and second order (directed connectivity) information in brain networks that are widely studied in a static sense, but not well-characterized dynamically. RNN-ICA predicts dynamics directly from the recurrent states of the RNN in both task and resting state fMRI. Our results show both task-related and group-differentiating directed connectivity.
Collapse
Affiliation(s)
- R Devon Hjelm
- Montréal Institute for Learning Algorithms, Montreal, QC, Canada.,Microsoft Research, Montreal, QC, Canada
| | - Eswar Damaraju
- The Mind Research Network, Albuquerque, NM, United States.,The University of New Mexico, Albuquerque, NM, United States
| | | | | | - Sergey M Plis
- The Mind Research Network, Albuquerque, NM, United States
| | - Vince D Calhoun
- The Mind Research Network, Albuquerque, NM, United States.,The University of New Mexico, Albuquerque, NM, United States
| |
Collapse
|
43
|
Dvornek NC, Yang D, Ventola P, Duncan JS. Learning Generalizable Recurrent Neural Networks from Small Task-fMRI Datasets. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2018; 11072:329-337. [PMID: 30873514 DOI: 10.1007/978-3-030-00931-1_38] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
Deep learning has become the new state-of-the-art for many problems in image analysis. However, large datasets are often required for such deep networks to learn effectively. This poses a difficult challenge for many medical image analysis problems in which only a small number of subjects are available, e.g., patients undergoing a new treatment. In this work, we propose a number of approaches for learning generalizable recurrent neural networks from smaller task-fMRI datasets: 1) a resampling method for ROI-based fMRI analysis to create augmented data; 2) inclusion of a small number of non-imaging variables to provide subject-specific initialization of the recurrent neural network; and 3) selection of the most generalizable model from multiple reinitialized training runs using criteria based on only training loss. Using cross-validation to assess model performance, we demonstrate the effectiveness of the proposed methods to train recurrent neural networks from small datasets to predict treatment outcome for children with autism spectrum disorder (N = 21) and classify autistic vs. typical control subjects (N = 40) from task-fMRI scans.
Collapse
Affiliation(s)
- Nicha C Dvornek
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Daniel Yang
- Autism and Neurodevelopmental Disorders Institute, George Washington University and Children's National Health System, Washington, DC, USA
| | - Pamela Ventola
- Child Study Center, Yale School of Medicine, New Haven, CT, USA
| | - James S Duncan
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA.,Department of Biomedical Engineering, Yale University, New Haven, CT, USA.,Department of Electrical Engineering, Yale University, New Haven, CT, USA
| |
Collapse
|
44
|
Wang Y, Wang Y, Lui YW. Generalized Recurrent Neural Network accommodating Dynamic Causal Modeling for functional MRI analysis. Neuroimage 2018; 178:385-402. [PMID: 29782993 DOI: 10.1016/j.neuroimage.2018.05.042] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 04/23/2018] [Accepted: 05/17/2018] [Indexed: 11/16/2022] Open
Abstract
Dynamic Causal Modeling (DCM) is an advanced biophysical model which explicitly describes the entire process from experimental stimuli to functional magnetic resonance imaging (fMRI) signals via neural activity and cerebral hemodynamics. To conduct a DCM study, one needs to represent the experimental stimuli as a compact vector-valued function of time, which is hard in complex tasks such as book reading and natural movie watching. Deep learning provides the state-of-the-art signal representation solution, encoding complex signals into compact dense vectors while preserving the essence of the original signals. There is growing interest in using Recurrent Neural Networks (RNNs), a major family of deep learning techniques, in fMRI modeling. However, the generic RNNs used in existing studies work as black boxes, making the interpretation of results in a neuroscience context difficult and obscure. In this paper, we propose a new biophysically interpretable RNN built on DCM, DCM-RNN. We generalize the vanilla RNN and show that DCM can be cast faithfully as a special form of the generalized RNN. DCM-RNN uses back propagation for parameter estimation. We believe DCM-RNN is a promising tool for neuroscience. It can fit seamlessly into classical DCM studies. We demonstrate face validity of DCM-RNN in two principal applications of DCM: causal brain architecture hypotheses testing and effective connectivity estimation. We also demonstrate construct validity of DCM-RNN in an attention-visual experiment. Moreover, DCM-RNN enables end-to-end training of DCM and representation learning deep neural networks, extending DCM studies to complex tasks.
Collapse
Affiliation(s)
- Yuan Wang
- NYU WIRELESS, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA.
| | - Yao Wang
- NYU WIRELESS, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA
| | - Yvonne W Lui
- Center for Advanced Imaging Innovation and Research (CAI2R), School of Medicine, New York University, 660 First Avenue, New York, NY 10016, USA; Bernard and Irene Schwartz Center for Biomedical Imaging, School of Medicine, New York University, 660 First Avenue, New York, NY 10016, USA
| |
Collapse
|
45
|
Shi J, Wen H, Zhang Y, Han K, Liu Z. Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision. Hum Brain Mapp 2018; 39:2269-2282. [PMID: 29436055 PMCID: PMC5895512 DOI: 10.1002/hbm.24006] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2017] [Revised: 12/15/2017] [Accepted: 02/06/2018] [Indexed: 02/05/2023] Open
Abstract
The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.
Collapse
Affiliation(s)
- Junxing Shi
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, 47906
- Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, Indiana, 47906
| | - Haiguang Wen
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, 47906
- Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, Indiana, 47906
| | - Yizhen Zhang
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, 47906
- Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, Indiana, 47906
| | - Kuan Han
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, 47906
- Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, Indiana, 47906
| | - Zhongming Liu
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, 47906
- Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, Indiana, 47906
- Weldon School of Biomedical Engineering, Purdue University, West Lafayette, Indiana, 47906
| |
Collapse
|
46
|
Chen S, Hu X. Individual Identification Using the Functional Brain Fingerprint Detected by the Recurrent Neural Network. Brain Connect 2018; 8:197-204. [PMID: 29634323 DOI: 10.1089/brain.2017.0561] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
Individual identification based on brain function has gained traction in literature. Investigating individual differences in brain function can provide additional insights into the brain. In this work, we introduce a recurrent neural network-based model for identifying individuals based on only a short segment of resting-state functional magnetic resonance imaging data. In addition, we demonstrate how the global signal and differences in atlases affect individual identifiability. Furthermore, we investigate neural network features that exhibit the uniqueness of each individual. The results indicate that our model is able to identify individuals based on neural features and provides additional information regarding brain dynamics.
Collapse
Affiliation(s)
- Shiyang Chen
- 1 Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University , Atlanta, Georgia
| | - Xiaoping Hu
- 2 Department of Bioengineering, University of California , Riverside, Riverside, California
| |
Collapse
|
47
|
van Gerven M. Computational Foundations of Natural Intelligence. Front Comput Neurosci 2017; 11:112. [PMID: 29375355 PMCID: PMC5770642 DOI: 10.3389/fncom.2017.00112] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2017] [Accepted: 11/22/2017] [Indexed: 01/14/2023] Open
Abstract
New developments in AI and neuroscience are revitalizing the quest to understanding natural intelligence, offering insight about how to equip machines with human-like capabilities. This paper reviews some of the computational principles relevant for understanding natural intelligence and, ultimately, achieving strong AI. After reviewing basic principles, a variety of computational modeling approaches is discussed. Subsequently, I concentrate on the use of artificial neural networks as a framework for modeling cognitive processes. This paper ends by outlining some of the challenges that remain to fulfill the promise of machines that show human-like intelligence.
Collapse
Affiliation(s)
- Marcel van Gerven
- Computational Cognitive Neuroscience Lab, Department of Artificial Intelligence, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
48
|
How Artificial Intelligence is Supporting Neuroscience Research: A Discussion About Foundations, Methods and Applications. COMMUNICATIONS IN COMPUTER AND INFORMATION SCIENCE 2017. [DOI: 10.1007/978-3-319-71011-2_6] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|