1
|
Pomi A, Lin J, Mizraji E. A memory access gate controlled by dynamic contexts. Biosystems 2024; 241:105232. [PMID: 38754622 DOI: 10.1016/j.biosystems.2024.105232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2024] [Revised: 05/08/2024] [Accepted: 05/09/2024] [Indexed: 05/18/2024]
Abstract
Temporary difficulties in accessing the contents of memories are a common experience in everyday life, for example, when we try to recognize a known person in an unusual context. In addition, recent experiments seem to indicate that retrograde amnesia in the early stages of Alzheimer's disease is due to disorders in accessing memories that were installed normally. These facts suggest the existence of an intermediate step between the stimulus arrival and the associative recognition. In this work, a multimodular neurocomputational model is presented postulating the existence of a neural gate that controls the access of the stimulus with its context to the consolidated memory. If recognition is not achieved, a random search is initiated in a contextual network aroused by the initial context. The search continues until the appropriate context that allows for recognition is found or until the process is turned off because the initial stimulus is no longer maintained in the working memory. The model is based on vector patterns of neural activity and context-dependent matrix memories. Simple Markov chain simulations are presented to exemplify possible search scenarios in the contextual network. Finally, we discuss some of the characteristics of the model and the phenomenon under study.
Collapse
Affiliation(s)
- Andrés Pomi
- Group of Cognitive Systems Modeling, Biophysics and Systems Biology Section, Facultad de Ciencias, Universidad de la República, Montevideo 11400, Uruguay.
| | - Juan Lin
- Group of Cognitive Systems Modeling, Biophysics and Systems Biology Section, Facultad de Ciencias, Universidad de la República, Montevideo 11400, Uruguay; Physics Department, Washington College, Chestertown, MD, 21620, USA
| | - Eduardo Mizraji
- Group of Cognitive Systems Modeling, Biophysics and Systems Biology Section, Facultad de Ciencias, Universidad de la República, Montevideo 11400, Uruguay
| |
Collapse
|
2
|
Kumar S, Sumers TR, Yamakoshi T, Goldstein A, Hasson U, Norman KA, Griffiths TL, Hawkins RD, Nastase SA. Shared functional specialization in transformer-based language models and the human brain. Nat Commun 2024; 15:5523. [PMID: 38951520 PMCID: PMC11217339 DOI: 10.1038/s41467-024-49173-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2023] [Accepted: 05/24/2024] [Indexed: 07/03/2024] Open
Abstract
When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space.
Collapse
Affiliation(s)
- Sreejan Kumar
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA.
| | - Theodore R Sumers
- Department of Computer Science, Princeton University, Princeton, NJ, 08540, USA.
| | - Takateru Yamakoshi
- Faculty of Medicine, The University of Tokyo, Bunkyo-ku, Tokyo, 113-0033, Japan
| | - Ariel Goldstein
- Department of Cognitive and Brain Sciences and Business School, Hebrew University, Jerusalem, 9190401, Israel
| | - Uri Hasson
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA
- Department of Psychology, Princeton University, Princeton, NJ, 08540, USA
| | - Kenneth A Norman
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA
- Department of Psychology, Princeton University, Princeton, NJ, 08540, USA
| | - Thomas L Griffiths
- Department of Computer Science, Princeton University, Princeton, NJ, 08540, USA
- Department of Psychology, Princeton University, Princeton, NJ, 08540, USA
| | - Robert D Hawkins
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA
- Department of Psychology, Princeton University, Princeton, NJ, 08540, USA
| | - Samuel A Nastase
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA.
| |
Collapse
|
3
|
Ferrante M, Boccato T, Passamonti L, Toschi N. Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model. J Neural Eng 2024; 21:046001. [PMID: 38885689 DOI: 10.1088/1741-2552/ad593c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 06/17/2024] [Indexed: 06/20/2024]
Abstract
Objective.Brain decoding is a field of computational neuroscience that aims to infer mental states or internal representations of perceptual inputs from measurable brain activity. This study proposes a novel approach to brain decoding that relies on semantic and contextual similarity.Approach.We use several functional magnetic resonance imaging (fMRI) datasets of natural images as stimuli and create a deep learning decoding pipeline inspired by the bottom-up and top-down processes in human vision. Our pipeline includes a linear brain-to-feature model that maps fMRI activity to semantic visual stimuli features. We assume that the brain projects visual information onto a space that is homeomorphic to the latent space of last layer of a pretrained neural network, which summarizes and highlights similarities and differences between concepts. These features are categorized in the latent space using a nearest-neighbor strategy, and the results are used to retrieve images or condition a generative latent diffusion model to create novel images.Main results.We demonstrate semantic classification and image retrieval on three different fMRI datasets: Generic Object Decoding (vision perception and imagination), BOLD5000, and NSD. In all cases, a simple mapping between fMRI and a deep semantic representation of the visual stimulus resulted in meaningful classification and retrieved or generated images. We assessed quality using quantitative metrics and a human evaluation experiment that reproduces the multiplicity of conscious and unconscious criteria that humans use to evaluate image similarity. Our method achieved correct evaluation in over 80% of the test set.Significance.Our study proposes a novel approach to brain decoding that relies on semantic and contextual similarity. The results demonstrate that measurable neural correlates can be linearly mapped onto the latent space of a neural network to synthesize images that match the original content. These findings have implications for both cognitive neuroscience and artificial intelligence.
Collapse
Affiliation(s)
- Matteo Ferrante
- Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Rome, Italy
| | - Tommaso Boccato
- Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Rome, Italy
| | - Luca Passamonti
- CNR, Istituto di Bioimmagini e Fisiologia Molecolare, Milan, Italy
| | - Nicola Toschi
- Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Rome, Italy
- Martinos Center for Biomedical Imaging, MGH and Harvard Medical School, Boston, MA, United States of America
| |
Collapse
|
4
|
Wang R, Chen ZS. Large-scale foundation models and generative AI for BigData neuroscience. Neurosci Res 2024:S0168-0102(24)00075-0. [PMID: 38897235 DOI: 10.1016/j.neures.2024.06.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2023] [Revised: 04/15/2024] [Accepted: 05/15/2024] [Indexed: 06/21/2024]
Abstract
Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.
Collapse
Affiliation(s)
- Ran Wang
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA
| | - Zhe Sage Chen
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Neuroscience and Physiology, Neuroscience Institute, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Biomedical Engineering, New York University Tandon School of Engineering, Brooklyn, NY 11201, USA.
| |
Collapse
|
5
|
Lützow Holm E, Fernández Slezak D, Tagliazucchi E. Contribution of low-level image statistics to EEG decoding of semantic content in multivariate and univariate models with feature optimization. Neuroimage 2024; 293:120626. [PMID: 38677632 DOI: 10.1016/j.neuroimage.2024.120626] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 04/23/2024] [Accepted: 04/24/2024] [Indexed: 04/29/2024] Open
Abstract
Spatio-temporal patterns of evoked brain activity contain information that can be used to decode and categorize the semantic content of visual stimuli. However, this procedure can be biased by low-level image features independently of the semantic content present in the stimuli, prompting the need to understand the robustness of different models regarding these confounding factors. In this study, we trained machine learning models to distinguish between concepts included in the publicly available THINGS-EEG dataset using electroencephalography (EEG) data acquired during a rapid serial visual presentation paradigm. We investigated the contribution of low-level image features to decoding accuracy in a multivariate model, utilizing broadband data from all EEG channels. Additionally, we explored a univariate model obtained through data-driven feature selection applied to the spatial and frequency domains. While the univariate models exhibited better decoding accuracy, their predictions were less robust to the confounding effect of low-level image statistics. Notably, some of the models maintained their accuracy even after random replacement of the training dataset with semantically unrelated samples that presented similar low-level content. In conclusion, our findings suggest that model optimization impacts sensitivity to confounding factors, regardless of the resulting classification performance. Therefore, the choice of EEG features for semantic decoding should ideally be informed by criteria beyond classifier performance, such as the neurobiological mechanisms under study.
Collapse
Affiliation(s)
- Eric Lützow Holm
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina.
| | - Diego Fernández Slezak
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Departamento de Computación, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina
| | - Enzo Tagliazucchi
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Latin American Brain Health (BrainLat), Universidad Adolfo Ibáñez, Av. Diag. Las Torres 2640, Peñalolén 7941169, Santiago Región Metropolitana, Chile.
| |
Collapse
|
6
|
Wajnerman-Paz A, Aboitiz F, Álamos F, Ramos Vergara P. A healthcare approach to mental integrity. JOURNAL OF MEDICAL ETHICS 2024:jme-2023-109682. [PMID: 38802142 DOI: 10.1136/jme-2023-109682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Accepted: 04/17/2024] [Indexed: 05/29/2024]
Abstract
The current human rights framework can shield people from many of the risks associated with neurotechnological applications. However, it has been argued that we need either to articulate new rights or reconceptualise existing ones in order to prevent some of these risks. In this paper, we would like to address the recent discussion about whether current reconceptualisations of the right to mental integrity identify an ethical dimension that is not covered by existing moral and/or legal rights. The main challenge of these proposals is that they make mental integrity indistinguishable from autonomy. They define mental integrity in terms of the control we can have over our mental states, which seems to be part of the authenticity condition for autonomous action. Based on a fairly comprehensive notion of mental health (ie, a notion that is not limited to the mere absence of illness), we propose an alternative view according to which mental integrity can be characterised both as a positive right to (medical and non-medical) interventions that restore and sustain mental and neural function, and promote its development and a negative right protecting people from interventions that threaten or undermine these functions or their development. We will argue that this notion is dissociated from cognitive control and therefore can be adequately distinguished from autonomy.
Collapse
Affiliation(s)
- Abel Wajnerman-Paz
- Instituto de Éticas Aplicadas, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Francisco Aboitiz
- Centro Interdisciplinario de Neurociencia, Departamento de Psiquiatría, Facultad de Medicina, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Florencia Álamos
- Centro de Bioética, Facultad de Medicina, Centro Interdisciplinario de Neurociencia, Pontificia Universidad Catolica de Chile, Santiago, Chile
| | - Paulina Ramos Vergara
- Centro de Bioética, Facultad de Medicina, Pontificia Universidad Catolica de Chile, Santiago, Chile
| |
Collapse
|
7
|
Caplette L, Turk-Browne NB. Computational reconstruction of mental representations using human behavior. Nat Commun 2024; 15:4183. [PMID: 38760341 PMCID: PMC11101448 DOI: 10.1038/s41467-024-48114-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Accepted: 04/19/2024] [Indexed: 05/19/2024] Open
Abstract
Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.
Collapse
Affiliation(s)
| | - Nicholas B Turk-Browne
- Department of Psychology, Yale University, New Haven, CT, USA
- Wu Tsai Institute, Yale University, New Haven, CT, USA
| |
Collapse
|
8
|
Bo K, Kraynak TE, Kwon M, Sun M, Gianaros PJ, Wager TD. A systems identification approach using Bayes factors to deconstruct the brain bases of emotion regulation. Nat Neurosci 2024; 27:975-987. [PMID: 38519748 DOI: 10.1038/s41593-024-01605-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 02/15/2024] [Indexed: 03/25/2024]
Abstract
Cognitive reappraisal is fundamental to cognitive therapies and everyday emotion regulation. Analyses using Bayes factors and an axiomatic systems identification approach identified four reappraisal-related components encompassing distributed neural activity patterns across two independent functional magnetic resonance imaging (fMRI) studies (n = 182 and n = 176): (1) an anterior prefrontal system selectively involved in cognitive reappraisal; (2) a fronto-parietal-insular system engaged by both reappraisal and emotion generation, demonstrating a general role in appraisal; (3) a largely subcortical system activated during negative emotion generation but unaffected by reappraisal, including amygdala, hypothalamus and periaqueductal gray; and (4) a posterior cortical system of negative emotion-related regions downregulated by reappraisal. These systems covaried with individual differences in reappraisal success and were differentially related to neurotransmitter binding maps, implicating cannabinoid and serotonin systems in reappraisal. These findings challenge 'limbic'-centric models of reappraisal and provide new systems-level targets for assessing and enhancing emotion regulation.
Collapse
Affiliation(s)
- Ke Bo
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Thomas E Kraynak
- Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA, USA
| | - Mijin Kwon
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Michael Sun
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Peter J Gianaros
- Department of Psychology, University of Pittsburgh, Pittsburgh, PA, USA.
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| |
Collapse
|
9
|
Heinen R, Bierbrauer A, Wolf OT, Axmacher N. Representational formats of human memory traces. Brain Struct Funct 2024; 229:513-529. [PMID: 37022435 PMCID: PMC10978732 DOI: 10.1007/s00429-023-02636-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 03/28/2023] [Indexed: 04/07/2023]
Abstract
Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.
Collapse
Affiliation(s)
- Rebekka Heinen
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany.
| | - Anne Bierbrauer
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
- Institute for Systems Neuroscience, Medical Center Hamburg-Eppendorf, Martinistraße 52, 20251, Hamburg, Germany
| | - Oliver T Wolf
- Department of Cognitive Psychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| | - Nikolai Axmacher
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| |
Collapse
|
10
|
Li R, Li J, Wang C, Liu H, Liu T, Wang X, Zou T, Huang W, Yan H, Chen H. Multi-Semantic Decoding of Visual Perception with Graph Neural Networks. Int J Neural Syst 2024; 34:2450016. [PMID: 38372016 DOI: 10.1142/s0129065724500163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Constructing computational decoding models to account for the cortical representation of semantic information plays a crucial role in understanding visual perception. The human visual system processes interactive relationships among different objects when perceiving the semantic contents of natural visions. However, the existing semantic decoding models commonly regard categories as completely separate and independent visually and semantically and rarely consider the relationships from prior information. In this work, a novel semantic graph learning model was proposed to decode multiple semantic categories of perceived natural images from brain activity. The proposed model was validated on the functional magnetic resonance imaging data collected from five normal subjects while viewing 2750 natural images comprising 52 semantic categories. The results showed that the Graph Neural Network-based decoding model achieved higher accuracies than other deep neural network models. Moreover, the co-occurrence probability among semantic categories showed a significant correlation with the decoding accuracy. Additionally, the results suggested that semantic content organized in a hierarchical way with higher visual areas was more closely related to the internal visual experience. Together, this study provides a superior computational framework for multi-semantic decoding that supports the visual integration mechanism of semantic processing.
Collapse
Affiliation(s)
- Rong Li
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Jiyi Li
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Chong Wang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Haoxiang Liu
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Tao Liu
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Xuyang Wang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Ting Zou
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Wei Huang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Hongmei Yan
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Huafu Chen
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| |
Collapse
|
11
|
Noda T, Aschauer DF, Chambers AR, Seiler JPH, Rumpel S. Representational maps in the brain: concepts, approaches, and applications. Front Cell Neurosci 2024; 18:1366200. [PMID: 38584779 PMCID: PMC10995314 DOI: 10.3389/fncel.2024.1366200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Accepted: 03/08/2024] [Indexed: 04/09/2024] Open
Abstract
Neural systems have evolved to process sensory stimuli in a way that allows for efficient and adaptive behavior in a complex environment. Recent technological advances enable us to investigate sensory processing in animal models by simultaneously recording the activity of large populations of neurons with single-cell resolution, yielding high-dimensional datasets. In this review, we discuss concepts and approaches for assessing the population-level representation of sensory stimuli in the form of a representational map. In such a map, not only are the identities of stimuli distinctly represented, but their relational similarity is also mapped onto the space of neuronal activity. We highlight example studies in which the structure of representational maps in the brain are estimated from recordings in humans as well as animals and compare their methodological approaches. Finally, we integrate these aspects and provide an outlook for how the concept of representational maps could be applied to various fields in basic and clinical neuroscience.
Collapse
Affiliation(s)
- Takahiro Noda
- Institute of Physiology, Focus Program Translational Neurosciences, University Medical Center, Johannes Gutenberg University-Mainz, Mainz, Germany
| | - Dominik F. Aschauer
- Institute of Physiology, Focus Program Translational Neurosciences, University Medical Center, Johannes Gutenberg University-Mainz, Mainz, Germany
| | - Anna R. Chambers
- Department of Otolaryngology – Head and Neck Surgery, Harvard Medical School, Boston, MA, United States
- Eaton Peabody Laboratories, Massachusetts Eye and Ear Infirmary, Boston, MA, United States
| | - Johannes P.-H. Seiler
- Institute of Physiology, Focus Program Translational Neurosciences, University Medical Center, Johannes Gutenberg University-Mainz, Mainz, Germany
| | - Simon Rumpel
- Institute of Physiology, Focus Program Translational Neurosciences, University Medical Center, Johannes Gutenberg University-Mainz, Mainz, Germany
| |
Collapse
|
12
|
Keller TA, Mason RA, Legg AE, Just MA. The neural and cognitive basis of expository text comprehension. NPJ SCIENCE OF LEARNING 2024; 9:21. [PMID: 38514702 PMCID: PMC10957871 DOI: 10.1038/s41539-024-00232-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 02/26/2024] [Indexed: 03/23/2024]
Abstract
As science and technology rapidly progress, it becomes increasingly important to understand how individuals comprehend expository technical texts that explain these advances. This study examined differences in individual readers' technical comprehension performance and differences among texts, using functional brain imaging to measure regional brain activity while students read passages on technical topics and then took a comprehension test. Better comprehension of the technical passages was related to higher activation in regions of the left inferior frontal gyrus, left superior parietal lobe, bilateral dorsolateral prefrontal cortex, and bilateral hippocampus. These areas are associated with the construction of a mental model of the passage and with the integration of new and prior knowledge in memory. Poorer comprehension of the passages was related to greater activation of the ventromedial prefrontal cortex and the precuneus, areas involved in autobiographical and episodic memory retrieval. More comprehensible passages elicited more brain activation associated with establishing links among different types of information in the text and activation associated with establishing conceptual coherence within the text representation. These findings converge with previous behavioral research in their implications for teaching technical learners to become better comprehenders and for improving the structure of instructional texts, to facilitate scientific and technological comprehension.
Collapse
Affiliation(s)
- Timothy A Keller
- Center for Cognitive Brain Imaging, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, 15213, USA.
| | - Robert A Mason
- Center for Cognitive Brain Imaging, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, 15213, USA
| | - Aliza E Legg
- Center for Cognitive Brain Imaging, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, 15213, USA
| | - Marcel Adam Just
- Center for Cognitive Brain Imaging, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, 15213, USA
| |
Collapse
|
13
|
Lu Z, Golomb JD. Human EEG and artificial neural networks reveal disentangled representations of object real-world size in natural images. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.08.19.553999. [PMID: 37662197 PMCID: PMC10473678 DOI: 10.1101/2023.08.19.553999] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Remarkably, human brains have the ability to accurately perceive and process the real-world size of objects, despite vast differences in distance and perspective. While previous studies have delved into this phenomenon, distinguishing this ability from other visual perceptions, like depth, has been challenging. Using the THINGS EEG2 dataset with high time-resolution human brain recordings and more ecologically valid naturalistic stimuli, our study uses an innovative approach to disentangle neural representations of object real-world size from retinal size and perceived real-world depth in a way that was not previously possible. Leveraging this state-of-the-art dataset, our EEG representational similarity results reveal a pure representation of object real-world size in human brains. We report a representational timeline of visual object processing: object real-world depth appeared first, then retinal size, and finally, real-world size. Additionally, we input both these naturalistic images and object-only images without natural background into artificial neural networks. Consistent with the human EEG findings, we also successfully disentangled representation of object real-world size from retinal size and real-world depth in all three types of artificial neural networks (visual-only ResNet, visual-language CLIP, and language-only Word2Vec). Moreover, our multi-modal representational comparison framework across human EEG and artificial neural networks reveals real-world size as a stable and higher-level dimension in object space incorporating both visual and semantic information. Our research provides a detailed and clear characterization of the object processing process, which offers further advances and insights into our understanding of object space and the construction of more brain-like visual models.
Collapse
|
14
|
Contier O, Baker CI, Hebart MN. Distributed representations of behaviorally-relevant object dimensions in the human visual system. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.08.23.553812. [PMID: 37662312 PMCID: PMC10473665 DOI: 10.1101/2023.08.23.553812] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of behaviorally-relevant dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behaviorally-relevant dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.
Collapse
Affiliation(s)
- O Contier
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Max Planck School of Cognition, Leipzig, Germany
| | - C I Baker
- Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda MD, USA
| | - M N Hebart
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Department of Medicine, Justus Liebig University Giessen, Giessen, Germany
| |
Collapse
|
15
|
Stoinski LM, Perkuhn J, Hebart MN. THINGSplus: New norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images. Behav Res Methods 2024; 56:1583-1603. [PMID: 37095326 PMCID: PMC10991023 DOI: 10.3758/s13428-023-02110-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/13/2023] [Indexed: 04/26/2023]
Abstract
To study visual and semantic object representations, the need for well-curated object concepts and images has grown significantly over the past years. To address this, we have previously developed THINGS, a large-scale database of 1854 systematically sampled object concepts with 26,107 high-quality naturalistic images of these concepts. With THINGSplus, we significantly extend THINGS by adding concept- and image-specific norms and metadata for all 1854 concepts and one copyright-free image example per concept. Concept-specific norms were collected for the properties of real-world size, manmadeness, preciousness, liveliness, heaviness, naturalness, ability to move or be moved, graspability, holdability, pleasantness, and arousal. Further, we provide 53 superordinate categories as well as typicality ratings for all their members. Image-specific metadata includes a nameability measure, based on human-generated labels of the objects depicted in the 26,107 images. Finally, we identified one new public domain image per concept. Property (M = 0.97, SD = 0.03) and typicality ratings (M = 0.97, SD = 0.01) demonstrate excellent consistency, with the subsequently collected arousal ratings as the only exception (r = 0.69). Our property (M = 0.85, SD = 0.11) and typicality (r = 0.72, 0.74, 0.88) data correlated strongly with external norms, again with the lowest validity for arousal (M = 0.41, SD = 0.08). To summarize, THINGSplus provides a large-scale, externally validated extension to existing object norms and an important extension to THINGS, allowing detailed selection of stimuli and control variables for a wide range of research interested in visual object processing, language, and semantic memory.
Collapse
Affiliation(s)
- Laura M Stoinski
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany.
| | - Jonas Perkuhn
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany
| | - Martin N Hebart
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany
- Justus Liebig University, Gießen, Germany
| |
Collapse
|
16
|
Eisenhauer S, Gonzalez Alam TRDJ, Cornelissen PL, Smallwood J, Jefferies E. Individual word representations dissociate from linguistic context along a cortical unimodal to heteromodal gradient. Hum Brain Mapp 2024; 45:e26607. [PMID: 38339897 PMCID: PMC10836172 DOI: 10.1002/hbm.26607] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Revised: 11/30/2023] [Accepted: 01/15/2024] [Indexed: 02/12/2024] Open
Abstract
Language comprehension involves multiple hierarchical processing stages across time, space, and levels of representation. When processing a word, the sensory input is transformed into increasingly abstract representations that need to be integrated with the linguistic context. Thus, language comprehension involves both input-driven as well as context-dependent processes. While neuroimaging research has traditionally focused on mapping individual brain regions to the distinct underlying processes, recent studies indicate that whole-brain distributed patterns of cortical activation might be highly relevant for cognitive functions, including language. One such pattern, based on resting-state connectivity, is the 'principal cortical gradient', which dissociates sensory from heteromodal brain regions. The present study investigated the extent to which this gradient provides an organizational principle underlying language function, using a multimodal neuroimaging dataset of functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) recordings from 102 participants during sentence reading. We found that the brain response to individual representations of a word (word length, orthographic distance, and word frequency), which reflect visual; orthographic; and lexical properties, gradually increases towards the sensory end of the gradient. Although these properties showed opposite effect directions in fMRI and MEG, their association with the sensory end of the gradient was consistent across both neuroimaging modalities. In contrast, MEG revealed that properties reflecting a word's relation to its linguistic context (semantic similarity and position within the sentence) involve the heteromodal end of the gradient to a stronger extent. This dissociation between individual word and contextual properties was stable across earlier and later time windows during word presentation, indicating interactive processing of word representations and linguistic context at opposing ends of the principal gradient. To conclude, our findings indicate that the principal gradient underlies the organization of a range of linguistic representations while supporting a gradual distinction between context-independent and context-dependent representations. Furthermore, the gradient reveals convergent patterns across neuroimaging modalities (similar location along the gradient) in the presence of divergent responses (opposite effect directions).
Collapse
Affiliation(s)
- Susanne Eisenhauer
- Department of PsychologyUniversity of YorkYorkUK
- York Neuroimaging Centre, Innovation WayYorkUK
| | | | | | | | - Elizabeth Jefferies
- Department of PsychologyUniversity of YorkYorkUK
- York Neuroimaging Centre, Innovation WayYorkUK
| |
Collapse
|
17
|
Waraich SA, Victor JD. The Geometry of Low- and High-Level Perceptual Spaces. J Neurosci 2024; 44:e1460232023. [PMID: 38267235 PMCID: PMC10860617 DOI: 10.1523/jneurosci.1460-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Revised: 11/27/2023] [Accepted: 11/28/2023] [Indexed: 01/26/2024] Open
Abstract
Low-level features are typically continuous (e.g., the gamut between two colors), but semantic information is often categorical (there is no corresponding gradient between dog and turtle) and hierarchical (animals live in land, water, or air). To determine the impact of these differences on cognitive representations, we characterized the geometry of perceptual spaces of five domains: a domain dominated by semantic information (animal names presented as words), a domain dominated by low-level features (colored textures), and three intermediate domains (animal images, lightly texturized animal images that were easy to recognize, and heavily texturized animal images that were difficult to recognize). Each domain had 37 stimuli derived from the same animal names. From 13 participants (9F), we gathered similarity judgments in each domain via an efficient psychophysical ranking paradigm. We then built geometric models of each domain for each participant, in which distances between stimuli accounted for participants' similarity judgments and intrinsic uncertainty. Remarkably, the five domains had similar global properties: each required 5-7 dimensions, and a modest amount of spherical curvature provided the best fit. However, the arrangement of the stimuli within these embeddings depended on the level of semantic information: dendrograms derived from semantic domains (word, image, and lightly texturized images) were more "tree-like" than those from feature-dominated domains (heavily texturized images and textures). Thus, the perceptual spaces of domains along this feature-dominated to semantic-dominated gradient shift to a tree-like organization when semantic information dominates, while retaining a similar global geometry.
Collapse
Affiliation(s)
| | - Jonathan D Victor
- Division of Systems Neurology and Neuroscience, Feil Family Brain and Mind Research Institute, Weill Cornell Medical College, New York 10065, New York
| |
Collapse
|
18
|
Yan Y, Zhan J, Garrod O, Cui X, Ince RAA, Schyns PG. Strength of predicted information content in the brain biases decision behavior. Curr Biol 2023; 33:5505-5514.e6. [PMID: 38065096 DOI: 10.1016/j.cub.2023.10.042] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Revised: 10/11/2023] [Accepted: 10/23/2023] [Indexed: 12/21/2023]
Abstract
Prediction-for-perception theories suggest that the brain predicts incoming stimuli to facilitate their categorization.1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 However, it remains unknown what the information contents of these predictions are, which hinders mechanistic explanations. This is because typical approaches cast predictions as an underconstrained contrast between two categories18,19,20,21,22,23,24-e.g., faces versus cars, which could lead to predictions of features specific to faces or cars, or features from both categories. Here, to pinpoint the information contents of predictions and thus their mechanistic processing in the brain, we identified the features that enable two different categorical perceptions of the same stimuli. We then trained multivariate classifiers to discern, from dynamic MEG brain responses, the features tied to each perception. With an auditory cueing design, we reveal where, when, and how the brain reactivates visual category features (versus the typical category contrast) before the stimulus is shown. We demonstrate that the predictions of category features have a more direct influence (bias) on subsequent decision behavior in participants than the typical category contrast. Specifically, these predictions are more precisely localized in the brain (lateralized), are more specifically driven by the auditory cues, and their reactivation strength before a stimulus presentation exerts a greater bias on how the individual participant later categorizes this stimulus. By characterizing the specific information contents that the brain predicts and then processes, our findings provide new insights into the brain's mechanisms of prediction for perception.
Collapse
Affiliation(s)
- Yuening Yan
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Jiayu Zhan
- School of Psychological and Cognitive Sciences, Peking University, 5 Yiheyuan Road, Beijing 100871, China
| | - Oliver Garrod
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Xuan Cui
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Robin A A Ince
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Philippe G Schyns
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK.
| |
Collapse
|
19
|
Ghazaryan G, van Vliet M, Lammi L, Lindh-Knuutila T, Kivisaari S, Hultén A, Salmelin R. Cortical time-course of evidence accumulation during semantic processing. Commun Biol 2023; 6:1242. [PMID: 38066098 PMCID: PMC10709650 DOI: 10.1038/s42003-023-05611-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Accepted: 11/20/2023] [Indexed: 12/18/2023] Open
Abstract
Our understanding of the surrounding world and communication with other people are tied to mental representations of concepts. In order for the brain to recognize an object, it must determine which concept to access based on information available from sensory inputs. In this study, we combine magnetoencephalography and machine learning to investigate how concepts are represented and accessed in the brain over time. Using brain responses from a silent picture naming task, we track the dynamics of visual and semantic information processing, and show that the brain gradually accumulates information on different levels before eventually reaching a plateau. The timing of this plateau point varies across individuals and feature models, indicating notable temporal variation in visual object recognition and semantic processing.
Collapse
Affiliation(s)
- Gayane Ghazaryan
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland.
| | - Marijn van Vliet
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
| | - Lotta Lammi
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
| | - Tiina Lindh-Knuutila
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
| | - Sasa Kivisaari
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
| | - Annika Hultén
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
- Aalto NeuroImaging, Aalto University, P.O. Box 12200, Aalto, FI-00076, Finland
| | - Riitta Salmelin
- Department of Neuroscience and Biomedical Engineering, Aalto University, P.O. Box 12200, FI-00076, Aalto, Finland
- Aalto NeuroImaging, Aalto University, P.O. Box 12200, Aalto, FI-00076, Finland
| |
Collapse
|
20
|
McMahon E, Bonner MF, Isik L. Hierarchical organization of social action features along the lateral visual pathway. Curr Biol 2023; 33:5035-5047.e8. [PMID: 37918399 PMCID: PMC10841461 DOI: 10.1016/j.cub.2023.10.015] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Revised: 09/01/2023] [Accepted: 10/10/2023] [Indexed: 11/04/2023]
Abstract
Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.
Collapse
Affiliation(s)
- Emalie McMahon
- Department of Cognitive Science, Zanvyl Krieger School of Arts & Sciences, Johns Hopkins University, 237 Krieger Hall, 3400 N. Charles Street, Baltimore, MD 21218, USA.
| | - Michael F Bonner
- Department of Cognitive Science, Zanvyl Krieger School of Arts & Sciences, Johns Hopkins University, 237 Krieger Hall, 3400 N. Charles Street, Baltimore, MD 21218, USA
| | - Leyla Isik
- Department of Cognitive Science, Zanvyl Krieger School of Arts & Sciences, Johns Hopkins University, 237 Krieger Hall, 3400 N. Charles Street, Baltimore, MD 21218, USA; Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Suite 400 West, Wyman Park Building, 3400 N. Charles Street, Baltimore, MD 21218, USA
| |
Collapse
|
21
|
Li D, Tian Y, Li J. SODFormer: Streaming Object Detection With Transformer Using Events and Frames. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:14020-14037. [PMID: 37494161 DOI: 10.1109/tpami.2023.3298925] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/28/2023]
Abstract
DAVIS camera, streaming two complementary sensing modalities of asynchronous events and frames, has gradually been used to address major object detection challenges (e.g., fast motion blur and low-light). However, how to effectively leverage rich temporal cues and fuse two heterogeneous visual streams remains a challenging endeavor. To address this challenge, we propose a novel streaming object detector with Transformer, namely SODFormer, which first integrates events and frames to continuously detect objects in an asynchronous manner. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-DAVIS-SOD) over 1080.1 k manual labels. Then, we design a spatiotemporal Transformer architecture to detect objects via an end-to-end sequence prediction problem, where the novel temporal Transformer module leverages rich temporal cues from two visual streams to improve the detection performance. Finally, an asynchronous attention-based fusion module is proposed to integrate two heterogeneous sensing modalities and take complementary advantages from each end, which can be queried at any time to locate objects and break through the limited output frequency from synchronized frame-based fusion strategies. The results show that the proposed SODFormer outperforms four state-of-the-art methods and our eight baselines by a significant margin. We also show that our unifying framework works well even in cases where the conventional frame-based camera fails, e.g., high-speed motion and low-light conditions. Our dataset and code can be available at https://github.com/dianzl/SODFormer.
Collapse
|
22
|
Yao M, Wen B, Yang M, Guo J, Jiang H, Feng C, Cao Y, He H, Chang L. High-dimensional topographic organization of visual features in the primate temporal lobe. Nat Commun 2023; 14:5931. [PMID: 37739988 PMCID: PMC10517140 DOI: 10.1038/s41467-023-41584-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2023] [Accepted: 09/07/2023] [Indexed: 09/24/2023] Open
Abstract
The inferotemporal cortex supports our supreme object recognition ability. Numerous studies have been conducted to elucidate the functional organization of this brain area, but there are still important questions that remain unanswered, including how this organization differs between humans and non-human primates. Here, we use deep neural networks trained on object categorization to construct a 25-dimensional space of visual features, and systematically measure the spatial organization of feature preference in both male monkey brains and human brains using fMRI. These feature maps allow us to predict the selectivity of a previously unknown region in monkey brains, which is corroborated by additional fMRI and electrophysiology experiments. These maps also enable quantitative analyses of the topographic organization of the temporal lobe, demonstrating the existence of a pair of orthogonal gradients that differ in spatial scale and revealing significant differences in the functional organization of high-level visual areas between monkey and human brains.
Collapse
Affiliation(s)
- Mengna Yao
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
- University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Bincheng Wen
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
- University of Chinese Academy of Sciences, Beijing, 100049, China
- Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
| | - Mingpo Yang
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Jiebin Guo
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Haozhou Jiang
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Chao Feng
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Yilei Cao
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Huiguang He
- University of Chinese Academy of Sciences, Beijing, 100049, China
- Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
| | - Le Chang
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China.
- University of Chinese Academy of Sciences, Beijing, 100049, China.
| |
Collapse
|
23
|
Du C, Fu K, Wen B, He H. Topographic representation of visually evoked emotional experiences in the human cerebral cortex. iScience 2023; 26:107571. [PMID: 37664621 PMCID: PMC10470388 DOI: 10.1016/j.isci.2023.107571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 07/03/2023] [Accepted: 08/07/2023] [Indexed: 09/05/2023] Open
Abstract
Affective neuroscience seeks to uncover the neural underpinnings of emotions that humans experience. However, it remains unclear whether an affective space underlies the discrete emotion categories in the human brain, and how it relates to the hypothesized affective dimensions. To address this question, we developed a voxel-wise encoding model to investigate the cortical organization of human emotions. Results revealed that the distributed emotion representations are constructed through a fundamental affective space. We further compared each dimension of this space to 14 hypothesized affective dimensions, and found that many affective dimensions are captured by the fundamental affective space. Our results suggest that emotional experiences are represented by broadly spatial overlapping cortical patterns and form smooth gradients across large areas of the cortex. This finding reveals the specific structure of the affective space and its relationship to hypothesized affective dimensions, while highlighting the distributed nature of emotional representations in the cortex.
Collapse
Affiliation(s)
- Changde Du
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
| | - Kaicheng Fu
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Bincheng Wen
- Center for Excellence in Brain Science and Intelligence Technology, Key Laboratory of Primate Neurobiology, Institute of Neuroscience, Chinese Academy of Sciences, Shanghai 200031, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Huiguang He
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| |
Collapse
|
24
|
Almeida J, Fracasso A, Kristensen S, Valério D, Bergström F, Chakravarthi R, Tal Z, Walbrin J. Neural and behavioral signatures of the multidimensionality of manipulable object processing. Commun Biol 2023; 6:940. [PMID: 37709924 PMCID: PMC10502059 DOI: 10.1038/s42003-023-05323-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 09/04/2023] [Indexed: 09/16/2023] Open
Abstract
Understanding how we recognize objects requires unravelling the variables that govern the way we think about objects and the neural organization of object representations. A tenable hypothesis is that the organization of object knowledge follows key object-related dimensions. Here, we explored, behaviorally and neurally, the multidimensionality of object processing. We focused on within-domain object information as a proxy for the decisions we typically engage in our daily lives - e.g., identifying a hammer in the context of other tools. We extracted object-related dimensions from subjective human judgments on a set of manipulable objects. We show that the extracted dimensions are cognitively interpretable and relevant - i.e., participants are able to consistently label them, and these dimensions can guide object categorization; and are important for the neural organization of knowledge - i.e., they predict neural signals elicited by manipulable objects. This shows that multidimensionality is a hallmark of the organization of manipulable object knowledge.
Collapse
Affiliation(s)
- Jorge Almeida
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal.
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal.
| | - Alessio Fracasso
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
| | - Stephanie Kristensen
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
| | - Daniela Valério
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
| | - Fredrik Bergström
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- Department of Psychology, University of Gothenburg, Gothenburg, Sweden
| | | | - Zohar Tal
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
| | - Jonathan Walbrin
- Proaction Lab, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
- CINEICC, Faculty of Psychology and Educational Sciences, University of Coimbra, Coimbra, Portugal
| |
Collapse
|
25
|
Reilly J, Finley AM, Litovsky CP, Kenett YN. Bigram semantic distance as an index of continuous semantic flow in natural language: Theory, tools, and applications. J Exp Psychol Gen 2023; 152:2578-2590. [PMID: 37079833 PMCID: PMC10790181 DOI: 10.1037/xge0001389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/22/2023]
Abstract
Much of our understanding of word meaning has been informed through studies of single words. High-dimensional semantic space models have recently proven instrumental in elucidating connections between words. Here we show how bigram semantic distance can yield novel insights into conceptual cohesion and topic flow when computed over continuous language samples. For example, "Cats drink milk" is comprised of an ordered vector of bigrams (cat-drink, drink-milk). Each of these bigrams has a unique semantic distance. These distances in turn may provide a metric of dispersion or the flow of concepts as language unfolds. We offer an R-package ("semdistflow") that transforms any user-specified language transcript into a vector of ordered bigrams, appending two metrics of semantic distance to each pair. We validated these distance metrics on a continuous stream of simulated verbal fluency data assigning predicted switch markers between alternating semantic clusters (animals, musical instruments, fruit). We then generated bigram distance norms on a large sample of text and demonstrated applications of the technique to a classic work of short fiction, To Build a Fire (London, 1908). In one application, we showed that bigrams spanning sentence boundaries are punctuated by jumps in the semantic distance. We discuss the promise of this technique for characterizing semantic processing in real-world narratives and for bridging findings at the single word level with macroscale discourse analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Collapse
Affiliation(s)
- Jamie Reilly
- Eleanor M. Saffran Center for Cognitive Neuroscience
- Department of Communication Sciences and Disorders, Temple University, Philadelphia, Pennsylvania USA
| | - Ann Marie Finley
- Eleanor M. Saffran Center for Cognitive Neuroscience
- Department of Communication Sciences and Disorders, Temple University, Philadelphia, Pennsylvania USA
| | - Celia P. Litovsky
- Eleanor M. Saffran Center for Cognitive Neuroscience
- Department of Communication Sciences and Disorders, Temple University, Philadelphia, Pennsylvania USA
| | - Yoed N. Kenett
- Faculty of Faculty of Data and Decision Sciences, Technion Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
26
|
LeBel A, Wagner L, Jain S, Adhikari-Desai A, Gupta B, Morgenthal A, Tang J, Xu L, Huth AG. A natural language fMRI dataset for voxelwise encoding models. Sci Data 2023; 10:555. [PMID: 37612332 PMCID: PMC10447563 DOI: 10.1038/s41597-023-02437-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 08/02/2023] [Indexed: 08/25/2023] Open
Abstract
Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.
Collapse
Affiliation(s)
- Amanda LeBel
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, 94704, USA
| | - Lauren Wagner
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA, 90095, USA
| | - Shailee Jain
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Aneesh Adhikari-Desai
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Bhavin Gupta
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Allyson Morgenthal
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Jerry Tang
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Lixiang Xu
- Department of Physics, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Alexander G Huth
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA.
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA.
| |
Collapse
|
27
|
Grossberg S. How children learn to understand language meanings: a neural model of adult-child multimodal interactions in real-time. Front Psychol 2023; 14:1216479. [PMID: 37599779 PMCID: PMC10435915 DOI: 10.3389/fpsyg.2023.1216479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 06/28/2023] [Indexed: 08/22/2023] Open
Abstract
This article describes a biological neural network model that can be used to explain how children learn to understand language meanings about the perceptual and affective events that they consciously experience. This kind of learning often occurs when a child interacts with an adult teacher to learn language meanings about events that they experience together. Multiple types of self-organizing brain processes are involved in learning language meanings, including processes that control conscious visual perception, joint attention, object learning and conscious recognition, cognitive working memory, cognitive planning, emotion, cognitive-emotional interactions, volition, and goal-oriented actions. The article shows how all of these brain processes interact to enable the learning of language meanings to occur. The article also contrasts these human capabilities with AI models such as ChatGPT. The current model is called the ChatSOME model, where SOME abbreviates Self-Organizing MEaning.
Collapse
Affiliation(s)
- Stephen Grossberg
- Center for Adaptive Systems, Boston University, Boston, MA, United States
| |
Collapse
|
28
|
Gong XL, Huth AG, Deniz F, Johnson K, Gallant JL, Theunissen FE. Phonemic segmentation of narrative speech in human cerebral cortex. Nat Commun 2023; 14:4309. [PMID: 37463907 PMCID: PMC10354060 DOI: 10.1038/s41467-023-39872-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Accepted: 06/29/2023] [Indexed: 07/20/2023] Open
Abstract
Speech processing requires extracting meaning from acoustic patterns using a set of intermediate representations based on a dynamic segmentation of the speech stream. Using whole brain mapping obtained in fMRI, we investigate the locus of cortical phonemic processing not only for single phonemes but also for short combinations made of diphones and triphones. We find that phonemic processing areas are much larger than previously described: they include not only the classical areas in the dorsal superior temporal gyrus but also a larger region in the lateral temporal cortex where diphone features are best represented. These identified phonemic regions overlap with the lexical retrieval region, but we show that short word retrieval is not sufficient to explain the observed responses to diphones. Behavioral studies have shown that phonemic processing and lexical retrieval are intertwined. Here, we also have identified candidate regions within the speech cortical network where this joint processing occurs.
Collapse
Affiliation(s)
- Xue L Gong
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, 94720, CA, USA.
| | - Alexander G Huth
- Departments of Neuroscience and Computer Science, University of Texas, Austin, Austin, 78712, TX, USA
| | - Fatma Deniz
- Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin, Berlin, 10587, Berlin, Germany
| | - Keith Johnson
- Department of Linguistics, University of California, Berkeley, Berkeley, 94720, CA, USA
| | - Jack L Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, 94720, CA, USA
- Department of Psychology, University of California, Berkeley, Berkeley, 94720, CA, USA
| | - Frédéric E Theunissen
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, 94720, CA, USA.
- Department of Psychology, University of California, Berkeley, Berkeley, 94720, CA, USA.
- Department of Integrative Biology, University of California, Berkeley, Berkeley, 94720, CA, USA.
| |
Collapse
|
29
|
Yang E, Milisav F, Kopal J, Holmes AJ, Mitsis GD, Misic B, Finn ES, Bzdok D. The default network dominates neural responses to evolving movie stories. Nat Commun 2023; 14:4197. [PMID: 37452058 PMCID: PMC10349102 DOI: 10.1038/s41467-023-39862-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Accepted: 06/27/2023] [Indexed: 07/18/2023] Open
Abstract
Neuroscientific studies exploring real-world dynamic perception often overlook the influence of continuous changes in narrative content. In our research, we utilize machine learning tools for natural language processing to examine the relationship between movie narratives and neural responses. By analyzing over 50,000 brain images of participants watching Forrest Gump from the studyforrest dataset, we find distinct brain states that capture unique semantic aspects of the unfolding story. The default network, associated with semantic information integration, is the most engaged during movie watching. Furthermore, we identify two mechanisms that underlie how the default network liaises with the amygdala and hippocampus. Our findings demonstrate effective approaches to understanding neural processes in everyday situations and their relation to conscious awareness.
Collapse
Affiliation(s)
- Enning Yang
- Department of Biomedical Engineering, TheNeuro-Montreal Neurological Institute (MNI), McConnell Brain Imaging Centre (BIC), McGill University, Montreal, QC, Canada
- Mila-Quebec Artificial Intelligence Institute, Montreal, QC, Canada
| | - Filip Milisav
- Department of Biomedical Engineering, TheNeuro-Montreal Neurological Institute (MNI), McConnell Brain Imaging Centre (BIC), McGill University, Montreal, QC, Canada
| | - Jakub Kopal
- Department of Biomedical Engineering, TheNeuro-Montreal Neurological Institute (MNI), McConnell Brain Imaging Centre (BIC), McGill University, Montreal, QC, Canada
- Mila-Quebec Artificial Intelligence Institute, Montreal, QC, Canada
| | - Avram J Holmes
- Department of Psychology and Psychiatry, Yale University, New Haven, CT, USA
| | - Georgios D Mitsis
- Department of Bioengineering, McGill University, Montreal, QC, Canada
| | - Bratislav Misic
- Department of Biomedical Engineering, TheNeuro-Montreal Neurological Institute (MNI), McConnell Brain Imaging Centre (BIC), McGill University, Montreal, QC, Canada
| | - Emily S Finn
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Danilo Bzdok
- Department of Biomedical Engineering, TheNeuro-Montreal Neurological Institute (MNI), McConnell Brain Imaging Centre (BIC), McGill University, Montreal, QC, Canada.
- Mila-Quebec Artificial Intelligence Institute, Montreal, QC, Canada.
| |
Collapse
|
30
|
Morton NW, Zippi EL, Preston AR. Memory reactivation and suppression modulate integration of the semantic features of related memories in hippocampus. Cereb Cortex 2023; 33:9020-9037. [PMID: 37264937 PMCID: PMC10350843 DOI: 10.1093/cercor/bhad179] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2019] [Revised: 04/27/2023] [Accepted: 04/28/2023] [Indexed: 06/03/2023] Open
Abstract
Encoding an event that overlaps with a previous experience may involve reactivating an existing memory and integrating it with new information or suppressing the existing memory to promote formation of a distinct, new representation. We used fMRI during overlapping event encoding to track reactivation and suppression of individual, related memories. We further used a model of semantic knowledge based on Wikipedia to quantify both reactivation of semantic knowledge related to a previous event and formation of integrated memories containing semantic features of both events. Representational similarity analysis revealed that reactivation of semantic knowledge related to a prior event in posterior medial prefrontal cortex (pmPFC) supported memory integration during new learning. Moreover, anterior hippocampus (aHPC) formed integrated representations combining the semantic features of overlapping events. We further found evidence that aHPC integration may be modulated on a trial-by-trial basis by interactions between ventrolateral PFC and anterior mPFC, with suppression of item-specific memory representations in anterior mPFC inhibiting hippocampal integration. These results suggest that PFC-mediated control processes determine the availability of specific relevant memories during new learning, thus impacting hippocampal memory integration.
Collapse
Affiliation(s)
- Neal W Morton
- Center for Learning and Memory, The University of Texas at Austin, Austin, TX 78712, United States
| | - Ellen L Zippi
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA 95064, United States
| | - Alison R Preston
- Center for Learning and Memory, The University of Texas at Austin, Austin, TX 78712, United States
- Department of Psychology, The University of Texas at Austin, Austin, TX 78712, United States
- Department of Neuroscience, The University of Texas at Austin, Austin, TX 78712, United States
| |
Collapse
|
31
|
Zhou M, Gong Z, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for human action recognition. Sci Data 2023; 10:415. [PMID: 37369643 DOI: 10.1038/s41597-023-02325-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 06/21/2023] [Indexed: 06/29/2023] Open
Abstract
Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.
Collapse
Affiliation(s)
- Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
32
|
Coggan DD, Tong F. Spikiness and animacy as potential organizing principles of human ventral visual cortex. Cereb Cortex 2023; 33:8194-8217. [PMID: 36958809 PMCID: PMC10321104 DOI: 10.1093/cercor/bhad108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Revised: 03/05/2023] [Accepted: 03/06/2023] [Indexed: 03/25/2023] Open
Abstract
Considerable research has been devoted to understanding the fundamental organizing principles of the ventral visual pathway. A recent study revealed a series of 3-4 topographical maps arranged along the macaque inferotemporal (IT) cortex. The maps articulated a two-dimensional space based on the spikiness and animacy of visual objects, with "inanimate-spiky" and "inanimate-stubby" regions of the maps constituting two previously unidentified cortical networks. The goal of our study was to determine whether a similar functional organization might exist in human IT. To address this question, we presented the same object stimuli and images from "classic" object categories (bodies, faces, houses) to humans while recording fMRI activity at 7 Tesla. Contrasts designed to reveal the spikiness-animacy object space evoked extensive significant activation across human IT. However, unlike the macaque, we did not observe a clear sequence of complete maps, and selectivity for the spikiness-animacy space was deeply and mutually entangled with category-selectivity. Instead, we observed multiple new stimulus preferences in category-selective regions, including functional sub-structure related to object spikiness in scene-selective cortex. Taken together, these findings highlight spikiness as a promising organizing principle of human IT and provide new insights into the role of category-selective regions in visual object processing.
Collapse
Affiliation(s)
- David D Coggan
- Department of Psychology, Vanderbilt University, 111 21st Ave S, Nashville, TN 37240, United States
| | - Frank Tong
- Department of Psychology, Vanderbilt University, 111 21st Ave S, Nashville, TN 37240, United States
| |
Collapse
|
33
|
Josephs EL, Hebart MN, Konkle T. Dimensions underlying human understanding of the reachable world. Cognition 2023; 234:105368. [PMID: 36641868 DOI: 10.1016/j.cognition.2023.105368] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 12/20/2022] [Accepted: 01/09/2023] [Indexed: 01/15/2023]
Abstract
Near-scale environments, like work desks, restaurant place settings or lab benches, are the interface of our hand-based interactions with the world. How are our conceptual representations of these environments organized? What properties distinguish among reachspaces, and why? We obtained 1.25 million similarity judgments on 990 reachspace images, and generated a 30-dimensional embedding which accurately predicts these judgments. Examination of the embedding dimensions revealed key properties underlying these judgments, such as reachspace layout, affordance, and visual appearance. Clustering performed over the embedding revealed four distinct interpretable classes of reachspaces, distinguishing among spaces related to food, electronics, analog activities, and storage or display. Finally, we found that reachspace similarity ratings were better predicted by the function of the spaces than their locations, suggesting that reachspaces are largely conceptualized in terms of the actions they support. Altogether, these results reveal the behaviorally-relevant principles that structure our internal representations of reach-relevant environments.
Collapse
Affiliation(s)
- Emilie L Josephs
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, USA; Psychology Department, Harvard University, Cambridge, USA.
| | - Martin N Hebart
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| | - Talia Konkle
- Psychology Department, Harvard University, Cambridge, USA.
| |
Collapse
|
34
|
Reber TP, Mackay S, Bausch M, Kehl MS, Borger V, Surges R, Mormann F. Single-neuron mechanisms of neural adaptation in the human temporal lobe. Nat Commun 2023; 14:2496. [PMID: 37120437 PMCID: PMC10148801 DOI: 10.1038/s41467-023-38190-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 04/13/2023] [Indexed: 05/01/2023] Open
Abstract
A central function of the human brain is to adapt to new situations based on past experience. Adaptation is reflected behaviorally by shorter reaction times to repeating or similar stimuli, and neurophysiologically by reduced neural activity in bulk-tissue measurements with fMRI or EEG. Several potential single-neuron mechanisms have been hypothesized to cause this reduction of activity at the macroscopic level. We here explore these mechanisms using an adaptation paradigm with visual stimuli bearing abstract semantic similarity. We recorded intracranial EEG (iEEG) simultaneously with spiking activity of single neurons in the medial temporal lobes of 25 neurosurgical patients. Recording from 4917 single neurons, we demonstrate that reduced event-related potentials in the macroscopic iEEG signal are associated with a sharpening of single-neuron tuning curves in the amygdala, but with an overall reduction of single-neuron activity in the hippocampus, entorhinal cortex, and parahippocampal cortex, consistent with fatiguing in these areas.
Collapse
Affiliation(s)
- Thomas P Reber
- Faculty of Psychology, UniDistance Suisse, Brig, Switzerland.
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany.
| | - Sina Mackay
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany
| | - Marcel Bausch
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany
| | - Marcel S Kehl
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany
| | - Valeri Borger
- Department of Neurosurgery, University of Bonn Medical Centre, Bonn, Germany
| | - Rainer Surges
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany
| | - Florian Mormann
- Department of Epileptology, University of Bonn Medical Centre, Bonn, Germany
| |
Collapse
|
35
|
Deniz F, Tseng C, Wehbe L, Dupré la Tour T, Gallant JL. Semantic Representations during Language Comprehension Are Affected by Context. J Neurosci 2023; 43:3144-3158. [PMID: 36973013 PMCID: PMC10146529 DOI: 10.1523/jneurosci.2459-21.2023] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 02/17/2023] [Accepted: 02/26/2023] [Indexed: 03/29/2023] Open
Abstract
The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. fMRI was used to record human brain activity while four subjects (two female) read words in four conditions that vary in context: narratives, isolated sentences, blocks of semantically similar words, and isolated words. We then compared the signal-to-noise ratio (SNR) of evoked brain responses, and we used a voxelwise encoding modeling approach to compare the representation of semantic information across the four conditions. We find four consistent effects of varying context. First, stimuli with more context evoke brain responses with higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared with stimuli with little context. Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. In individual subjects, only natural language stimuli consistently evoke widespread representation of semantic information. Third, context affects voxel semantic tuning. Finally, models estimated using stimuli with little context do not generalize well to natural language. These results show that context has large effects on the quality of neuroimaging data and on the representation of meaning in the brain. Thus, neuroimaging studies that use stimuli with little context may not generalize well to the natural regime.SIGNIFICANCE STATEMENT Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here, we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuro-imaging data and changes where and how semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.
Collapse
Affiliation(s)
- Fatma Deniz
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
- Institute of Software Engineering and Theoretical Computer Science, Technische Universität Berlin, Berlin 10623, Germany
| | - Christine Tseng
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
| | - Leila Wehbe
- Machine Learning Department, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213
| | - Tom Dupré la Tour
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
| | - Jack L Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
- Department of Psychology, University of California, Berkeley, California 94720
| |
Collapse
|
36
|
Nakai T, Nishimoto S. Artificial neural network modelling of the neural population code underlying mathematical operations. Neuroimage 2023; 270:119980. [PMID: 36848969 DOI: 10.1016/j.neuroimage.2023.119980] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2022] [Revised: 02/10/2023] [Accepted: 02/23/2023] [Indexed: 02/28/2023] Open
Abstract
Mathematical operations have long been regarded as a sparse, symbolic process in neuroimaging studies. In contrast, advances in artificial neural networks (ANN) have enabled extracting distributed representations of mathematical operations. Recent neuroimaging studies have compared distributed representations of the visual, auditory and language domains in ANNs and biological neural networks (BNNs). However, such a relationship has not yet been examined in mathematics. Here we hypothesise that ANN-based distributed representations can explain brain activity patterns of symbolic mathematical operations. We used the fMRI data of a series of mathematical problems with nine different combinations of operators to construct voxel-wise encoding/decoding models using both sparse operator and latent ANN features. Representational similarity analysis demonstrated shared representations between ANN and BNN, an effect particularly evident in the intraparietal sulcus. Feature-brain similarity (FBS) analysis served to reconstruct a sparse representation of mathematical operations based on distributed ANN features in each cortical voxel. Such reconstruction was more efficient when using features from deeper ANN layers. Moreover, latent ANN features allowed the decoding of novel operators not used during model training from brain activity. The current study provides novel insights into the neural code underlying mathematical thought.
Collapse
Affiliation(s)
- Tomoya Nakai
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan; Lyon Neuroscience Research Center (CRNL), INSERM U1028 - CNRS UMR5292, University of Lyon, Bron, France.
| | - Shinji Nishimoto
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan; Graduate School of Frontier Biosciences, Osaka University, Suita, Japan; Graduate School of Medicine, Osaka University, Suita, Japan
| |
Collapse
|
37
|
Zhang H, Zhang K, Zhang Z, Zhao M, Liu Q, Luo W, Wu H. Social conformity is associated with inter-trial electroencephalogram variability. Ann N Y Acad Sci 2023; 1523:104-118. [PMID: 36964981 DOI: 10.1111/nyas.14983] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/27/2023]
Abstract
Human society encompasses diverse social influences, and people experience events differently and may behave differently under such influence, including in forming an impression of others. However, little is known about the underlying neural relevance of individual differences in following others' opinions or social norms. In the present study, we designed a series of tasks centered on social influence to investigate the underlying relevance between an individual's degree of social conformity and their neural variability. We found that individual differences under the social influence are associated with the amount of inter-trial electroencephalogram (EEG) variability over multiple stages in a conformity task (making face judgments and receiving social influence). This association was robust in the alpha band over the frontal and occipital electrodes for negative social influence. We also found that inter-trial EEG variability is a very stable, participant-driven internal state measurement and could be interpreted as mindset instability. Overall, these findings support the hypothesis that higher inter-trial EEG variability may be related to higher mindset instability, which makes participants more vulnerable to exposed external social influence. The present study provides a novel approach that considers the stability of one's endogenous neural signal during tasks and links it to human social behaviors.
Collapse
Affiliation(s)
- Haoming Zhang
- Centre for Cognitive and Brain Sciences and Department of Psychology, University of Macau, Macau, China
| | - Kunkun Zhang
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian, China
| | - Ziqi Zhang
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian, China
| | - Mingqi Zhao
- Research Center for Motor Control and Neuroplasticity, KU Leuven, Leuven, Belgium
| | - Quanying Liu
- Shenzhen Key Laboratory of Smart Healthcare Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Wenbo Luo
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian, China
| | - Haiyan Wu
- Centre for Cognitive and Brain Sciences and Department of Psychology, University of Macau, Macau, China
| |
Collapse
|
38
|
Santavirta S, Karjalainen T, Nazari-Farsani S, Hudson M, Putkinen V, Seppälä K, Sun L, Glerean E, Hirvonen J, Karlsson HK, Nummenmaa L. Functional organization of social perception in the human brain. Neuroimage 2023; 272:120025. [PMID: 36958619 PMCID: PMC10112277 DOI: 10.1016/j.neuroimage.2023.120025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 03/07/2023] [Accepted: 03/11/2023] [Indexed: 03/25/2023] Open
Abstract
Humans rapidly extract diverse and complex information from ongoing social interactions, but the perceptual and neural organization of the different aspects of social perception remains unresolved. We showed short movie clips with rich social content to 97 healthy participants while their haemodynamic brain activity was measured with fMRI. The clips were annotated moment-to-moment for a large set of social features and 45 of the features were evaluated reliably between annotators. Cluster analysis of the social features revealed that 13 dimensions were sufficient for describing the social perceptual space. Three different analysis methods were used to map the social perceptual processes in the human brain. Regression analysis mapped regional neural response profiles for different social dimensions. Multivariate pattern analysis then established the spatial specificity of the responses and intersubject correlation analysis connected social perceptual processing with neural synchronization. The results revealed a gradient in the processing of social information in the brain. Posterior temporal and occipital regions were broadly tuned to most social dimensions and the classifier revealed that these responses showed spatial specificity for social dimensions; in contrast Heschl gyri and parietal areas were also broadly associated with different social signals, yet the spatial patterns of responses did not differentiate social dimensions. Frontal and subcortical regions responded only to a limited number of social dimensions and the spatial response patterns did not differentiate social dimension. Altogether these results highlight the distributed nature of social processing in the brain.
Collapse
Affiliation(s)
- Severi Santavirta
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland.
| | - Tomi Karjalainen
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland
| | - Sanaz Nazari-Farsani
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland
| | - Matthew Hudson
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland; School of Psychology, University of Plymouth, Plymouth, United Kingdom
| | - Vesa Putkinen
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland
| | - Kerttu Seppälä
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland; Department of Medical Physics, Turku University Hospital, Turku, Finland
| | - Lihua Sun
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland; Department of Nuclear Medicine, Pudong Hospital, Fudan University, Shanghai, China
| | - Enrico Glerean
- Department of Neuroscience and Biomedical Engineering, Aalto University School of Science, Espoo, Finland
| | - Jussi Hirvonen
- Department of Radiology, University of Turku and Turku University Hospital, Turku, Finland; Medical Imaging Center, Department of Radiology, Tampere University and Tampere University Hospital, Tampere, Finland
| | - Henry K Karlsson
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland
| | - Lauri Nummenmaa
- Turku PET Centre, University of Turku, Turku, Finland; Turku PET Centre, Turku University Hospital, Turku, Finland; Department of Psychology, University of Turku, Turku, Finland
| |
Collapse
|
39
|
Matsumoto Y, Nishida S, Hayashi R, Son S, Murakami A, Yoshikawa N, Ito H, Oishi N, Masuda N, Murai T, Friston K, Nishimoto S, Takahashi H. Disorganization of Semantic Brain Networks in Schizophrenia Revealed by fMRI. Schizophr Bull 2023; 49:498-506. [PMID: 36542452 PMCID: PMC10016409 DOI: 10.1093/schbul/sbac157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
OBJECTIVES Schizophrenia is a mental illness that presents with thought disorders including delusions and disorganized speech. Thought disorders have been regarded as a consequence of the loosening of associations between semantic concepts since the term "schizophrenia" was first coined by Bleuler. However, a mechanistic account of this cardinal disturbance in terms of functional dysconnection has been lacking. To evaluate how aberrant semantic connections are expressed through brain activity, we characterized large-scale network structures of concept representations using functional magnetic resonance imaging (fMRI). STUDY DESIGN We quantified various concept representations in patients' brains from fMRI activity evoked by movie scenes using encoding modeling. We then constructed semantic brain networks by evaluating the similarity of these semantic representations and conducted graph theory-based network analyses. STUDY RESULTS Neurotypical networks had small-world properties similar to those of natural languages, suggesting small-worldness as a universal property in semantic knowledge networks. Conversely, small-worldness was significantly reduced in networks of schizophrenia patients and was correlated with psychological measures of delusions. Patients' semantic networks were partitioned into more distinct categories and had more random within-category structures than those of controls. CONCLUSIONS The differences in conceptual representations manifest altered semantic clustering and associative intrusions that underlie thought disorders. This is the first study to provide pathophysiological evidence for the loosening of associations as reflected in randomization of semantic networks in schizophrenia. Our method provides a promising approach for understanding the neural basis of altered or creative inner experiences of individuals with mental illness or exceptional abilities, respectively.
Collapse
Affiliation(s)
- Yukiko Matsumoto
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Satoshi Nishida
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology (NICT), Suita, Osaka, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Osaka, Japan
| | - Ryusuke Hayashi
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Shuraku Son
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Akio Murakami
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | | | - Hiroyoshi Ito
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan.,Faculty of Library, Information and Media Science, University of Tsukuba, Tsukuba, Ibaraki, Japan
| | - Naoya Oishi
- Medical Innovation Center, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Naoki Masuda
- Department of Mathematics, University at Buffalo, State University of New York, Buffalo, USA
| | - Toshiya Murai
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Karl Friston
- The Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London, UK
| | - Shinji Nishimoto
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology (NICT), Suita, Osaka, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Osaka, Japan.,Graduate School of Medicine, Osaka University, Suita, Osaka, Japan
| | - Hidehiko Takahashi
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan.,Center for Brain Integration Research, Tokyo Medical and Dental University, Tokyo, Japan
| |
Collapse
|
40
|
Russ BE, Koyano KW, Day-Cooney J, Perwez N, Leopold DA. Temporal continuity shapes visual responses of macaque face patch neurons. Neuron 2023; 111:903-914.e3. [PMID: 36630962 PMCID: PMC10023462 DOI: 10.1016/j.neuron.2022.12.021] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Revised: 09/09/2022] [Accepted: 12/13/2022] [Indexed: 01/12/2023]
Abstract
Macaque inferior temporal cortex neurons respond selectively to complex visual images, with recent work showing that they are also entrained reliably by the evolving content of natural movies. To what extent does temporal continuity itself shape the responses of high-level visual neurons? We addressed this question by measuring how cells in face-selective regions of the macaque visual cortex were affected by the manipulation of a movie's temporal structure. Sampling a 5-min movie at 1 s intervals, we measured neural responses to randomized, brief stimuli of different lengths, ranging from 800 ms dynamic movie snippets to 100 ms static frames. We found that the disruption of temporal continuity strongly altered neural response profiles, particularly in the early response period after stimulus onset. The results suggest that models of visual system function based on discrete and randomized visual presentations may not translate well to the brain's natural modes of operation.
Collapse
Affiliation(s)
- Brian E Russ
- Section on Cognitive Neurophysiology and Imaging, National Institute of Mental Health, Bethesda, MD 20814, USA; Center for Biomedical Imaging and Neuromodulation, Nathan Kline Institute, Orangeburg, NY 10962, USA; Nash Family Department of Neuroscience and Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Department of Psychiatry, New York University at Langone, New York City, NY 10016, USA.
| | - Kenji W Koyano
- Section on Cognitive Neurophysiology and Imaging, National Institute of Mental Health, Bethesda, MD 20814, USA
| | - Julian Day-Cooney
- Section on Cognitive Neurophysiology and Imaging, National Institute of Mental Health, Bethesda, MD 20814, USA
| | - Neda Perwez
- Section on Cognitive Neurophysiology and Imaging, National Institute of Mental Health, Bethesda, MD 20814, USA
| | - David A Leopold
- Section on Cognitive Neurophysiology and Imaging, National Institute of Mental Health, Bethesda, MD 20814, USA; Neurophysiology Imaging Facility, National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, National Eye Institute, Bethesda, MD 20814, USA
| |
Collapse
|
41
|
Jozwik KM, Kietzmann TC, Cichy RM, Kriegeskorte N, Mur M. Deep Neural Networks and Visuo-Semantic Models Explain Complementary Components of Human Ventral-Stream Representational Dynamics. J Neurosci 2023; 43:1731-1741. [PMID: 36759190 PMCID: PMC10010451 DOI: 10.1523/jneurosci.1424-22.2022] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Revised: 11/08/2022] [Accepted: 12/20/2022] [Indexed: 02/11/2023] Open
Abstract
Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants (nine females, six males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.SIGNIFICANCE STATEMENT When we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. DNNs have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as 'eye', 'wheel', and 'face', can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.
Collapse
Affiliation(s)
- Kamila M Jozwik
- Department of Psychology, University of Cambridge, Cambridge CB2 3EB, United Kingdom
| | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
| | - Radoslaw M Cichy
- Department of Education and Psychology, Freie Universität Berlin, 14195 Berlin, Germany
| | - Nikolaus Kriegeskorte
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York 10027
| | - Marieke Mur
- Department of Psychology, Western University, London, Ontario N6A 3K7, Canada
- Department of Computer Science, Western University, London, Ontario N6A 3K7, Canada
| |
Collapse
|
42
|
Speer SPH, Keysers C, Barrios JC, Teurlings CJS, Smidts A, Boksem MAS, Wager TD, Gazzola V. A multivariate brain signature for reward. Neuroimage 2023; 271:119990. [PMID: 36878456 DOI: 10.1016/j.neuroimage.2023.119990] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 02/20/2023] [Accepted: 02/25/2023] [Indexed: 03/07/2023] Open
Abstract
The processing of reinforcers and punishers is crucial to adapt to an ever changing environment and its dysregulation is prevalent in mental health and substance use disorders. While many human brain measures related to reward have been based on activity in individual brain regions, recent studies indicate that many affective and motivational processes are encoded in distributed systems that span multiple regions. Consequently, decoding these processes using individual regions yields small effect sizes and limited reliability, whereas predictive models based on distributed patterns yield larger effect sizes and excellent reliability. To create such a predictive model for the processes of rewards and losses, termed the Brain Reward Signature (BRS), we trained a model to predict the signed magnitude of monetary rewards on the Monetary Incentive Delay task (MID; N = 39) and achieved a highly significant decoding performance (92% for decoding rewards versus losses). We subsequently demonstrate the generalizability of our signature on another version of the MID in a different sample (92% decoding accuracy; N = 12) and on a gambling task from a large sample (73% decoding accuracy, N = 1084). We further provided preliminary data to characterize the specificity of the signature by illustrating that the signature map generates estimates that significantly differ between rewarding and negative feedback (92% decoding accuracy) but do not differ for conditions that differ in disgust rather than reward in a novel Disgust-Delay Task (N = 39). Finally, we show that passively viewing positive and negatively valenced facial expressions loads positively on our signature, in line with previous studies on morbid curiosity. We thus created a BRS that can accurately predict brain responses to rewards and losses in active decision making tasks, and that possibly relates to information seeking in passive observational tasks.
Collapse
Affiliation(s)
- Sebastian P H Speer
- Social Brain Lab, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands; Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Christian Keysers
- Social Brain Lab, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands; Brain and Cognition, Department of Psychology, University of Amsterdam, The Netherlands
| | | | - Cas J S Teurlings
- Social Brain Lab, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands
| | - Ale Smidts
- Rotterdam School of Management, Erasmus University, 3062 PA Rotterdam, The Netherlands
| | - Maarten A S Boksem
- Rotterdam School of Management, Erasmus University, 3062 PA Rotterdam, The Netherlands
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755, USA
| | - Valeria Gazzola
- Social Brain Lab, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands.
| |
Collapse
|
43
|
Bianchi B, Loredo R, Fonseca MD, Carden J, Jaichenco V, der Malsburg TV, Shalom DE, Kamienkowski J. Neural bases of predictions during natural reading of known statements: An EEG and eye movements co-registration study. Neuroscience 2023; 519:131-146. [PMID: 37003544 DOI: 10.1016/j.neuroscience.2023.03.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 01/30/2023] [Accepted: 03/22/2023] [Indexed: 04/03/2023]
Abstract
Predictions of incoming words performed during reading have an impact on how the reader moves their eyes and on the electrical brain potentials. Eye tracking (ET) experiments show that less predictable words are fixated for longer periods of times. Electroencephalography (EEG) experiments show that these words elicit a more negative potential around 400ms (N400) after the word onset when reading one word at a time (foveated reading). Nevertheless, there was no N400 potential during the foveated reading of previously known sentences (memory-encoded), which suggests that the prediction of words from memory-encoded sentences is based on different mechanisms than predictions performed on common sentences. Here, we performed an ET-EEG co-registration experiment where participants read common and memory-encoded sentences. Our results show that the N400 potential disappear when the reader recognises the sentence. Furthermore, time-frequency analyses show a larger alpha lateralisation and a beta power increase for memory-encoded sentences. This suggests a more distributed attention and an active maintenance of the cognitive set, in concordance to the predictive coding framework.
Collapse
|
44
|
Frisby SL, Halai AD, Cox CR, Lambon Ralph MA, Rogers TT. Decoding semantic representations in mind and brain. Trends Cogn Sci 2023; 27:258-281. [PMID: 36631371 DOI: 10.1016/j.tics.2022.12.006] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 12/12/2022] [Accepted: 12/13/2022] [Indexed: 01/11/2023]
Abstract
A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition.
Collapse
Affiliation(s)
- Saskia L Frisby
- Medical Research Council (MRC) Cognition and Brain Sciences Unit, Chaucer Road, Cambridge CB2 7EF, UK.
| | - Ajay D Halai
- Medical Research Council (MRC) Cognition and Brain Sciences Unit, Chaucer Road, Cambridge CB2 7EF, UK
| | - Christopher R Cox
- Department of Psychology, Louisiana State University, Baton Rouge, LA 70803, USA
| | - Matthew A Lambon Ralph
- Medical Research Council (MRC) Cognition and Brain Sciences Unit, Chaucer Road, Cambridge CB2 7EF, UK
| | - Timothy T Rogers
- Department of Psychology, University of Wisconsin-Madison, 1202 West Johnson Street, Madison, WI 53706, USA.
| |
Collapse
|
45
|
Nakai T, Nishimoto S. Quantitative modelling demonstrates format-invariant representations of mathematical problems in the brain. Eur J Neurosci 2023; 57:1003-1017. [PMID: 36710081 DOI: 10.1111/ejn.15925] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 01/19/2023] [Accepted: 01/23/2023] [Indexed: 01/31/2023]
Abstract
Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modelling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modelling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.
Collapse
Affiliation(s)
- Tomoya Nakai
- Lyon Neuroscience Research Center (CRNL), INSERM U1028-CNRS UMR5292, University of Lyon, Bron, France.,Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan
| | - Shinji Nishimoto
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Japan.,Graduate School of Medicine, Osaka University, Suita, Japan
| |
Collapse
|
46
|
Intracerebral Electrophysiological Recordings to Understand the Neural Basis of Human Face Recognition. Brain Sci 2023; 13:brainsci13020354. [PMID: 36831897 PMCID: PMC9954066 DOI: 10.3390/brainsci13020354] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2022] [Revised: 02/09/2023] [Accepted: 02/10/2023] [Indexed: 02/22/2023] Open
Abstract
Understanding how the human brain recognizes faces is a primary scientific goal in cognitive neuroscience. Given the limitations of the monkey model of human face recognition, a key approach in this endeavor is the recording of electrophysiological activity with electrodes implanted inside the brain of human epileptic patients. However, this approach faces a number of challenges that must be overcome for meaningful scientific knowledge to emerge. Here we synthesize a 10 year research program combining the recording of intracerebral activity (StereoElectroEncephaloGraphy, SEEG) in the ventral occipito-temporal cortex (VOTC) of large samples of participants and fast periodic visual stimulation (FPVS), to objectively define, quantify, and characterize the neural basis of human face recognition. These large-scale studies reconcile the wide distribution of neural face recognition activity with its (right) hemispheric and regional specialization and extend face-selectivity to anterior regions of the VOTC, including the ventral anterior temporal lobe (VATL) typically affected by magnetic susceptibility artifacts in functional magnetic resonance imaging (fMRI). Clear spatial dissociations in category-selectivity between faces and other meaningful stimuli such as landmarks (houses, medial VOTC regions) or written words (left lateralized VOTC) are found, confirming and extending neuroimaging observations while supporting the validity of the clinical population tested to inform about normal brain function. The recognition of face identity - arguably the ultimate form of recognition for the human brain - beyond mere differences in physical features is essentially supported by selective populations of neurons in the right inferior occipital gyrus and the lateral portion of the middle and anterior fusiform gyrus. In addition, low-frequency and high-frequency broadband iEEG signals of face recognition appear to be largely concordant in the human association cortex. We conclude by outlining the challenges of this research program to understand the neural basis of human face recognition in the next 10 years.
Collapse
|
47
|
Pexman PM, Diveica V, Binney RJ. Social semantics: the organization and grounding of abstract concepts. Philos Trans R Soc Lond B Biol Sci 2023; 378:20210363. [PMID: 36571120 PMCID: PMC9791475 DOI: 10.1098/rstb.2021.0363] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022] Open
Abstract
concepts, like justice and friendship, are central features of our daily lives. Traditionally, abstract concepts are distinguished from other concepts in that they cannot be directly experienced through the senses. As such, they pose a challenge for strongly embodied models of semantic representation that assume a central role for sensorimotor information. There is growing recognition, however, that it is possible for meaning to be 'grounded' via cognitive systems, including those involved in processing language and emotion. In this article, we focus on the specific proposal that social significance is a key feature in the representation of some concepts. We begin by reviewing recent evidence in favour of this proposal from the fields of psycholinguistics and neuroimaging. We then discuss the limited extent to which there is consensus about the definition of 'socialness' and propose essential next steps for research in this domain. Taking one such step, we describe preliminary data from an unprecedented large-scale rating study that can help determine how socialness is distinct from other facets of word meaning. We provide a backdrop of contemporary theories regarding semantic representation and social cognition and highlight important predictions for both brain and behaviour. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.
Collapse
Affiliation(s)
- Penny M. Pexman
- Department of Psychology and Hotchkiss Brain Institute, University of Calgary, Calgary, Canada, T2N 1N4
| | - Veronica Diveica
- School of Human and Behavioural Sciences, Bangor University, Bangor LL57 2AS, UK
| | - Richard J. Binney
- School of Human and Behavioural Sciences, Bangor University, Bangor LL57 2AS, UK
| |
Collapse
|
48
|
Abstract
Research on concepts has focused on categorization. Categorization starts with a stimulus. Equally important are episodes that start with a thought. We engage in thinking to draw out new consequences from stored information, or to work out how to act. Each of the concepts out of which thought is constructed provides access to a large body of stored information. Access is not always just a matter of retrieving a stored belief (semantic memory). Often it depends on running a simulation. Simulation allows conceptual thought to draw on information in special-purpose systems, information stored in special-purpose computational dispositions and special-purpose representational structures. While the utility of simulation, prospection or imagination is widely appreciated, the role of concepts in the process is not well understood. This paper turns to cognitive and computational neuroscience for a model of how simulations enable thinkers to reach novel conclusions. Carried over to conceptual thought, the model suggests that concepts are 'plug & play' devices. The distinctive power of thought-driven simulation derives from the ability of concepts to plug into two kinds of structure at once: the combinatorial structure of a thought at one end and special-purpose structural representations at the other. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.
Collapse
Affiliation(s)
- Nicholas Shea
- Faculty of Philosophy, University of Oxford, Radcliffe Humanities, Woodstock Road, Oxford OX2 6GG, UK,Institute of Philosophy, University of London School of Advanced Study, Senate House, Malet Street, London WC1E 7HU, UK
| |
Collapse
|
49
|
Palomar-García MÁ, Villar-Rodríguez E, Pérez-Lozano C, Sanjuán A, Bueichekú E, Miró-Padilla A, Costumero V, Adrián-Ventura J, Parcet MA, Ávila C. Two different brain networks underlying picture naming with familiar pre-existing native words and new vocabulary. BRAIN AND LANGUAGE 2023; 237:105231. [PMID: 36716643 DOI: 10.1016/j.bandl.2023.105231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2021] [Revised: 12/18/2022] [Accepted: 01/18/2023] [Indexed: 06/18/2023]
Abstract
The present research used fMRI to longitudinally investigate the impact of learning new vocabulary on the activation pattern of the language control network by measuring BOLD signal changes during picture naming tasks with familiar pre-existing native words (old words) and new vocabulary. Nineteen healthy participants successfully learned new synonyms for already known Spanish words, and they performed a picture naming task using the old words and the new words immediately after learning and two weeks after learning. The results showed that naming with old words, compared to naming with newly learned words, produced activations in a cortical network involving frontal and parietal regions, whereas the opposite contrast showed activation in a broader cortical/subcortical network, including the SMA/ACC, the hippocampus, and the midbrain. These two networks are maintained two weeks after learning. These results suggest that the language control network can be separated into two functional circuits for diverse cognitive purposes.
Collapse
Affiliation(s)
| | - Esteban Villar-Rodríguez
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | - Cristina Pérez-Lozano
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | - Ana Sanjuán
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | - Elisenda Bueichekú
- Gordon Center for Medical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Anna Miró-Padilla
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | - Victor Costumero
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | | | - María-Antonia Parcet
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| | - César Ávila
- Neuropsychology and Functional Neuroimaging Group, Department of Basic Psychology, Clinical Psychology and Psychobiology, University Jaume I, 12071 Castellón, Spain
| |
Collapse
|
50
|
Koban L, Wager TD, Kober H. A neuromarker for drug and food craving distinguishes drug users from non-users. Nat Neurosci 2023; 26:316-325. [PMID: 36536243 DOI: 10.1038/s41593-022-01228-w] [Citation(s) in RCA: 35] [Impact Index Per Article: 35.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2021] [Accepted: 11/01/2022] [Indexed: 12/24/2022]
Abstract
Craving is a core feature of substance use disorders. It is a strong predictor of substance use and relapse and is linked to overeating, gambling, and other maladaptive behaviors. Craving is measured via self-report, which is limited by introspective access and sociocultural contexts. Neurobiological markers of craving are both needed and lacking, and it remains unclear whether craving for drugs and food involve similar mechanisms. Across three functional magnetic resonance imaging studies (n = 99), we used machine learning to identify a cross-validated neuromarker that predicts self-reported intensity of cue-induced drug and food craving (P < 0.0002). This pattern, which we term the Neurobiological Craving Signature (NCS), includes ventromedial prefrontal and cingulate cortices, ventral striatum, temporal/parietal association areas, mediodorsal thalamus and cerebellum. Importantly, NCS responses to drug versus food cues discriminate drug users versus non-users with 82% accuracy. The NCS is also modulated by a self-regulation strategy. Transfer between separate neuromarkers for drug and food craving suggests shared neurobiological mechanisms. Future studies can assess the discriminant and convergent validity of the NCS and test whether it responds to clinical interventions and predicts long-term clinical outcomes.
Collapse
Affiliation(s)
- Leonie Koban
- Paris Brain Institute (ICM), Inserm, CNRS, Sorbonne Université, Paris, France.
- Centre de Recherche en Neurosciences de Lyon (CRNL), CNRS, INSERM, Université Claude Bernard Lyon 1, Bron, France.
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| | - Hedy Kober
- Department of Psychiatry and Psychology, Yale University, New Haven, CT, USA.
| |
Collapse
|