1
|
Bezsudnova Y, Quinn AJ, Wynn SC, Jensen O. Spatiotemporal Properties of Common Semantic Categories for Words and Pictures. J Cogn Neurosci 2024; 36:1760-1769. [PMID: 38739567 DOI: 10.1162/jocn_a_02182] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
The timing of semantic processing during object recognition in the brain is a topic of ongoing discussion. One way of addressing this question is by applying multivariate pattern analysis to human electrophysiological responses to object images of different semantic categories. However, although multivariate pattern analysis can reveal whether neuronal activity patterns are distinct for different stimulus categories, concerns remain on whether low-level visual features also contribute to the classification results. To circumvent this issue, we applied a cross-decoding approach to magnetoencephalography data from stimuli from two different modalities: images and their corresponding written words. We employed items from three categories and presented them in a randomized order. We show that if the classifier is trained on words, pictures are classified between 150 and 430 msec after stimulus onset, and when training on pictures, words are classified between 225 and 430 msec. The topographical map, identified using a searchlight approach for cross-modal activation in both directions, showed left lateralization, confirming the involvement of linguistic representations. These results point to semantic activation of pictorial stimuli occurring at ∼150 msec, whereas for words, the semantic activation occurs at ∼230 msec.
Collapse
Affiliation(s)
| | | | - Syanah C Wynn
- University of Birmingham
- Gutenberg University Medical Center Mainz
| | | |
Collapse
|
2
|
Mou X, He C, Tan L, Yu J, Liang H, Zhang J, Tian Y, Yang YF, Xu T, Wang Q, Cao M, Chen Z, Hu CP, Wang X, Liu Q, Wu H. ChineseEEG: A Chinese Linguistic Corpora EEG Dataset for Semantic Alignment and Neural Decoding. Sci Data 2024; 11:550. [PMID: 38811613 PMCID: PMC11137001 DOI: 10.1038/s41597-024-03398-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2024] [Accepted: 05/21/2024] [Indexed: 05/31/2024] Open
Abstract
An Electroencephalography (EEG) dataset utilizing rich text stimuli can advance the understanding of how the brain encodes semantic information and contribute to semantic decoding in brain-computer interface (BCI). Addressing the scarcity of EEG datasets featuring Chinese linguistic stimuli, we present the ChineseEEG dataset, a high-density EEG dataset complemented by simultaneous eye-tracking recordings. This dataset was compiled while 10 participants silently read approximately 13 hours of Chinese text from two well-known novels. This dataset provides long-duration EEG recordings, along with pre-processed EEG sensor-level data and semantic embeddings of reading materials extracted by a pre-trained natural language processing (NLP) model. As a pilot EEG dataset derived from natural Chinese linguistic stimuli, ChineseEEG can significantly support research across neuroscience, NLP, and linguistics. It establishes a benchmark dataset for Chinese semantic decoding, aids in the development of BCIs, and facilitates the exploration of alignment between large language models and human cognitive processes. It can also aid research into the brain's mechanisms of language processing within the context of the Chinese natural language.
Collapse
Affiliation(s)
- Xinyu Mou
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Cuilin He
- Centre for Cognitive and Brain Sciences, Department of Psychology, Faculty of Social Sciences, University of Macau, Taipa, Macau SAR, China
| | - Liwei Tan
- Centre for Cognitive and Brain Sciences, Department of Psychology, Faculty of Social Sciences, University of Macau, Taipa, Macau SAR, China
| | - Junjie Yu
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Huadong Liang
- AI Research Institute, iFLYTEK Co., LTD, Hefei, China
| | - Jianyu Zhang
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Yan Tian
- Centre for Cognitive and Brain Sciences, Department of Psychology, Faculty of Social Sciences, University of Macau, Taipa, Macau SAR, China
| | - Yu-Fang Yang
- Division of Experimental Psychology and Neuropsychology, Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Ting Xu
- Center for the Integrative Developmental Neuroscience, Child Mind Institute, New York, NY, USA
| | - Qing Wang
- Shanghai Mental Health Center, School of Medicine, Shanghai Jiao Tong University, 600 S. Wanping Rd., Shanghai, 200030, China
| | - Miao Cao
- Australian National Imaging Facility and Swinburne Neuroimaging Facility, Swinburne University of Technology, Victoria, Australia
| | - Zijiao Chen
- Centre for Cognitive and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Kent Ridge, Singapore
| | - Chuan-Peng Hu
- School of Psychology, Nanjing Normal University, Nanjing, China
| | - Xindi Wang
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Quanying Liu
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China.
| | - Haiyan Wu
- Centre for Cognitive and Brain Sciences, Department of Psychology, Faculty of Social Sciences, University of Macau, Taipa, Macau SAR, China.
| |
Collapse
|
3
|
Faghel-Soubeyrand S, Richoz AR, Waeber D, Woodhams J, Caldara R, Gosselin F, Charest I. Neural computations in prosopagnosia. Cereb Cortex 2024; 34:bhae211. [PMID: 38795358 PMCID: PMC11127037 DOI: 10.1093/cercor/bhae211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Revised: 04/30/2024] [Accepted: 05/03/2024] [Indexed: 05/27/2024] Open
Abstract
We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a well-documented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in a format that facilitates direct comparisons across time points, different individuals, and computational models. To understand how the lesions in PS's ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalization of her brain representations. We uncovered that PS's early brain representations exhibit an unusual similarity to later representations, implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS' brain representations with those of deep neural networks (DNN). We found that the computations underlying PS' brain activity bore a closer resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became more akin to those of the later layers of the model compared to PS. We confirmed PS's deficits in high-level brain representations by demonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.
Collapse
Affiliation(s)
- Simon Faghel-Soubeyrand
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
- Department of Experimental Psychology, University of Oxford, Anna Watts Building, Woodstock Rd, Oxford OX2 6GG
| | - Anne-Raphaelle Richoz
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Delphine Waeber
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Jessica Woodhams
- School of Psychology, University of Birmingham, Hills Building, Edgbaston Park Rd, Birmingham B15 2TT, UK
| | - Roberto Caldara
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Frédéric Gosselin
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
| | - Ian Charest
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
| |
Collapse
|
4
|
Magnabosco F, Hauk O. Decoding Semantics from Dynamic Brain Activation Patterns: From Trials to Task in EEG/MEG Source Space. eNeuro 2024; 11:ENEURO.0277-23.2023. [PMID: 38320767 PMCID: PMC10913025 DOI: 10.1523/eneuro.0277-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 10/21/2023] [Accepted: 11/30/2023] [Indexed: 03/06/2024] Open
Abstract
The temporal dynamics within the semantic brain network and its dependence on stimulus and task parameters are still not well understood. Here, we addressed this by decoding task as well as stimulus information from source-estimated EEG/MEG human data. We presented the same visual word stimuli in a lexical decision (LD) and three semantic decision (SD) tasks. The meanings of the presented words varied across five semantic categories. Source space decoding was applied over time in five ROIs in the left hemisphere (anterior and posterior temporal lobe, inferior frontal gyrus, primary visual areas, and angular gyrus) and one in the right hemisphere (anterior temporal lobe). Task decoding produced sustained significant effects in all ROIs from 50 to 100 ms, both when categorizing tasks with different semantic demands (LD-SD) as well as for similar semantic tasks (SD-SD). In contrast, a semantic word category could only be decoded in lATL, rATL, PTC, and IFG, between 250 and 500 ms. Furthermore, we compared two approaches to source space decoding: conventional ROI-by-ROI decoding and combined-ROI decoding with back-projected activation patterns. The former produced more reliable results for word category decoding while the latter was more informative for task decoding. This indicates that task effects are distributed across the whole semantic network while stimulus effects are more focal. Our results demonstrate that the semantic network is widely distributed but that bilateral anterior temporal lobes together with control regions are particularly relevant for the processing of semantic information.
Collapse
Affiliation(s)
- Federica Magnabosco
- MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge CB2 7EF, United Kingdom
| | - Olaf Hauk
- MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge CB2 7EF, United Kingdom
| |
Collapse
|
5
|
Faghel-Soubeyrand S, Ramon M, Bamps E, Zoia M, Woodhams J, Richoz AR, Caldara R, Gosselin F, Charest I. Decoding face recognition abilities in the human brain. PNAS NEXUS 2024; 3:pgae095. [PMID: 38516275 PMCID: PMC10957238 DOI: 10.1093/pnasnexus/pgae095] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 02/20/2024] [Indexed: 03/23/2024]
Abstract
Why are some individuals better at recognizing faces? Uncovering the neural mechanisms supporting face recognition ability has proven elusive. To tackle this challenge, we used a multimodal data-driven approach combining neuroimaging, computational modeling, and behavioral tests. We recorded the high-density electroencephalographic brain activity of individuals with extraordinary face recognition abilities-super-recognizers-and typical recognizers in response to diverse visual stimuli. Using multivariate pattern analyses, we decoded face recognition abilities from 1 s of brain activity with up to 80% accuracy. To better understand the mechanisms subtending this decoding, we compared representations in the brains of our participants with those in artificial neural network models of vision and semantics, as well as with those involved in human judgments of shape and meaning similarity. Compared to typical recognizers, we found stronger associations between early brain representations of super-recognizers and midlevel representations of vision models as well as shape similarity judgments. Moreover, we found stronger associations between late brain representations of super-recognizers and representations of the artificial semantic model as well as meaning similarity judgments. Overall, these results indicate that important individual variations in brain processing, including neural computations extending beyond purely visual processes, support differences in face recognition abilities. They provide the first empirical evidence for an association between semantic computations and face recognition abilities. We believe that such multimodal data-driven approaches will likely play a critical role in further revealing the complex nature of idiosyncratic face recognition in the human brain.
Collapse
Affiliation(s)
- Simon Faghel-Soubeyrand
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6GG, UK
- Département de psychologie, Université de Montréal, Montréal, Québec H2V 2S9, Canada
| | - Meike Ramon
- Institute of Psychology, University of Lausanne, Lausanne CH-1015, Switzerland
| | - Eva Bamps
- Center for Contextual Psychiatry, Department of Neurosciences, KU Leuven, Leuven ON5, Belgium
| | - Matteo Zoia
- Department for Biomedical Research, University of Bern, Bern 3008, Switzerland
| | - Jessica Woodhams
- Département de psychologie, Université de Montréal, Montréal, Québec H2V 2S9, Canada
- School of Psychology, University of Birmingham, Hills Building, Edgbaston Park Rd, Birmingham B15 2TT, UK
| | | | - Roberto Caldara
- Département de Psychology, Université de Fribourg, Fribourg CH-1700, Switzerland
| | - Frédéric Gosselin
- Département de psychologie, Université de Montréal, Montréal, Québec H2V 2S9, Canada
| | - Ian Charest
- Département de psychologie, Université de Montréal, Montréal, Québec H2V 2S9, Canada
| |
Collapse
|
6
|
Zheng XY, Hebart MN, Grill F, Dolan RJ, Doeller CF, Cools R, Garvert MM. Parallel cognitive maps for multiple knowledge structures in the hippocampal formation. Cereb Cortex 2024; 34:bhad485. [PMID: 38204296 PMCID: PMC10839836 DOI: 10.1093/cercor/bhad485] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Revised: 11/27/2023] [Accepted: 11/30/2023] [Indexed: 01/12/2024] Open
Abstract
The hippocampal-entorhinal system uses cognitive maps to represent spatial knowledge and other types of relational information. However, objects can often be characterized by different types of relations simultaneously. How does the hippocampal formation handle the embedding of stimuli in multiple relational structures that differ vastly in their mode and timescale of acquisition? Does the hippocampal formation integrate different stimulus dimensions into one conjunctive map or is each dimension represented in a parallel map? Here, we reanalyzed human functional magnetic resonance imaging data from Garvert et al. (2017) that had previously revealed a map in the hippocampal formation coding for a newly learnt transition structure. Using functional magnetic resonance imaging adaptation analysis, we found that the degree of representational similarity in the bilateral hippocampus also decreased as a function of the semantic distance between presented objects. Importantly, while both map-like structures localized to the hippocampal formation, the semantic map was located in more posterior regions of the hippocampal formation than the transition structure and thus anatomically distinct. This finding supports the idea that the hippocampal-entorhinal system forms parallel cognitive maps that reflect the embedding of objects in diverse relational structures.
Collapse
Affiliation(s)
- Xiaochen Y Zheng
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6525 EN, Nijmegen, the Netherlands
| | - Martin N Hebart
- Max-Planck-Institute for Human Cognitive and Brain Sciences, 04103, Leipzig, Germany
- Department of Medicine, Justus Liebig University, 35390, Giessen, Germany
| | - Filip Grill
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6525 EN, Nijmegen, the Netherlands
- Radboud University Medical Center, Department of Neurology, 6525 GA, Nijmegen, the Netherlands
| | - Raymond J Dolan
- Wellcome Centre for Human Neuroimaging, University College London, London WC1N 3AR, United Kingdom
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London WC1B 5EH, United Kingdom
| | - Christian F Doeller
- Max-Planck-Institute for Human Cognitive and Brain Sciences, 04103, Leipzig, Germany
- Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer's Disease, NTNU, 7491, Trondheim, Norway
- Wilhelm Wundt Institute of Psychology, Leipzig University, 04109, Leipzig, Germany
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6525 EN, Nijmegen, the Netherlands
- Radboud University Medical Center, Department of Psychiatry, 6525 GA, Nijmegen, the Netherlands
| | - Mona M Garvert
- Max-Planck-Institute for Human Cognitive and Brain Sciences, 04103, Leipzig, Germany
- Max Planck Research Group NeuroCode, Max Planck Institute for Human Development, 14195, Berlin, Germany
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany
- Faculty of Human Sciences, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
| |
Collapse
|
7
|
Duran ND, Paige A, D'Mello SK. Multi-Level Linguistic Alignment in a Dynamic Collaborative Problem-Solving Task. Cogn Sci 2024; 48:e13398. [PMID: 38212897 DOI: 10.1111/cogs.13398] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 10/13/2023] [Accepted: 12/08/2023] [Indexed: 01/13/2024]
Abstract
Cocreating meaning in collaboration is challenging. Success is often determined by people's abilities to coordinate their language to converge upon shared mental representations. Here we explore one set of low-level linguistic behaviors, linguistic alignment, that both emerges from, and facilitates, outcomes of high-level convergence. Linguistic alignment captures the ways people reuse, that is, "align to," the lexical, syntactic, and semantic forms of others' utterances. Our focus is on the temporal change of multi-level linguistic alignment, as well as how alignment is related to communicative outcomes within a unique collaborative problem-solving paradigm. The primary task, situated within a virtual educational video game, requires creative thinking between three people where the paths for possible solutions are highly variable. We find that over time interactions are marked by decreasing lexical and syntactic alignment, with a trade-off of increasing semantic alignment. However, greater semantic alignment does not translate into better team performance. Overall, these findings provide greater clarity on the role of linguistic coordination within complex and dynamic collaborative problem-solving tasks.
Collapse
Affiliation(s)
- Nicholas D Duran
- School of Social and Behavioral Sciences, Arizona State University
| | - Amie Paige
- Department of Psychology, Stony Brook University
| | - Sidney K D'Mello
- Institute of Cognitive Science and Department of Computer Science, University of Colorado Boulder
| |
Collapse
|