1
|
Thye M, Hoffman P, Mirman D. The neural basis of naturalistic semantic and social cognition. Sci Rep 2024; 14:6796. [PMID: 38514738 PMCID: PMC10957894 DOI: 10.1038/s41598-024-56897-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Accepted: 03/11/2024] [Indexed: 03/23/2024] Open
Abstract
Decoding social environments and engaging meaningfully with other people are critical aspects of human cognition. Multiple cognitive systems, including social and semantic cognition, work alongside each other to support these processes. This study investigated shared processing between social and semantic systems using neuroimaging data collected during movie-viewing, which captures the multimodal environment in which social knowledge is exchanged. Semantic and social content from movie events (event-level) and movie transcripts (word-level) were used in parametric modulation analyses to test (1) the degree to which semantic and social information is processed within each respective network and (2) engagement of the same cross-network regions or the same domain-general hub located within the semantic network during semantic and social processing. Semantic word and event-level content engaged the same fronto-temporo-parietal network and a portion of the semantic hub in the anterior temporal lobe (ATL). Social word and event-level content engaged the supplementary motor area and right angular gyrus within the social network, but only social words engaged the domain-general semantic hub in left ATL. There was evidence of shared processing between the social and semantic systems in the dorsolateral portion of right ATL which was engaged by word and event-level semantic and social content. Overlap between the semantic and social word and event results was highly variable within and across participants, with the most consistent loci of overlap occurring in left inferior frontal, bilateral precentral and supramarginal gyri for social and semantic words and in bilateral superior temporal gyrus extending from ATL posteriorly into supramarginal gyri for social and semantic events. These results indicate a complex pattern of shared and distinct regions for social and semantic cognition during naturalistic processing. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on October 11, 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/ACWQY .
Collapse
Affiliation(s)
- Melissa Thye
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK.
| | - Paul Hoffman
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK
| | - Daniel Mirman
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK
| |
Collapse
|
2
|
Lin N, Zhang X, Wang X, Wang S. The organization of the semantic network as reflected by the neural correlates of six semantic dimensions. BRAIN AND LANGUAGE 2024; 250:105388. [PMID: 38295716 DOI: 10.1016/j.bandl.2024.105388] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 01/23/2024] [Accepted: 01/26/2024] [Indexed: 03/03/2024]
Abstract
Multiple sensory-motor and non-sensory-motor dimensions have been proposed for semantic representation, but it remains unclear how the semantic system is organized along them in the human brain. Using naturalistic fMRI data and large-scale semantic ratings, we investigated the overlaps and dissociations between the neural correlates of six semantic dimensions: vision, motor, socialness, emotion, space, and time. Our findings revealed a more complex semantic atlas than what is predicted by current neurobiological models of semantic representation. Brain regions that are selectively sensitive to specific semantic dimensions were found both within and outside the brain networks assumed to represent multimodal general and/or abstract semantics. Overlaps between the neural correlates of different semantic dimensions were mainly found inside the default mode network, concentrated in the left anterior superior temporal gyrus and angular gyrus, which have been proposed as two connector hubs that bridge the multimodal experiential semantic system and the language-supported semantic system.
Collapse
Affiliation(s)
- Nan Lin
- CAS Key Laboratory of Behavioural Sciences, Institute of Psychology, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China.
| | - Xiaohan Zhang
- State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Xiuyi Wang
- CAS Key Laboratory of Behavioural Sciences, Institute of Psychology, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Shaonan Wang
- State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
3
|
Desai RH, Hackett CT, Johari K, Lai VT, Riccardi N. Spatiotemporal characteristics of the neural representation of event concepts. BRAIN AND LANGUAGE 2023; 246:105328. [PMID: 37847931 PMCID: PMC10873121 DOI: 10.1016/j.bandl.2023.105328] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 09/25/2023] [Accepted: 10/01/2023] [Indexed: 10/19/2023]
Abstract
Events are a fundamentally important part of our understanding of the world. How lexical concepts denoting events are represented in the brain remains controversial. We conducted two experiments using event and object nouns matched on a range of psycholinguistic variables, including concreteness, to examine spatial and temporal characteristics of event concepts. Both experiments used magnitude and valence tasks on event and object nouns. The fMRI experiment revealed a distributed set of regions for events, including the angular gyrus, anterior temporal lobe, and posterior cingulate across tasks. In the EEG experiment, events and objects differed in amplitude within the 300-500 ms window. Together these results shed light into the spatiotemporal characteristics of event concept representation and show that event concepts are represented in the putative hubs of the semantic system. While these hubs are typically associated with object semantics, they also represent events, and have a likely role in temporal integration.
Collapse
Affiliation(s)
- Rutvik H Desai
- Department of Psychology, University of South Carolina, United States; Institute for Mind and Brain, University of South Carolina, United States.
| | | | - Karim Johari
- Department of Communication Sciences & Disorders, Louisiana State University, United States
| | - Vicky T Lai
- Department of Psychology, University of Arizona, United States
| | - Nicholas Riccardi
- Department of Psychology, University of South Carolina, United States
| |
Collapse
|
4
|
Zhang G, Xu Y, Wang X, Li J, Shi W, Bi Y, Lin N. A social-semantic working-memory account for two canonical language areas. Nat Hum Behav 2023; 7:1980-1997. [PMID: 37735521 DOI: 10.1038/s41562-023-01704-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 08/22/2023] [Indexed: 09/23/2023]
Abstract
Language and social cognition are traditionally studied as separate cognitive domains, yet accumulative studies reveal overlapping neural correlates at the left ventral temporoparietal junction (vTPJ) and the left lateral anterior temporal lobe (lATL), which have been attributed to sentence processing and social concept activation. We propose a common cognitive component underlying both effects: social-semantic working memory. We confirmed two key predictions of our hypothesis using functional MRI. First, the left vTPJ and lATL showed sensitivity to sentences only when the sentences conveyed social meaning; second, these regions showed persistent social-semantic-selective activity after the linguistic stimuli disappeared. We additionally found that both regions were sensitive to the socialness of non-linguistic stimuli and were more tightly connected with the social-semantic-processing areas than with the sentence-processing areas. The converging evidence indicates the social-semantic working-memory function of the left vTPJ and lATL and challenges the general-semantic and/or syntactic accounts for the neural activity of these regions.
Collapse
Affiliation(s)
- Guangyao Zhang
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China
- IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
| | - Yangwen Xu
- Center for Mind/Brain Sciences, University of Trento, Trento, Italy
| | - Xiuyi Wang
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Jixing Li
- Department of Linguistics and Translation, City University of Hong Kong, Hong Kong SAR, China
| | - Weiting Shi
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Yanchao Bi
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China
- IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
- Chinese Institute for Brain Research, Beijing, China
| | - Nan Lin
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing, China.
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China.
| |
Collapse
|
5
|
Landsiedel J, Koldewyn K. Auditory dyadic interactions through the "eye" of the social brain: How visual is the posterior STS interaction region? IMAGING NEUROSCIENCE (CAMBRIDGE, MASS.) 2023; 1:1-20. [PMID: 37719835 PMCID: PMC10503480 DOI: 10.1162/imag_a_00003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Accepted: 05/17/2023] [Indexed: 09/19/2023]
Abstract
Human interactions contain potent social cues that meet not only the eye but also the ear. Although research has identified a region in the posterior superior temporal sulcus as being particularly sensitive to visually presented social interactions (SI-pSTS), its response to auditory interactions has not been tested. Here, we used fMRI to explore brain response to auditory interactions, with a focus on temporal regions known to be important in auditory processing and social interaction perception. In Experiment 1, monolingual participants listened to two-speaker conversations (intact or sentence-scrambled) and one-speaker narrations in both a known and an unknown language. Speaker number and conversational coherence were explored in separately localised regions-of-interest (ROI). In Experiment 2, bilingual participants were scanned to explore the role of language comprehension. Combining univariate and multivariate analyses, we found initial evidence for a heteromodal response to social interactions in SI-pSTS. Specifically, right SI-pSTS preferred auditory interactions over control stimuli and represented information about both speaker number and interactive coherence. Bilateral temporal voice areas (TVA) showed a similar, but less specific, profile. Exploratory analyses identified another auditory-interaction sensitive area in anterior STS. Indeed, direct comparison suggests modality specific tuning, with SI-pSTS preferring visual information while aSTS prefers auditory information. Altogether, these results suggest that right SI-pSTS is a heteromodal region that represents information about social interactions in both visual and auditory domains. Future work is needed to clarify the roles of TVA and aSTS in auditory interaction perception and further probe right SI-pSTS interaction-selectivity using non-semantic prosodic cues.
Collapse
Affiliation(s)
- Julia Landsiedel
- Department of Psychology, School of Human and Behavioural Sciences, Bangor University, Bangor, United Kingdom
| | - Kami Koldewyn
- Department of Psychology, School of Human and Behavioural Sciences, Bangor University, Bangor, United Kingdom
| |
Collapse
|
6
|
Wang S, Zhang Y, Shi W, Zhang G, Zhang J, Lin N, Zong C. A large dataset of semantic ratings and its computational extension. Sci Data 2023; 10:106. [PMID: 36823158 PMCID: PMC9950052 DOI: 10.1038/s41597-023-01995-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 01/31/2023] [Indexed: 02/25/2023] Open
Abstract
Evidence from psychology and cognitive neuroscience indicates that the human brain's semantic system contains several specific subsystems, each representing a particular dimension of semantic information. Word ratings on these different semantic dimensions can help investigate the behavioral and neural impacts of semantic dimensions on language processes and build computational representations of language meaning according to the semantic space of the human cognitive system. Existing semantic rating databases provide ratings for hundreds to thousands of words, which can hardly support a comprehensive semantic analysis of natural texts or speech. This article reports a large database, the Six Semantic Dimension Database (SSDD), which contains subjective ratings for 17,940 commonly used Chinese words on six major semantic dimensions: vision, motor, socialness, emotion, time, and space. Furthermore, using computational models to learn the mapping relations between subjective ratings and word embeddings, we include the estimated semantic ratings for 1,427,992 Chinese and 1,515,633 English words in the SSDD. The SSDD will aid studies on natural language processing, text analysis, and semantic representation in the brain.
Collapse
Affiliation(s)
- Shaonan Wang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Yunhao Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Weiting Shi
- CAS Key Laboratory of Behavioural Sciences, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Guangyao Zhang
- CAS Key Laboratory of Behavioural Sciences, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Jiajun Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Nan Lin
- CAS Key Laboratory of Behavioural Sciences, Institute of Psychology, Beijing, China.
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China.
| | - Chengqing Zong
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
7
|
Pexman PM, Diveica V, Binney RJ. Social semantics: the organization and grounding of abstract concepts. Philos Trans R Soc Lond B Biol Sci 2023; 378:20210363. [PMID: 36571120 PMCID: PMC9791475 DOI: 10.1098/rstb.2021.0363] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022] Open
Abstract
concepts, like justice and friendship, are central features of our daily lives. Traditionally, abstract concepts are distinguished from other concepts in that they cannot be directly experienced through the senses. As such, they pose a challenge for strongly embodied models of semantic representation that assume a central role for sensorimotor information. There is growing recognition, however, that it is possible for meaning to be 'grounded' via cognitive systems, including those involved in processing language and emotion. In this article, we focus on the specific proposal that social significance is a key feature in the representation of some concepts. We begin by reviewing recent evidence in favour of this proposal from the fields of psycholinguistics and neuroimaging. We then discuss the limited extent to which there is consensus about the definition of 'socialness' and propose essential next steps for research in this domain. Taking one such step, we describe preliminary data from an unprecedented large-scale rating study that can help determine how socialness is distinct from other facets of word meaning. We provide a backdrop of contemporary theories regarding semantic representation and social cognition and highlight important predictions for both brain and behaviour. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.
Collapse
Affiliation(s)
- Penny M. Pexman
- Department of Psychology and Hotchkiss Brain Institute, University of Calgary, Calgary, Canada, T2N 1N4
| | - Veronica Diveica
- School of Human and Behavioural Sciences, Bangor University, Bangor LL57 2AS, UK
| | - Richard J. Binney
- School of Human and Behavioural Sciences, Bangor University, Bangor LL57 2AS, UK
| |
Collapse
|
8
|
Coexistence of the social semantic effect and non-semantic effect in the default mode network. Brain Struct Funct 2023; 228:321-339. [PMID: 35394555 DOI: 10.1007/s00429-022-02476-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2021] [Accepted: 02/23/2022] [Indexed: 01/07/2023]
Abstract
Neuroimaging studies have found both semantic and non-semantic effects in the default mode network (DMN), leading to an intense debate on the role of the DMN in semantic processes. Four different views have been proposed: (1) the general semantic view holds that the DMN contains several hub regions supporting general semantic processes; (2) the non-semantic view holds that the semantic effects observed in the DMN (especially the ventral angular gyrus) are confounded by difficulty and do not reflect semantic processing per se; (3) the multifunction view holds that the same areas in the DMN can support both semantic and non-semantic functions; and (4) the multisystem view holds that the DMN contains multiple subnetworks supporting different aspects of semantic processes separately. Using an fMRI experiment, we found that in one of the subnetworks of the DMN, called the social semantic network, all areas showed social semantic activation and difficulty-induced deactivation. The distributions of two non-semantic effects, that is, difficulty-induced and task-induced deactivations, showed dissociation in the DMN. In the bilateral angular gyri, the ventral subdivisions showed social semantic activation independent of difficulty, while the dorsal subdivisions showed no semantic effect but difficulty-induced activation. Our findings provide two insights into the semantic and non-semantic functions of the DMN, which are consistent with both the multisystem and multifunction views: first, the same areas of the DMN can support both social semantic and non-semantic functions; second, similar to the multiple semantic effects of the DMN, the non-semantic effects also vary across its subsystems.
Collapse
|
9
|
From words to phrases: neural basis of social event semantic composition. Brain Struct Funct 2022; 227:1683-1695. [PMID: 35184222 PMCID: PMC9098591 DOI: 10.1007/s00429-022-02465-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 01/25/2022] [Indexed: 11/13/2022]
Abstract
Events are typically composed of at least actions and entities. Both actions and entities have been shown to be represented by neural structures respecting domain organizations in the brain, including those of social/animate (face and body; person-directed action) versus inanimate (man-made object or tool; object-directed action) concepts. It is unclear whether the brain combines actions and entities into events in a (relative) domain-specific fashion or via domain-general mechanisms in regions that have been shown to support semantic and syntactic composition. We tested these hypotheses in a functional magnetic resonance imaging experiment where two domains of verb-noun event phrases (social-person versus manipulation-artifact, e.g., “hug mother” versus “fold napkin”) and their component words were contrasted. We found a series of brain region supporting social-composition effects more strongly than the manipulation phrase composition—the bilateral inferior occipital gyrus (IOG), inferior temporal gyrus (ITG) and anterior temporal lobe (ATL)—which either showed stronger activation strength tested by univariate contrast, stronger content representation tested by representation similarity analysis, or stronger relationship between the neural activation patterns of phrases and synthesis (additive and multiplication) of the neural activity patterns of the word constituents. No regions were observed showing evidence of phrase composition for both domains or stronger effects of manipulation phrases. These findings highlight the roles of the visual cortex and ATL in social event compositions, suggesting a domain-preferring, rather than domain-general, mechanisms of verbal event composition.
Collapse
|