1
|
Song M, Wang J, Cai Q. The unique contribution of uncertainty reduction during naturalistic language comprehension. Cortex 2024; 181:12-25. [PMID: 39447486 DOI: 10.1016/j.cortex.2024.09.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Revised: 07/21/2024] [Accepted: 09/24/2024] [Indexed: 10/26/2024]
Abstract
Language comprehension is an incremental process with prediction. Delineating various mental states during such a process is critical to understanding the relationship between human cognition and the properties of language. Entropy reduction, which indicates the dynamic decrease of uncertainty as language input unfolds, has been recognized as effective in predicting neural responses during comprehension. According to the entropy reduction hypothesis (Hale, 2006), entropy reduction is related to the processing difficulty of a word, the effect of which may overlap with other well-documented information-theoretical metrics such as surprisal or next-word entropy. However, the processing difficulty was often confused with the information conveyed by a word, especially lacking neural differentiation. We propose that entropy reduction represents the cognitive neural process of information gain that can be dissociated from processing difficulty. This study characterized various information-theoretical metrics using GPT-2 and identified the unique effects of entropy reduction in predicting fMRI time series acquired during language comprehension. In addition to the effects of surprisal and entropy, entropy reduction was associated with activations in the left inferior frontal gyrus, bilateral ventromedial prefrontal cortex, insula, thalamus, basal ganglia, and middle cingulate cortex. The reduction of uncertainty, rather than its fluctuation, proved to be an effective factor in modeling neural responses. The neural substrates underlying the reduction in uncertainty might imply the brain's desire for information regardless of processing difficulty.
Collapse
Affiliation(s)
- Ming Song
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), Affiliated Mental Health Center (ECNU), Institute of Brain and Education Innovation, School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China; Shanghai Changning Mental Health Center, Shanghai, China; Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China
| | - Jing Wang
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), Affiliated Mental Health Center (ECNU), Institute of Brain and Education Innovation, School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China; Shanghai Changning Mental Health Center, Shanghai, China; Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China.
| | - Qing Cai
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), Affiliated Mental Health Center (ECNU), Institute of Brain and Education Innovation, School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China; Shanghai Changning Mental Health Center, Shanghai, China; Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China.
| |
Collapse
|
2
|
Yu S, Gu C, Huang K, Li P. Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension. SCIENCE ADVANCES 2024; 10:eadn7744. [PMID: 38781343 PMCID: PMC11114233 DOI: 10.1126/sciadv.adn7744] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Accepted: 04/18/2024] [Indexed: 05/25/2024]
Abstract
Current large language models (LLMs) rely on word prediction as their backbone pretraining task. Although word prediction is an important mechanism underlying language processing, human language comprehension occurs at multiple levels, involving the integration of words and sentences to achieve a full understanding of discourse. This study models language comprehension by using the next sentence prediction (NSP) task to investigate mechanisms of discourse-level comprehension. We show that NSP pretraining enhanced a model's alignment with brain data especially in the right hemisphere and in the multiple demand network, highlighting the contributions of nonclassical language regions to high-level language understanding. Our results also suggest that NSP can enable the model to better capture human comprehension performance and to better encode contextual information. Our study demonstrates that the inclusion of diverse learning objectives in a model leads to more human-like representations, and investigating the neurocognitive plausibility of pretraining tasks in LLMs can shed light on outstanding questions in language neuroscience.
Collapse
Affiliation(s)
- Shaoyun Yu
- Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Chanyuan Gu
- Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Kexin Huang
- Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Ping Li
- Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong SAR, China
- Centre for Immersive Learning and Metaverse in Education, The Hong Kong Polytechnic University, Hong Kong SAR, China
| |
Collapse
|
3
|
Fedorenko E, Ivanova AA, Regev TI. The language network as a natural kind within the broader landscape of the human brain. Nat Rev Neurosci 2024; 25:289-312. [PMID: 38609551 DOI: 10.1038/s41583-024-00802-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/23/2024] [Indexed: 04/14/2024]
Abstract
Language behaviour is complex, but neuroscientific evidence disentangles it into distinct components supported by dedicated brain areas or networks. In this Review, we describe the 'core' language network, which includes left-hemisphere frontal and temporal areas, and show that it is strongly interconnected, independent of input and output modalities, causally important for language and language-selective. We discuss evidence that this language network plausibly stores language knowledge and supports core linguistic computations related to accessing words and constructions from memory and combining them to interpret (decode) or generate (encode) linguistic messages. We emphasize that the language network works closely with, but is distinct from, both lower-level - perceptual and motor - mechanisms and higher-level systems of knowledge and reasoning. The perceptual and motor mechanisms process linguistic signals, but, in contrast to the language network, are sensitive only to these signals' surface properties, not their meanings; the systems of knowledge and reasoning (such as the system that supports social reasoning) are sometimes engaged during language use but are not language-selective. This Review lays a foundation both for in-depth investigations of these different components of the language processing pipeline and for probing inter-component interactions.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- The Program in Speech and Hearing in Bioscience and Technology, Harvard University, Cambridge, MA, USA.
| | - Anna A Ivanova
- School of Psychology, Georgia Institute of Technology, Atlanta, GA, USA
| | - Tamar I Regev
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
4
|
Billot A, Kiran S. Disentangling neuroplasticity mechanisms in post-stroke language recovery. BRAIN AND LANGUAGE 2024; 251:105381. [PMID: 38401381 PMCID: PMC10981555 DOI: 10.1016/j.bandl.2024.105381] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 11/28/2023] [Accepted: 01/12/2024] [Indexed: 02/26/2024]
Abstract
A major objective in post-stroke aphasia research is to gain a deeper understanding of neuroplastic mechanisms that drive language recovery, with the ultimate goal of enhancing treatment outcomes. Subsequent to recent advances in neuroimaging techniques, we now have the ability to examine more closely how neural activity patterns change after a stroke. However, the way these neural activity changes relate to language impairments and language recovery is still debated. The aim of this review is to provide a theoretical framework to better investigate and interpret neuroplasticity mechanisms underlying language recovery in post-stroke aphasia. We detail two sets of neuroplasticity mechanisms observed at the synaptic level that may explain functional neuroimaging findings in post-stroke aphasia recovery at the network level: feedback-based homeostatic plasticity and associative Hebbian plasticity. In conjunction with these plasticity mechanisms, higher-order cognitive control processes dynamically modulate neural activity in other regions to meet communication demands, despite reduced neural resources. This work provides a network-level neurobiological framework for understanding neural changes observed in post-stroke aphasia and can be used to define guidelines for personalized treatment development.
Collapse
Affiliation(s)
- Anne Billot
- Center for Brain Recovery, Boston University, Boston, USA; Department of Psychology, Center for Brain Science, Harvard University, Cambridge, Massachusetts, USA; Department of Neurology, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Swathi Kiran
- Center for Brain Recovery, Boston University, Boston, USA.
| |
Collapse
|
5
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. Nat Hum Behav 2024; 8:544-561. [PMID: 38172630 DOI: 10.1038/s41562-023-01783-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 11/10/2023] [Indexed: 01/05/2024]
Abstract
Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of the brain response associated with each sentence. We then use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress the activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also non-invasively control neural activity in higher-level cortical areas, such as the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- MIT-IBM Watson AI Lab, Cambridge, MA, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA, USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
6
|
Malik-Moraleda S, Jouravlev O, Taliaferro M, Mineroff Z, Cucu T, Mahowald K, Blank IA, Fedorenko E. Functional characterization of the language network of polyglots and hyperpolyglots with precision fMRI. Cereb Cortex 2024; 34:bhae049. [PMID: 38466812 PMCID: PMC10928488 DOI: 10.1093/cercor/bhae049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Revised: 01/24/2024] [Accepted: 01/25/2024] [Indexed: 03/13/2024] Open
Abstract
How do polyglots-individuals who speak five or more languages-process their languages, and what can this population tell us about the language system? Using fMRI, we identified the language network in each of 34 polyglots (including 16 hyperpolyglots with knowledge of 10+ languages) and examined its response to the native language, non-native languages of varying proficiency, and unfamiliar languages. All language conditions engaged all areas of the language network relative to a control condition. Languages that participants rated as higher proficiency elicited stronger responses, except for the native language, which elicited a similar or lower response than a non-native language of similar proficiency. Furthermore, unfamiliar languages that were typologically related to the participants' high-to-moderate-proficiency languages elicited a stronger response than unfamiliar unrelated languages. The results suggest that the language network's response magnitude scales with the degree of engagement of linguistic computations (e.g. related to lexical access and syntactic-structure building). We also replicated a prior finding of weaker responses to native language in polyglots than non-polyglot bilinguals. These results contribute to our understanding of how multiple languages coexist within a single brain and provide new evidence that the language network responds more strongly to stimuli that more fully engage linguistic computations.
Collapse
Affiliation(s)
- Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114, United States
| | - Olessia Jouravlev
- Department of Cognitive Science, Carleton University, Ottawa K1S 5B6, Canada
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
| | - Zachary Mineroff
- Eberly Center, Carnegie Mellon University, Pittsburgh, PA 15289, United States
| | - Theodore Cucu
- Department of Psychology, Carnegie Mellon University, Pittsburgh, PA 15289, United States
| | - Kyle Mahowald
- Department of Linguistics, The University of Texas at Austin, Austin, TX 78712, United States
| | - Idan A Blank
- Department of Psychology, University of California Los Angeles, Los Angeles, CA 90095, United States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114, United States
| |
Collapse
|
7
|
Arvidsson C, Torubarova E, Pereira A, Uddén J. Conversational production and comprehension: fMRI-evidence reminiscent of but deviant from the classical Broca-Wernicke model. Cereb Cortex 2024; 34:bhae073. [PMID: 38501383 PMCID: PMC10949358 DOI: 10.1093/cercor/bhae073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 02/02/2024] [Accepted: 02/03/2024] [Indexed: 03/20/2024] Open
Abstract
A key question in research on the neurobiology of language is to which extent the language production and comprehension systems share neural infrastructure, but this question has not been addressed in the context of conversation. We utilized a public fMRI dataset where 24 participants engaged in unscripted conversations with a confederate outside the scanner, via an audio-video link. We provide evidence indicating that the two systems share neural infrastructure in the left-lateralized perisylvian language network, but diverge regarding the level of activation in regions within the network. Activity in the left inferior frontal gyrus was stronger in production compared to comprehension, while comprehension showed stronger recruitment of the left anterior middle temporal gyrus and superior temporal sulcus, compared to production. Although our results are reminiscent of the classical Broca-Wernicke model, the anterior (rather than posterior) temporal activation is a notable difference from that model. This is one of the findings that may be a consequence of the conversational setting, another being that conversational production activated what we interpret as higher-level socio-pragmatic processes. In conclusion, we present evidence for partial overlap and functional asymmetry of the neural infrastructure of production and comprehension, in the above-mentioned frontal vs temporal regions during conversation.
Collapse
Affiliation(s)
- Caroline Arvidsson
- Department of Linguistics, Stockholm University, Universitetsvägen 10 C, 114 18 Stockholm, Sweden
| | - Ekaterina Torubarova
- Division of Speech, Music, and Hearing, KTH Royal Institute of Technology, Lindstedtsvägen 24, 114 28 Stockholm, Sweden
| | - André Pereira
- Division of Speech, Music, and Hearing, KTH Royal Institute of Technology, Lindstedtsvägen 24, 114 28 Stockholm, Sweden
| | - Julia Uddén
- Department of Linguistics, Stockholm University, Universitetsvägen 10 C, 114 18 Stockholm, Sweden
- Department of Psychology, Stockholm University, Albanovägen 12, 114 19 Stockholm, Sweden
| |
Collapse
|
8
|
Wat EK, Jangraw DC, Finn ES, Bandettini PA, Preston JL, Landi N, Hoeft F, Frost SJ, Lau A, Chen G, Pugh KR, Molfese PJ. Will you read how I will read? Naturalistic fMRI predictors of emergent reading. Neuropsychologia 2024; 193:108763. [PMID: 38141965 PMCID: PMC11370251 DOI: 10.1016/j.neuropsychologia.2023.108763] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Revised: 10/07/2023] [Accepted: 12/16/2023] [Indexed: 12/25/2023]
Abstract
Despite reading being an essential and almost universal skill in the developed world, reading proficiency varies substantially from person to person. To study why, the fMRI field is beginning to turn from single-word or nonword reading tasks to naturalistic stimuli like connected text and listening to stories. To study reading development in children just beginning to read, listening to stories is an appropriate paradigm because speech perception and phonological processing are important for, and are predictors of, reading proficiency. Our study examined the relationship between behavioral reading-related skills and the neural response to listening to stories in the fMRI environment. Functional MRI were gathered in a 3T TIM-Trio scanner. During the fMRI scan, children aged approximately 7 years listened to professionally narrated common short stories and answered comprehension questions following the narration. Analyses of the data used inter-subject correlation (ISC), and representational similarity analysis (RSA). Our primary finding is that ISC reveals areas of increased synchrony in both high- and low-performing emergent readers previously implicated in reading ability/disability. Of particular interest are that several previously identified brain regions (medial temporal gyrus (MTG), inferior frontal gyrus (IFG), inferior temporal gyrus (ITG)) were found to "synchronize" across higher reading ability participants, while lower reading ability participants had idiosyncratic activation patterns in these regions. Additionally, two regions (superior frontal gyrus (SFG) and another portion of ITG) were recruited by all participants, but their specific timecourse of activation depended on reading performance. These analyses support the idea that different brain regions involved in reading follow different developmental trajectories that correlate with reading proficiency on a spectrum rather than the usual dichotomy of poor readers versus strong readers.
Collapse
Affiliation(s)
| | - David C Jangraw
- Department of Electrical and Biomedical Engineering, University of Vermont, Burlington, VT, USA
| | - Emily S Finn
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Peter A Bandettini
- Section on Functional Imaging Methods, NIMH, Bethesda, MD, USA; Center for Multimodal Neuroimaging, NIMH, Bethesda, MD, USA
| | - Jonathan L Preston
- Haskins Laboratories, New Haven, CT, USA; Syracuse University, Syracuse, NY, USA
| | - Nicole Landi
- Haskins Laboratories, New Haven, CT, USA; Department of Psychological Sciences, University of Connecticut, USA
| | - Fumiko Hoeft
- Haskins Laboratories, New Haven, CT, USA; Department of Psychological Sciences, University of Connecticut, USA
| | | | - Airey Lau
- Haskins Laboratories, New Haven, CT, USA
| | - Gang Chen
- Statistical Computing Core, NIMH, Bethesda, MD, USA
| | - Kenneth R Pugh
- Haskins Laboratories, New Haven, CT, USA; Department of Psychological Sciences, University of Connecticut, USA; Department of Linguistics, Yale University School of Medicine, New Haven, CT, USA
| | - Peter J Molfese
- Center for Multimodal Neuroimaging, NIMH, Bethesda, MD, USA; Haskins Laboratories, New Haven, CT, USA.
| |
Collapse
|
9
|
Wolna A, Szewczyk J, Diaz M, Domagalik A, Szwed M, Wodniecka Z. Domain-general and language-specific contributions to speech production in a second language: an fMRI study using functional localizers. Sci Rep 2024; 14:57. [PMID: 38168139 PMCID: PMC10761726 DOI: 10.1038/s41598-023-49375-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Accepted: 12/07/2023] [Indexed: 01/05/2024] Open
Abstract
For bilinguals, speaking in a second language (L2) compared to the native language (L1) is usually more difficult. In this study we asked whether the difficulty in L2 production reflects increased demands imposed on domain-general or core language mechanisms. We compared the brain response to speech production in L1 and L2 within two functionally-defined networks in the brain: the Multiple Demand (MD) network and the language network. We found that speech production in L2 was linked to a widespread increase of brain activity in the domain-general MD network. The language network did not show a similarly robust differences in processing speech in the two languages, however, we found increased response to L2 production in the language-specific portion of the left inferior frontal gyrus (IFG). To further explore our results, we have looked at domain-general and language-specific response within the brain structures postulated to form a Bilingual Language Control (BLC) network. Within this network, we found a robust increase in response to L2 in the domain-general, but also in some language-specific voxels including in the left IFG. Our findings show that L2 production strongly engages domain-general mechanisms, but only affects language sensitive portions of the left IFG. These results put constraints on the current model of bilingual language control by precisely disentangling the domain-general and language-specific contributions to the difficulty in speech production in L2.
Collapse
Affiliation(s)
- Agata Wolna
- Institute of Psychology, Jagiellonian University, Ul. Ingardena 6, 30-060, Kraków, Poland.
| | - Jakub Szewczyk
- Institute of Psychology, Jagiellonian University, Ul. Ingardena 6, 30-060, Kraków, Poland
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Donders Institute for Brain Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Michele Diaz
- Social, Life, and Engineering Sciences Imaging Center, The Pennsylvania State University, Pennsylvania, USA
| | | | - Marcin Szwed
- Institute of Psychology, Jagiellonian University, Ul. Ingardena 6, 30-060, Kraków, Poland
| | - Zofia Wodniecka
- Institute of Psychology, Jagiellonian University, Ul. Ingardena 6, 30-060, Kraków, Poland.
| |
Collapse
|
10
|
Olson HA, Chen EM, Lydic KO, Saxe RR. Left-Hemisphere Cortical Language Regions Respond Equally to Observed Dialogue and Monologue. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2023; 4:575-610. [PMID: 38144236 PMCID: PMC10745132 DOI: 10.1162/nol_a_00123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 09/20/2023] [Indexed: 12/26/2023]
Abstract
Much of the language we encounter in our everyday lives comes in the form of conversation, yet the majority of research on the neural basis of language comprehension has used input from only one speaker at a time. Twenty adults were scanned while passively observing audiovisual conversations using functional magnetic resonance imaging. In a block-design task, participants watched 20 s videos of puppets speaking either to another puppet (the dialogue condition) or directly to the viewer (the monologue condition), while the audio was either comprehensible (played forward) or incomprehensible (played backward). Individually functionally localized left-hemisphere language regions responded more to comprehensible than incomprehensible speech but did not respond differently to dialogue than monologue. In a second task, participants watched videos (1-3 min each) of two puppets conversing with each other, in which one puppet was comprehensible while the other's speech was reversed. All participants saw the same visual input but were randomly assigned which character's speech was comprehensible. In left-hemisphere cortical language regions, the time course of activity was correlated only among participants who heard the same character speaking comprehensibly, despite identical visual input across all participants. For comparison, some individually localized theory of mind regions and right-hemisphere homologues of language regions responded more to dialogue than monologue in the first task, and in the second task, activity in some regions was correlated across all participants regardless of which character was speaking comprehensibly. Together, these results suggest that canonical left-hemisphere cortical language regions are not sensitive to differences between observed dialogue and monologue.
Collapse
|
11
|
Branzi FM, Lambon Ralph MA. Semantic-specific and domain-general mechanisms for integration and update of contextual information. Hum Brain Mapp 2023; 44:5547-5566. [PMID: 37787648 PMCID: PMC10619409 DOI: 10.1002/hbm.26454] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Revised: 07/26/2023] [Accepted: 08/02/2023] [Indexed: 10/04/2023] Open
Abstract
Recent research has highlighted the importance of domain-general processes and brain regions for language and semantic cognition. Yet, this has been mainly observed in executively demanding tasks, leaving open the question of the contribution of domain-general processes to natural language and semantic cognition. Using fMRI, we investigated whether neural processes reflecting context integration and context update-two key aspects of naturalistic language and semantic processing-are domain-specific versus domain-general. Thus, we compared neural responses during the integration of contextual information across semantic and non-semantic tasks. Whole-brain results revealed both shared (left posterior-dorsal inferior frontal gyrus, left posterior inferior temporal gyrus, and left dorsal angular gyrus/intraparietal sulcus) and distinct (left anterior-ventral inferior frontal gyrus, left anterior ventral angular gyrus, left posterior middle temporal gyrus for semantic control only) regions involved in context integration and update. Furthermore, data-driven functional connectivity analysis clustered domain-specific versus domain-general brain regions into distinct but interacting functional neural networks. These results provide a first characterisation of the neural processes required for context-dependent integration during language processing along the domain-specificity dimension, and at the same time, they bring new insights into the role of left posterior lateral temporal cortex and left angular gyrus for semantic cognition.
Collapse
Affiliation(s)
- Francesca M. Branzi
- Department of Psychological SciencesInstitute of Population Health, University of LiverpoolLiverpoolUK
- MRC Cognition & Brain Sciences UnitThe University of CambridgeCambridgeUK
| | | |
Collapse
|
12
|
Ryskin R, Nieuwland MS. Prediction during language comprehension: what is next? Trends Cogn Sci 2023; 27:1032-1052. [PMID: 37704456 DOI: 10.1016/j.tics.2023.08.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 08/03/2023] [Accepted: 08/04/2023] [Indexed: 09/15/2023]
Abstract
Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it. We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input. The nature of these predictions may vary between individuals owing to differences in language experience, among other factors. We then turn to unresolved questions which may guide the search for the underlying mechanisms. (i) Is prediction essential to language processing or an optional strategy? (ii) Are predictions generated from within the language system or by domain-general processes? (iii) What is the relationship between prediction and memory? (iv) Does prediction in comprehension require simulation via the production system? We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.
Collapse
Affiliation(s)
- Rachel Ryskin
- Department of Cognitive and Information Sciences, University of California Merced, 5200 Lake Road, Merced, CA 95343, USA.
| | - Mante S Nieuwland
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands; Donders Institute for Brain, Cognition, and Behaviour, Nijmegen, The Netherlands
| |
Collapse
|
13
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.16.537080. [PMID: 37090673 PMCID: PMC10120732 DOI: 10.1101/2023.04.16.537080] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/25/2023]
Abstract
Transformer models such as GPT generate human-like language and are highly predictive of human brain responses to language. Here, using fMRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of brain response associated with each sentence. Then, we use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also noninvasively control neural activity in higher-level cortical areas, like the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- MIT-IBM Watson AI Lab, Cambridge, MA 02142, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455 USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138 USA
| |
Collapse
|
14
|
Philips M, Schneck SM, Levy DF, Wilson SM. Modality-Specificity of the Neural Correlates of Linguistic and Non-Linguistic Demand. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2023; 4:516-535. [PMID: 37841966 PMCID: PMC10575553 DOI: 10.1162/nol_a_00114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Accepted: 06/28/2023] [Indexed: 10/17/2023]
Abstract
Imaging studies of language processing in clinical populations can be complicated to interpret for several reasons, one being the difficulty of matching the effortfulness of processing across individuals or tasks. To better understand how effortful linguistic processing is reflected in functional activity, we investigated the neural correlates of task difficulty in linguistic and non-linguistic contexts in the auditory modality and then compared our findings to a recent analogous experiment in the visual modality in a different cohort. Nineteen neurologically normal individuals were scanned with fMRI as they performed a linguistic task (semantic matching) and a non-linguistic task (melodic matching), each with two levels of difficulty. We found that left hemisphere frontal and temporal language regions, as well as the right inferior frontal gyrus, were modulated by linguistic demand and not by non-linguistic demand. This was broadly similar to what was previously observed in the visual modality. In contrast, the multiple demand (MD) network, a set of brain regions thought to support cognitive flexibility in many contexts, was modulated neither by linguistic demand nor by non-linguistic demand in the auditory modality. This finding was in striking contradistinction to what was previously observed in the visual modality, where the MD network was robustly modulated by both linguistic and non-linguistic demand. Our findings suggest that while the language network is modulated by linguistic demand irrespective of modality, modulation of the MD network by linguistic demand is not inherent to linguistic processing, but rather depends on specific task factors.
Collapse
Affiliation(s)
- Mackenzie Philips
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Sarah M. Schneck
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Deborah F. Levy
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Stephen M. Wilson
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
- School of Health and Rehabilitation Sciences, University of Queensland, Brisbane, Australia
| |
Collapse
|
15
|
Wang J, Wang X, Zou J, Duan J, Shen Z, Xu N, Chen Y, Zhang J, He H, Bi Y, Ding N. Neural substrate underlying the learning of a passage with unfamiliar vocabulary and syntax. Cereb Cortex 2023; 33:10036-10046. [PMID: 37491998 DOI: 10.1093/cercor/bhad263] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 06/26/2023] [Accepted: 06/27/2023] [Indexed: 07/27/2023] Open
Abstract
Speech comprehension is a complex process involving multiple stages, such as decoding of phonetic units, recognizing words, and understanding sentences and passages. In this study, we identify cortical networks beyond basic phonetic processing using a novel passage learning paradigm. Participants learn to comprehend a story composed of syllables of their native language, but containing unfamiliar vocabulary and syntax. Three learning methods are employed, each resulting in some degree of learning within a 12-min learning session. Functional magnetic resonance imaging results reveal that, when listening to the same story, the classic temporal-frontal language network is significantly enhanced by learning. Critically, activation of the left anterior and posterior temporal lobe correlates with the learning outcome that is assessed behaviorally through, e.g. word recognition and passage comprehension tests. This study demonstrates that a brief learning session is sufficient to induce neural plasticity in the left temporal lobe, which underlies the transformation from phonetic units to the units of meaning, such as words and sentences.
Collapse
Affiliation(s)
- Jing Wang
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Xiaosha Wang
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Jiajie Zou
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Jipeng Duan
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Zhuowen Shen
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Nannan Xu
- School of Linguistic Sciences and Arts, Jiangsu Normal University, Xuzhou 221009, China
| | - Yan Chen
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Jianfeng Zhang
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Hongjian He
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
| | - Yanchao Bi
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Nai Ding
- Key Laboratory for Biomedical Engineering of Ministry of Education, Center for Brain Imaging Science and Technology, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China
- MOE Frontier Science Center for Brain Science & Brain-machine Integration, Zhejiang University, Hangzhou 310027, China
| |
Collapse
|
16
|
Zhang Y, Rennig J, Magnotti JF, Beauchamp MS. Multivariate fMRI responses in superior temporal cortex predict visual contributions to, and individual differences in, the intelligibility of noisy speech. Neuroimage 2023; 278:120271. [PMID: 37442310 PMCID: PMC10460966 DOI: 10.1016/j.neuroimage.2023.120271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Revised: 06/20/2023] [Accepted: 07/06/2023] [Indexed: 07/15/2023] Open
Abstract
Humans have the unique ability to decode the rapid stream of language elements that constitute speech, even when it is contaminated by noise. Two reliable observations about noisy speech perception are that seeing the face of the talker improves intelligibility and the existence of individual differences in the ability to perceive noisy speech. We introduce a multivariate BOLD fMRI measure that explains both observations. In two independent fMRI studies, clear and noisy speech was presented in visual, auditory and audiovisual formats to thirty-seven participants who rated intelligibility. An event-related design was used to sort noisy speech trials by their intelligibility. Individual-differences multidimensional scaling was applied to fMRI response patterns in superior temporal cortex and the dissimilarity between responses to clear speech and noisy (but intelligible) speech was measured. Neural dissimilarity was less for audiovisual speech than auditory-only speech, corresponding to the greater intelligibility of noisy audiovisual speech. Dissimilarity was less in participants with better noisy speech perception, corresponding to individual differences. These relationships held for both single word and entire sentence stimuli, suggesting that they were driven by intelligibility rather than the specific stimuli tested. A neural measure of perceptual intelligibility may aid in the development of strategies for helping those with impaired speech perception.
Collapse
Affiliation(s)
- Yue Zhang
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States; Department of Neurosurgery, Baylor College of Medicine, Houston, TX, United States
| | - Johannes Rennig
- Division of Neuropsychology, Center of Neurology, Hertie-Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany
| | - John F Magnotti
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Michael S Beauchamp
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States.
| |
Collapse
|
17
|
Thye M, Hoffman P, Mirman D. The words that little by little revealed everything: Neural response to lexical-semantic content during narrative comprehension. Neuroimage 2023; 276:120204. [PMID: 37257674 DOI: 10.1016/j.neuroimage.2023.120204] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Revised: 04/19/2023] [Accepted: 05/27/2023] [Indexed: 06/02/2023] Open
Abstract
The ease with which narratives are understood belies the complexity of the information being conveyed and the cognitive processes that support comprehension. The meanings of the words must be rapidly accessed and integrated with the reader's mental representation of the overarching, unfolding scenario. A broad, bilateral brain network is engaged by this process, but it is not clear how words that vary on specific semantic dimensions, such as ambiguity, emotion, or socialness, engage the semantic, semantic control, or social cognition systems. In the present study, data from 48 participants who listened to The Little Prince audiobook during MRI scanning were selected from the Le Petit Prince dataset. The lexical and semantic content within the narrative was quantified from the transcript words with factor scores capturing Word Length, Semantic Flexibility, Emotional Strength, and Social Impact. These scores, along with word quantity variables, were used to investigate where these predictors co-vary with activation across the brain. In contrast to studies of isolated word processing, large networks were found to co-vary with the lexical and semantic content within the narrative. An increase in semantic content engaged the ventral portion of ventrolateral ATL, consistent with its role as a semantic hub. Decreased semantic content engaged temporal pole and inferior parietal lobule, which may reflect semantic integration. The semantic control network was engaged by words with low Semantic Flexibility, perhaps due to the demand required to process infrequent, less semantically diverse language. Activation in ATL co-varied with an increase in Social Impact, which is consistent with the claim that social knowledge is housed within the neural architecture of the semantic system. These results suggest that current models of language processing may present an impoverished estimate of the neural systems that coordinate to support narrative comprehension, and, by extension, real-world language processing.
Collapse
Affiliation(s)
- Melissa Thye
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom.
| | - Paul Hoffman
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom
| | - Daniel Mirman
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom
| |
Collapse
|
18
|
Zhang H, Diaz MT. Resting State Network Segregation Modulates Age-Related Differences in Language Production. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2023; 4:382-403. [PMID: 37546689 PMCID: PMC10403275 DOI: 10.1162/nol_a_00106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Accepted: 03/28/2023] [Indexed: 08/08/2023]
Abstract
Older adults typically exhibit decline in language production. However, how the brain supports or fails to support these processes is unclear. Moreover, there are competing hypotheses about the nature of age-related neural changes and whether age-related increases in neural activity reflect compensation or a decline in neural efficiency. In the current study, we investigated the neural bases of language production focusing on resting state functional connectivity. We hypothesized that language production performance, functional connectivity, and their relationship would differ as a function of age. Consistent with prior work, older age was associated with worse language production performance. Functional connectivity analyses showed that network segregation within the left hemisphere language network was maintained across adulthood. However, increased age was associated with lower whole brain network segregation. Moreover, network segregation was related to language production ability. In both network analyses, there were significant interactions with age-higher network segregation was associated with better language production abilities for younger and middle-aged adults, but not for older adults. Interestingly, there was a stronger relationship between language production and the whole brain network segregation than between production and the language network. These results highlight the utility of network segregation measures as an index of brain function, with higher network segregation associated with better language production ability. Moreover, these results are consistent with stability in the left hemisphere language network across adulthood and suggest that dedifferentiation among brain networks, outside of the language network, is a hallmark of aging and may contribute to age-related language production difficulties.
Collapse
Affiliation(s)
- Haoyun Zhang
- Centre for Cognitive and Brain Sciences, University of Macau, Macau SAR, China
- Department of Psychology, Pennsylvania State University, University Park, PA, USA
| | - Michele T. Diaz
- Department of Psychology, Pennsylvania State University, University Park, PA, USA
| |
Collapse
|
19
|
Chen X, Affourtit J, Ryskin R, Regev TI, Norman-Haignere S, Jouravlev O, Malik-Moraleda S, Kean H, Varley R, Fedorenko E. The human language system, including its inferior frontal component in "Broca's area," does not support music perception. Cereb Cortex 2023; 33:7904-7929. [PMID: 37005063 PMCID: PMC10505454 DOI: 10.1093/cercor/bhad087] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 01/02/2023] [Accepted: 01/03/2023] [Indexed: 04/04/2023] Open
Abstract
Language and music are two human-unique capacities whose relationship remains debated. Some have argued for overlap in processing mechanisms, especially for structure processing. Such claims often concern the inferior frontal component of the language system located within "Broca's area." However, others have failed to find overlap. Using a robust individual-subject fMRI approach, we examined the responses of language brain regions to music stimuli, and probed the musical abilities of individuals with severe aphasia. Across 4 experiments, we obtained a clear answer: music perception does not engage the language system, and judgments about music structure are possible even in the presence of severe damage to the language network. In particular, the language regions' responses to music are generally low, often below the fixation baseline, and never exceed responses elicited by nonmusic auditory conditions, like animal sounds. Furthermore, the language regions are not sensitive to music structure: they show low responses to both intact and structure-scrambled music, and to melodies with vs. without structural violations. Finally, in line with past patient investigations, individuals with aphasia, who cannot judge sentence grammaticality, perform well on melody well-formedness judgments. Thus, the mechanisms that process structure in language do not appear to process music, including music syntax.
Collapse
Affiliation(s)
- Xuanyi Chen
- Department of Cognitive Sciences, Rice University, TX 77005, United States
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Rachel Ryskin
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Cognitive & Information Sciences, University of California, Merced, Merced, CA 95343, United States
| | - Tamar I Regev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Samuel Norman-Haignere
- Department of Biostatistics & Computational Biology, University of Rochester Medical Center, Rochester, NY, United States
- Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, United States
- Department of Biomedical Engineering, University of Rochester, Rochester, NY, United States
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, United States
| | - Olessia Jouravlev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| | - Hope Kean
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Rosemary Varley
- Psychology & Language Sciences, UCL, London, WCN1 1PF, United Kingdom
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| |
Collapse
|
20
|
Hauptman M, Blank I, Fedorenko E. Non-literal language processing is jointly supported by the language and theory of mind networks: Evidence from a novel meta-analytic fMRI approach. Cortex 2023; 162:96-114. [PMID: 37023480 PMCID: PMC10210011 DOI: 10.1016/j.cortex.2023.01.013] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Revised: 11/08/2022] [Accepted: 01/11/2023] [Indexed: 03/12/2023]
Abstract
Going beyond the literal meaning of language is key to communicative success. However, the mechanisms that support non-literal inferences remain debated. Using a novel meta-analytic approach, we evaluate the contribution of linguistic, social-cognitive, and executive mechanisms to non-literal interpretation. We identified 74 fMRI experiments (n = 1,430 participants) from 2001 to 2021 that contrasted non-literal language comprehension with a literal control condition, spanning ten phenomena (e.g., metaphor, irony, indirect speech). Applying the activation likelihood estimation approach to the 825 activation peaks yielded six left-lateralized clusters. We then evaluated the locations of both the individual-study peaks and the clusters against probabilistic functional atlases (cf. anatomical locations, as is typically done) for three candidate brain networks-the language-selective network (Fedorenko, Behr, & Kanwisher, 2011), which supports language processing, the Theory of Mind (ToM) network (Saxe & Kanwisher, 2003), which supports social inferences, and the domain-general Multiple-Demand (MD) network (Duncan, 2010), which supports executive control. These atlases were created by overlaying individual activation maps of participants who performed robust and extensively validated 'localizer' tasks that selectively target each network in question (n = 806 for language; n = 198 for ToM; n = 691 for MD). We found that both the individual-study peaks and the ALE clusters fell primarily within the language network and the ToM network. These results suggest that non-literal processing is supported by both i) mechanisms that process literal linguistic meaning, and ii) mechanisms that support general social inference. They thus undermine a strong divide between literal and non-literal aspects of language and challenge the claim that non-literal processing requires additional executive resources.
Collapse
Affiliation(s)
- Miriam Hauptman
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Department of Psychological & Brain Sciences, Johns Hopkins University, Baltimore, MD 21218, USA.
| | - Idan Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Department of Psychology, UCLA, Los Angeles, CA 90095, USA; Department of Linguistics, UCLA, Los Angeles, CA 90095, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Program in Speech and Hearing in Bioscience and Technology, Harvard University, Boston, MA 02114, USA.
| |
Collapse
|
21
|
Yang X, Lin N, Wang L. Situation updating during discourse comprehension recruits right posterior portion of the multiple-demand network. Hum Brain Mapp 2023; 44:2129-2141. [PMID: 36602295 PMCID: PMC10028651 DOI: 10.1002/hbm.26198] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Revised: 12/01/2022] [Accepted: 12/17/2022] [Indexed: 01/06/2023] Open
Abstract
Discourse comprehension involves the construction of a mental representation of the situation model as well as a continuous update of this representation. This mental update is cognitively demanding and likely engages the multiple-demand network. However, there is little evidence for the involvement of the multiple-demand network during situation updating. In this study, we used fMRI to test whether situation updating based on the change of spatial location activated the multiple-demand network. In a discourse comprehension task, readers read two-sentence discourses in which the second sentence either continues or introduces a shift of the spatial location information presented in the first sentence. Compared to situation continuation, situation updating reliably activated the right superior parietal lobule. This area is a part of the multiple-demand network as defined by a digit N-back localizer task and locates within the dorsal attention network as defined in the previous study by Yeo et al. in 2011. Our results provide evidence for the reliable involvement of a specific area of the multiple-demand network in situation updating during high-level discourse processing.
Collapse
Affiliation(s)
- XiaoHong Yang
- Department of Psychology, Renmin University of China, Beijing, China
| | - Nan Lin
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Lin Wang
- Department of Psychology, Tufts University, Medford, Massachusetts, USA
- Department of Psychiatry and the Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Charlestown, Massachusetts, USA
| |
Collapse
|
22
|
Meyer AS. Timing in Conversation. J Cogn 2023; 6:20. [PMID: 37033404 PMCID: PMC10077995 DOI: 10.5334/joc.268] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2022] [Accepted: 03/17/2023] [Indexed: 04/08/2023] Open
Abstract
Turn-taking in everyday conversation is fast, with median latencies in corpora of conversational speech often reported to be under 300 ms. This seems like magic, given that experimental research on speech planning has shown that speakers need much more time to plan and produce even the shortest of utterances. This paper reviews how language scientists have combined linguistic analyses of conversations and experimental work to understand the skill of swift turn-taking and proposes a tentative solution to the riddle of fast turn-taking.
Collapse
Affiliation(s)
- Antje S. Meyer
- Radboud University Nijmegen and Max Planck Institute for Psycholinguistics and Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
23
|
MacGregor LJ, Gilbert RA, Balewski Z, Mitchell DJ, Erzinçlioğlu SW, Rodd JM, Duncan J, Fedorenko E, Davis MH. Causal Contributions of the Domain-General (Multiple Demand) and the Language-Selective Brain Networks to Perceptual and Semantic Challenges in Speech Comprehension. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:665-698. [PMID: 36742011 PMCID: PMC9893226 DOI: 10.1162/nol_a_00081] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Accepted: 09/07/2022] [Indexed: 06/18/2023]
Abstract
Listening to spoken language engages domain-general multiple demand (MD; frontoparietal) regions of the human brain, in addition to domain-selective (frontotemporal) language regions, particularly when comprehension is challenging. However, there is limited evidence that the MD network makes a functional contribution to core aspects of understanding language. In a behavioural study of volunteers (n = 19) with chronic brain lesions, but without aphasia, we assessed the causal role of these networks in perceiving, comprehending, and adapting to spoken sentences made more challenging by acoustic-degradation or lexico-semantic ambiguity. We measured perception of and adaptation to acoustically degraded (noise-vocoded) sentences with a word report task before and after training. Participants with greater damage to MD but not language regions required more vocoder channels to achieve 50% word report, indicating impaired perception. Perception improved following training, reflecting adaptation to acoustic degradation, but adaptation was unrelated to lesion location or extent. Comprehension of spoken sentences with semantically ambiguous words was measured with a sentence coherence judgement task. Accuracy was high and unaffected by lesion location or extent. Adaptation to semantic ambiguity was measured in a subsequent word association task, which showed that availability of lower-frequency meanings of ambiguous words increased following their comprehension (word-meaning priming). Word-meaning priming was reduced for participants with greater damage to language but not MD regions. Language and MD networks make dissociable contributions to challenging speech comprehension: Using recent experience to update word meaning preferences depends on language-selective regions, whereas the domain-general MD network plays a causal role in reporting words from degraded speech.
Collapse
Affiliation(s)
- Lucy J. MacGregor
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| | - Rebecca A. Gilbert
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| | - Zuzanna Balewski
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA
| | - Daniel J. Mitchell
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| | | | - Jennifer M. Rodd
- Psychology and Language Sciences, University College London, London, UK
| | - John Duncan
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA
| | - Matthew H. Davis
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| |
Collapse
|
24
|
Explaining neural activity in human listeners with deep learning via natural language processing of narrative text. Sci Rep 2022; 12:17838. [PMID: 36284195 PMCID: PMC9596412 DOI: 10.1038/s41598-022-21782-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2022] [Accepted: 10/04/2022] [Indexed: 01/20/2023] Open
Abstract
Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.
Collapse
|
25
|
Chiou R, Jefferies E, Duncan J, Humphreys GF, Lambon Ralph MA. A middle ground where executive control meets semantics: the neural substrates of semantic control are topographically sandwiched between the multiple-demand and default-mode systems. Cereb Cortex 2022; 33:4512-4526. [PMID: 36130101 PMCID: PMC10110444 DOI: 10.1093/cercor/bhac358] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2021] [Revised: 08/15/2022] [Accepted: 08/16/2022] [Indexed: 11/12/2022] Open
Abstract
Semantic control is the capability to operate on meaningful representations, selectively focusing on certain aspects of meaning while purposefully ignoring other aspects based on one's behavioral aim. This ability is especially vital for comprehending figurative/ambiguous language. It remains unclear why and how regions involved in semantic control seem reliably juxtaposed alongside other functionally specialized regions in the association cortex, prompting speculation about the relationship between topography and function. We investigated this issue by characterizing how semantic control regions topographically relate to the default-mode network (associated with memory and abstract cognition) and multiple-demand network (associated with executive control). Topographically, we established that semantic control areas were sandwiched by the default-mode and multi-demand networks, forming an orderly arrangement observed both at the individual and group level. Functionally, semantic control regions exhibited "hybrid" responses, fusing generic preferences for cognitively demanding operation (multiple-demand) and for meaningful representations (default-mode) into a domain-specific preference for difficult operation on meaningful representations. When projected onto the principal gradient of human connectome, the neural activity of semantic control showed a robustly dissociable trajectory from visuospatial control, implying different roles in the functional transition from sensation to cognition. We discuss why the hybrid functional profile of semantic control regions might result from their intermediate topographical positions on the cortex.
Collapse
Affiliation(s)
- Rocco Chiou
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF, UK.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, UK
| | | | - John Duncan
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF, UK.,Department of Experimental Psychology, University of Oxford, OX2 6GG, UK
| | - Gina F Humphreys
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF, UK
| | | |
Collapse
|
26
|
Lipkin B, Tuckute G, Affourtit J, Small H, Mineroff Z, Kean H, Jouravlev O, Rakocevic L, Pritchett B, Siegelman M, Hoeflin C, Pongos A, Blank IA, Struhl MK, Ivanova A, Shannon S, Sathe A, Hoffmann M, Nieto-Castañón A, Fedorenko E. Probabilistic atlas for the language network based on precision fMRI data from >800 individuals. Sci Data 2022; 9:529. [PMID: 36038572 PMCID: PMC9424256 DOI: 10.1038/s41597-022-01645-3] [Citation(s) in RCA: 45] [Impact Index Per Article: 22.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2022] [Accepted: 08/09/2022] [Indexed: 11/13/2022] Open
Abstract
Two analytic traditions characterize fMRI language research. One relies on averaging activations across individuals. This approach has limitations: because of inter-individual variability in the locations of language areas, any given voxel/vertex in a common brain space is part of the language network in some individuals but in others, may belong to a distinct network. An alternative approach relies on identifying language areas in each individual using a functional 'localizer'. Because of its greater sensitivity, functional resolution, and interpretability, functional localization is gaining popularity, but it is not always feasible, and cannot be applied retroactively to past studies. To bridge these disjoint approaches, we created a probabilistic functional atlas using fMRI data for an extensively validated language localizer in 806 individuals. This atlas enables estimating the probability that any given location in a common space belongs to the language network, and thus can help interpret group-level activation peaks and lesion locations, or select voxels/electrodes for analysis. More meaningful comparisons of findings across studies should increase robustness and replicability in language research.
Collapse
Affiliation(s)
- Benjamin Lipkin
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Hannah Small
- Department of Cognitive Science, Johns Hopkins University, Baltimore, MD, USA
| | - Zachary Mineroff
- Human-computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Hope Kean
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Olessia Jouravlev
- Department of Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Lara Rakocevic
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Brianna Pritchett
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | | | - Caitlyn Hoeflin
- Harris School of Public Policy, University of Chicago, Chicago, IL, USA
| | - Alvincé Pongos
- Department of Bioengineering, University of California, Berkeley, CA, USA
| | - Idan A Blank
- Department of Psychology, University of California, Los Angeles, CA, USA
| | - Melissa Kline Struhl
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Anna Ivanova
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Steven Shannon
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Malte Hoffmann
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Cambridge, MA, USA
| | - Alfonso Nieto-Castañón
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Speech, Language, and Hearing Sciences, Boston University, Boston, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Department of Speech, Hearing, Bioscience, and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
27
|
Wu W, Morales M, Patel T, Pickering MJ, Hoffman P. Modulation of brain activity by psycholinguistic information during naturalistic speech comprehension and production. Cortex 2022; 155:287-306. [DOI: 10.1016/j.cortex.2022.08.002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Revised: 06/23/2022] [Accepted: 08/01/2022] [Indexed: 11/25/2022]
|
28
|
Russo AG, De Martino M, Elia A, Di Salle F, Esposito F. Negative correlation between word-level surprisal and intersubject neural synchronization during narrative listening. Cortex 2022; 155:132-149. [DOI: 10.1016/j.cortex.2022.07.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2021] [Revised: 02/10/2022] [Accepted: 07/06/2022] [Indexed: 11/30/2022]
|
29
|
Meier EL, Kelly CR, Hillis AE. Dissociable language and executive control deficits and recovery in post-stroke aphasia: An exploratory observational and case series study. Neuropsychologia 2022; 172:108270. [PMID: 35597266 PMCID: PMC9728463 DOI: 10.1016/j.neuropsychologia.2022.108270] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2021] [Revised: 04/30/2022] [Accepted: 05/13/2022] [Indexed: 01/04/2023]
Abstract
A growing body of evidence indicates many, but not all, individuals with post-stroke aphasia experience executive dysfunction. Relationships between language and executive function skills are often reported in the literature, but the degree of interdependence between these abilities remains largely unanswered. Therefore, in this study, we investigated the extent to which language and executive control deficits dissociated in 1) acute stroke and 2) longitudinal aphasia recovery. Twenty-three individuals admitted to Johns Hopkins Hospital with a new left hemisphere stroke completed the Western Aphasia Battery-Revised (WAB-R), several additional language measures (of naming, semantics, spontaneous speech, and oral reading), and three non-linguistic cognitive tasks from the NIH Toolbox (i.e., Pattern Comparison Processing Speed Test, Flanker Inhibitory Control and Attention Test, and Dimensional Change Card Sort Test). Two participants with aphasia (PWA) with temporoparietal lesions, one of whom (PWA1) had greater temporal but less frontal and superior parietal damage than the other (PWA2), also completed testing at subacute (three months post-onset) and early chronic (six months post-onset) time points. In aim 1, principal component analysis on the acute test data (excluding the WAB-R) revealed language and non-linguistic executive control tasks largely loaded onto separate components. Both components were significant predictors of acute aphasia severity per the WAB-R Aphasia Quotient (AQ). Crucially, executive dysfunction explained an additional 17% of the variance in AQ beyond the explanatory power of language impairments alone. In aim 2, both case patients exhibited language and executive control deficits at the acute post-stroke stage. A dissociation was observed in longitudinal recovery of these patients. By the early chronic time point, PWA1 exhibited improved (but persistent) deficits in several language domains and recovered executive control. In contrast, PWA2 demonstrated mostly recovered language but persistent executive dysfunction. Greater damage to language and attention networks in these respective patients may explain the observed behavioral patterns. These results demonstrate that language and executive control can dissociate (at least to a degree), but both contribute to early post-stroke presentation of aphasia and likely influence longitudinal aphasia recovery.
Collapse
Affiliation(s)
| | | | - Argye E Hillis
- Department of Neurology, USA; Physical Medicine and Rehabilitation, USA; Cognitive Science, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
30
|
Paunov AM, Blank IA, Jouravlev O, Mineroff Z, Gallée J, Fedorenko E. Differential Tracking of Linguistic vs. Mental State Content in Naturalistic Stimuli by Language and Theory of Mind (ToM) Brain Networks. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:413-440. [PMID: 37216061 PMCID: PMC10158571 DOI: 10.1162/nol_a_00071] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Accepted: 04/11/2022] [Indexed: 05/24/2023]
Abstract
Language and social cognition, especially the ability to reason about mental states, known as theory of mind (ToM), are deeply related in development and everyday use. However, whether these cognitive faculties rely on distinct, overlapping, or the same mechanisms remains debated. Some evidence suggests that, by adulthood, language and ToM draw on largely distinct-though plausibly interacting-cortical networks. However, the broad topography of these networks is similar, and some have emphasized the importance of social content / communicative intent in the linguistic signal for eliciting responses in the language areas. Here, we combine the power of individual-subject functional localization with the naturalistic-cognition inter-subject correlation approach to illuminate the language-ToM relationship. Using functional magnetic resonance imaging (fMRI), we recorded neural activity as participants (n = 43) listened to stories and dialogues with mental state content (+linguistic, +ToM), viewed silent animations and live action films with mental state content but no language (-linguistic, +ToM), or listened to an expository text (+linguistic, -ToM). The ToM network robustly tracked stimuli rich in mental state information regardless of whether mental states were conveyed linguistically or non-linguistically, while tracking a +linguistic / -ToM stimulus only weakly. In contrast, the language network tracked linguistic stimuli more strongly than (a) non-linguistic stimuli, and than (b) the ToM network, and showed reliable tracking even for the linguistic condition devoid of mental state content. These findings suggest that in spite of their indisputably close links, language and ToM dissociate robustly in their neural substrates-and thus plausibly cognitive mechanisms-including during the processing of rich naturalistic materials.
Collapse
Affiliation(s)
- Alexander M. Paunov
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, Université Paris-Saclay, NeuroSpin Center, 91191Gif/Yvette, France
| | - Idan A. Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Department of Psychology, UCLA, Los Angeles, CA, USA
| | - Olessia Jouravlev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Institute for Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Eberly Center for Teaching Excellence & Educational Innovation, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Jeanne Gallée
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| |
Collapse
|
31
|
Williams KA, Numssen O, Hartwigsen G. Task-specific network interactions across key cognitive domains. Cereb Cortex 2022; 32:5050-5071. [PMID: 35158372 PMCID: PMC9667178 DOI: 10.1093/cercor/bhab531] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 12/21/2021] [Accepted: 12/22/2021] [Indexed: 12/27/2022] Open
Abstract
Human cognition is organized in distributed networks in the brain. Although distinct specialized networks have been identified for different cognitive functions, previous work also emphasizes the overlap of key cognitive domains in higher level association areas. The majority of previous studies focused on network overlap and dissociation during resting states whereas task-related network interactions across cognitive domains remain largely unexplored. A better understanding of network overlap and dissociation during different cognitive tasks may elucidate flexible (re-)distribution of resources during human cognition. The present study addresses this issue by providing a broad characterization of large-scale network dynamics in three key cognitive domains. Combining prototypical tasks of the larger domains of attention, language, and social cognition with whole-brain multivariate activity and connectivity approaches, we provide a spatiotemporal characterization of multiple large-scale, overlapping networks that differentially interact across cognitive domains. We show that network activity and interactions increase with increased cognitive complexity across domains. Interaction patterns reveal a common core structure across domains as well as dissociable domain-specific network activity. The observed patterns of activation and deactivation of overlapping and strongly coupled networks provide insight beyond region-specific activity within a particular cognitive domain toward a network perspective approach across diverse key cognitive functions.
Collapse
Affiliation(s)
- Kathleen A Williams
- Address correspondence to Kathleen A. Williams, Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, 04103 Leipzig, Germany.
| | - Ole Numssen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, 04103 Leipzig, Germany
| | - Gesa Hartwigsen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, 04103 Leipzig, Germany
| |
Collapse
|
32
|
Fedorenko E, Shain C. Similarity of computations across domains does not imply shared implementation: The case of language comprehension. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 2021; 30:526-534. [PMID: 35295820 PMCID: PMC8923525 DOI: 10.1177/09637214211046955] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2023]
Abstract
Understanding language requires applying cognitive operations (e.g., memory retrieval, prediction, structure building)-relevant across many cognitive domains-to specialized knowledge structures (a particular language's phonology, lexicon, and syntax). Are these computations carried out by domain-general circuits or by circuits that store domain-specific representations? Recent work has characterized the roles in language comprehension of the language-selective network and the multiple demand (MD) network, which has been implicated in executive functions and linked to fluid intelligence, making it a prime candidate for implementing computations that support information processing across domains. The language network responds robustly to diverse aspects of comprehension, but the MD network shows no sensitivity to linguistic variables. We therefore argue that the MD network does not play a core role in language comprehension, and that past claims to the contrary are likely due to methodological artifacts. Although future studies may discover some aspects of language that require the MD network, evidence to date suggests that those will not be related to core linguistic processes like lexical access or composition. The finding that the circuits that store linguistic knowledge carry out computations on those representations aligns with general arguments against the separation between memory and computation in the mind and brain.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Brain & Cognitive Sciences Department and McGovern Institute for Brain Research, MIT
| | - Cory Shain
- Brain & Cognitive Sciences Department and McGovern Institute for Brain Research, MIT
| |
Collapse
|
33
|
Schrimpf M, Blank IA, Tuckute G, Kauf C, Hosseini EA, Kanwisher N, Tenenbaum JB, Fedorenko E. The neural architecture of language: Integrative modeling converges on predictive processing. Proc Natl Acad Sci U S A 2021; 118:e2105646118. [PMID: 34737231 PMCID: PMC8694052 DOI: 10.1073/pnas.2105646118] [Citation(s) in RCA: 125] [Impact Index Per Article: 41.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/03/2021] [Indexed: 01/30/2023] Open
Abstract
The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species' signature cognitive skill. We find that the most powerful "transformer" models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models' neural fits ("brain score") and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.
Collapse
Affiliation(s)
- Martin Schrimpf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Idan Asher Blank
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- Department of Psychology, University of California, Los Angeles, CA 90095
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Carina Kauf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Eghbal A Hosseini
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Joshua B Tenenbaum
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| |
Collapse
|
34
|
Cotosck KR, Meltzer JA, Nucci MP, Lukasova K, Mansur LL, Amaro E. Engagement of Language and Domain General Networks during Word Monitoring in a Native and Unknown Language. Brain Sci 2021; 11:brainsci11081063. [PMID: 34439682 PMCID: PMC8393423 DOI: 10.3390/brainsci11081063] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Revised: 08/06/2021] [Accepted: 08/09/2021] [Indexed: 12/27/2022] Open
Abstract
Functional neuroimaging studies have highlighted the roles of three networks in processing language, all of which are typically left-lateralized: a ventral stream involved in semantics, a dorsal stream involved in phonology and speech production, and a more dorsal "multiple demand" network involved in many effortful tasks. As lateralization in all networks may be affected by life factors such as age, literacy, education, and brain pathology, we sought to develop a task paradigm with which to investigate the engagement of these networks, including manipulations to selectively emphasize semantic and phonological processing within a single task performable by almost anyone regardless of literacy status. In young healthy participants, we administered an auditory word monitoring task, in which participants had to note the occurrence of a target word within a continuous story presented in either their native language, Portuguese, or the unknown language, Japanese. Native language task performance activated ventral stream language networks, left lateralized but bilateral in the anterior temporal lobe. Unfamiliar language performance, being more difficult, activated left hemisphere dorsal stream structures and the multiple demand network bilaterally, but predominantly in the right hemisphere. These findings suggest that increased demands on phonological processing to accomplish word monitoring in the absence of semantic support may result in the bilateral recruitment of networks involved in speech perception under more challenging conditions.
Collapse
Affiliation(s)
- Kelly R. Cotosck
- FNI–Functional Neuroimaging, LIM 4–Laboratório de Investigação Médica 44 (Laboratory of Medical Investigation 44), Department of Radiology, Universidade de São Paulo, São Paulo 05403-000, Brazil; (M.P.N.); (K.L.); (E.A.J.)
- Correspondence: or ; Tel.: +55-11-95131-2225
| | | | - Mariana P. Nucci
- FNI–Functional Neuroimaging, LIM 4–Laboratório de Investigação Médica 44 (Laboratory of Medical Investigation 44), Department of Radiology, Universidade de São Paulo, São Paulo 05403-000, Brazil; (M.P.N.); (K.L.); (E.A.J.)
| | - Katerina Lukasova
- FNI–Functional Neuroimaging, LIM 4–Laboratório de Investigação Médica 44 (Laboratory of Medical Investigation 44), Department of Radiology, Universidade de São Paulo, São Paulo 05403-000, Brazil; (M.P.N.); (K.L.); (E.A.J.)
- Center of Mathematics, Computing and Cognition, Universidade Federal do ABC, Santo André 09210-580, Brazil
| | | | - Edson Amaro
- FNI–Functional Neuroimaging, LIM 4–Laboratório de Investigação Médica 44 (Laboratory of Medical Investigation 44), Department of Radiology, Universidade de São Paulo, São Paulo 05403-000, Brazil; (M.P.N.); (K.L.); (E.A.J.)
| |
Collapse
|
35
|
Continuous-time deconvolutional regression for psycholinguistic modeling. Cognition 2021; 215:104735. [PMID: 34303182 DOI: 10.1016/j.cognition.2021.104735] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2019] [Revised: 04/01/2021] [Accepted: 04/11/2021] [Indexed: 12/28/2022]
Abstract
The influence of stimuli in psycholinguistic experiments diffuses across time because the human response to language is not instantaneous. The linear models typically used to analyze psycholinguistic data are unable to account for this phenomenon due to strong temporal independence assumptions, while existing deconvolutional methods for estimating diffuse temporal structure model time discretely and therefore cannot be directly applied to natural language stimuli where events (words) have variable duration. In light of evidence that continuous-time deconvolutional regression (CDR) can address these issues (Shain & Schuler, 2018), this article motivates the use of CDR for many experimental settings, exposits some of its mathematical properties, and empirically evaluates the influence of various experimental confounds (noise, multicollinearity, and impulse response misspecification), hyperparameter settings, and response types (behavioral and fMRI). Results show that CDR (1) yields highly consistent estimates across a variety of hyperparameter configurations, (2) faithfully recovers the data-generating model on synthetic data, even under adverse training conditions, and (3) outperforms widely-used statistical approaches when applied to naturalistic reading and fMRI data. In addition, procedures for testing scientific hypotheses using CDR are defined and demonstrated, and empirically-motivated best-practices for CDR modeling are proposed. Results support the use of CDR for analyzing psycholinguistic time series, especially in a naturalistic experimental paradigm.
Collapse
|