1
|
Regev TI, Casto C, Hosseini EA, Adamek M, Ritaccio AL, Willie JT, Brunner P, Fedorenko E. Neural populations in the language network differ in the size of their temporal receptive windows. Nat Hum Behav 2024:10.1038/s41562-024-01944-2. [PMID: 39187713 DOI: 10.1038/s41562-024-01944-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Accepted: 07/03/2024] [Indexed: 08/28/2024]
Abstract
Despite long knowing what brain areas support language comprehension, our knowledge of the neural computations that these frontal and temporal regions implement remains limited. One important unresolved question concerns functional differences among the neural populations that comprise the language network. Here we leveraged the high spatiotemporal resolution of human intracranial recordings (n = 22) to examine responses to sentences and linguistically degraded conditions. We discovered three response profiles that differ in their temporal dynamics. These profiles appear to reflect different temporal receptive windows, with average windows of about 1, 4 and 6 words, respectively. Neural populations exhibiting these profiles are interleaved across the language network, which suggests that all language regions have direct access to distinct, multiscale representations of linguistic input-a property that may be critical for the efficiency and robustness of language processing.
Collapse
Affiliation(s)
- Tamar I Regev
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Colton Casto
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology (SHBT), Harvard University, Boston, MA, USA.
- Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Allston, MA, USA.
| | - Eghbal A Hosseini
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Markus Adamek
- National Center for Adaptive Neurotechnologies, Albany, NY, USA
- Department of Neurosurgery, Washington University School of Medicine, St Louis, MO, USA
| | | | - Jon T Willie
- National Center for Adaptive Neurotechnologies, Albany, NY, USA
- Department of Neurosurgery, Washington University School of Medicine, St Louis, MO, USA
| | - Peter Brunner
- National Center for Adaptive Neurotechnologies, Albany, NY, USA
- Department of Neurosurgery, Washington University School of Medicine, St Louis, MO, USA
- Department of Neurology, Albany Medical College, Albany, NY, USA
| | - Evelina Fedorenko
- Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology (SHBT), Harvard University, Boston, MA, USA.
| |
Collapse
|
2
|
Tuckute G, Kanwisher N, Fedorenko E. Language in Brains, Minds, and Machines. Annu Rev Neurosci 2024; 47:277-301. [PMID: 38669478 DOI: 10.1146/annurev-neuro-120623-101142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/28/2024]
Abstract
It has long been argued that only humans could produce and understand language. But now, for the first time, artificial language models (LMs) achieve this feat. Here we survey the new purchase LMs are providing on the question of how language is implemented in the brain. We discuss why, a priori, LMs might be expected to share similarities with the human language system. We then summarize evidence that LMs represent linguistic information similarly enough to humans to enable relatively accurate brain encoding and decoding during language processing. Finally, we examine which LM properties-their architecture, task performance, or training-are critical for capturing human neural responses to language and review studies using LMs as in silico model organisms for testing hypotheses about language. These ongoing investigations bring us closer to understanding the representations and processes that underlie our ability to comprehend sentences and express thoughts in language.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| |
Collapse
|
3
|
Shain C, Kean H, Casto C, Lipkin B, Affourtit J, Siegelman M, Mollica F, Fedorenko E. Distributed Sensitivity to Syntax and Semantics throughout the Language Network. J Cogn Neurosci 2024; 36:1427-1471. [PMID: 38683732 DOI: 10.1162/jocn_a_02164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/02/2024]
Abstract
Human language is expressive because it is compositional: The meaning of a sentence (semantics) can be inferred from its structure (syntax). It is commonly believed that language syntax and semantics are processed by distinct brain regions. Here, we revisit this claim using precision fMRI methods to capture separation or overlap of function in the brains of individual participants. Contrary to prior claims, we find distributed sensitivity to both syntax and semantics throughout a broad frontotemporal brain network. Our results join a growing body of evidence for an integrated network for language in the human brain within which internal specialization is primarily a matter of degree rather than kind, in contrast with influential proposals that advocate distinct specialization of different brain areas for different types of linguistic functions.
Collapse
Affiliation(s)
| | - Hope Kean
- Massachusetts Institute of Technology
| | | | | | | | | | | | | |
Collapse
|
4
|
Kauf C, Tuckute G, Levy R, Andreas J, Fedorenko E. Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2024; 5:7-42. [PMID: 38645614 PMCID: PMC11025651 DOI: 10.1162/nol_a_00116] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Accepted: 07/11/2023] [Indexed: 04/23/2024]
Abstract
Representations from artificial neural network (ANN) language models have been shown to predict human brain activity in the language network. To understand what aspects of linguistic stimuli contribute to ANN-to-brain similarity, we used an fMRI data set of responses to n = 627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated the stimuli for which ANN representations were extracted. In particular, we (i) perturbed sentences' word order, (ii) removed different subsets of words, or (iii) replaced sentences with other sentences of varying semantic similarity. We found that the lexical-semantic content of the sentence (largely carried by content words) rather than the sentence's syntactic form (conveyed via word order or function words) is primarily responsible for the ANN-to-brain similarity. In follow-up analyses, we found that perturbation manipulations that adversely affect brain predictivity also lead to more divergent representations in the ANN's embedding space and decrease the ANN's ability to predict upcoming tokens in those stimuli. Further, results are robust as to whether the mapping model is trained on intact or perturbed stimuli and whether the ANN sentence representations are conditioned on the same linguistic context that humans saw. The critical result-that lexical-semantic content is the main contributor to the similarity between ANN representations and neural ones-aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.
Collapse
Affiliation(s)
- Carina Kauf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Roger Levy
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Jacob Andreas
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA, USA
| |
Collapse
|
5
|
Schneider JM, Scott TL, Legault J, Qi Z. Limited but specific engagement of the mature language network during linguistic statistical learning. Cereb Cortex 2024; 34:bhae123. [PMID: 38566510 PMCID: PMC10987970 DOI: 10.1093/cercor/bhae123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Revised: 03/04/2024] [Accepted: 03/05/2024] [Indexed: 04/04/2024] Open
Abstract
Statistical learning (SL) is the ability to detect and learn regularities from input and is foundational to language acquisition. Despite the dominant role of SL as a theoretical construct for language development, there is a lack of direct evidence supporting the shared neural substrates underlying language processing and SL. It is also not clear whether the similarities, if any, are related to linguistic processing, or statistical regularities in general. The current study tests whether the brain regions involved in natural language processing are similarly recruited during auditory, linguistic SL. Twenty-two adults performed an auditory linguistic SL task, an auditory nonlinguistic SL task, and a passive story listening task as their neural activation was monitored. Within the language network, the left posterior temporal gyrus showed sensitivity to embedded speech regularities during auditory, linguistic SL, but not auditory, nonlinguistic SL. Using a multivoxel pattern similarity analysis, we uncovered similarities between the neural representation of auditory, linguistic SL, and language processing within the left posterior temporal gyrus. No other brain regions showed similarities between linguistic SL and language comprehension, suggesting that a shared neurocomputational process for auditory SL and natural language processing within the left posterior temporal gyrus is specific to linguistic stimuli.
Collapse
Affiliation(s)
- Julie M Schneider
- Department of Communication Sciences and Disorders, Louisiana State University, 77 Hatcher Hall, Field House Dr., Baton Rouge, LA 70803, United States
- Department of Linguistics & Cognitive Science, University of Delaware, 125 E Main St, Newark, DE 19716, United States
| | - Terri L Scott
- School of Medicine, University of California San Francisco, 533 Parnassus Ave, San Francisco, CA 94143, United States
| | - Jennifer Legault
- Department of Psychology, Elizabethtown College, One Alpha Dr, Elizabethtown, PA 17022, United States
| | - Zhenghan Qi
- Department of Linguistics & Cognitive Science, University of Delaware, 125 E Main St, Newark, DE 19716, United States
- Bouvé College of Health Sciences, Northeastern University, 360 Huntington Ave, Boston, MA 02115, United States
| |
Collapse
|
6
|
Regev TI, Kim HS, Chen X, Affourtit J, Schipper AE, Bergen L, Mahowald K, Fedorenko E. High-level language brain regions process sublexical regularities. Cereb Cortex 2024; 34:bhae077. [PMID: 38494886 DOI: 10.1093/cercor/bhae077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2023] [Revised: 02/05/2024] [Accepted: 02/07/2024] [Indexed: 03/19/2024] Open
Abstract
A network of left frontal and temporal brain regions supports language processing. This "core" language network stores our knowledge of words and constructions as well as constraints on how those combine to form sentences. However, our linguistic knowledge additionally includes information about phonemes and how they combine to form phonemic clusters, syllables, and words. Are phoneme combinatorics also represented in these language regions? Across five functional magnetic resonance imaging experiments, we investigated the sensitivity of high-level language processing brain regions to sublexical linguistic regularities by examining responses to diverse nonwords-sequences of phonemes that do not constitute real words (e.g. punes, silory, flope). We establish robust responses in the language network to visually (experiment 1a, n = 605) and auditorily (experiments 1b, n = 12, and 1c, n = 13) presented nonwords. In experiment 2 (n = 16), we find stronger responses to nonwords that are more well-formed, i.e. obey the phoneme-combinatorial constraints of English. Finally, in experiment 3 (n = 14), we provide suggestive evidence that the responses in experiments 1 and 2 are not due to the activation of real words that share some phonology with the nonwords. The results suggest that sublexical regularities are stored and processed within the same fronto-temporal network that supports lexical and syntactic processes.
Collapse
Affiliation(s)
- Tamar I Regev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Hee So Kim
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Xuanyi Chen
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Cognitive Sciences, Rice University, Houston, TX 77005, United States
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Abigail E Schipper
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
| | - Leon Bergen
- Department of Linguistics, University of California San Diego, San Diego CA 92093, United States
| | - Kyle Mahowald
- Department of Linguistics, University of Texas at Austin, Austin, TX 78712, United States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- The Harvard Program in Speech and Hearing Bioscience and Technology, Boston, MA 02115, United States
| |
Collapse
|
7
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. Nat Hum Behav 2024; 8:544-561. [PMID: 38172630 DOI: 10.1038/s41562-023-01783-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 11/10/2023] [Indexed: 01/05/2024]
Abstract
Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of the brain response associated with each sentence. We then use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress the activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also non-invasively control neural activity in higher-level cortical areas, such as the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- MIT-IBM Watson AI Lab, Cambridge, MA, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA, USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
8
|
Malik-Moraleda S, Jouravlev O, Taliaferro M, Mineroff Z, Cucu T, Mahowald K, Blank IA, Fedorenko E. Functional characterization of the language network of polyglots and hyperpolyglots with precision fMRI. Cereb Cortex 2024; 34:bhae049. [PMID: 38466812 PMCID: PMC10928488 DOI: 10.1093/cercor/bhae049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Revised: 01/24/2024] [Accepted: 01/25/2024] [Indexed: 03/13/2024] Open
Abstract
How do polyglots-individuals who speak five or more languages-process their languages, and what can this population tell us about the language system? Using fMRI, we identified the language network in each of 34 polyglots (including 16 hyperpolyglots with knowledge of 10+ languages) and examined its response to the native language, non-native languages of varying proficiency, and unfamiliar languages. All language conditions engaged all areas of the language network relative to a control condition. Languages that participants rated as higher proficiency elicited stronger responses, except for the native language, which elicited a similar or lower response than a non-native language of similar proficiency. Furthermore, unfamiliar languages that were typologically related to the participants' high-to-moderate-proficiency languages elicited a stronger response than unfamiliar unrelated languages. The results suggest that the language network's response magnitude scales with the degree of engagement of linguistic computations (e.g. related to lexical access and syntactic-structure building). We also replicated a prior finding of weaker responses to native language in polyglots than non-polyglot bilinguals. These results contribute to our understanding of how multiple languages coexist within a single brain and provide new evidence that the language network responds more strongly to stimuli that more fully engage linguistic computations.
Collapse
Affiliation(s)
- Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114, United States
| | - Olessia Jouravlev
- Department of Cognitive Science, Carleton University, Ottawa K1S 5B6, Canada
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
| | - Zachary Mineroff
- Eberly Center, Carnegie Mellon University, Pittsburgh, PA 15289, United States
| | - Theodore Cucu
- Department of Psychology, Carnegie Mellon University, Pittsburgh, PA 15289, United States
| | - Kyle Mahowald
- Department of Linguistics, The University of Texas at Austin, Austin, TX 78712, United States
| | - Idan A Blank
- Department of Psychology, University of California Los Angeles, Los Angeles, CA 90095, United States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114, United States
| |
Collapse
|
9
|
Malik-Moraleda S, Jouravlev O, Taliaferro M, Mineroff Z, Cucu T, Mahowald K, Blank IA, Fedorenko E. Functional characterization of the language network of polyglots and hyperpolyglots with precision fMRI. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.01.19.524657. [PMID: 36711949 PMCID: PMC9882290 DOI: 10.1101/2023.01.19.524657] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
How do polyglots-individuals who speak five or more languages-process their languages, and what can this population tell us about the language system? Using fMRI, we identified the language network in each of 34 polyglots (including 16 hyperpolyglots with knowledge of 10+ languages) and examined its response to the native language, non-native languages of varying proficiency, and unfamiliar languages. All language conditions engaged all areas of the language network relative to a control condition. Languages that participants rated as higher-proficiency elicited stronger responses, except for the native language, which elicited a similar or lower response than a non-native language of similar proficiency. Furthermore, unfamiliar languages that were typologically related to the participants' high-to-moderate-proficiency languages elicited a stronger response than unfamiliar unrelated languages. The results suggest that the language network's response magnitude scales with the degree of engagement of linguistic computations (e.g., related to lexical access and syntactic-structure building). We also replicated a prior finding of weaker responses to native language in polyglots than non-polyglot bilinguals. These results contribute to our understanding of how multiple languages co-exist within a single brain and provide new evidence that the language network responds more strongly to stimuli that more fully engage linguistic computations.
Collapse
Affiliation(s)
- Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114
| | - Olessia Jouravlev
- Department of Cognitive Science, Carleton University, Ottawa, Canada, K1S 5B6
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | | | - Theodore Cucu
- Department of Psychology, Carnegie Mellon University, Pittsburgh, PA 15289
| | - Kyle Mahowald
- Department of Linguistics, The University of Texas at Austin, Austin, TX 78712
| | - Idan A. Blank
- Department of Psychology, University of California Los Angeles, CA 90095
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114
| |
Collapse
|
10
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.16.537080. [PMID: 37090673 PMCID: PMC10120732 DOI: 10.1101/2023.04.16.537080] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/25/2023]
Abstract
Transformer models such as GPT generate human-like language and are highly predictive of human brain responses to language. Here, using fMRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of brain response associated with each sentence. Then, we use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also noninvasively control neural activity in higher-level cortical areas, like the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- MIT-IBM Watson AI Lab, Cambridge, MA 02142, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455 USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138 USA
| |
Collapse
|
11
|
Malik-Moraleda S, Taliaferro M, Shannon S, Jhingan N, Swords S, Peterson DJ, Frommer P, Okrand M, Sams J, Cardwell R, Freeman C, Fedorenko E. Constructed languages are processed by the same brain mechanisms as natural languages. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.07.28.550667. [PMID: 37546901 PMCID: PMC10402139 DOI: 10.1101/2023.07.28.550667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/08/2023]
Abstract
What constitutes a language? Natural languages share some features with other domains: from math, to music, to gesture. However, the brain mechanisms that process linguistic input are highly specialized, showing little or no response to diverse non-linguistic tasks. Here, we examine constructed languages (conlangs) to ask whether they draw on the same neural mechanisms as natural languages, or whether they instead pattern with domains like math and logic. Using individual-subject fMRI analyses, we show that understanding conlangs recruits the same brain areas as natural language comprehension. This result holds for Esperanto (n=19 speakers)- created to resemble natural languages-and fictional conlangs (Klingon (n=10), Na'vi (n=9), High Valyrian (n=3), and Dothraki (n=3)), created to differ from natural languages, and suggests that conlangs and natural languages share critical features and that the notable differences between conlangs and natural language are not consequential for the cognitive and neural mechanisms that they engage.
Collapse
Affiliation(s)
- Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Steve Shannon
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Niharika Jhingan
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Sara Swords
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | | | - Paul Frommer
- Marshall School of Business, University of Southern California, Los Angeles, CA 90089
| | | | | | | | | | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA 02114
| |
Collapse
|
12
|
Chen X, Affourtit J, Ryskin R, Regev TI, Norman-Haignere S, Jouravlev O, Malik-Moraleda S, Kean H, Varley R, Fedorenko E. The human language system, including its inferior frontal component in "Broca's area," does not support music perception. Cereb Cortex 2023; 33:7904-7929. [PMID: 37005063 PMCID: PMC10505454 DOI: 10.1093/cercor/bhad087] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 01/02/2023] [Accepted: 01/03/2023] [Indexed: 04/04/2023] Open
Abstract
Language and music are two human-unique capacities whose relationship remains debated. Some have argued for overlap in processing mechanisms, especially for structure processing. Such claims often concern the inferior frontal component of the language system located within "Broca's area." However, others have failed to find overlap. Using a robust individual-subject fMRI approach, we examined the responses of language brain regions to music stimuli, and probed the musical abilities of individuals with severe aphasia. Across 4 experiments, we obtained a clear answer: music perception does not engage the language system, and judgments about music structure are possible even in the presence of severe damage to the language network. In particular, the language regions' responses to music are generally low, often below the fixation baseline, and never exceed responses elicited by nonmusic auditory conditions, like animal sounds. Furthermore, the language regions are not sensitive to music structure: they show low responses to both intact and structure-scrambled music, and to melodies with vs. without structural violations. Finally, in line with past patient investigations, individuals with aphasia, who cannot judge sentence grammaticality, perform well on melody well-formedness judgments. Thus, the mechanisms that process structure in language do not appear to process music, including music syntax.
Collapse
Affiliation(s)
- Xuanyi Chen
- Department of Cognitive Sciences, Rice University, TX 77005, United States
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Rachel Ryskin
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Cognitive & Information Sciences, University of California, Merced, Merced, CA 95343, United States
| | - Tamar I Regev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Samuel Norman-Haignere
- Department of Biostatistics & Computational Biology, University of Rochester Medical Center, Rochester, NY, United States
- Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, United States
- Department of Biomedical Engineering, University of Rochester, Rochester, NY, United States
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, United States
| | - Olessia Jouravlev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Saima Malik-Moraleda
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| | - Hope Kean
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Rosemary Varley
- Psychology & Language Sciences, UCL, London, WCN1 1PF, United Kingdom
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| |
Collapse
|
13
|
Kauf C, Tuckute G, Levy R, Andreas J, Fedorenko E. Lexical semantic content, not syntactic structure, is the main contributor to ANN-brain similarity of fMRI responses in the language network. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.05.05.539646. [PMID: 37205405 PMCID: PMC10187317 DOI: 10.1101/2023.05.05.539646] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Representations from artificial neural network (ANN) language models have been shown to predict human brain activity in the language network. To understand what aspects of linguistic stimuli contribute to ANN-to-brain similarity, we used an fMRI dataset of responses to n=627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated the stimuli for which ANN representations were extracted. In particular, we i) perturbed sentences' word order, ii) removed different subsets of words, or iii) replaced sentences with other sentences of varying semantic similarity. We found that the lexical semantic content of the sentence (largely carried by content words) rather than the sentence's syntactic form (conveyed via word order or function words) is primarily responsible for the ANN-to-brain similarity. In follow-up analyses, we found that perturbation manipulations that adversely affect brain predictivity also lead to more divergent representations in the ANN's embedding space and decrease the ANN's ability to predict upcoming tokens in those stimuli. Further, results are robust to whether the mapping model is trained on intact or perturbed stimuli, and whether the ANN sentence representations are conditioned on the same linguistic context that humans saw. The critical result-that lexical-semantic content is the main contributor to the similarity between ANN representations and neural ones-aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.
Collapse
Affiliation(s)
- Carina Kauf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - Roger Levy
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Jacob Andreas
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Program in Speech and Hearing Bioscience and Technology, Harvard University
| |
Collapse
|
14
|
Hauptman M, Blank I, Fedorenko E. Non-literal language processing is jointly supported by the language and theory of mind networks: Evidence from a novel meta-analytic fMRI approach. Cortex 2023; 162:96-114. [PMID: 37023480 PMCID: PMC10210011 DOI: 10.1016/j.cortex.2023.01.013] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Revised: 11/08/2022] [Accepted: 01/11/2023] [Indexed: 03/12/2023]
Abstract
Going beyond the literal meaning of language is key to communicative success. However, the mechanisms that support non-literal inferences remain debated. Using a novel meta-analytic approach, we evaluate the contribution of linguistic, social-cognitive, and executive mechanisms to non-literal interpretation. We identified 74 fMRI experiments (n = 1,430 participants) from 2001 to 2021 that contrasted non-literal language comprehension with a literal control condition, spanning ten phenomena (e.g., metaphor, irony, indirect speech). Applying the activation likelihood estimation approach to the 825 activation peaks yielded six left-lateralized clusters. We then evaluated the locations of both the individual-study peaks and the clusters against probabilistic functional atlases (cf. anatomical locations, as is typically done) for three candidate brain networks-the language-selective network (Fedorenko, Behr, & Kanwisher, 2011), which supports language processing, the Theory of Mind (ToM) network (Saxe & Kanwisher, 2003), which supports social inferences, and the domain-general Multiple-Demand (MD) network (Duncan, 2010), which supports executive control. These atlases were created by overlaying individual activation maps of participants who performed robust and extensively validated 'localizer' tasks that selectively target each network in question (n = 806 for language; n = 198 for ToM; n = 691 for MD). We found that both the individual-study peaks and the ALE clusters fell primarily within the language network and the ToM network. These results suggest that non-literal processing is supported by both i) mechanisms that process literal linguistic meaning, and ii) mechanisms that support general social inference. They thus undermine a strong divide between literal and non-literal aspects of language and challenge the claim that non-literal processing requires additional executive resources.
Collapse
Affiliation(s)
- Miriam Hauptman
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Department of Psychological & Brain Sciences, Johns Hopkins University, Baltimore, MD 21218, USA.
| | - Idan Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Department of Psychology, UCLA, Los Angeles, CA 90095, USA; Department of Linguistics, UCLA, Los Angeles, CA 90095, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA; Program in Speech and Hearing in Bioscience and Technology, Harvard University, Boston, MA 02114, USA.
| |
Collapse
|
15
|
Hu J, Small H, Kean H, Takahashi A, Zekelman L, Kleinman D, Ryan E, Nieto-Castañón A, Ferreira V, Fedorenko E. Precision fMRI reveals that the language-selective network supports both phrase-structure building and lexical access during language production. Cereb Cortex 2023; 33:4384-4404. [PMID: 36130104 PMCID: PMC10110436 DOI: 10.1093/cercor/bhac350] [Citation(s) in RCA: 17] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Revised: 08/01/2022] [Accepted: 08/02/2022] [Indexed: 11/13/2022] Open
Abstract
A fronto-temporal brain network has long been implicated in language comprehension. However, this network's role in language production remains debated. In particular, it remains unclear whether all or only some language regions contribute to production, and which aspects of production these regions support. Across 3 functional magnetic resonance imaging experiments that rely on robust individual-subject analyses, we characterize the language network's response to high-level production demands. We report 3 novel results. First, sentence production, spoken or typed, elicits a strong response throughout the language network. Second, the language network responds to both phrase-structure building and lexical access demands, although the response to phrase-structure building is stronger and more spatially extensive, present in every language region. Finally, contra some proposals, we find no evidence of brain regions-within or outside the language network-that selectively support phrase-structure building in production relative to comprehension. Instead, all language regions respond more strongly during production than comprehension, suggesting that production incurs a greater cost for the language network. Together, these results align with the idea that language comprehension and production draw on the same knowledge representations, which are stored in a distributed manner within the language-selective network and are used to both interpret and generate linguistic utterances.
Collapse
Affiliation(s)
- Jennifer Hu
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
| | - Hannah Small
- Department of Cognitive Science, Johns Hopkins University, Baltimore, MD 21218, United States
| | - Hope Kean
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Atsushi Takahashi
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
| | - Leo Zekelman
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| | | | - Elizabeth Ryan
- St. George’s Medical School, St. George’s University, Grenada, West Indies
| | - Alfonso Nieto-Castañón
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Department of Speech, Language, and Hearing Sciences, Boston University, Boston, MA 02215, United States
| | - Victor Ferreira
- Department of Psychology, UCSD, La Jolla, CA 92093, United States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, United States
- McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, United States
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138, United States
| |
Collapse
|
16
|
Fedorenko E, Ryskin R, Gibson E. Agrammatic output in non-fluent, including Broca's, aphasia as a rational behavior. APHASIOLOGY 2022; 37:1981-2000. [PMID: 38213953 PMCID: PMC10782888 DOI: 10.1080/02687038.2022.2143233] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/11/2022] [Accepted: 10/31/2022] [Indexed: 01/13/2024]
Abstract
Background Speech of individuals with non-fluent, including Broca's, aphasia is often characterized as "agrammatic" because their output mostly consists of nouns and, to a lesser extent, verbs and lacks function words, like articles and prepositions, and correct morphological endings. Among the earliest accounts of agrammatic output in the early 1900s was the "economy of effort" idea whereby agrammatic output is construed as a way of coping with increases in the cost of language production. This idea resurfaced in the 1980s, but in general, the field of language research has largely focused on accounts of agrammatism that postulated core deficits in syntactic knowledge. Aims We here revisit the economy of effort hypothesis in light of increasing emphasis in cognitive science on rational and efficient behavior. Main contribution The critical idea is as follows: there is a cost per unit of linguistic output, and this cost is greater for patients with non-fluent aphasia. For a rational agent, this increase leads to shorter messages. Critically, the informative parts of the message should be preserved and the redundant ones (like the function words and inflectional markers) should be omitted. Although economy of effort is unlikely to provide a unifying account of agrammatic output in all patients-the relevant population is too heterogeneous and the empirical landscape too complex for any single-factor explanation-we argue that the idea of agrammatic output as a rational behavior was dismissed prematurely and appears to provide a plausible explanation for a large subset of the reported cases of expressive aphasia. Conclusions The rational account of expressive agrammatism should be evaluated more carefully and systematically. On the basic research side, pursuing this hypothesis may reveal how the human mind and brain optimize communicative efficiency in the presence of production difficulties. And on the applied side, this construal of expressive agrammatism emphasizes the strengths of some patients to flexibly adapt utterances in order to communicate in spite of grammatical difficulties; and focusing on these strengths may be more effective than trying to "fix" their grammar.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Massachusetts Institute of Technology, Brain & Cognitive Sciences Department
- Massachusetts Institute of Technology, McGovern Institute for Brain Research
- Speech and Hearing in Bioscience and Technology program at Harvard University
| | - Rachel Ryskin
- University of California at Merced, Cognitive & Information Sciences Department
| | - Edward Gibson
- Massachusetts Institute of Technology, Brain & Cognitive Sciences Department
| |
Collapse
|
17
|
Kasai C, Sumiya M, Koike T, Yoshimoto T, Maki H, Sadato N. Neural underpinning of Japanese particle processing in non-native speakers. Sci Rep 2022; 12:18740. [PMID: 36335170 PMCID: PMC9637203 DOI: 10.1038/s41598-022-23382-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2021] [Accepted: 10/31/2022] [Indexed: 11/08/2022] Open
Abstract
Grammar acquisition by non-native learners (L2) is typically less successful and may produce fundamentally different grammatical systems than that by native speakers (L1). The neural representation of grammatical processing between L1 and L2 speakers remains controversial. We hypothesized that working memory is the primary source of L1/L2 differences, by considering working memory within the predictive coding account, which models grammatical processes as higher-level neuronal representations of cortical hierarchies, generating predictions (forward model) of lower-level representations. A functional MRI study was conducted with L1 Japanese speakers and highly proficient Japanese learners requiring oral production of grammatically correct Japanese particles. We assumed selecting proper particles requires forward model-dependent processes of working memory as their functions are highly context-dependent. As a control, participants read out a visually designated mora indicated by underlining. Particle selection by L1/L2 groups commonly activated the bilateral inferior frontal gyrus/insula, pre-supplementary motor area, left caudate, middle temporal gyrus, and right cerebellum, which constituted the core linguistic production system. In contrast, the left inferior frontal sulcus, known as the neural substrate of verbal working memory, showed more prominent activation in L2 than in L1. Thus, the working memory process causes L1/L2 differences even in highly proficient L2 learners.
Collapse
Affiliation(s)
- Chise Kasai
- grid.256342.40000 0004 0370 4927Faculty of Regional Studies, Gifu University, Yanagido, 501-1193 Japan ,grid.467811.d0000 0001 2272 1771Division of Cerebral Integration, National Institute for Physiological Sciences, Okazaki, 444-8585 Japan
| | - Motofumi Sumiya
- grid.505613.40000 0000 8937 6696Research Center for Child Mental Development, Hamamatsu University School of Medicine, Hamamatsu, 431-3192 Japan
| | - Takahiko Koike
- grid.467811.d0000 0001 2272 1771Division of Cerebral Integration, National Institute for Physiological Sciences, Okazaki, 444-8585 Japan ,grid.275033.00000 0004 1763 208XDepartment of Physiological Sciences, SOKENDAI (The Graduate University for Advanced Studies), Hayama, 240-0193 Japan
| | - Takaaki Yoshimoto
- grid.467811.d0000 0001 2272 1771Division of Cerebral Integration, National Institute for Physiological Sciences, Okazaki, 444-8585 Japan ,grid.262576.20000 0000 8863 9909Research Organization of Science and Technology, Ritsumeikan University, Kusatsu, 525-8577 Japan
| | - Hideki Maki
- grid.256342.40000 0004 0370 4927Faculty of Regional Studies, Gifu University, Yanagido, 501-1193 Japan
| | - Norihiro Sadato
- grid.467811.d0000 0001 2272 1771Division of Cerebral Integration, National Institute for Physiological Sciences, Okazaki, 444-8585 Japan ,grid.275033.00000 0004 1763 208XDepartment of Physiological Sciences, SOKENDAI (The Graduate University for Advanced Studies), Hayama, 240-0193 Japan ,grid.262576.20000 0000 8863 9909Research Organization of Science and Technology, Ritsumeikan University, Kusatsu, 525-8577 Japan
| |
Collapse
|
18
|
Broderick MP, Zuk NJ, Anderson AJ, Lalor EC. More than words: Neurophysiological correlates of semantic dissimilarity depend on comprehension of the speech narrative. Eur J Neurosci 2022; 56:5201-5214. [PMID: 35993240 DOI: 10.1111/ejn.15805] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2022] [Revised: 08/15/2022] [Accepted: 08/18/2022] [Indexed: 12/14/2022]
Abstract
Speech comprehension relies on the ability to understand words within a coherent context. Recent studies have attempted to obtain electrophysiological indices of this process by modelling how brain activity is affected by a word's semantic dissimilarity to preceding words. Although the resulting indices appear robust and are strongly modulated by attention, it remains possible that, rather than capturing the contextual understanding of words, they may actually reflect word-to-word changes in semantic content without the need for a narrative-level understanding on the part of the listener. To test this, we recorded electroencephalography from subjects who listened to speech presented in either its original, narrative form, or after scrambling the word order by varying amounts. This manipulation affected the ability of subjects to comprehend the speech narrative but not the ability to recognise individual words. Neural indices of semantic understanding and low-level acoustic processing were derived for each scrambling condition using the temporal response function. Signatures of semantic processing were observed when speech was unscrambled or minimally scrambled and subjects understood the speech. The same markers were absent for higher scrambling levels as speech comprehension dropped. In contrast, word recognition remained high and neural measures related to envelope tracking did not vary significantly across scrambling conditions. This supports the previous claim that electrophysiological indices based on the semantic dissimilarity of words to their context reflect a listener's understanding of those words relative to that context. It also highlights the relative insensitivity of neural measures of low-level speech processing to speech comprehension.
Collapse
Affiliation(s)
- Michael P Broderick
- School of Engineering, Trinity Centre for Biomedical Engineering and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
| | - Nathaniel J Zuk
- School of Engineering, Trinity Centre for Biomedical Engineering and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
| | - Andrew J Anderson
- Del Monte Institute for Neuroscience, Department of Neuroscience, Department of Biomedical Engineering, University of Rochester, Rochester, New York, USA
| | - Edmund C Lalor
- School of Engineering, Trinity Centre for Biomedical Engineering and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland.,Del Monte Institute for Neuroscience, Department of Neuroscience, Department of Biomedical Engineering, University of Rochester, Rochester, New York, USA
| |
Collapse
|
19
|
Lipkin B, Tuckute G, Affourtit J, Small H, Mineroff Z, Kean H, Jouravlev O, Rakocevic L, Pritchett B, Siegelman M, Hoeflin C, Pongos A, Blank IA, Struhl MK, Ivanova A, Shannon S, Sathe A, Hoffmann M, Nieto-Castañón A, Fedorenko E. Probabilistic atlas for the language network based on precision fMRI data from >800 individuals. Sci Data 2022; 9:529. [PMID: 36038572 PMCID: PMC9424256 DOI: 10.1038/s41597-022-01645-3] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2022] [Accepted: 08/09/2022] [Indexed: 11/13/2022] Open
Abstract
Two analytic traditions characterize fMRI language research. One relies on averaging activations across individuals. This approach has limitations: because of inter-individual variability in the locations of language areas, any given voxel/vertex in a common brain space is part of the language network in some individuals but in others, may belong to a distinct network. An alternative approach relies on identifying language areas in each individual using a functional ‘localizer’. Because of its greater sensitivity, functional resolution, and interpretability, functional localization is gaining popularity, but it is not always feasible, and cannot be applied retroactively to past studies. To bridge these disjoint approaches, we created a probabilistic functional atlas using fMRI data for an extensively validated language localizer in 806 individuals. This atlas enables estimating the probability that any given location in a common space belongs to the language network, and thus can help interpret group-level activation peaks and lesion locations, or select voxels/electrodes for analysis. More meaningful comparisons of findings across studies should increase robustness and replicability in language research. Measurement(s) | Brain activity measurement | Technology Type(s) | fMRI | Sample Characteristic - Organism | Homo sapiens |
Collapse
Affiliation(s)
- Benjamin Lipkin
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA. .,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Hannah Small
- Department of Cognitive Science, Johns Hopkins University, Baltimore, MD, USA
| | - Zachary Mineroff
- Human-computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Hope Kean
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Olessia Jouravlev
- Department of Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Lara Rakocevic
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Brianna Pritchett
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | | | - Caitlyn Hoeflin
- Harris School of Public Policy, University of Chicago, Chicago, IL, USA
| | - Alvincé Pongos
- Department of Bioengineering, University of California, Berkeley, CA, USA
| | - Idan A Blank
- Department of Psychology, University of California, Los Angeles, CA, USA
| | - Melissa Kline Struhl
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Anna Ivanova
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Steven Shannon
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Malte Hoffmann
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Cambridge, MA, USA
| | - Alfonso Nieto-Castañón
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.,Department of Speech, Language, and Hearing Sciences, Boston University, Boston, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA. .,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA. .,Department of Speech, Hearing, Bioscience, and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
20
|
Paunov AM, Blank IA, Jouravlev O, Mineroff Z, Gallée J, Fedorenko E. Differential Tracking of Linguistic vs. Mental State Content in Naturalistic Stimuli by Language and Theory of Mind (ToM) Brain Networks. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:413-440. [PMID: 37216061 PMCID: PMC10158571 DOI: 10.1162/nol_a_00071] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Accepted: 04/11/2022] [Indexed: 05/24/2023]
Abstract
Language and social cognition, especially the ability to reason about mental states, known as theory of mind (ToM), are deeply related in development and everyday use. However, whether these cognitive faculties rely on distinct, overlapping, or the same mechanisms remains debated. Some evidence suggests that, by adulthood, language and ToM draw on largely distinct-though plausibly interacting-cortical networks. However, the broad topography of these networks is similar, and some have emphasized the importance of social content / communicative intent in the linguistic signal for eliciting responses in the language areas. Here, we combine the power of individual-subject functional localization with the naturalistic-cognition inter-subject correlation approach to illuminate the language-ToM relationship. Using functional magnetic resonance imaging (fMRI), we recorded neural activity as participants (n = 43) listened to stories and dialogues with mental state content (+linguistic, +ToM), viewed silent animations and live action films with mental state content but no language (-linguistic, +ToM), or listened to an expository text (+linguistic, -ToM). The ToM network robustly tracked stimuli rich in mental state information regardless of whether mental states were conveyed linguistically or non-linguistically, while tracking a +linguistic / -ToM stimulus only weakly. In contrast, the language network tracked linguistic stimuli more strongly than (a) non-linguistic stimuli, and than (b) the ToM network, and showed reliable tracking even for the linguistic condition devoid of mental state content. These findings suggest that in spite of their indisputably close links, language and ToM dissociate robustly in their neural substrates-and thus plausibly cognitive mechanisms-including during the processing of rich naturalistic materials.
Collapse
Affiliation(s)
- Alexander M. Paunov
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, Université Paris-Saclay, NeuroSpin Center, 91191Gif/Yvette, France
| | - Idan A. Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Department of Psychology, UCLA, Los Angeles, CA, USA
| | - Olessia Jouravlev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Institute for Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Eberly Center for Teaching Excellence & Educational Innovation, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Jeanne Gallée
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| |
Collapse
|
21
|
Distinct spatiotemporal patterns of syntactic and semantic processing in human inferior frontal gyrus. Nat Hum Behav 2022; 6:1104-1111. [PMID: 35618778 DOI: 10.1038/s41562-022-01334-6] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2020] [Accepted: 03/09/2022] [Indexed: 11/08/2022]
Abstract
Human languages are based on syntax, a set of rules which allow an infinite number of meaningful sentences to be constructed from a finite set of words. A theory associated with Chomsky and others holds that syntax is a mind-internal, universal structure independent of semantics. This theory, however, has been challenged by studies of the Chinese language showing that syntax is processed under the semantic umbrella, and is secondary and not independent. Here, using intracranial high-density electrocorticography, we find distinct spatiotemporal patterns of neural activity in the left inferior frontal gyrus that are specifically associated with syntactic and semantic processing of Chinese sentences. These results suggest that syntactic processing may occur before semantic processing. Our findings are consistent with the view that the human brain implements syntactic structures in a manner that is independent of semantics.
Collapse
|
22
|
Tuckute G, Paunov A, Kean H, Small H, Mineroff Z, Blank I, Fedorenko E. Frontal language areas do not emerge in the absence of temporal language areas: A case study of an individual born without a left temporal lobe. Neuropsychologia 2022; 169:108184. [DOI: 10.1016/j.neuropsychologia.2022.108184] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Revised: 12/07/2021] [Accepted: 02/15/2022] [Indexed: 10/19/2022]
|
23
|
Wehbe L, Blank IA, Shain C, Futrell R, Levy R, von der Malsburg T, Smith N, Gibson E, Fedorenko E. Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network. Cereb Cortex 2021. [PMID: 33895807 DOI: 10.1101/2020.04.15.043844] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/11/2023] Open
Abstract
What role do domain-general executive functions play in human language comprehension? To address this question, we examine the relationship between behavioral measures of comprehension and neural activity in the domain-general "multiple demand" (MD) network, which has been linked to constructs like attention, working memory, inhibitory control, and selection, and implicated in diverse goal-directed behaviors. Specifically, functional magnetic resonance imaging data collected during naturalistic story listening are compared with theory-neutral measures of online comprehension difficulty and incremental processing load (reading times and eye-fixation durations). Critically, to ensure that variance in these measures is driven by features of the linguistic stimulus rather than reflecting participant- or trial-level variability, the neuroimaging and behavioral datasets were collected in nonoverlapping samples. We find no behavioral-neural link in functionally localized MD regions; instead, this link is found in the domain-specific, fronto-temporal "core language network," in both left-hemispheric areas and their right hemispheric homotopic areas. These results argue against strong involvement of domain-general executive circuits in language comprehension.
Collapse
Affiliation(s)
- Leila Wehbe
- Carnegie Mellon University, Machine Learning Department PA 15213, USA
| | - Idan Asher Blank
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
- University of California Los Angeles, Department of Psychology CA 90095, USA
| | - Cory Shain
- Ohio State University, Department of Linguistics OH 43210, USA
| | - Richard Futrell
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
- University of California Irvine, Department of Linguistics CA 92697, USA
| | - Roger Levy
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
- University of California San Diego, Department of Linguistics CA 92161, USA
| | - Titus von der Malsburg
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
- University of Stuttgart, Institute of Linguistics, 70049 Stuttgart, Germany
| | - Nathaniel Smith
- University of California San Diego, Department of Linguistics CA 92161, USA
| | - Edward Gibson
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
| | - Evelina Fedorenko
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
- Massachusetts Institute of Technology, McGovern Institute for Brain ResearchMA 02139, USA
| |
Collapse
|
24
|
Wehbe L, Blank IA, Shain C, Futrell R, Levy R, von der Malsburg T, Smith N, Gibson E, Fedorenko E. Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network. Cereb Cortex 2021; 31:4006-4023. [PMID: 33895807 DOI: 10.1093/cercor/bhab065] [Citation(s) in RCA: 29] [Impact Index Per Article: 9.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Revised: 01/15/2021] [Accepted: 02/21/2021] [Indexed: 12/28/2022] Open
Abstract
What role do domain-general executive functions play in human language comprehension? To address this question, we examine the relationship between behavioral measures of comprehension and neural activity in the domain-general "multiple demand" (MD) network, which has been linked to constructs like attention, working memory, inhibitory control, and selection, and implicated in diverse goal-directed behaviors. Specifically, functional magnetic resonance imaging data collected during naturalistic story listening are compared with theory-neutral measures of online comprehension difficulty and incremental processing load (reading times and eye-fixation durations). Critically, to ensure that variance in these measures is driven by features of the linguistic stimulus rather than reflecting participant- or trial-level variability, the neuroimaging and behavioral datasets were collected in nonoverlapping samples. We find no behavioral-neural link in functionally localized MD regions; instead, this link is found in the domain-specific, fronto-temporal "core language network," in both left-hemispheric areas and their right hemispheric homotopic areas. These results argue against strong involvement of domain-general executive circuits in language comprehension.
Collapse
Affiliation(s)
- Leila Wehbe
- Carnegie Mellon University, Machine Learning Department PA 15213, USA
| | - Idan Asher Blank
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA.,University of California Los Angeles, Department of Psychology CA 90095, USA
| | - Cory Shain
- Ohio State University, Department of Linguistics OH 43210, USA
| | - Richard Futrell
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA.,University of California Irvine, Department of Linguistics CA 92697, USA
| | - Roger Levy
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA.,University of California San Diego, Department of Linguistics CA 92161, USA
| | - Titus von der Malsburg
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA.,University of Stuttgart, Institute of Linguistics, 70049 Stuttgart, Germany
| | - Nathaniel Smith
- University of California San Diego, Department of Linguistics CA 92161, USA
| | - Edward Gibson
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA
| | - Evelina Fedorenko
- Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences MA 02139, USA.,Massachusetts Institute of Technology, McGovern Institute for Brain ResearchMA 02139, USA
| |
Collapse
|
25
|
Jouravlev O, Mineroff Z, Blank IA, Fedorenko E. The Small and Efficient Language Network of Polyglots and Hyper-polyglots. Cereb Cortex 2021; 31:62-76. [PMID: 32820332 DOI: 10.1093/cercor/bhaa205] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2019] [Revised: 07/06/2020] [Accepted: 07/06/2020] [Indexed: 11/13/2022] Open
Abstract
Acquiring a foreign language is challenging for many adults. Yet certain individuals choose to acquire sometimes dozens of languages and often just for fun. Is there something special about the minds and brains of such polyglots? Using robust individual-level markers of language activity, measured with fMRI, we compared native language processing in polyglots versus matched controls. Polyglots (n = 17, including nine "hyper-polyglots" with proficiency in 10-55 languages) used fewer neural resources to process language: Their activations were smaller in both magnitude and extent. This difference was spatially and functionally selective: The groups were similar in their activation of two other brain networks-the multiple demand network and the default mode network. We hypothesize that the activation reduction in the language network is experientially driven, such that the acquisition and use of multiple languages makes language processing generally more efficient. However, genetic and longitudinal studies will be critical to distinguish this hypothesis from the one whereby polyglots' brains already differ at birth or early in development. This initial characterization of polyglots' language network opens the door to future investigations of the cognitive and neural architecture of individuals who gain mastery of multiple languages, including changes in this architecture with linguistic experiences.
Collapse
Affiliation(s)
- Olessia Jouravlev
- Brain & Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.,Department of Cognitive Science, Carleton University, Ottawa, ON K1S5B6, Canada
| | - Zachary Mineroff
- Brain & Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Idan A Blank
- Brain & Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.,Department of Psychology, University of California Los Angeles, Los Angeles, CA 90095, USA
| | - Evelina Fedorenko
- Brain & Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| |
Collapse
|
26
|
Ivanova AA, Srikant S, Sueoka Y, Kean HH, Dhamala R, O'Reilly UM, Bers MU, Fedorenko E. Comprehension of computer code relies primarily on domain-general executive brain regions. eLife 2020; 9:e58906. [PMID: 33319744 PMCID: PMC7738192 DOI: 10.7554/elife.58906] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2020] [Accepted: 11/06/2020] [Indexed: 12/22/2022] Open
Abstract
Computer programming is a novel cognitive tool that has transformed modern society. What cognitive and neural mechanisms support this skill? Here, we used functional magnetic resonance imaging to investigate two candidate brain systems: the multiple demand (MD) system, typically recruited during math, logic, problem solving, and executive tasks, and the language system, typically recruited during linguistic processing. We examined MD and language system responses to code written in Python, a text-based programming language (Experiment 1) and in ScratchJr, a graphical programming language (Experiment 2); for both, we contrasted responses to code problems with responses to content-matched sentence problems. We found that the MD system exhibited strong bilateral responses to code in both experiments, whereas the language system responded strongly to sentence problems, but weakly or not at all to code problems. Thus, the MD system supports the use of novel cognitive tools even when the input is structurally similar to natural language.
Collapse
Affiliation(s)
- Anna A Ivanova
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Shashank Srikant
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Yotaro Sueoka
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Hope H Kean
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Riva Dhamala
- Eliot-Pearson Department of Child Study and Human Development, Tufts UniversityMedfordUnited States
| | - Una-May O'Reilly
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Marina U Bers
- Eliot-Pearson Department of Child Study and Human Development, Tufts UniversityMedfordUnited States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| |
Collapse
|
27
|
Movies and narratives as naturalistic stimuli in neuroimaging. Neuroimage 2020; 224:117445. [PMID: 33059053 PMCID: PMC7805386 DOI: 10.1016/j.neuroimage.2020.117445] [Citation(s) in RCA: 54] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2020] [Revised: 10/06/2020] [Accepted: 10/09/2020] [Indexed: 01/06/2023] Open
Abstract
Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.
Collapse
|
28
|
Fedorenko E, Blank IA, Siegelman M, Mineroff Z. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition 2020; 203:104348. [PMID: 32569894 DOI: 10.1101/477851] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2018] [Revised: 05/14/2020] [Accepted: 05/31/2020] [Indexed: 05/25/2023]
Abstract
To understand what you are reading now, your mind retrieves the meanings of words and constructions from a linguistic knowledge store (lexico-semantic processing) and identifies the relationships among them to construct a complex meaning (syntactic or combinatorial processing). Do these two sets of processes rely on distinct, specialized mechanisms or, rather, share a common pool of resources? Linguistic theorizing, empirical evidence from language acquisition and processing, and computational modeling have jointly painted a picture whereby lexico-semantic and syntactic processing are deeply inter-connected and perhaps not separable. In contrast, many current proposals of the neural architecture of language continue to endorse a view whereby certain brain regions selectively support syntactic/combinatorial processing, although the locus of such "syntactic hub", and its nature, vary across proposals. Here, we searched for selectivity for syntactic over lexico-semantic processing using a powerful individual-subjects fMRI approach across three sentence comprehension paradigms that have been used in prior work to argue for such selectivity: responses to lexico-semantic vs. morpho-syntactic violations (Experiment 1); recovery from neural suppression across pairs of sentences differing in only lexical items vs. only syntactic structure (Experiment 2); and same/different meaning judgments on such sentence pairs (Experiment 3). Across experiments, both lexico-semantic and syntactic conditions elicited robust responses throughout the left fronto-temporal language network. Critically, however, no regions were more strongly engaged by syntactic than lexico-semantic processing, although some regions showed the opposite pattern. Thus, contra many current proposals of the neural architecture of language, syntactic/combinatorial processing is not separable from lexico-semantic processing at the level of brain regions-or even voxel subsets-within the language network, in line with strong integration between these two processes that has been consistently observed in behavioral and computational language research. The results further suggest that the language network may be generally more strongly concerned with meaning than syntactic form, in line with the primary function of language-to share meanings across minds.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA.
| | - Idan Asher Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Department of Psychology, UCLA, Los Angeles, CA 90095, USA
| | - Matthew Siegelman
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Department of Psychology, Columbia University, New York, NY 10027, USA
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Eberly Center for Teaching Excellence & Educational Innovation, CMU, Pittsburgh, PA 15213, USA
| |
Collapse
|
29
|
Fedorenko E, Blank IA, Siegelman M, Mineroff Z. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition 2020; 203:104348. [PMID: 32569894 PMCID: PMC7483589 DOI: 10.1016/j.cognition.2020.104348] [Citation(s) in RCA: 72] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2018] [Revised: 05/14/2020] [Accepted: 05/31/2020] [Indexed: 12/31/2022]
Abstract
To understand what you are reading now, your mind retrieves the meanings of words and constructions from a linguistic knowledge store (lexico-semantic processing) and identifies the relationships among them to construct a complex meaning (syntactic or combinatorial processing). Do these two sets of processes rely on distinct, specialized mechanisms or, rather, share a common pool of resources? Linguistic theorizing, empirical evidence from language acquisition and processing, and computational modeling have jointly painted a picture whereby lexico-semantic and syntactic processing are deeply inter-connected and perhaps not separable. In contrast, many current proposals of the neural architecture of language continue to endorse a view whereby certain brain regions selectively support syntactic/combinatorial processing, although the locus of such "syntactic hub", and its nature, vary across proposals. Here, we searched for selectivity for syntactic over lexico-semantic processing using a powerful individual-subjects fMRI approach across three sentence comprehension paradigms that have been used in prior work to argue for such selectivity: responses to lexico-semantic vs. morpho-syntactic violations (Experiment 1); recovery from neural suppression across pairs of sentences differing in only lexical items vs. only syntactic structure (Experiment 2); and same/different meaning judgments on such sentence pairs (Experiment 3). Across experiments, both lexico-semantic and syntactic conditions elicited robust responses throughout the left fronto-temporal language network. Critically, however, no regions were more strongly engaged by syntactic than lexico-semantic processing, although some regions showed the opposite pattern. Thus, contra many current proposals of the neural architecture of language, syntactic/combinatorial processing is not separable from lexico-semantic processing at the level of brain regions-or even voxel subsets-within the language network, in line with strong integration between these two processes that has been consistently observed in behavioral and computational language research. The results further suggest that the language network may be generally more strongly concerned with meaning than syntactic form, in line with the primary function of language-to share meanings across minds.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, MIT, Cambridge, MA 02139, USA.
| | - Idan Asher Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Department of Psychology, UCLA, Los Angeles, CA 90095, USA
| | - Matthew Siegelman
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Department of Psychology, Columbia University, New York, NY 10027, USA
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA; Eberly Center for Teaching Excellence & Educational Innovation, CMU, Pittsburgh, PA 15213, USA
| |
Collapse
|
30
|
Jouravlev O, Kell AJE, Mineroff Z, Haskins AJ, Ayyash D, Kanwisher N, Fedorenko E. Reduced Language Lateralization in Autism and the Broader Autism Phenotype as Assessed with Robust Individual-Subjects Analyses. Autism Res 2020; 13:1746-1761. [PMID: 32935455 DOI: 10.1002/aur.2393] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2020] [Revised: 07/28/2020] [Accepted: 08/25/2020] [Indexed: 12/13/2022]
Abstract
One of the few replicated functional brain differences between individuals with autism spectrum disorders (ASD) and neurotypical (NT) controls is reduced language lateralization. However, most prior reports relied on comparisons of group-level activation maps or functional markers that had not been validated at the individual-subject level, and/or used tasks that do not isolate language processing from other cognitive processes, complicating interpretation. Furthermore, few prior studies have examined functional responses in other brain networks, as needed to determine the spatial selectivity of the effect. Using functional magnetic resonance imaging (fMRI), we compared language lateralization between 28 adult ASD participants and carefully pairwise-matched controls, with the language regions defined individually using a well-validated language "localizer" task. Across two language comprehension paradigms, ASD participants showed less lateralized responses due to stronger right hemisphere activity. Furthermore, this effect did not stem from a ubiquitous reduction in lateralization of function across the brain: ASD participants did not differ from controls in the lateralization of two other large-scale networks-the Theory of Mind network and the Multiple Demand network. Finally, in an exploratory study, we tested whether reduced language lateralization may also be present in NT individuals with high autism-like traits. Indeed, autistic trait load in a large set of NT participants (n = 189) was associated with less lateralized language responses. These results suggest that reduced language lateralization is robustly associated with autism and, to some extent, with autism-like traits in the general population, and this lateralization reduction appears to be restricted to the language system. LAY SUMMARY: How do brains of individuals with autism spectrum disorders (ASD) differ from those of neurotypical (NT) controls? One of the most consistently reported differences is the reduction of lateralization during language processing in individuals with ASD. However, most prior studies have used methods that made this finding difficult to interpret, and perhaps even artifactual. Using robust individual-level markers of lateralization, we found that indeed, ASD individuals show reduced lateralization for language due to stronger right-hemisphere activity. We further show that this reduction is not due to a general reduction of lateralization of function across the brain. Finally, we show that greater autistic trait load is associated with less lateralized language responses in the NT population. These results suggest that reduced language lateralization is robustly associated with autism and, to some extent, with autism-like traits in the general population. Autism Res 2020, 13: 1746-1761. © 2020 International Society for Autism Research and Wiley Periodicals LLC.
Collapse
Affiliation(s)
- Olessia Jouravlev
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Department of Cognitive Science, Carleton University, Ottawa, Ontario, Canada
| | - Alexander J E Kell
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Zuckerman Institute, Columbia University, New York, New York, USA
| | - Zachary Mineroff
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Eberly Center, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
| | - Amanda J Haskins
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, USA
| | - Dima Ayyash
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| | - Nancy Kanwisher
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| | - Evelina Fedorenko
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| |
Collapse
|
31
|
Diachek E, Blank I, Siegelman M, Affourtit J, Fedorenko E. The Domain-General Multiple Demand (MD) Network Does Not Support Core Aspects of Language Comprehension: A Large-Scale fMRI Investigation. J Neurosci 2020; 40:4536-4550. [PMID: 32317387 PMCID: PMC7275862 DOI: 10.1523/jneurosci.2036-19.2020] [Citation(s) in RCA: 81] [Impact Index Per Article: 20.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2019] [Revised: 03/02/2020] [Accepted: 04/05/2020] [Indexed: 11/21/2022] Open
Abstract
Aside from the language-selective left-lateralized frontotemporal network, language comprehension sometimes recruits a domain-general bilateral frontoparietal network implicated in executive functions: the multiple demand (MD) network. However, the nature of the MD network's contributions to language comprehension remains debated. To illuminate the role of this network in language processing in humans, we conducted a large-scale fMRI investigation using data from 30 diverse word and sentence comprehension experiments (481 unique participants [female and male], 678 scanning sessions). In line with prior findings, the MD network was active during many language tasks. Moreover, similar to the language-selective network, which is robustly lateralized to the left hemisphere, these responses were stronger in the left-hemisphere MD regions. However, in contrast with the language-selective network, the MD network responded more strongly (1) to lists of unconnected words than to sentences, and (2) in paradigms with an explicit task compared with passive comprehension paradigms. Indeed, many passive comprehension tasks failed to elicit a response above the fixation baseline in the MD network, in contrast to strong responses in the language-selective network. Together, these results argue against a role for the MD network in core aspects of sentence comprehension, such as inhibiting irrelevant meanings or parses, keeping intermediate representations active in working memory, or predicting upcoming words or structures. These results align with recent evidence of relatively poor tracking of the linguistic signal by the MD regions during naturalistic comprehension, and instead suggest that the MD network's engagement during language processing reflects effort associated with extraneous task demands.SIGNIFICANCE STATEMENT Domain-general executive processes, such as working memory and cognitive control, have long been implicated in language comprehension, including in neuroimaging studies that have reported activation in domain-general multiple demand (MD) regions for linguistic manipulations. However, much prior evidence has come from paradigms where language interpretation is accompanied by extraneous tasks. Using a large fMRI dataset (30 experiments/481 participants/678 sessions), we demonstrate that MD regions are engaged during language comprehension in the presence of task demands, but not during passive reading/listening, conditions that strongly activate the frontotemporal language network. These results present a fundamental challenge to proposals whereby linguistic computations, such as inhibiting irrelevant meanings, keeping representations active in working memory, or predicting upcoming elements, draw on domain-general executive resources.
Collapse
Affiliation(s)
- Evgeniia Diachek
- Department of Psychology, Vanderbilt University, Nashville, Tennessee 37203
| | - Idan Blank
- Department of Psychology, University of California at Los Angeles, Los Angeles, California 90095
| | - Matthew Siegelman
- Department of Psychology, Columbia University, New York, New York 10027
| | - Josef Affourtit
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
- Department of Psychiatry, Massachusetts General Hospital, Charlestown, Massachusetts 02129
| |
Collapse
|
32
|
Blank IA, Fedorenko E. No evidence for differences among language regions in their temporal receptive windows. Neuroimage 2020; 219:116925. [PMID: 32407994 DOI: 10.1016/j.neuroimage.2020.116925] [Citation(s) in RCA: 36] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2019] [Revised: 03/20/2020] [Accepted: 05/06/2020] [Indexed: 10/24/2022] Open
Abstract
The "core language network" consists of left frontal and temporal regions that are selectively engaged in linguistic processing. Whereas functional differences among these regions have long been debated, many accounts propose distinctions in terms of representational grain-size-e.g., words vs. phrases/sentences-or processing time-scale, i.e., operating on local linguistic features vs. larger spans of input. Indeed, the topography of language regions appears to overlap with a cortical hierarchy reported by Lerner et al. (2011) wherein mid-posterior temporal regions are sensitive to low-level features of speech, surrounding areas-to word-level information, and inferior frontal areas-to sentence-level information and beyond. However, the correspondence between the language network and this hierarchy of "temporal receptive windows" (TRWs) is difficult to establish because the precise anatomical locations of language regions vary across individuals. To directly test this correspondence, we first identified language regions in each participant with a well-validated task-based localizer, which confers high functional resolution to the study of TRWs (traditionally based on stereotactic coordinates); then, we characterized regional TRWs with the naturalistic story listening paradigm of Lerner et al. (2011), which augments task-based characterizations of the language network by more closely resembling comprehension "in the wild". We find no region-by-TRW interactions across temporal and inferior frontal regions, which are all sensitive to both word-level and sentence-level information. Therefore, the language network as a whole constitutes a unique stage of information integration within a broader cortical hierarchy.
Collapse
Affiliation(s)
- Idan A Blank
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| |
Collapse
|
33
|
Honbolygó F, Kóbor A, Hermann P, Kettinger ÁO, Vidnyánszky Z, Kovács G, Csépe V. Expectations about word stress modulate neural activity in speech-sensitive cortical areas. Neuropsychologia 2020; 143:107467. [PMID: 32305299 DOI: 10.1016/j.neuropsychologia.2020.107467] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2019] [Revised: 03/06/2020] [Accepted: 04/12/2020] [Indexed: 10/24/2022]
Abstract
A recent dual-stream model of language processing proposed that the postero-dorsal stream performs predictive sequential processing of linguistic information via hierarchically organized internal models. However, it remains unexplored whether the prosodic segmentation of linguistic information involves predictive processes. Here, we addressed this question by investigating the processing of word stress, a major component of speech segmentation, using probabilistic repetition suppression (RS) modulation as a marker of predictive processing. In an event-related acoustic fMRI RS paradigm, we presented pairs of pseudowords having the same (Rep) or different (Alt) stress patterns, in blocks with varying Rep and Alt trial probabilities. We found that the BOLD signal was significantly lower for Rep than for Alt trials, indicating RS in the posterior and middle superior temporal gyrus (STG) bilaterally, and in the anterior STG in the left hemisphere. Importantly, the magnitude of RS was modulated by repetition probability in the posterior and middle STG. These results reveal the predictive processing of word stress in the STG areas and raise the possibility that words stress processing is related to the dorsal "where" auditory stream.
Collapse
Affiliation(s)
- Ferenc Honbolygó
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Institute of Psychology, Eötvös Loránd University, Budapest, Hungary.
| | - Andrea Kóbor
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Petra Hermann
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Ádám Ottó Kettinger
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Department of Nuclear Techniques, Budapest University of Technology and Economics, Budapest, Hungary
| | - Zoltán Vidnyánszky
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Gyula Kovács
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Department of Biological Psychology and Cognitive Neuroscience, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| | - Valéria Csépe
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Faculty of Modern Philology and Social Sciences, University of Pannonia, Veszprém, Hungary
| |
Collapse
|
34
|
Fedorenko E, Blank IA. Broca's Area Is Not a Natural Kind. Trends Cogn Sci 2020; 24:270-284. [PMID: 32160565 PMCID: PMC7211504 DOI: 10.1016/j.tics.2020.01.001] [Citation(s) in RCA: 129] [Impact Index Per Article: 32.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2019] [Revised: 12/21/2019] [Accepted: 01/09/2020] [Indexed: 01/09/2023]
Abstract
Theories of human cognition prominently feature 'Broca's area', which causally contributes to a myriad of mental functions. However, Broca's area is not a monolithic, multipurpose unit - it is structurally and functionally heterogeneous. Some functions engaging (subsets of) this area share neurocognitive resources, whereas others rely on separable circuits. A decade of converging evidence has now illuminated a fundamental distinction between two subregions of Broca's area that likely play computationally distinct roles in cognition: one belongs to the domain-specific 'language network', the other to the domain-general 'multiple-demand (MD) network'. Claims about Broca's area should be (re)cast in terms of these (and other, as yet undetermined) functional components, to establish a cumulative research enterprise where empirical findings can be replicated and theoretical proposals can be meaningfully compared and falsified.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Brain and Cognitive Sciences Department, and McGovern Institute for Brain Research, Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA.
| | - Idan A Blank
- Department of Psychology, University of California at Los Angeles (UCLA), Los Angeles, CA 90095, USA.
| |
Collapse
|
35
|
Shain C, Blank IA, van Schijndel M, Schuler W, Fedorenko E. fMRI reveals language-specific predictive coding during naturalistic sentence comprehension. Neuropsychologia 2020; 138:107307. [PMID: 31874149 PMCID: PMC7140726 DOI: 10.1016/j.neuropsychologia.2019.107307] [Citation(s) in RCA: 87] [Impact Index Per Article: 21.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2019] [Revised: 12/02/2019] [Accepted: 12/13/2019] [Indexed: 11/19/2022]
Abstract
Much research in cognitive neuroscience supports prediction as a canonical computation of cognition across domains. Is such predictive coding implemented by feedback from higher-order domain-general circuits, or is it locally implemented in domain-specific circuits? What information sources are used to generate these predictions? This study addresses these two questions in the context of language processing. We present fMRI evidence from a naturalistic comprehension paradigm (1) that predictive coding in the brain's response to language is domain-specific, and (2) that these predictions are sensitive both to local word co-occurrence patterns and to hierarchical structure. Using a recently developed continuous-time deconvolutional regression technique that supports data-driven hemodynamic response function discovery from continuous BOLD signal fluctuations in response to naturalistic stimuli, we found effects of prediction measures in the language network but not in the domain-general multiple-demand network, which supports executive control processes and has been previously implicated in language comprehension. Moreover, within the language network, surface-level and structural prediction effects were separable. The predictability effects in the language network were substantial, with the model capturing over 37% of explainable variance on held-out data. These findings indicate that human sentence processing mechanisms generate predictions about upcoming words using cognitive processes that are sensitive to hierarchical structure and specialized for language processing, rather than via feedback from high-level executive control mechanisms.
Collapse
Affiliation(s)
| | - Idan Asher Blank
- University of California Los Angeles, 90024, USA; Massachusetts Institute of Technology, 02139, USA.
| | | | - William Schuler
- The Ohio State University, 43210, USA; Massachusetts General Hospital, Program in Speech and Hearing Bioscience and Technology, 02115, USA.
| | - Evelina Fedorenko
- Massachusetts General Hospital, Program in Speech and Hearing Bioscience and Technology, 02115, USA.
| |
Collapse
|
36
|
Mollica F, Siegelman M, Diachek E, Piantadosi ST, Mineroff Z, Futrell R, Kean H, Qian P, Fedorenko E. Composition is the Core Driver of the Language-selective Network. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2020; 1:104-134. [PMID: 36794007 PMCID: PMC9923699 DOI: 10.1162/nol_a_00005] [Citation(s) in RCA: 39] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/09/2019] [Accepted: 12/19/2019] [Indexed: 05/11/2023]
Abstract
The frontotemporal language network responds robustly and selectively to sentences. But the features of linguistic input that drive this response and the computations that these language areas support remain debated. Two key features of sentences are typically confounded in natural linguistic input: words in sentences (a) are semantically and syntactically combinable into phrase- and clause-level meanings, and (b) occur in an order licensed by the language's grammar. Inspired by recent psycholinguistic work establishing that language processing is robust to word order violations, we hypothesized that the core linguistic computation is composition, and, thus, can take place even when the word order violates the grammatical constraints of the language. This hypothesis predicts that a linguistic string should elicit a sentence-level response in the language network provided that the words in that string can enter into dependency relationships as in typical sentences. We tested this prediction across two fMRI experiments (total N = 47) by introducing a varying number of local word swaps into naturalistic sentences, leading to progressively less syntactically well-formed strings. Critically, local dependency relationships were preserved because combinable words remained close to each other. As predicted, word order degradation did not decrease the magnitude of the blood oxygen level-dependent response in the language network, except when combinable words were so far apart that composition among nearby words was highly unlikely. This finding demonstrates that composition is robust to word order violations, and that the language regions respond as strongly as they do to naturalistic linguistic input, providing that composition can take place.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Hope Kean
- Brain & Cognitive Sciences Department, MIT
| | - Peng Qian
- Brain & Cognitive Sciences Department, MIT
| | | |
Collapse
|
37
|
Jouravlev O, Zheng D, Balewski Z, Le Arnz Pongos A, Levan Z, Goldin-Meadow S, Fedorenko E. Speech-accompanying gestures are not processed by the language-processing mechanisms. Neuropsychologia 2019; 132:107132. [PMID: 31276684 PMCID: PMC6708375 DOI: 10.1016/j.neuropsychologia.2019.107132] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2018] [Revised: 06/01/2019] [Accepted: 06/30/2019] [Indexed: 12/15/2022]
Abstract
Speech-accompanying gestures constitute one information channel during communication. Some have argued that processing gestures engages the brain regions that support language comprehension. However, studies that have been used as evidence for shared mechanisms suffer from one or more of the following limitations: they (a) have not directly compared activations for gesture and language processing in the same study and relied on the fallacious reverse inference (Poldrack, 2006) for interpretation, (b) relied on traditional group analyses, which are bound to overestimate overlap (e.g., Nieto-Castañon and Fedorenko, 2012), (c) failed to directly compare the magnitudes of response (e.g., Chen et al., 2017), and (d) focused on gestures that may have activated the corresponding linguistic representations (e.g., "emblems"). To circumvent these limitations, we used fMRI to examine responses to gesture processing in language regions defined functionally in individual participants (e.g., Fedorenko et al., 2010), including directly comparing effect sizes, and covering a broad range of spontaneously generated co-speech gestures. Whenever speech was present, language regions responded robustly (and to a similar degree regardless of whether the video contained gestures or grooming movements). In contrast, and critically, responses in the language regions were low - at or slightly above the fixation baseline - when silent videos were processed (again, regardless of whether they contained gestures or grooming movements). Brain regions outside of the language network, including some in close proximity to its regions, differentiated between gestures and grooming movements, ruling out the possibility that the gesture/grooming manipulation was too subtle. Behavioral studies on the critical video materials further showed robust differentiation between the gesture and grooming conditions. In summary, contra prior claims, language-processing regions do not respond to co-speech gestures in the absence of speech, suggesting that these regions are selectively driven by linguistic input (e.g., Fedorenko et al., 2011). Although co-speech gestures are uncontroversially important in communication, they appear to be processed in brain regions distinct from those that support language comprehension, similar to other extra-linguistic communicative signals, like facial expressions and prosody.
Collapse
Affiliation(s)
- Olessia Jouravlev
- Massachusetts Institute of Technology, Cambridge, MA, 02139, USA; Carleton University, Ottawa, ON K1S 5B6, Canada.
| | - David Zheng
- Princeton University, Princeton, NJ, 08544, USA
| | - Zuzanna Balewski
- Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | | | - Zena Levan
- University of Chicago, Chicago, IL, 60637, USA
| | | | - Evelina Fedorenko
- Massachusetts Institute of Technology, Cambridge, MA, 02139, USA; McGovern Institute for Brain Research, Cambridge, MA, 02139, USA; Massachusetts General Hospital, Boston, MA, 02114, USA.
| |
Collapse
|
38
|
Siegelman M, Blank IA, Mineroff Z, Fedorenko E. An Attempt to Conceptually Replicate the Dissociation between Syntax and Semantics during Sentence Comprehension. Neuroscience 2019; 413:219-229. [PMID: 31200104 PMCID: PMC6661197 DOI: 10.1016/j.neuroscience.2019.06.003] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2018] [Revised: 05/31/2019] [Accepted: 06/03/2019] [Indexed: 11/24/2022]
Abstract
Is sentence structure processed by the same neural and cognitive resources that are recruited for processing word meanings, or do structure and meaning rely on distinct resources? Linguistic theorizing and much behavioral evidence suggest tight integration between lexico-semantic and syntactic representations and processing. However, most current proposals of the neural architecture of language continue to postulate a distinction between the two. One of the earlier and most cited pieces of neuroimaging evidence in favor of this dissociation comes from a paper by Dapretto and Bookheimer (1999). Using a sentence-meaning judgment task, Dapretto & Bookheimer observed two distinct peaks within the left inferior frontal gyrus (LIFG): one was more active during a lexico-semantic manipulation, and the other during a syntactic manipulation. Although the paper is highly cited, no attempt has been made, to our knowledge, to replicate the original finding. We report an fMRI study that attempts to do so. Using a combination of whole-brain, group-level ROI, and participant-specific functional ROI approaches, we fail to replicate the original dissociation. In particular, whereas parts of LIFG respond reliably more strongly during lexico-semantic than syntactic processing, no part of LIFG (including in the region defined around the peak reported by Dapretto & Bookheimer) shows the opposite pattern. We speculate that the original result was a false positive, possibly driven by a small subset of participants or items that biased a fixed-effects analysis with low power.
Collapse
Affiliation(s)
- Matthew Siegelman
- MIT, Department of Brain and Cognitive Sciences; Columbia University, Department of Psychology
| | - Idan A Blank
- MIT, Department of Brain and Cognitive Sciences; UCLA, Department of Psychology
| | | | - Evelina Fedorenko
- MIT, Department of Brain and Cognitive Sciences; MIT, McGovern Institute for Brain Research; MGH, Department of Psychiatry.
| |
Collapse
|
39
|
Paunov AM, Blank IA, Fedorenko E. Functionally distinct language and Theory of Mind networks are synchronized at rest and during language comprehension. J Neurophysiol 2019; 121:1244-1265. [PMID: 30601693 PMCID: PMC6485726 DOI: 10.1152/jn.00619.2018] [Citation(s) in RCA: 41] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2018] [Revised: 12/26/2018] [Accepted: 12/30/2018] [Indexed: 12/30/2022] Open
Abstract
Communication requires the abilities to generate and interpret utterances and to infer the beliefs, desires, and goals of others ("Theory of Mind"; ToM). These two abilities have been shown to dissociate: individuals with aphasia retain the ability to think about others' mental states; and individuals with autism are impaired in social reasoning, but their basic language processing is often intact. In line with this evidence from brain disorders, functional MRI (fMRI) studies have shown that linguistic and ToM abilities recruit distinct sets of brain regions. And yet, language is a social tool that allows us to share thoughts with one another. Thus, the language and ToM brain networks must share information despite being implemented in distinct neural circuits. Here, we investigated potential interactions between these networks during naturalistic cognition using functional correlations in fMRI. The networks were functionally defined in individual participants, in terms of preference for sentences over nonwords for language, and for belief inference over physical-event processing for ToM, with both a verbal and a nonverbal paradigm. Although, across experiments, interregion correlations within each network were higher than between-network correlations, we also observed above-baseline synchronization of blood oxygenation level-dependent signal fluctuations between the two networks during rest and story comprehension. This synchronization was functionally specific: neither network was synchronized with the executive control network (functionally defined in terms of preference for a harder over easier version of an executive task). Thus, coordination between the language and ToM networks appears to be an inherent and specific characteristic of their functional architecture. NEW & NOTEWORTHY Humans differ from nonhuman primates in their abilities to communicate linguistically and to infer others' mental states. Although linguistic and social abilities appear to be interlinked onto- and phylogenetically, they are dissociated in the adult human brain. Yet successful communication requires language and social reasoning to work in concert. Using functional MRI, we show that language regions are synchronized with social regions during rest and language comprehension, pointing to a possible mechanism for internetwork interaction.
Collapse
Affiliation(s)
- Alexander M Paunov
- Massachusetts Institute of Technology, Brain & Cognitive Sciences Department , Cambridge, Massachusetts
| | - Idan A Blank
- Massachusetts Institute of Technology, Brain & Cognitive Sciences Department , Cambridge, Massachusetts
| | - Evelina Fedorenko
- Massachusetts Institute of Technology, Brain & Cognitive Sciences Department , Cambridge, Massachusetts
- Harvard Medical School, Psychiatry Department , Boston, Massachusetts
- Massachusetts General Hospital, Psychiatry Department , Boston, Massachusetts
| |
Collapse
|
40
|
Fedorenko E, Williams ZM, Ferreira VS. Remaining Puzzles about Morpheme Production in the Posterior Temporal Lobe. Neuroscience 2018; 392:160-163. [PMID: 30278250 DOI: 10.1016/j.neuroscience.2018.09.032] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2018] [Revised: 09/18/2018] [Accepted: 09/22/2018] [Indexed: 11/19/2022]
Abstract
Using data from time-resolved cortical stimulation, intracranial neural recordings, and focal surgical resections, Lee et al. (2018) demonstrate that a small area within left posterior superior temporal gyrus (pSTG) supports the ability to produce functional morphemes but not other basic aspects of language production or comprehension. These findings are intriguing because they raise important questions about the functional architecture of language processing, including critically, the relationship between production and comprehension. Here, we highlight some of the puzzles that remain and that we hope will guide future empirical explorations of the cognitive and neural mechanisms that support our capacity for language.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Department of Psychiatry, Massachusetts General Hospital/Harvard Medical School, Boston, MA, United States; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, United States.
| | - Ziv M Williams
- Department of Neurosurgery, Massachusetts General Hospital/Harvard Medical School, Boston, MA, United States; Program in Neuroscience, Harvard Medical School, Boston, MA, United States
| | | |
Collapse
|
41
|
Pritchett BL, Hoeflin C, Koldewyn K, Dechter E, Fedorenko E. High-level language processing regions are not engaged in action observation or imitation. J Neurophysiol 2018; 120:2555-2570. [PMID: 30156457 DOI: 10.1152/jn.00222.2018] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022] Open
Abstract
A set of left frontal, temporal, and parietal brain regions respond robustly during language comprehension and production (e.g., Fedorenko E, Hsieh PJ, Nieto-Castañón A, Whitfield-Gabrieli S, Kanwisher N. J Neurophysiol 104: 1177-1194, 2010; Menenti L, Gierhan SM, Segaert K, Hagoort P. Psychol Sci 22: 1173-1182, 2011). These regions have been further shown to be selective for language relative to other cognitive processes, including arithmetic, aspects of executive function, and music perception (e.g., Fedorenko E, Behr MK, Kanwisher N. Proc Natl Acad Sci USA 108: 16428-16433, 2011; Monti MM, Osherson DN. Brain Res 1428: 33-42, 2012). However, one claim about overlap between language and nonlinguistic cognition remains prominent. In particular, some have argued that language processing shares computational demands with action observation and/or execution (e.g., Rizzolatti G, Arbib MA. Trends Neurosci 21: 188-194, 1998; Koechlin E, Jubault T. Neuron 50: 963-974, 2006; Tettamanti M, Weniger D. Cortex 42: 491-494, 2006). However, the evidence for these claims is indirect, based on observing activation for language and action tasks within the same broad anatomical areas (e.g., on the lateral surface of the left frontal lobe). To test whether language indeed shares machinery with action observation/execution, we examined the responses of language brain regions, defined functionally in each individual participant (Fedorenko E, Hsieh PJ, Nieto-Castañón A, Whitfield-Gabrieli S, Kanwisher N. J Neurophysiol 104: 1177-1194, 2010) to action observation ( experiments 1, 2, and 3a) and action imitation ( experiment 3b). With the exception of the language region in the angular gyrus, all language regions, including those in the inferior frontal gyrus (within "Broca's area"), showed little or no response during action observation/imitation. These results add to the growing body of literature suggesting that high-level language regions are highly selective for language processing (see Fedorenko E, Varley R. Ann NY Acad Sci 1369: 132-153, 2016 for a review). NEW & NOTEWORTHY Many have argued for overlap in the machinery used to interpret language and others' actions, either because action observation was a precursor to linguistic communication or because both require interpreting hierarchically-structured stimuli. However, existing evidence is indirect, relying on group analyses or reverse inference. We examined responses to action observation in language regions defined functionally in individual participants and found no response. Thus language comprehension and action observation recruit distinct circuits in the modern brain.
Collapse
Affiliation(s)
- Brianna L Pritchett
- Department of Brain and Cognitive Sciences/McGovern Institute for Brain Research, Massachusetts Institute of Technology , Cambridge, Massachusetts
| | - Caitlyn Hoeflin
- Department of Brain and Cognitive Sciences/McGovern Institute for Brain Research, Massachusetts Institute of Technology , Cambridge, Massachusetts
| | - Kami Koldewyn
- School of Psychology, Bangor University, Gwynedd, United Kingdom
| | - Eyal Dechter
- Department of Brain and Cognitive Sciences/McGovern Institute for Brain Research, Massachusetts Institute of Technology , Cambridge, Massachusetts
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences/McGovern Institute for Brain Research, Massachusetts Institute of Technology , Cambridge, Massachusetts.,Department of Psychiatry, Massachusetts General Hospital, Charlestown, Massachusetts.,Department of Psychiatry, Harvard Medical School , Boston, Massachusetts
| |
Collapse
|
42
|
Wilson SM, Yen M, Eriksson DK. An adaptive semantic matching paradigm for reliable and valid language mapping in individuals with aphasia. Hum Brain Mapp 2018; 39:3285-3307. [PMID: 29665223 PMCID: PMC6045968 DOI: 10.1002/hbm.24077] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2018] [Revised: 03/26/2018] [Accepted: 03/30/2018] [Indexed: 11/08/2022] Open
Abstract
Research on neuroplasticity in recovery from aphasia depends on the ability to identify language areas of the brain in individuals with aphasia. However, tasks commonly used to engage language processing in people with aphasia, such as narrative comprehension and picture naming, are limited in terms of reliability (test-retest reproducibility) and validity (identification of language regions, and not other regions). On the other hand, paradigms such as semantic decision that are effective in identifying language regions in people without aphasia can be prohibitively challenging for people with aphasia. This paper describes a new semantic matching paradigm that uses an adaptive staircase procedure to present individuals with stimuli that are challenging yet within their competence, so that language processing can be fully engaged in people with and without language impairments. The feasibility, reliability and validity of the adaptive semantic matching paradigm were investigated in sixteen individuals with chronic post-stroke aphasia and fourteen neurologically normal participants, in comparison to narrative comprehension and picture naming paradigms. All participants succeeded in learning and performing the semantic paradigm. Test-retest reproducibility of the semantic paradigm in people with aphasia was good (Dice coefficient = 0.66), and was superior to the other two paradigms. The semantic paradigm revealed known features of typical language organization (lateralization; frontal and temporal regions) more consistently in neurologically normal individuals than the other two paradigms, constituting evidence for validity. In sum, the adaptive semantic matching paradigm is a feasible, reliable and valid method for mapping language regions in people with aphasia.
Collapse
Affiliation(s)
- Stephen M. Wilson
- Department of Hearing and Speech SciencesVanderbilt University Medical CenterNashvilleTennessee
| | - Melodie Yen
- Department of Hearing and Speech SciencesVanderbilt University Medical CenterNashvilleTennessee
| | - Dana K. Eriksson
- Department of SpeechLanguage, and Hearing Sciences, University of ArizonaTucsonArizona
| |
Collapse
|
43
|
Lee DK, Fedorenko E, Simon MV, Curry WT, Nahed BV, Cahill DP, Williams ZM. Neural encoding and production of functional morphemes in the posterior temporal lobe. Nat Commun 2018; 9:1877. [PMID: 29760465 PMCID: PMC5951905 DOI: 10.1038/s41467-018-04235-3] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2017] [Accepted: 04/13/2018] [Indexed: 11/17/2022] Open
Abstract
Morphemes are the smallest meaning-carrying units in human language, and are among the most basic building blocks through which humans express specific ideas and concepts. By using time-resolved cortical stimulations, neural recordings, and focal lesion evaluations, we show that inhibition of a small cortical area within the left dominant posterior–superior temporal lobe selectively impairs the ability to produce appropriate functional morphemes but does not distinctly affect semantic and lexical retrieval, comprehension, or articulation. Additionally, neural recordings within this area reveal the localized encoding of morphological properties and their planned production prior to speech onset. Finally, small lesions localized to the gray matter in this area result in a selective functional morpheme-production deficit. Collectively, these findings reveal a detailed division of linguistic labor within the posterior–superior temporal lobe and suggest that functional morpheme processing constitutes an operationally discrete step in the series of computations essential to language production. Functional morphemes allow us to express details about objects, events, and their relationships. Here, authors show that inhibiting a small cortical area within left posterior superior temporal lobe selectively impairs the ability to produce functional morphemes but does not impair other linguistic abilities.
Collapse
Affiliation(s)
- Daniel K Lee
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Evelina Fedorenko
- Department of Psychiatry, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA.,Harvard Program in Speech and Hearing Bioscience, Cambridge, 02138, MA, USA.,Massachusetts Institute of Technology, McGovern Institute for Brain Research, Cambridge, 02139, MA, USA
| | - Mirela V Simon
- Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - William T Curry
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Brian V Nahed
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Dan P Cahill
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Ziv M Williams
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA. .,Harvard-MIT Division of Health Sciences and Technology, Boston, 02115, MA, USA. .,Program in Neuroscience, Harvard Medical School, Boston, 02115, MA, USA.
| |
Collapse
|
44
|
Domain-General Brain Regions Do Not Track Linguistic Input as Closely as Language-Selective Regions. J Neurosci 2017; 37:9999-10011. [PMID: 28871034 DOI: 10.1523/jneurosci.3642-16.2017] [Citation(s) in RCA: 50] [Impact Index Per Article: 7.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2016] [Revised: 08/16/2017] [Accepted: 08/18/2017] [Indexed: 01/05/2023] Open
Abstract
Language comprehension engages a cortical network of left frontal and temporal regions. Activity in this network is language-selective, showing virtually no modulation by nonlinguistic tasks. In addition, language comprehension engages a second network consisting of bilateral frontal, parietal, cingulate, and insular regions. Activity in this "multiple demand" (MD) network scales with comprehension difficulty, but also with cognitive effort across a wide range of nonlinguistic tasks in a domain-general fashion. Given the functional dissociation between the language and MD networks, their respective contributions to comprehension are likely distinct, yet such differences remain elusive. Prior neuroimaging studies have suggested that activity in each network covaries with some linguistic features that, behaviorally, influence on-line processing and comprehension. This sensitivity of the language and MD networks to local input characteristics has often been interpreted, implicitly or explicitly, as evidence that both networks track linguistic input closely, and in a manner consistent across individuals. Here, we used fMRI to directly test this assumption by comparing the BOLD signal time courses in each network across different people (n = 45, men and women) listening to the same story. Language network activity showed fewer individual differences, indicative of closer input tracking, whereas MD network activity was more idiosyncratic and, moreover, showed lower reliability within an individual across repetitions of a story. These findings constrain cognitive models of language comprehension by suggesting a novel distinction between the processes implemented in the language and MD networks.SIGNIFICANCE STATEMENT Language comprehension recruits both language-specific mechanisms and domain-general mechanisms that are engaged in many cognitive processes. In the human cortex, language-selective mechanisms are implemented in the left-lateralized "core language network", whereas domain-general mechanisms are implemented in the bilateral "multiple demand" (MD) network. Here, we report the first direct comparison of the respective contributions of these networks to naturalistic story comprehension. Using a novel combination of neuroimaging approaches we find that MD regions track stories less closely than language regions. This finding constrains the possible contributions of the MD network to comprehension, contrasts with accounts positing that this network has continuous access to linguistic input, and suggests a new typology of comprehension processes based on their extent of input tracking.
Collapse
|
45
|
Abstract
The neural processes that underlie your ability to read and understand this sentence are unknown. Sentence comprehension occurs very rapidly, and can only be understood at a mechanistic level by discovering the precise sequence of underlying computational and neural events. However, we have no continuous and online neural measure of sentence processing with high spatial and temporal resolution. Here we report just such a measure: intracranial recordings from the surface of the human brain show that neural activity, indexed by γ-power, increases monotonically over the course of a sentence as people read it. This steady increase in activity is absent when people read and remember nonword-lists, despite the higher cognitive demand entailed, ruling out accounts in terms of generic attention, working memory, and cognitive load. Response increases are lower for sentence structure without meaning ("Jabberwocky" sentences) and word meaning without sentence structure (word-lists), showing that this effect is not explained by responses to syntax or word meaning alone. Instead, the full effect is found only for sentences, implicating compositional processes of sentence understanding, a striking and unique feature of human language not shared with animal communication systems. This work opens up new avenues for investigating the sequence of neural events that underlie the construction of linguistic meaning.
Collapse
|
46
|
Fedorenko E, Varley R. Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Ann N Y Acad Sci 2016; 1369:132-53. [PMID: 27096882 PMCID: PMC4874898 DOI: 10.1111/nyas.13046] [Citation(s) in RCA: 80] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2015] [Revised: 02/18/2016] [Accepted: 02/25/2016] [Indexed: 01/29/2023]
Abstract
Is thought possible without language? Individuals with global aphasia, who have almost no ability to understand or produce language, provide a powerful opportunity to find out. Surprisingly, despite their near-total loss of language, these individuals are nonetheless able to add and subtract, solve logic problems, think about another person's thoughts, appreciate music, and successfully navigate their environments. Further, neuroimaging studies show that healthy adults strongly engage the brain's language areas when they understand a sentence, but not when they perform other nonlinguistic tasks such as arithmetic, storing information in working memory, inhibiting prepotent responses, or listening to music. Together, these two complementary lines of evidence provide a clear answer: many aspects of thought engage distinct brain regions from, and do not depend on, language.
Collapse
Affiliation(s)
- Evelina Fedorenko
- Psychiatry Department, Massachusetts General Hospital, Charlestown, Massachusetts
- Harvard Medical School, Boston, Massachusetts
- Center for Academic Research and Training in Anthropogeny (CARTA), University of California, San Diego, La Jolla, California
| | | |
Collapse
|