1
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. Nat Hum Behav 2024; 8:544-561. [PMID: 38172630 DOI: 10.1038/s41562-023-01783-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 11/10/2023] [Indexed: 01/05/2024]
Abstract
Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of the brain response associated with each sentence. We then use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress the activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also non-invasively control neural activity in higher-level cortical areas, such as the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- MIT-IBM Watson AI Lab, Cambridge, MA, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA, USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
2
|
Fairhall SL. Sentence-level embeddings reveal dissociable word- and sentence-level cortical representation across coarse- and fine-grained levels of meaning. BRAIN AND LANGUAGE 2024; 250:105389. [PMID: 38306958 DOI: 10.1016/j.bandl.2024.105389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Revised: 01/09/2024] [Accepted: 01/26/2024] [Indexed: 02/04/2024]
Abstract
In this large-sample (N = 64) fMRI study, sentence embeddings (text-embedding-ada-002, OpenAI) and representational similarity analysis were used to contrast sentence-level and word-level semantic representation. Overall, sentence-level information resulted in a 20-25 % increase in the model's ability to captures neural representation when compared to word-level only information (word-order scrambled embeddings). This increase was relatively undifferentiated across the cortex. However, when coarse-grained (across thematic category) and fine-grained (within thematic category) combinatorial meaning were separately assessed, word- and sentence-level representations were seen to strongly dissociate across the cortex and to do so differently as a function of grain. Coarse-grained sentence-level representations were evident in occipitotemporal, ventral temporal and medial prefrontal cortex, while fine-grained differences were seen in lateral prefrontal and parietal cortex, middle temporal gyrus, the precuneus, and medial prefrontal cortex. This result indicates dissociable cortical substrates underly single concept versus combinatorial meaning and that different cortical regions specialise for fine- and coarse-grained meaning.
Collapse
Affiliation(s)
- Scott L Fairhall
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Italy.
| |
Collapse
|
3
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.16.537080. [PMID: 37090673 PMCID: PMC10120732 DOI: 10.1101/2023.04.16.537080] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/25/2023]
Abstract
Transformer models such as GPT generate human-like language and are highly predictive of human brain responses to language. Here, using fMRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of brain response associated with each sentence. Then, we use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also noninvasively control neural activity in higher-level cortical areas, like the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- MIT-IBM Watson AI Lab, Cambridge, MA 02142, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455 USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
- The Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA 02138 USA
| |
Collapse
|
4
|
Paunov AM, Blank IA, Jouravlev O, Mineroff Z, Gallée J, Fedorenko E. Differential Tracking of Linguistic vs. Mental State Content in Naturalistic Stimuli by Language and Theory of Mind (ToM) Brain Networks. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:413-440. [PMID: 37216061 PMCID: PMC10158571 DOI: 10.1162/nol_a_00071] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Accepted: 04/11/2022] [Indexed: 05/24/2023]
Abstract
Language and social cognition, especially the ability to reason about mental states, known as theory of mind (ToM), are deeply related in development and everyday use. However, whether these cognitive faculties rely on distinct, overlapping, or the same mechanisms remains debated. Some evidence suggests that, by adulthood, language and ToM draw on largely distinct-though plausibly interacting-cortical networks. However, the broad topography of these networks is similar, and some have emphasized the importance of social content / communicative intent in the linguistic signal for eliciting responses in the language areas. Here, we combine the power of individual-subject functional localization with the naturalistic-cognition inter-subject correlation approach to illuminate the language-ToM relationship. Using functional magnetic resonance imaging (fMRI), we recorded neural activity as participants (n = 43) listened to stories and dialogues with mental state content (+linguistic, +ToM), viewed silent animations and live action films with mental state content but no language (-linguistic, +ToM), or listened to an expository text (+linguistic, -ToM). The ToM network robustly tracked stimuli rich in mental state information regardless of whether mental states were conveyed linguistically or non-linguistically, while tracking a +linguistic / -ToM stimulus only weakly. In contrast, the language network tracked linguistic stimuli more strongly than (a) non-linguistic stimuli, and than (b) the ToM network, and showed reliable tracking even for the linguistic condition devoid of mental state content. These findings suggest that in spite of their indisputably close links, language and ToM dissociate robustly in their neural substrates-and thus plausibly cognitive mechanisms-including during the processing of rich naturalistic materials.
Collapse
Affiliation(s)
- Alexander M. Paunov
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, Université Paris-Saclay, NeuroSpin Center, 91191Gif/Yvette, France
| | - Idan A. Blank
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- Department of Psychology, UCLA, Los Angeles, CA, USA
| | - Olessia Jouravlev
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Institute for Cognitive Science, Carleton University, Ottawa, ON, Canada
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Eberly Center for Teaching Excellence & Educational Innovation, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Jeanne Gallée
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, MIT, Cambridge, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Boston, MA, USA
| |
Collapse
|
5
|
Acunzo DJ, Low DM, Fairhall SL. Deep neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus. Neuroimage 2022; 251:119005. [PMID: 35176493 PMCID: PMC10184870 DOI: 10.1016/j.neuroimage.2022.119005] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2021] [Revised: 02/11/2022] [Accepted: 02/13/2022] [Indexed: 11/17/2022] Open
Abstract
When reading a sentence, individual words can be combined to create more complex meaning. In this study, we sought to uncover brain regions that reflect the representation of the meaning of sentences at the topic level, as opposed to the meaning of their individual constituent words when considered irrespective of their context. Using fMRI, we recorded the neural activity of participants while reading sentences. We constructed a topic-level sentence representations using the final layer of a convolutional neural network (CNN) trained to classify Wikipedia sentences into broad semantic categories. This model was contrasted with word-level sentence representations constructed using the average of the word embeddings constituting the sentence. Using representational similarity analysis, we found that the medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus more strongly represent sentence topic-level, compared to word-level, meaning, uncovering the important role of these semantic system regions in the representation of topic-level meaning. Results were comparable when sentence meaning was modelled with a multilayer perceptron that was not sensitive to word order within a sentence, suggesting that the learning objective, in the terms of the topic being modelled, is the critical factor in capturing these neural representational spaces.
Collapse
Affiliation(s)
- David J Acunzo
- CIMeC/University of Trento, Corso Bettini 31, Rovereto 38068, Italy
| | - Daniel M Low
- Program in Speech and Hearing Bioscience and Technology, Harvard Medical School, United States; Brain and Cognitive Sciences Department, MIT, United States
| | - Scott L Fairhall
- CIMeC/University of Trento, Corso Bettini 31, Rovereto 38068, Italy.
| |
Collapse
|
6
|
Aguirre-Celis N, Miikkulainen R. How the Brain Dynamically Constructs Sentence-Level Meanings From Word-Level Features. Front Artif Intell 2022; 5:733163. [PMID: 35527795 PMCID: PMC9069966 DOI: 10.3389/frai.2022.733163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Accepted: 02/09/2022] [Indexed: 11/24/2022] Open
Abstract
How are words connected to the thoughts they help to express? Recent brain imaging studies suggest that word representations are embodied in different neural systems through which the words are experienced. Building on this idea, embodied approaches such as the Concept Attribute Representations (CAR) theory represents concepts as a set of semantic features (attributes) mapped to different brain systems. An intriguing challenge to this theory is that people weigh concept attributes differently based on context, i.e., they construct meaning dynamically according to the combination of concepts that occur in the sentence. This research addresses this challenge through the Context-dEpendent meaning REpresentations in the BRAin (CEREBRA) neural network model. Based on changes in the brain images, CEREBRA quantifies the effect of sentence context on word meanings. Computational experiments demonstrated that words in different contexts have different representations, the changes observed in the concept attributes reveal unique conceptual combinations, and that the new representations are more similar to the other words in the sentence than to the original representations. Behavioral analysis further confirmed that the changes produced by CEREBRA are actionable knowledge that can be used to predict human responses. These experiments constitute a comprehensive evaluation of CEREBRA's context-based representations, showing that CARs can be dynamic and change based on context. Thus, CEREBRA is a useful tool for understanding how word meanings are represented in the brain, providing a framework for future interdisciplinary research on the mental lexicon.
Collapse
Affiliation(s)
- Nora Aguirre-Celis
- Department of Computer Science, ITESM, Monterrey, Mexico
- Department of Computer Science, The University of Texas in Austin, Austin, TX, United States
- *Correspondence: Nora Aguirre-Celis
| | - Risto Miikkulainen
- Department of Computer Science, The University of Texas in Austin, Austin, TX, United States
| |
Collapse
|
7
|
Caucheteux C, King JR. Brains and algorithms partially converge in natural language processing. Commun Biol 2022; 5:134. [PMID: 35173264 PMCID: PMC8850612 DOI: 10.1038/s42003-022-03036-1] [Citation(s) in RCA: 64] [Impact Index Per Article: 32.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 12/29/2021] [Indexed: 11/29/2022] Open
Abstract
Deep learning algorithms trained to predict masked words from large amount of text have recently been shown to generate activations similar to those of the human brain. However, what drives this similarity remains currently unknown. Here, we systematically compare a variety of deep language models to identify the computational principles that lead them to generate brain-like representations of sentences. Specifically, we analyze the brain responses to 400 isolated sentences in a large cohort of 102 subjects, each recorded for two hours with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We then test where and when each of these algorithms maps onto the brain responses. Finally, we estimate how the architecture, training, and performance of these models independently account for the generation of brain-like representations. Our analyses reveal two main findings. First, the similarity between the algorithms and the brain primarily depends on their ability to predict words from context. Second, this similarity reveals the rise and maintenance of perceptual, lexical, and compositional representations within each cortical region. Overall, this study shows that modern language algorithms partially converge towards brain-like solutions, and thus delineates a promising path to unravel the foundations of natural language processing.
Collapse
Affiliation(s)
- Charlotte Caucheteux
- Facebook AI Research, Paris, France.
- Université Paris-Saclay, Inria, CEA, Palaiseau, France.
| | - Jean-Rémi King
- Facebook AI Research, Paris, France.
- École normale supérieure, PSL University, CNRS, Paris, France.
| |
Collapse
|
8
|
Anderson AJ, Kiela D, Binder JR, Fernandino L, Humphries CJ, Conant LL, Raizada RDS, Grimm S, Lalor EC. Deep Artificial Neural Networks Reveal a Distributed Cortical Network Encoding Propositional Sentence-Level Meaning. J Neurosci 2021; 41:4100-4119. [PMID: 33753548 PMCID: PMC8176751 DOI: 10.1523/jneurosci.1152-20.2021] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2020] [Revised: 02/03/2021] [Accepted: 02/22/2021] [Indexed: 11/21/2022] Open
Abstract
Understanding how and where in the brain sentence-level meaning is constructed from words presents a major scientific challenge. Recent advances have begun to explain brain activation elicited by sentences using vector models of word meaning derived from patterns of word co-occurrence in text corpora. These studies have helped map out semantic representation across a distributed brain network spanning temporal, parietal, and frontal cortex. However, it remains unclear whether activation patterns within regions reflect unified representations of sentence-level meaning, as opposed to superpositions of context-independent component words. This is because models have typically represented sentences as "bags-of-words" that neglect sentence-level structure. To address this issue, we interrogated fMRI activation elicited as 240 sentences were read by 14 participants (9 female, 5 male), using sentences encoded by a recurrent deep artificial neural-network trained on a sentence inference task (InferSent). Recurrent connections and nonlinear filters enable InferSent to transform sequences of word vectors into unified "propositional" sentence representations suitable for evaluating intersentence entailment relations. Using voxelwise encoding modeling, we demonstrate that InferSent predicts elements of fMRI activation that cannot be predicted by bag-of-words models and sentence models using grammatical rules to assemble word vectors. This effect occurs throughout a distributed network, which suggests that propositional sentence-level meaning is represented within and across multiple cortical regions rather than at any single site. In follow-up analyses, we place results in the context of other deep network approaches (ELMo and BERT) and estimate the degree of unpredicted neural signal using an "experiential" semantic model and cross-participant encoding.SIGNIFICANCE STATEMENT A modern-day scientific challenge is to understand how the human brain transforms word sequences into representations of sentence meaning. A recent approach, emerging from advances in functional neuroimaging, big data, and machine learning, is to computationally model meaning, and use models to predict brain activity. Such models have helped map a cortical semantic information-processing network. However, how unified sentence-level information, as opposed to word-level units, is represented throughout this network remains unclear. This is because models have typically represented sentences as unordered "bags-of-words." Using a deep artificial neural network that recurrently and nonlinearly combines word representations into unified propositional sentence representations, we provide evidence that sentence-level information is encoded throughout a cortical network, rather than in a single region.
Collapse
Affiliation(s)
- Andrew James Anderson
- Department of Neuroscience, University of Rochester, Rochester, New York 14642
- Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York 14642
| | - Douwe Kiela
- Facebook AI Research, New York, New York 10003
| | - Jeffrey R Binder
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Leonardo Fernandino
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Colin J Humphries
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Lisa L Conant
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Rajeev D S Raizada
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York 14627
| | - Scott Grimm
- Department of Linguistics, University of Rochester, Rochester, New York 14627
| | - Edmund C Lalor
- Department of Neuroscience, University of Rochester, Rochester, New York 14642
- Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York 14642
- Department of Biomedical Engineering, University of Rochester, Rochester, New York 14627
| |
Collapse
|
9
|
Reilly J, Finley AM, Kelly A, Zuckerman B, Flurie M. Olfactory language and semantic processing in anosmia: a neuropsychological case control study. Neurocase 2021; 27:86-96. [PMID: 33400623 PMCID: PMC8026498 DOI: 10.1080/13554794.2020.1871491] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
A longstanding debate within philosophy and neuroscience involves the extent to which sensory information is a necessary condition for conceptual knowledge. Much of our understanding of this relationship has been informed by examining the impact of congenital blindness and deafness on language and cognitive development. Relatively little is known about the "lesser" senses of smell and taste. Here we report a neuropsychological case-control study contrasting a young adult male (P01) diagnosed with anosmia (i.e. no olfaction) during early childhood relative to an age- and sex-matched control group. A structural MRI of P01's brain revealed profoundly atrophic/aplastic olfactory bulbs, and standardized smell testing confirmed his prior pediatric diagnosis of anosmia. Participants completed three language experiments examining comprehension, production, and subjective experiential ratings of odor salient words (e.g. sewer) and scenarios (e.g. fish market). P01's ratings of odor salience of single words were lower than all control participants, whereas his ratings on five other perceptual and affective dimensions were similar to controls. P01 produced unusual associations when cued to generate words that smelled similar to odor-neutral target words (e.g. ink → plant). In narrative picture description for odor salient scenes (e.g. bakery), P01 was indistinguishable from controls. These results suggest that odor deprivation does not overtly impair functional language use. However, subtle lexical-semantic effects of anosmia may be revealed using sensitive linguistic measures.
Collapse
Affiliation(s)
- Jamie Reilly
- Temple University, Eleanor M. Saffran Center for Cognitive Neuroscience, Philadelphia PA, USA.,Department of Communication Sciences and Disorders.,Department of Psychology, Temple University, Philadelphia, PA, USA
| | - Ann Marie Finley
- Temple University, Eleanor M. Saffran Center for Cognitive Neuroscience, Philadelphia PA, USA.,Department of Communication Sciences and Disorders
| | - Alexandra Kelly
- Department of Psychology, Drexel University, Philadelphia, PA, USA
| | - Bonnie Zuckerman
- Temple University, Eleanor M. Saffran Center for Cognitive Neuroscience, Philadelphia PA, USA.,Department of Communication Sciences and Disorders
| | - Maurice Flurie
- Temple University, Eleanor M. Saffran Center for Cognitive Neuroscience, Philadelphia PA, USA.,Department of Communication Sciences and Disorders
| |
Collapse
|
10
|
Ivanova AA, Srikant S, Sueoka Y, Kean HH, Dhamala R, O'Reilly UM, Bers MU, Fedorenko E. Comprehension of computer code relies primarily on domain-general executive brain regions. eLife 2020; 9:e58906. [PMID: 33319744 PMCID: PMC7738192 DOI: 10.7554/elife.58906] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2020] [Accepted: 11/06/2020] [Indexed: 12/22/2022] Open
Abstract
Computer programming is a novel cognitive tool that has transformed modern society. What cognitive and neural mechanisms support this skill? Here, we used functional magnetic resonance imaging to investigate two candidate brain systems: the multiple demand (MD) system, typically recruited during math, logic, problem solving, and executive tasks, and the language system, typically recruited during linguistic processing. We examined MD and language system responses to code written in Python, a text-based programming language (Experiment 1) and in ScratchJr, a graphical programming language (Experiment 2); for both, we contrasted responses to code problems with responses to content-matched sentence problems. We found that the MD system exhibited strong bilateral responses to code in both experiments, whereas the language system responded strongly to sentence problems, but weakly or not at all to code problems. Thus, the MD system supports the use of novel cognitive tools even when the input is structurally similar to natural language.
Collapse
Affiliation(s)
- Anna A Ivanova
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Shashank Srikant
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Yotaro Sueoka
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Hope H Kean
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Riva Dhamala
- Eliot-Pearson Department of Child Study and Human Development, Tufts UniversityMedfordUnited States
| | - Una-May O'Reilly
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Marina U Bers
- Eliot-Pearson Department of Child Study and Human Development, Tufts UniversityMedfordUnited States
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- McGovern Institute for Brain Research, Massachusetts Institute of TechnologyCambridgeUnited States
| |
Collapse
|
11
|
Anderson AJ, McDermott K, Rooks B, Heffner KL, Dodell-Feder D, Lin FV. Decoding individual identity from brain activity elicited in imagining common experiences. Nat Commun 2020; 11:5916. [PMID: 33219210 PMCID: PMC7679397 DOI: 10.1038/s41467-020-19630-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Accepted: 10/19/2020] [Indexed: 11/19/2022] Open
Abstract
Everyone experiences common events differently. This leads to personal memories that presumably provide neural signatures of individual identity when events are reimagined. We present initial evidence that these signatures can be read from brain activity. To do this, we progress beyond previous work that has deployed generic group-level computational semantic models to distinguish between neural representations of different events, but not revealed interpersonal differences in event representations. We scanned 26 participants' brain activity using functional Magnetic Resonance Imaging as they vividly imagined themselves personally experiencing 20 common scenarios (e.g., dancing, shopping, wedding). Rather than adopting a one-size-fits-all approach to generically model scenarios, we constructed personal models from participants' verbal descriptions and self-ratings of sensory/motor/cognitive/spatiotemporal and emotional characteristics of the imagined experiences. We demonstrate that participants' neural representations are better predicted by their own models than other peoples'. This showcases how neuroimaging and personalized models can quantify individual-differences in imagined experiences.
Collapse
Affiliation(s)
- Andrew James Anderson
- Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA.
- Del Monte Institute for Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA.
| | - Kelsey McDermott
- Elaine C. Hubbard Center for Nursing Research on Aging, School of Nursing, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Neuroscience, University of Arizona, Tucson, AZ, 85721, USA
| | - Brian Rooks
- Elaine C. Hubbard Center for Nursing Research on Aging, School of Nursing, University of Rochester Medical Center, Rochester, NY, 14642, USA
| | - Kathi L Heffner
- Elaine C. Hubbard Center for Nursing Research on Aging, School of Nursing, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Department of Psychiatry, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Division of Geriatrics and Aging, Department of Medicine, University of Rochester Medical Center, Rochester, NY, 14642, USA
| | - David Dodell-Feder
- Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Del Monte Institute for Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Department of Psychology, University of Rochester, Rochester, NY, 14642, USA
| | - Feng V Lin
- Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Del Monte Institute for Neuroscience, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Elaine C. Hubbard Center for Nursing Research on Aging, School of Nursing, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Department of Neurology, University of Rochester Medical Center, Rochester, NY, 14642, USA
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, 14642, USA
| |
Collapse
|
12
|
Blank IA, Fedorenko E. No evidence for differences among language regions in their temporal receptive windows. Neuroimage 2020; 219:116925. [PMID: 32407994 PMCID: PMC9392830 DOI: 10.1016/j.neuroimage.2020.116925] [Citation(s) in RCA: 36] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2019] [Revised: 03/20/2020] [Accepted: 05/06/2020] [Indexed: 10/24/2022] Open
Abstract
The "core language network" consists of left frontal and temporal regions that are selectively engaged in linguistic processing. Whereas functional differences among these regions have long been debated, many accounts propose distinctions in terms of representational grain-size-e.g., words vs. phrases/sentences-or processing time-scale, i.e., operating on local linguistic features vs. larger spans of input. Indeed, the topography of language regions appears to overlap with a cortical hierarchy reported by Lerner et al. (2011) wherein mid-posterior temporal regions are sensitive to low-level features of speech, surrounding areas-to word-level information, and inferior frontal areas-to sentence-level information and beyond. However, the correspondence between the language network and this hierarchy of "temporal receptive windows" (TRWs) is difficult to establish because the precise anatomical locations of language regions vary across individuals. To directly test this correspondence, we first identified language regions in each participant with a well-validated task-based localizer, which confers high functional resolution to the study of TRWs (traditionally based on stereotactic coordinates); then, we characterized regional TRWs with the naturalistic story listening paradigm of Lerner et al. (2011), which augments task-based characterizations of the language network by more closely resembling comprehension "in the wild". We find no region-by-TRW interactions across temporal and inferior frontal regions, which are all sensitive to both word-level and sentence-level information. Therefore, the language network as a whole constitutes a unique stage of information integration within a broader cortical hierarchy.
Collapse
Affiliation(s)
- Idan A Blank
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| |
Collapse
|
13
|
Nelson MJ, Moeller S, Basu A, Christopher L, Rogalski EJ, Greicius M, Weintraub S, Bonakdarpour B, Hurley RS, Mesulam MM. Taxonomic Interference Associated with Phonemic Paraphasias in Agrammatic Primary Progressive Aphasia. Cereb Cortex 2020; 30:2529-2541. [PMID: 31800048 PMCID: PMC7174997 DOI: 10.1093/cercor/bhz258] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2019] [Accepted: 09/17/2019] [Indexed: 11/14/2022] Open
Abstract
Phonemic paraphasias are thought to reflect phonological (post-semantic) deficits in language production. Here we present evidence that phonemic paraphasias in non-semantic primary progressive aphasia (PPA) may be associated with taxonomic interference. Agrammatic and logopenic PPA patients and control participants performed a word-to-picture visual search task where they matched a stimulus noun to 1 of 16 object pictures as their eye movements were recorded. Participants were subsequently asked to name the same items. We measured taxonomic interference (ratio of time spent viewing related vs. unrelated foils) during the search task for each item. Target items that elicited a phonemic paraphasia during object naming elicited increased taxonomic interference during the search task in agrammatic but not logopenic PPA patients. These results could reflect either very subtle sub-clinical semantic distortions of word representations or partial degradation of specific phonological word forms in agrammatic PPA during both word-to-picture matching (input stage) and picture naming (output stage). The mechanism for phonemic paraphasias in logopenic patients seems to be different and to be operative at the pre-articulatory stage of phonological retrieval. Glucose metabolic imaging suggests that degeneration in the left posterior frontal lobe and left temporo-parietal junction, respectively, might underlie these different patterns of phonemic paraphasia.
Collapse
Affiliation(s)
- M J Nelson
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Neurological Surgery, Feinberg School of Medicine , Northwestern University, Chicago, IL 60611, USA
- Department of Neurosurgery, School of Medicine, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - S Moeller
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - A Basu
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - L Christopher
- Department of Neurology and Neurological Sciences, FIND Lab, Stanford University, Stanford, CA 94304, USA
| | - E J Rogalski
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - M Greicius
- Department of Neurology and Neurological Sciences, FIND Lab, Stanford University, Stanford, CA 94304, USA
| | - S Weintraub
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Neurology, Feinberg School of Medicine , Northwestern University, Chicago, IL 60611, USA
| | - B Bonakdarpour
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - R S Hurley
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Psychology, Cleveland State University, Cleveland, OH 44115, USA
| | - M-M Mesulam
- Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Neurology, Feinberg School of Medicine , Northwestern University, Chicago, IL 60611, USA
| |
Collapse
|
14
|
Anderson AJ, Binder JR, Fernandino L, Humphries CJ, Conant LL, Raizada RDS, Lin F, Lalor EC. An Integrated Neural Decoder of Linguistic and Experiential Meaning. J Neurosci 2019; 39:8969-8987. [PMID: 31570538 PMCID: PMC6832686 DOI: 10.1523/jneurosci.2575-18.2019] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2018] [Revised: 08/26/2019] [Accepted: 08/31/2019] [Indexed: 11/21/2022] Open
Abstract
The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.
Collapse
Affiliation(s)
- Andrew James Anderson
- Department of Neuroscience, University of Rochester, Rochester, New York 14642,
- Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY 14642, and
| | - Jeffrey R Binder
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Leonardo Fernandino
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Colin J Humphries
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Lisa L Conant
- Department of Neurology, Medical College of Wisconsin, Milwaukee, Wisconsin 53226
| | - Rajeev D S Raizada
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York 14627
| | - Feng Lin
- Department of Neuroscience, University of Rochester, Rochester, New York 14642
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York 14627
- School of Nursing, University of Rochester, Rochester, New York 14642
- Department of Psychiatry, University of Rochester, Rochester, New York 14642
- Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY 14642, and
- Department of Neurology, University of Rochester, Rochester, NY 14642
| | - Edmund C Lalor
- Department of Neuroscience, University of Rochester, Rochester, New York 14642
- Department of Biomedical Engineering, University of Rochester, Rochester, New York 14627
- School of Engineering, Trinity Centre for Bioengineering, and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin 2, Ireland
- Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY 14642, and
| |
Collapse
|
15
|
Church JA, Cirino PT, Miciak J, Juranek J, Vaughn S, Fletcher JM. Cognitive, Intervention, and Neuroimaging Perspectives on Executive Function in Children With Reading Disabilities. New Dir Child Adolesc Dev 2019; 2019:25-54. [PMID: 31046202 PMCID: PMC6522302 DOI: 10.1002/cad.20292] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
The role of executive function (EF) in the reading process, and in those with reading difficulties, remains unclear. As members of the Texas Center for Learning Disabilities, we review multiple perspectives regarding EF in reading and then summarize some of our recent studies of struggling and typical readers in grades 3-5. Study 1a found that a bi-factor structure best represented a comprehensive assessment of EF. Study 1b found that cognitive and behavioral measures of EF related independently to math and reading. Study 1c found that EF related to reading, above and beyond other variables, but Study 1d found no evidence that adding an EF training component improved intervention response. Study 1e found that pretest EF abilities did not relate to intervention response. Neuroimaging studies examined EF-related brain activity during both reading and nonlexical EF tasks. In Study 2a, the EF task evoked control activity, but generated no differences between struggling and typical readers. The reading task, however, had group differences in both EF and reading regions. In Study 2b, EF activity during reading at pretest was related to intervention response. Across studies, EF appears involved in the reading process. There is less evidence for general EF predicting or improving intervention outcomes.
Collapse
|
16
|
Anderson AJ, Lin F. How pattern information analyses of semantic brain activity elicited in language comprehension could contribute to the early identification of Alzheimer's Disease. Neuroimage Clin 2019; 22:101788. [PMID: 30991624 PMCID: PMC6451171 DOI: 10.1016/j.nicl.2019.101788] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2018] [Revised: 01/28/2019] [Accepted: 03/22/2019] [Indexed: 12/19/2022]
Abstract
Alzheimer's disease (AD) is associated with a loss of semantic knowledge reflecting brain pathophysiology that begins years before dementia. Identifying early signs of pathophysiology induced dysfunction in the neural systems that access and process words' meaning could therefore help forecast dementia. This article reviews pioneering studies demonstrating that abnormal functional Magnetic Resonance Imaging (fMRI) response patterns elicited in semantic tasks reflect both AD-pathophysiology and the hereditary risk of AD, and also can help forecast cognitive decline. However, to bring current semantic task-based fMRI research up to date with new AD research guidelines the relationship with different types of AD-pathophysiology needs to be more thoroughly examined. We shall argue that new analytic techniques and experimental paradigms will be critical for this. Previous work has relied on specialized tests of specific components of semantic knowledge/processing (e.g. famous name recognition) to reveal coarse AD-related changes in activation across broad brain regions. Recent computational advances now enable more detailed tests of the semantic information that is represented within brain regions during more natural language comprehension. These new methods stand to more directly index how pathophysiology alters neural information processing, whilst using language comprehension as the basis for a more comprehensive examination of semantic brain function. We here connect the semantic pattern information analysis literature up with AD research to raise awareness to potential cross-disciplinary research opportunities.
Collapse
Affiliation(s)
- Andrew James Anderson
- Department of Neuroscience, University of Rochester Medical Center, United States of America.
| | - Feng Lin
- Department of Neuroscience, University of Rochester Medical Center, United States of America; School of Nursing, University of Rochester Medical Center, United States of America; Department of Psychiatry, University of Rochester Medical Center, United States of America; Department of Neurology, University of Rochester Medical Center, United States of America; Department of Brain and Cognitive Sciences, University of Rochester, United States of America.
| |
Collapse
|
17
|
|