The Neuronal Correlates of Indeterminate Sentence Comprehension: An fMRI Study

Modeling Brain Representations of Words' Concreteness in Context Using GPT-2 and Human Ratings

The meaning of most words in language depends on their context. Understanding how the human brain extracts contextualized meaning, and identifying where in the brain this takes place, remain important scientific challenges. But technological and computational advances in neuroscience and artificial intelligence now provide unprecedented opportunities to study the human brain in action as language is read and understood. Recent contextualized language models seem to be able to capture homonymic meaning variation ("bat", in a baseball vs. a vampire context), as well as more nuanced differences of meaning-for example, polysemous words such as "book", which can be interpreted in distinct but related senses ("explain a book", information, vs. "open a book", object) whose differences are fine-grained. We study these subtle differences in lexical meaning along the concrete/abstract dimension, as they are triggered by verb-noun semantic composition. We analyze functional magnetic resonance imaging (fMRI) activations elicited by Italian verb phrases containing nouns whose interpretation is affected by the verb to different degrees. By using a contextualized language model and human concreteness ratings, we shed light on where in the brain such fine-grained meaning variation takes place and how it is coded. Our results show that phrase concreteness judgments and the contextualized model can predict BOLD activation associated with semantic composition within the language network. Importantly, representations derived from a complex, nonlinear composition process consistently outperform simpler composition approaches. This is compatible with a holistic view of semantic composition in the brain, where semantic representations are modified by the process of composition itself. When looking at individual brain areas, we find that encoding performance is statistically significant, although with differing patterns of results, suggesting differential involvement, in the posterior superior temporal sulcus, inferior frontal gyrus and anterior temporal lobe, and in motor areas previously associated with processing of concreteness/abstractness.

Indeterminate and Enriched Propositions in Context Linger: Evidence From an Eye-Tracking False Memory Paradigm

A sentence such as We finished the paper is indeterminate with regards to what we finished doing with the paper. Indeterminate sentences constitute a test case for two major issues regarding language comprehension: (1) how we compose sentence meaning; and (2) what is retained in memory about what we read in context over time. In an eye-tracking experiment, participants read short stories that were unexpectedly followed by one of three recognition probes: (a) an indeterminate sentence (Lisa began the book), that is identical to the one in the story; (b) an enriched but false probe (Lisa began reading the book); and (c) a contextually unrelated probe (Lisa began writing the book). The probes were presented either at the offset of the original indeterminate sentence in context or following additional neutral discourse. We measured accuracy, probe recognition time, and reading times of the probe sentences. Results showed that, at the immediate time point, participants correctly accepted the identical probes with high accuracy and short recognition times, but that this effect reversed to chance-level accuracy and significantly longer recognition times at the delayed time point. We also found that participants falsely accept the enriched probe at both time points 50% of the time. There were no reading-time differences between identical and enriched probes, suggesting that enrichment might not be an early, mandatory process for indeterminate sentences. Overall, results suggest that while context produces an enriched proposition, an unenriched proposition true to the indeterminate sentence also lingers in memory.

Context Breeds False Memories for Indeterminate Sentences

What are the roles of semantic and pragmatic processes in the interpretation of sentences in context? And how do we attain such interpretations when sentences are deemed indeterminate? Consider a sentence such as "Lisa began the book" which does not overtly express the activity that Lisa began doing with the book. Although it is believed that individuals compute a specified event to enrich the sentential representation - yielding, e.g., "began [reading] the book" - there is no evidence that a default event meaning is attained. Moreover, if indeterminate sentences are enriched, it is not clear where the information required to generate enriched interpretations come from. Experiment 1 showed that, in isolation, there is no default interpretation for indeterminate sentences. The experiment also showed that biasing contexts constrain event interpretations and improve plausibility judgments, suggesting that event representations for indeterminate sentences are generated by context. In Experiment 2, participants heard biasing discourse contexts and later falsely recognized foil sentences containing the biased events ("Lisa began reading the book") at the same proportion and with the same confidence as the original indeterminate sentence ("Lisa began the book"). We suggest that indeterminate sentences trigger event-enriching inferences but only in sufficiently constraining contexts. We also suggest that indeterminate sentences create two memory traces, one for the proposition consistent with the denotational, compositional meaning, and another for the proposition that is enriched pragmatically over time.

Differential contributions of left-hemispheric language regions to basic semantic composition

Semantic composition, the ability to combine single words to form complex meanings, is a core feature of human language. Despite growing interest in the basis of semantic composition, the neural correlates and the interaction of regions within this network remain a matter of debate. We designed a well-controlled two-word fMRI paradigm in which phrases only differed along the semantic dimension while keeping syntactic information alike. Healthy participants listened to meaningful ("fresh apple"), anomalous ("awake apple") and pseudoword phrases ("awake gufel") while performing an implicit and an explicit semantic task. We identified neural signatures for distinct processes during basic semantic composition. When lexical information is kept constant across conditions and the evaluation of phrasal plausibility is examined (meaningful vs. anomalous phrases), a small set of mostly left-hemispheric semantic regions, including the anterior part of the left angular gyrus, is found active. Conversely, when the load of lexical information-independently of phrasal plausibility-is varied (meaningful or anomalous vs. pseudoword phrases), conceptual combination involves a wide-spread left-hemispheric network comprising executive semantic control regions and general conceptual representation regions. Within this network, the functional coupling between the left anterior inferior frontal gyrus, the bilateral pre-supplementary motor area and the posterior angular gyrus specifically increases for meaningful phrases relative to pseudoword phrases. Stronger effects in the explicit task further suggest task-dependent neural recruitment. Overall, we provide a separation between distinct nodes of the semantic network, whose functional contributions depend on the type of compositional process under analysis.

Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation

Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.

How right hemisphere damage after stroke can impair speech comprehension

Acquired language disorders after stroke are strongly associated with left hemisphere damage. When language difficulties are observed in the context of right hemisphere strokes, patients are usually considered to have atypical functional anatomy. By systematically integrating behavioural and lesion data from brain damaged patients with functional MRI data from neurologically normal participants, we investigated when and why right hemisphere strokes cause language disorders. Experiment 1 studied right-handed patients with unilateral strokes that damaged the right (n = 109) or left (n = 369) hemispheres. The most frequently impaired language task was: auditory sentence-to-picture matching after right hemisphere strokes; and spoken picture description after left hemisphere strokes. For those with auditory sentence-to-picture matching impairments after right hemisphere strokes, the majority (n = 9) had normal performance on tests of perceptual (visual or auditory) and linguistic (semantic, phonological or syntactic) processing. Experiment 2 found that these nine patients had significantly more damage to dorsal parts of the superior longitudinal fasciculus and the right inferior frontal sulcus compared to 75 other patients who also had right hemisphere strokes but were not impaired on the auditory sentence-to-picture matching task. Damage to these right hemisphere regions caused long-term speech comprehension difficulties in 67% of patients. Experiments 3 and 4 used functional MRI in two groups of 25 neurologically normal individuals to show that within the regions identified by Experiment 2, the right inferior frontal sulcus was normally activated by (i) auditory sentence-to-picture matching; and (ii) one-back matching when the demands on linguistic and non-linguistic working memory were high. Together, these experiments demonstrate that the right inferior frontal cortex contributes to linguistic and non-linguistic working memory capacity (executive function) that is needed for normal speech comprehension. Our results link previously unrelated literatures on the role of the right inferior frontal cortex in executive processing and the role of executive processing in sentence comprehension; which in turn helps to explain why right inferior frontal activity has previously been reported to increase during recovery of language function after left hemisphere stroke. The clinical relevance of our findings is that the detrimental effect of right hemisphere strokes on language is (i) much greater than expected; (ii) frequently observed after damage to the right inferior frontal sulcus; (iii) task dependent; (iv) different to the type of impairments observed after left hemisphere strokes; and (v) can result in long-lasting deficits that are (vi) not the consequence of atypical language lateralization.