Spatiotemporal dynamics of argument retrieval and reordering: an FMRI and EEG study on sentence processing

Revealing the spatiotemporal brain dynamics of covert speech compared with overt speech: A simultaneous EEG-fMRI study

Covert speech (CS) refers to speaking internally to oneself without producing any sound or movement. CS is involved in multiple cognitive functions and disorders. Reconstructing CS content by brain-computer interface (BCI) is also an emerging technique. However, it is still controversial whether CS is a truncated neural process of overt speech (OS) or involves independent patterns. Here, we performed a word-speaking experiment with simultaneous EEG-fMRI. It involved 32 participants, who generated words both overtly and covertly. By integrating spatial constraints from fMRI into EEG source localization, we precisely estimated the spatiotemporal dynamics of neural activity. During CS, EEG source activity was localized in three regions: the left precentral gyrus, the left supplementary motor area, and the left putamen. Although OS involved more brain regions with stronger activations, CS was characterized by an earlier event-locked activation in the left putamen (peak at 262 ms versus 1170 ms). The left putamen was also identified as the only hub node within the functional connectivity (FC) networks of both OS and CS, while showing weaker FC strength towards speech-related regions in the dominant hemisphere during CS. Path analysis revealed significant multivariate associations, indicating an indirect association between the earlier activation in the left putamen and CS, which was mediated by reduced FC towards speech-related regions. These findings revealed the specific spatiotemporal dynamics of CS, offering insights into CS mechanisms that are potentially relevant for future treatment of self-regulation deficits, speech disorders, and development of BCI speech applications.

Robust Effects of Working Memory Demand during Naturalistic Language Comprehension in Language-Selective Cortex

To understand language, we must infer structured meanings from real-time auditory or visual signals. Researchers have long focused on word-by-word structure building in working memory as a mechanism that might enable this feat. However, some have argued that language processing does not typically involve rich word-by-word structure building, and/or that apparent working memory effects are underlyingly driven by surprisal (how predictable a word is in context). Consistent with this alternative, some recent behavioral studies of naturalistic language processing that control for surprisal have not shown clear working memory effects. In this fMRI study, we investigate a range of theory-driven predictors of word-by-word working memory demand during naturalistic language comprehension in humans of both sexes under rigorous surprisal controls. In addition, we address a related debate about whether the working memory mechanisms involved in language comprehension are language specialized or domain general. To do so, in each participant, we functionally localize (1) the language-selective network and (2) the "multiple-demand" network, which supports working memory across domains. Results show robust surprisal-independent effects of memory demand in the language network and no effect of memory demand in the multiple-demand network. Our findings thus support the view that language comprehension involves computationally demanding word-by-word structure building operations in working memory, in addition to any prediction-related mechanisms. Further, these memory operations appear to be primarily conducted by the same neural resources that store linguistic knowledge, with no evidence of involvement of brain regions known to support working memory across domains.SIGNIFICANCE STATEMENT This study uses fMRI to investigate signatures of working memory (WM) demand during naturalistic story listening, using a broad range of theoretically motivated estimates of WM demand. Results support a strong effect of WM demand in the brain that is distinct from effects of word predictability. Further, these WM demands register primarily in language-selective regions, rather than in "multiple-demand" regions that have previously been associated with WM in nonlinguistic domains. Our findings support a core role for WM in incremental language processing, using WM resources that are specialized for language.

The interaction between language and working memory: a systematic review of fMRI studies in the past two decades

Language processing involves other cognitive domains, including Working Memory (WM). Much detail about the neural correlates of language and WM interaction remains unclear. This review summarizes the evidence for the interaction between WM and language obtained via functional Magnetic Resonance Imaging (fMRI) in the past two decades. The search was limited to PubMed, Google Scholar, Science direct and Neurosynth for working memory, language, fMRI, neuroimaging, cognition, attention, network, connectome keywords. The exclusion criteria consisted of studies including children, older adults, bilingual or multilingual population, clinical cases, music, sign language, speech, motor processing, review papers, meta-analyses, electroencephalography/event-related potential, and positron emission tomography. A total of 20 articles were included and discussed in four categories: language comprehension, language production, syntax, and networks. Studies on neural correlates of WM and language interaction are rare. Language tasks that involve WM activate common neural systems. Activated areas can be associated with cognitive concepts proposed by Baddeley and Hitch (1974), including the phonological loop of WM (mainly Broca and Wernicke's areas), other prefrontal cortex and right hemispheric regions linked to the visuospatial sketchpad. There is a clear, dynamic interaction between language and WM, reflected in the involvement of subcortical structures, particularly the basal ganglia (caudate), and of widespread right hemispheric regions. WM involvement is levered by cognitive demand in response to task complexity. High WM capacity readers draw upon buffer memory systems in midline cortical areas to decrease the WM demands for efficiency. Different dynamic networks are involved in WM and language interaction in response to the task in hand for an ultimate brain function efficiency, modulated by language modality and attention.

Young children's sentence comprehension: Neural correlates of syntax-semantic competition

Sentence comprehension requires the assignment of thematic relations between the verb and its noun arguments in order to determine who is doing what to whom. In some languages, such as English, word order is the primary syntactic cue. In other languages, such as German, case-marking is additionally used to assign thematic roles. During development children have to acquire the thematic relevance of these syntactic cues and weigh them against semantic cues. Here we investigated the processing of syntactic cues and semantic cues in 2- and 3-year-old children by analyzing their behavioral and neurophysiological responses. Case-marked subject-first and object-first sentences (syntactic cue) including animate and inanimate nouns (semantic cue) were presented auditorily. The semantic animacy cue either conflicted with or supported the thematic roles assigned by syntactic case-marking. In contrast to adults, for whom semantics did not interfere with case-marking, children attended to both syntactic and to semantic cues with a stronger reliance on semantic cues in early development. Children's event-related brain potentials indicated sensitivity to syntactic information but increased processing costs when case-marking and animacy assigned conflicting thematic roles. These results demonstrate an early developmental sensitivity and ongoing shift towards the use of syntactic cues during sentence comprehension.

Linguistic Bias Modulates Interpretation of Speech via Neural Delta-Band Oscillations

Language comprehension requires that single words be grouped into syntactic phrases, as words in sentences are too many to memorize individually. In speech, acoustic and syntactic grouping patterns mostly align. However, when ambiguous sentences allow for alternative grouping patterns, comprehenders may form phrases that contradict speech prosody. While delta-band oscillations are known to track prosody, we hypothesized that linguistic grouping bias can modulate the interpretational impact of speech prosody in ambiguous situations, which should surface in delta-band oscillations when grouping patterns chosen by comprehenders differ from those indicated by prosody. In our auditory electroencephalography study, the interpretation of ambiguous sentences depended on whether an identical word was either followed by a prosodic boundary or not, thereby signaling the ending or continuation of the current phrase. Delta-band oscillatory phase at the critical word should reflect whether participants terminate a phrase despite a lack of acoustic boundary cues. Crossing speech prosody with participants' grouping choice, we observed a main effect of grouping choice-independent of prosody. An internal linguistic bias for grouping words into phrases can thus modulate the interpretational impact of speech prosody via delta-band oscillatory phase.

Working Memory in the Processing of Long-Distance Dependencies: Interference and Filler Maintenance

During the temporal delay between the filler and gap sites in long-distance dependencies, the "active filler" strategy can be implemented in two ways: the filler phrase can be actively maintained in working memory ("maintenance account"), or it can be retrieved only when the parser posits a gap ("retrieval account"). The current study tested whether filler content is maintained during the processing of dependencies. Using a self-paced reading paradigm, we compared reading times on a noun phrase (NP) between the filler and gap sites in object relative clauses, to reading times on an NP between the antecedent and ellipsis sites in ellipsis sentences. While in the former type of dependency a filler by hypothesis can be maintained, in the latter there is no indication for the existence of a dependency prior to the ellipsis site, and hence no maintenance. By varying the amount of similarity-based interference between the antecedent and integration sites, we tested the influence of holding an unresolved dependency on reading times. Significantly increased reading times due to interference were found only in the object relative condition, and not in the ellipsis condition, demonstrating filler maintenance costs. The fact that these costs were measured as an effect on similarity-based interference indicates that the maintained representation of the filler must include at least some of the features shared by the interfering NP.

The neural oscillations of speech processing and language comprehension: state of the art and emerging mechanisms

Neural oscillations subserve a broad range of functions in speech processing and language comprehension. On the one hand, speech contains-somewhat-repetitive trains of air pressure bursts that occur at three dominant amplitude modulation frequencies, physically marking the linguistically meaningful progressions of phonemes, syllables and intonational phrase boundaries. To these acoustic events, neural oscillations of isomorphous operating frequencies are thought to synchronise, presumably resulting in an implicit temporal alignment of periods of neural excitability to linguistically meaningful spectral information on the three low-level linguistic description levels. On the other hand, speech is a carrier signal that codes for high-level linguistic meaning, such as syntactic structure and semantic information-which cannot be read from stimulus acoustics, but must be acquired during language acquisition and decoded for language comprehension. Neural oscillations subserve the processing of both syntactic structure and semantic information. Here, I synthesise a mapping from each linguistic processing domain to a unique set of subserving oscillatory mechanisms-the mapping is plausible given the role ascribed to different oscillatory mechanisms in different subfunctions of cortical information processing and faithful to the underlying electrophysiology. In sum, the present article provides an accessible and extensive review of the functional mechanisms that neural oscillations subserve in speech processing and language comprehension.

Temporally and spatially distinct theta oscillations dissociate a language-specific from a domain-general processing mechanism across the age trajectory

The cognitive functionality of neural oscillations is still highly debated, as different functions have been associated with identical frequency ranges. Theta band oscillations, for instance, were proposed to underlie both language comprehension and domain-general cognitive abilities. Here we show that the ageing brain can provide an answer to the open question whether it is one and the same theta oscillation underlying those functions, thereby resolving a long-standing paradox. While better cognitive functioning is predicted by low theta power in the brain at rest, resting state (RS) theta power declines with age, but sentence comprehension deteriorates in old age. We resolve this paradox showing that sentence comprehension declines due to changes in RS theta power within domain-general brain networks known to support successful sentence comprehension, while low RS theta power within the left-hemispheric dorso-frontal language network predicts intact sentence comprehension. The two RS theta networks were also found to functionally decouple relative to their independent internal coupling. Thus, both temporally and spatially distinct RS theta oscillations dissociate a language-specific from a domain-general processing mechanism.

Task-dependent neural and behavioral effects of verb argument structure features

Understanding which verb argument structure (VAS) features (if any) are part of verbs' lexical entries and under which conditions they are accessed provides information on the nature of lexical representations and sentence construction. We investigated neural and behavioral effects of three understudied VAS characteristics (number of subcategorization options, number of thematic options and overall number of valency frames) in lexical decision and sentence well-formedness judgment in healthy adults. VAS effects showed strong dependency on processing conditions. As reflected by behavioral performance and neural recruitment patterns, increased VAS complexity in terms of subcategorization options and thematic options had a detrimental effect on sentence processing, but facilitated lexical access to single words, possibly by providing more lexico-semantic associations and access routes (facilitation through complexity). Effects of the number of valency frames are equivocal. We suggest that VAS effects may be mediated semantically rather than by a dedicated VAS module in verbs' representations.

Frontal-posterior theta oscillations reflect memory retrieval during sentence comprehension

Successful working-memory retrieval requires that items be retained as distinct units. At the neural level, it has been shown that theta-band oscillatory power increases with the number of to-be-distinguished items during working-memory retrieval. Here we hypothesized that during sentence comprehension, verbal-working-memory retrieval demands lead to increased theta power over frontal cortex, supposedly supporting the distinction amongst stored items during verbal-working-memory retrieval. Also, synchronicity may increase between the frontal cortex and the posterior cortex, with the latter supposedly supporting item retention. We operationalized retrieval by using pronouns, which refer to and trigger the retrieval of antecedent nouns from a preceding sentence part. Retrieval demand was systematically varied by changing the pronoun antecedent: Either, it was non-embedded in the preceding main clause, and thus easy-to-retrieve across a single clause boundary, or embedded in the preceding subordinate clause, and thus hard-to-retrieve across a double clause boundary. We combined electroencephalography (EEG), scalp-level time-frequency analysis, source localization, and source-level coherence analysis, observing a frontal-midline and broad left-hemispheric theta-power increase for embedded-antecedent compared to non-embedded-antecedent retrieval. Sources were localized to left-frontal, left-parietal, and bilateral-inferior-temporal cortices. Coherence analyses suggested synchronicity between left-frontal and left-parietal and between left-frontal and right-inferior-temporal cortices. Activity of an array of left-frontal, left-parietal, and bilateral-inferior-temporal cortices may thus assist retrieval during sentence comprehension, potentially indexing the orchestration of item distinction, verbal working memory, and long-term memory. Our results extend prior findings by mapping prior knowledge on the functional role of theta oscillations onto processes genuine to human sentence comprehension.

Distinguishable neural correlates of verbs and nouns: a MEG study on homonyms

The dissociability of nouns and verbs and of their morphosyntactic operations has been firmly established by lesion data. However, the hypothesis that they are processed by distinct neural substrates is inconsistently supported by neuroimaging studies. We tackled this issue in a silent reading experiment during MEG. Participants silently read noun/verb homonyms in minimal syntactic context: article-noun (NPs), pronoun-verb (VPs) (e.g., il ballo/i balli, the dance/the dances; io ballo/tu balli, I dance/you dance). Homonyms allow to rule out prelexical or postlexical nuisance factors-they are orthographically and phonologically identical, but serve different grammatical functions depending on context. Under these experimental conditions, different activity to nouns and verbs can be confidently attributed to representational/processing distinctions. At the sensor level, three components of event-related magnetic fields were observed for the function word and four for the content word, but Global Field Power (GFP) analysis only showed differences between VPs and NPs at several but very short time windows. By contrast, source level analysis based on Minimum Norm Estimates (MNE) yielded significantly greater activity for VPs in left frontal areas and in a left frontoparietal network at late time windows (380-397 and 393-409 ms). These results are fully consistent with lesion data, and show that verbs and nouns are processed differently in the brain. Frontal and parietal activation to verbs might correspond to morphosyntactic processes and to working memory recruitment (or thematic role assignment), respectively. Findings are consistent with the view that nouns and verbs and their morphosyntactic operations involve at least partially distinct neural substrates. However, they do not entirely rule out that nouns and verbs are processed in a shared neural substrate, and that differences result from greater complexity of verbal morphosyntax.

EEG-fMRI integration for the study of human brain function

Electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) have proved to be extremely valuable tools for the non-invasive study of human brain function. Moreover, due to a notable degree of complementarity between the two modalities, the combination of EEG and fMRI data has been actively sought in the last two decades. Although initially focused on epilepsy, EEG-fMRI applications were rapidly extended to the study of healthy brain function, yielding new insights into its underlying mechanisms and pathways. Nevertheless, EEG and fMRI have markedly different spatial and temporal resolutions, and probe neuronal activity through distinct biophysical processes, many aspects of which are still poorly understood. The remarkable conceptual and methodological challenges associated with EEG-fMRI integration have motivated the development of a wide range of analysis approaches over the years, each relying on more or less restrictive assumptions, and aiming to shed further light on the mechanisms of brain function along with those of the EEG-fMRI coupling itself. Here, we present a review of the most relevant EEG-fMRI integration approaches yet proposed for the study of brain function, supported by a general overview of our current understanding of the biophysical mechanisms coupling the signals obtained from the two modalities.