Electrophysiology Reveals the Neural Dynamics of Naturalistic Auditory Language Processing: Event-Related Potentials Reflect Continuous Model Updates

Understanding words in context: A naturalistic EEG study of children's lexical processing

When listening to speech, adults rely on context to anticipate upcoming words. Evidence for this comes from studies demonstrating that the N400, an event-related potential (ERP) that indexes ease of lexical-semantic processing, is influenced by the predictability of a word in context. We know far less about the role of context in children's speech comprehension. The present study explored lexical processing in adults and 5-10-year-old children as they listened to a story. ERPs time-locked to the onset of every word were recorded. Each content word was coded for frequency, semantic association, and predictability. In both children and adults, N400s reflect word predictability, even when controlling for frequency and semantic association. These findings suggest that both adults and children use top-down constraints from context to anticipate upcoming words when listening to stories.

Flexible multi-step hypothesis testing of human ECoG data using cluster-based permutation tests with GLMEs

BACKGROUND

Time series analysis is critical for understanding brain signals and their relationship to behavior and cognition. Cluster-based permutation tests (CBPT) are commonly used to analyze a variety of electrophysiological signals including EEG, MEG, ECoG, and sEEG data without a priori assumptions about specific temporal effects. However, two major limitations of CBPT include the inability to directly analyze experiments with multiple fixed effects and the inability to account for random effects (e.g. variability across subjects). Here, we propose a flexible multi-step hypothesis testing strategy using CBPT with Linear Mixed Effects Models (LMEs) and Generalized Linear Mixed Effects Models (GLMEs) that can be applied to a wide range of experimental designs and data types.

METHODS

We first evaluate the statistical robustness of LMEs and GLMEs using simulated data distributions. Second, we apply a multi-step hypothesis testing strategy to analyze ERPs and broadband power signals extracted from human ECoG recordings collected during a simple image viewing experiment with image category and novelty as fixed effects. Third, we assess the statistical power differences between analyzing signals with CBPT using LMEs compared to CBPT using separate t-tests run on each fixed effect through simulations that emulate broadband power signals. Finally, we apply CBPT using GLMEs to high-gamma burst data to demonstrate the extension of the proposed method to the analysis of nonlinear data.

RESULTS

First, we found that LMEs and GLMEs are robust statistical models. In simple simulations LMEs produced highly congruent results with other appropriately applied linear statistical models, but LMEs outperformed many linear statistical models in the analysis of "suboptimal" data and maintained power better than analyzing individual fixed effects with separate t-tests. GLMEs also performed similarly to other nonlinear statistical models. Second, in real world human ECoG data, LMEs performed at least as well as separate t-tests when applied to predefined time windows or when used in conjunction with CBPT. Additionally, fixed effects time courses extracted with CBPT using LMEs from group-level models of pseudo-populations replicated latency effects found in individual category-selective channels. Third, analysis of simulated broadband power signals demonstrated that CBPT using LMEs was superior to CBPT using separate t-tests in identifying time windows with significant fixed effects especially for small effect sizes. Lastly, the analysis of high-gamma burst data using CBPT with GLMEs produced results consistent with CBPT using LMEs applied to broadband power data.

CONCLUSIONS

We propose a general approach for statistical analysis of electrophysiological data using CBPT in conjunction with LMEs and GLMEs. We demonstrate that this method is robust for experiments with multiple fixed effects and applicable to the analysis of linear and nonlinear data. Our methodology maximizes the statistical power available in a dataset across multiple experimental variables while accounting for hierarchical random effects and controlling FWER across fixed effects. This approach substantially improves power leading to better reproducibility. Additionally, CBPT using LMEs and GLMEs can be used to analyze individual channels or pseudo-population data for the comparison of functional or anatomical groups of data.

Inhibiting presynaptic calcium channel motility in the auditory cortex suppresses synchronized input processing

Introduction

The emergent coherent population activity from thousands of stochastic neurons in the brain is believed to constitute a key neuronal mechanism for salient processing of external stimuli and its link to internal states like attention and perception. In the sensory cortex, functional cell assemblies are formed by recurrent excitation and inhibitory influences. The stochastic dynamics of each cell involved is largely orchestrated by presynaptic CAV2.1 voltage-gated calcium channels (VGCCs). Cav2.1 VGCCs initiate the release of neurotransmitters from the presynaptic compartment and are therefore able to add variability into synaptic transmission which can be partly explained by their mobile organization around docked vesicles.

Methods

To investigate the relevance of Cav2.1 channel surface motility for the input processing in the primary auditory cortex (A1) in vivo, we make use of a new optogenetic system which allows for acute, reversable cross-linking Cav2.1 VGCCs via a photo-cross-linkable cryptochrome mutant, CRY2olig. In order to map neuronal activity across all cortical layers of the A1, we performed laminar current-source density (CSD) recordings with varying auditory stimulus sets in transgenic mice with a citrine tag on the N-terminus of the VGCCs.

Results

Clustering VGCCs suppresses overall sensory-evoked population activity, particularly when stimuli lead to a highly synchronized distribution of synaptic inputs.

Discussion

Our findings reveal the importance of membrane dynamics of presynaptic calcium channels for sensory encoding by dynamically adjusting network activity across a wide range of synaptic input strength.

Language prediction in monolingual and bilingual speakers: an EEG study

Prediction of upcoming words is thought to be crucial for language comprehension. Here, we are asking whether bilingualism entails changes to the electrophysiological substrates of prediction. Prior findings leave it open whether monolingual and bilingual speakers predict upcoming words to the same extent and in the same manner. We address this issue with a naturalistic approach, employing an information-theoretic metric, surprisal, to predict and contrast the N400 brain potential in monolingual and bilingual speakers. We recruited 18 Iranian Azeri-Persian bilingual speakers and 22 Persian monolingual speakers. Subjects listened to a story in Persian while their electroencephalogram (EEG) was recorded. Bayesian item-level analysis was used. While in monolingual speakers N400 was sensitive to information-theoretic properties of both the current and previous words, in bilingual speakers N400 reflected the properties of the previous word only. Our findings show evidence for a processing delay in bilingual speakers which is consistent with prior research.

Diverging neural dynamics for syntactic structure building in naturalistic speaking and listening

The neural correlates of sentence production are typically studied using task paradigms that differ considerably from the experience of speaking outside of an experimental setting. In this fMRI study, we aimed to gain a better understanding of syntactic processing in spontaneous production versus naturalistic comprehension in three regions of interest (BA44, BA45, and left posterior middle temporal gyrus). A group of participants (n = 16) was asked to speak about the events of an episode of a TV series in the scanner. Another group of participants (n = 36) listened to the spoken recall of a participant from the first group. To model syntactic processing, we extracted word-by-word metrics of phrase-structure building with a top-down and a bottom-up parser that make different hypotheses about the timing of structure building. While the top-down parser anticipates syntactic structure, sometimes before it is obvious to the listener, the bottom-up parser builds syntactic structure in an integratory way after all of the evidence has been presented. In comprehension, neural activity was found to be better modeled by the bottom-up parser, while in production, it was better modeled by the top-down parser. We additionally modeled structure building in production with two strategies that were developed here to make different predictions about the incrementality of structure building during speaking. We found evidence for highly incremental and anticipatory structure building in production, which was confirmed by a converging analysis of the pausing patterns in speech. Overall, this study shows the feasibility of studying the neural dynamics of spontaneous language production.

The Access to Grammatical Number in Spanish Children and Adults

In Spanish, the plural form in plural dominant frequency pairs, like "diente/dientes" [tooth/teeth], occurs more frequently than the corresponding singular form. On the other hand, for the singular dominant frequency pairs such as "cometa/cometas" [kite/kites], the singular form is more common than the plural. The recognition of singular forms by adult readers is dependent on the dominance factor, while the identification of plural forms relies on the frequency of the stem. Given that age and reading experience may influence morphological processing of words, we investigate the representation of singulars and plurals in Spanish primary school children in Third Grade (8/9) and Sixth Grade (11/12) and adults through a lexical decision task. Though children's lexical decisions were twice as slow as adults, the pattern of morphological processing was consistent across ages: dominant plural forms resulted in decision times that were comparable to those of non-dominant singular forms, while recognition of singular-dominant forms was quicker than recognition of plural non-dominant forms. It appears that singulars are accessed and stored in the lexical memory as separate entities, while plurals depend on their morphological closer relatives, in this case, the singular forms.

The role of multimodal cues in second language comprehension

In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues - meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal - while using multimodal cues to a lesser extent than L1 comprehenders overall.

Flexible Multi-Step Hypothesis Testing of Human ECoG Data using Cluster-based Permutation Tests with GLMEs

Background

Time series analysis is critical for understanding brain signals and their relationship to behavior and cognition. Cluster-based permutation tests (CBPT) are commonly used to analyze a variety of electrophysiological signals including EEG, MEG, ECoG, and sEEG data without a priori assumptions about specific temporal effects. However, two major limitations of CBPT include the inability to directly analyze experiments with multiple fixed effects and the inability to account for random effects (e.g. variability across subjects). Here, we propose a flexible multi-step hypothesis testing strategy using CBPT with Linear Mixed Effects Models (LMEs) and Generalized Linear Mixed Effects Models (GLMEs) that can be applied to a wide range of experimental designs and data types.

Methods

We first evaluate the statistical robustness of LMEs and GLMEs using simulated data distributions. Second, we apply a multi-step hypothesis testing strategy to analyze ERPs and broadband power signals extracted from human ECoG recordings collected during a simple image viewing experiment with image category and novelty as fixed effects. Third, we assess the statistical power differences between analyzing signals with CBPT using LMEs compared to CBPT using separate t-tests run on each fixed effect through simulations that emulate broadband power signals. Finally, we apply CBPT using GLMEs to high-gamma burst data to demonstrate the extension of the proposed method to the analysis of nonlinear data.

Results

First, we found that LMEs and GLMEs are robust statistical models. In simple simulations LMEs produced highly congruent results with other appropriately applied linear statistical models, but LMEs outperformed many linear statistical models in the analysis of "suboptimal" data and maintained power better than analyzing individual fixed effects with separate t-tests. GLMEs also performed similarly to other nonlinear statistical models. Second, in real world human ECoG data, LMEs performed at least as well as separate t-tests when applied to predefined time windows or when used in conjunction with CBPT. Additionally, fixed effects time courses extracted with CBPT using LMEs from group-level models of pseudo-populations replicated latency effects found in individual category-selective channels. Third, analysis of simulated broadband power signals demonstrated that CBPT using LMEs was superior to CBPT using separate t-tests in identifying time windows with significant fixed effects especially for small effect sizes. Lastly, the analysis of high-gamma burst data using CBPT with GLMEs produced results consistent with CBPT using LMEs applied to broadband power data.

Conclusions

We propose a general approach for statistical analysis of electrophysiological data using CBPT in conjunction with LMEs and GLMEs. We demonstrate that this method is robust for experiments with multiple fixed effects and applicable to the analysis of linear and nonlinear data. Our methodology maximizes the statistical power available in a dataset across multiple experimental variables while accounting for hierarchical random effects and controlling FWER across fixed effects. This approach substantially improves power and accuracy leading to better reproducibility. Additionally, CBPT using LMEs and GLMEs can be used to analyze individual channels or pseudo-population data for the comparison of functional or anatomical groups of data.

What naturalistic stimuli tell us about pronoun resolution in real-time processing

Studies on pronoun resolution have mostly utilized short texts consisting of a context and a target sentence. In the current study we presented participants with nine chapters of an audio book while recording their EEG to investigate the real-time resolution of personal and demonstrative pronouns in a more naturalistic setting. The annotation of the features of the pronouns and their antecedents registered a surprising pattern: demonstrative pronouns showed an interpretive preference for subject/agent antecedents, although they are described to have an anti-subject or anti-agent preference. Given the presence of perspectival centers in the audio book, this however confirmed proposals that demonstrative pronouns are sensitive to perspectival centers. The ERP results revealed a biphasic N400–Late Positivity pattern at posterior electrodes for the demonstrative pronoun relative to the personal pronoun, thereby confirming previous findings with highly controlled stimuli. We take the observed N400 for the demonstrative pronoun as an indication for more demanding processing costs that occur due to the relative unexpectedness of this referential expression. The Late Positivity is taken to reflect the consequences of attentional reorientation: since the demonstrative pronoun indicates a possible shift in the discourse structure, it induces updating of the discourse structure. In addition to the biphasic pattern, the data showed an enhanced positivity at frontal electrode sites for the demonstrative pronoun relative to the personal pronoun. We suggest that this frontal positivity reflects self-relevant engagement and identification with the perspective holder. Our study suggests that by using naturalistic stimuli, we get one step closer to understanding the implementation of language processing in the brain during real life language processing.

Language switching may facilitate the processing of negative responses

It has been proposed that processing sentential negation recruits the neural network of inhibitory control (de Vega et al., 2016; Beltrán et al., 2021). In addition, inhibition mechanisms also play a role in switching languages for bilinguals (Kroll et al., 2015). Since both processes may share inhibitory resources, the current study explored for the first time whether and how language-switching influences the processing of negation. To this end, two groups of Spanish-English bilinguals participated in an encoding-verification memory task. They read short stories involving the same two protagonists (Montse and Jordi), referring to their activities in four different scenarios in Spanish or English. Following each story, the participants received verification questions requiring “yes” or “no” responses depending on whether a given fact was correctly referred to one of the protagonists. Some of the verification questions were in the story’s original language (non-switch condition) and others in the alternate language (switch condition). Results revealed that language-switching facilitated negative responses compared to affirmative responses, exclusively for questions switching from dominant language (L1) to non-dominant language (L2). This effect might reflect that the domain-general mechanisms of inhibitory control are recruited at least partially for both language switch and negation process simultaneously, although this phenomenon is modulated by language dominance.

Rapid adaptation of predictive models during language comprehension: Aperiodic EEG slope, individual alpha frequency and idea density modulate individual differences in real-time model updating

Predictive coding provides a compelling, unified theory of neural information processing, including for language. However, there is insufficient understanding of how predictive models adapt to changing contextual and environmental demands and the extent to which such adaptive processes differ between individuals. Here, we used electroencephalography (EEG) to track prediction error responses during a naturalistic language processing paradigm. In Experiment 1, 45 native speakers of English listened to a series of short passages. Via a speaker manipulation, we introduced changing intra-experimental adjective order probabilities for two-adjective noun phrases embedded within the passages and investigated whether prediction error responses adapt to reflect these intra-experimental predictive contingencies. To this end, we calculated a novel measure of speaker-based, intra-experimental surprisal (“speaker-based surprisal”) as defined on a trial-by-trial basis and by clustering together adjectives with a similar meaning. N400 amplitude at the position of the critical second adjective was used as an outcome measure of prediction error. Results showed that N400 responses attuned to speaker-based surprisal over the course of the experiment, thus indicating that listeners rapidly adapt their predictive models to reflect local environmental contingencies (here: the probability of one type of adjective following another when uttered by a particular speaker). Strikingly, this occurs in spite of the wealth of prior linguistic experience that participants bring to the laboratory. Model adaptation effects were strongest for participants with a steep aperiodic (1/f) slope in resting EEG and low individual alpha frequency (IAF), with idea density (ID) showing a more complex pattern. These results were replicated in a separate sample of 40 participants in Experiment 2, which employed a highly similar design to Experiment 1. Overall, our results suggest that individuals with a steep aperiodic slope adapt their predictive models most strongly to context-specific probabilistic information. Steep aperiodic slope is thought to reflect low neural noise, which in turn may be associated with higher neural gain control and better cognitive control. Individuals with a steep aperiodic slope may thus be able to more effectively and dynamically reconfigure their prediction-related neural networks to meet current task demands. We conclude that predictive mechanisms in language are highly malleable and dynamic, reflecting both the affordances of the present environment as well as intrinsic information processing capabilities of the individual.

Twelve- and Fourteen-Year-Old School Children Differentially Benefit from Sensorimotor- and Multisensory-Enriched Vocabulary Training

Both children and adults have been shown to benefit from the integration of multisensory and sensorimotor enrichment into pedagogy. For example, integrating pictures or gestures into foreign language (L2) vocabulary learning can improve learning outcomes relative to unisensory learning. However, whereas adults seem to benefit to a greater extent from sensorimotor enrichment such as the performance of gestures in contrast to multisensory enrichment with pictures, this is not the case in elementary school children. Here, we compared multisensory- and sensorimotor-enriched learning in an intermediate age group that falls between the age groups tested in previous studies (elementary school children and young adults), in an attempt to determine the developmental time point at which children’s responses to enrichment mature from a child-like pattern into an adult-like pattern. Twelve-year-old and fourteen-year-old German children were trained over 5 consecutive days on auditorily presented, concrete and abstract, Spanish vocabulary. The vocabulary was learned under picture-enriched, gesture-enriched, and non-enriched (auditory-only) conditions. The children performed vocabulary recall and translation tests at 3 days, 2 months, and 6 months post-learning. Both picture and gesture enrichment interventions were found to benefit children’s L2 learning relative to non-enriched learning up to 6 months post-training. Interestingly, gesture-enriched learning was even more beneficial than picture-enriched learning for the 14-year-olds, while the 12-year-olds benefitted equivalently from learning enriched with pictures and gestures. These findings provide evidence for opting to integrate gestures rather than pictures into L2 pedagogy starting at 14 years of age.

Lexical Frequency and Sentence Context Influence the Brain's Response to Single Words

Typical adults read remarkably quickly. Such fast reading is facilitated by brain processes that are sensitive to both word frequency and contextual constraints. It is debated as to whether these attributes have additive or interactive effects on language processing in the brain. We investigated this issue by analysing existing magnetoencephalography data from 99 participants reading intact and scrambled sentences. Using a cross-validated model comparison scheme, we found that lexical frequency predicted the word-by-word elicited MEG signal in a widespread cortical network, irrespective of sentential context. In contrast, index (ordinal word position) was more strongly encoded in sentence words, in left front-temporal areas. This confirms that frequency influences word processing independently of predictability, and that contextual constraints affect word-by-word brain responses. With a conservative multiple comparisons correction, only the interaction between lexical frequency and surprisal survived, in anterior temporal and frontal cortex, and not between lexical frequency and entropy, nor between lexical frequency and index. However, interestingly, the uncorrected index × frequency interaction revealed an effect in left frontal and temporal cortex that reversed in time and space for intact compared to scrambled sentences. Finally, we provide evidence to suggest that, in sentences, lexical frequency and predictability may independently influence early (<150 ms) and late stages of word processing, but also interact during late stages of word processing (>150-250 ms), thus helping to converge previous contradictory eye-tracking and electrophysiological literature. Current neurocognitive models of reading would benefit from accounting for these differing effects of lexical frequency and predictability on different stages of word processing.

Noun-phrase production as a window to language selection: An ERP study

Characterising the time course of non-native language production is critical in understanding the mechanisms behind successful communication. Yet, little is known about the modulating role of cross-linguistic influence (CLI) on the temporal unfolding of non-native production and the locus of target language selection. In this study, we explored CLI effects on non-native noun phrase production with behavioural and neural methods. We were particularly interested in the modulation of the P300 as an index for inhibitory control, and the N400 as an index for co-activation and CLI. German late learners of Spanish overtly named pictures while their EEG was monitored. Our results indicate traceable CLI effects at the behavioural and neural level in both early and late production stages. This suggests that speakers faced competition between the target and non-target language until advanced production stages. Our findings add important behavioural and neural evidence to the underpinnings of non-native production processes, in particular for late learners.

Cross-linguistic interference in late language learners: An ERP study

This study investigated cross-linguistic interference in German low-proficient late learners of Spanish. We examined the modulating influence of gender congruency and cognate status using a syntactic violation paradigm. Behavioural results demonstrated that participants were more sensitive to similarities at the syntactic level (gender congruency) than to phonological and orthographic overlap (cognate status). Electrophysiological data showed that they were sensitive to syntactic violations (P600 effect) already in early acquisition stages. However, P600 effect sizes were not modulated by gender congruency or cognate status. Therefore, our late learners of Spanish did not seem to be susceptible to influences from inherent noun properties when processing non-native noun phrases at the neural level. Our results contribute to the discussion about the neural correlates of grammatical gender processing and sensitivity to syntactic violations in early acquisition stages.

Referential Chains Reveal Predictive Processes and Form-to-Function Mapping: An Electroencephalographic Study Using Naturalistic Story Stimuli

In discourse pragmatics, different referential forms are claimed to be indicative of the cognitive status of a referent in the current discourse. Referential expressions thereby possess a double function: They point back to an (existing) referent (form-to-function mapping), and they are used to derive predictions about a referent’s subsequent recurrence in discourse. Existing event-related potential (ERP) research has mainly focused on the form-to-function mapping of referential expression. In the present ERP study, we explore the relationship of form-to-function mapping and prediction derived from the antecedent of referential expressions in naturalistic auditory language comprehension. Specifically, the study investigates the relationship between the form of a referential expression (pronoun vs. noun) and the form of its antecedent (pronoun vs. noun); i.e., it examines the influence of the interplay of predictions derived from an antecedent (forward-looking function) and the form-to-function mapping of an anaphor (backward-looking function) on the ERPs time-locked to anaphoric expressions. The results in the time range of the P300 and N400 allow for a dissociation of these two functions during online language comprehension.

Visual recognition of words learned with gestures induces motor resonance in the forearm muscles

According to theories of Embodied Cognition, memory for words is related to sensorimotor experiences collected during learning. At a neural level, words encoded with self-performed gestures are represented in distributed sensorimotor networks that resonate during word recognition. Here, we ask whether muscles involved in gesture execution also resonate during word recognition. Native German speakers encoded words by reading them (baseline condition) or by reading them in tandem with picture observation, gesture observation, or gesture observation and execution. Surface electromyogram (EMG) activity from both arms was recorded during the word recognition task and responses were detected using eye-tracking. The recognition of words encoded with self-performed gestures coincided with an increase in arm muscle EMG activity compared to the recognition of words learned under other conditions. This finding suggests that sensorimotor networks resonate into the periphery and provides new evidence for a strongly embodied view of recognition memory.

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Electrophysiological evidence for the coexistence of expectancy fulfillment and semantic integration during the processing of binding and compound nouns

This study employed electrophysiological measures to investigate the processes of expectancy fulfillment and semantic integration during the processing of binding and compound nouns in Chinese. Sequential expectancy and cloze probability between the two constituents of a two-character noun were manipulated in two experiments, resulting in four types of target characters (i.e., the final characters of two-character nouns): (i) high-cloze, binding character (HB); (ii) high-cloze, compound character (HC); (iii) low-cloze, binding character (LB); (iv) low-cloze, compound character (LC). Participants were asked to judge as quickly and accurately as possible whether the two characters, presented sequentially, formed a real word. The two experiments varied in stimulus onset asynchrony (SOA) between the initial and the target characters: 1000 ms and 300 ms. ERPs acquired at the target characters revealed a significant semantic integration effect in both experiments whereby low-cloze target characters elicited more negative-going activities than high-cloze target characters (L > H) with both long and short SOAs. Importantly, there was a graded N400 effect (LC = LB > HC > HB) at the central region: a clearly visible positive deflection for HB relative to HC in synchrony with an equivalence in negativity for LB and LC. This graded pattern was observed with an SOA of 1000 ms but not 300 ms. These results are discussed in terms of the possible coexistence of an expectancy fulfillment mechanism indexed by the P300 that monitors incoming information for realization of expectancies based on stored mental representations and a semantic integration mechanism indexed by the N400 that incorporates incoming information into the preceding context based on semantic-pragmatic knowledge. Manifestation of this coexistence appears sensitive to SOA-modulated attention orientation.

Situating language in a minimal social context: how seeing a picture of the speaker's face affects language comprehension

Natural use of language involves at least two individuals. Some studies have focused on the interaction between senders in communicative situations and how the knowledge about the speaker can bias language comprehension. However, the mere effect of a face as a social context on language processing remains unknown. In the present study, we used event-related potentials to investigate the semantic and morphosyntactic processing of speech in the presence of a photographic portrait of the speaker. In Experiment 1, we show that the N400, a component related to semantic comprehension, increased its amplitude when processed within this minimal social context compared to a scrambled face control condition. Hence, the semantic neural processing of speech is sensitive to the concomitant perception of a picture of the speaker's face, even if irrelevant to the content of the sentences. Moreover, a late posterior negativity effect was found to the presentation of the speaker's face compared to control stimuli. In contrast, in Experiment 2, we found that morphosyntactic processing, as reflected in left anterior negativity and P600 effects, is not notably affected by the presence of the speaker's portrait. Overall, the present findings suggest that the mere presence of the speaker's image seems to trigger a minimal communicative context, increasing processing resources for language comprehension at the semantic level.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

EEG cross-frequency phase synchronization as an index of memory matching in visual search

Visual perception is influenced by our expectancies about incoming sensory information. It is assumed that mental templates of expected sensory input are created and compared to actual input, which can be matching or not. When such mental templates are held in working memory, cross-frequency phase synchronization (CFS) between theta and gamma band activity has been proposed to serve matching processes between prediction and sensation. We investigated how this is affected by the number of activated templates that could be matched by comparing conditions where participants had to keep either one or multiple templates in mind for successful visual search. We found a transient CFS between EEG theta and gamma activity in an early time window around 150 ms after search display presentation, in right hemispheric parietal cortex. Our results suggest that for single template conditions, stronger transient theta-gamma CFS at posterior sites contralateral to target presentation can be observed than for multiple templates. This can be interpreted as evidence to the idea of sequential attentional templates. But mainly, it is understood in line with previous theoretical accounts strongly arguing for transient synchronization between posterior theta and gamma phase as a neural correlate of matching incoming sensory information with contents from working memory and as evidence for limitations in memory matching during multiple template search.

Phasic alerting increases visual processing speed in amnestic mild cognitive impairment

External warning cues temporarily increase the brain's sensitivity for upcoming events, helping individuals to flexibly adapt their reactions to the requirements of complex visual environments. Previous studies reported that younger and cognitively normal older adults profit from phasic alerting cues. Such an intact phasic alerting mechanism could be even more relevant in individuals with Alzheimer's disease who are characterized by reduced processing capacities. The present study employed a theory of visual attention based verbal whole report paradigm with auditory cues in order to investigate phasic alerting effects in amnestic mild cognitive impairment (aMCI). Patients with aMCI were also compared to a previously reported sample of cognitively normal older adults. In patients with aMCI, visual processing speed was higher in the cue compared to the no-cue condition. Further, visual processing speed was reduced in patients with aMCI compared to cognitively normal older adults. Taken together, the results suggest that the processing system of patients with aMCI exhibits general declines but can still integrate auditory warning signals on a perceptual level.

Individual Differences in Peripheral Hearing and Cognition Reveal Sentence Processing Differences in Healthy Older Adults

When viewed cross-sectionally, aging seems to negatively affect speech comprehension. However, aging is a heterogeneous process, and variability among older adults is typically large. In this study, we investigated language comprehension as a function of individual differences in older adults. Specifically, we tested whether hearing thresholds, working memory, inhibition, and individual alpha frequency would predict event-related potential amplitudes in response to classic psycholinguistic manipulations at the sentence level. Twenty-nine healthy older adults (age range 61-76 years) listened to English sentences containing reduced relative clauses and object-relative clauses while their electroencephalogram was recorded. We found that hearing thresholds and working memory predicted P600 amplitudes early during reduced relative clause processing, while individual alpha frequency predicted P600 amplitudes at a later point in time. The results suggest that participants with better hearing and larger working memory capacity simultaneously activated both the preferred and the dispreferred interpretation of reduced relative clauses, while participants with worse hearing and smaller working memory capacity only activated the preferred interpretation. They also suggest that participants with a higher individual alpha frequency had a higher likelihood of successfully reanalysing the sentence toward the reduced relative clause reading than participants with a lower individual alpha frequency. By contrast, we found no relationship between object-relative clause processing and working memory or hearing thresholds. Taken together, the results support the view that older adults employ different strategies during auditory sentence processing dependent on their hearing and cognitive abilities and that there is no single ability that uniformly predicts sentence processing outcomes.

Semantic reversal anomalies under the microscope: Task and modality influences on language-associated event-related potentials

Semantic reversal anomalies (SRAs)-sentences where an implausibility is created by reversing participant roles-have attracted much attention in the literature on the electrophysiology of language. In spite of being syntactically well formed but semantically implausible, these sentences unexpectedly elicited a monophasic P600 effect in English and Dutch rather than an N400 effect. Subsequent research revealed variability in the presence/absence of an N400 effect to SRAs depending on the language examined and the choice of verb type in English. However, most previous studies employed the same presentation modality (visual) and task (acceptability judgement). Here, we conducted two experiments and three statistical analyses to investigate the influence of stimulus modality, task demand and statistical choices on event-related potential (ERP) response patterns to SRAs in English. We reproduced a previous study's procedure and analysis (N. Bourguignon et al. (2012) Brain and Language, 122, 179-189) and further introduced between-subjects factors of task type and modality, using mixed-effects modelling to analyse the data. We observed an N400 effect to typical English SRAs (agent subject verbs, e.g. "the fries will eat the boys"), which contrasts existing literature and was not predicted by existing theories that account for SRA processing. Task demand modulated the ERPs elicited by SRAs, while auditory presentation led to increased comprehension accuracy and a more broadly distributed ERP. Finally, the statistical methods used influenced the presence/absence of ERP effects. Our results suggest a sensitivity of language-related ERP patterns to methodological parameters, and we conclude that future experiments should take this into careful consideration.

Hyperconnectivity during screen-based stories listening is associated with lower narrative comprehension in preschool children exposed to screens vs dialogic reading: An EEG study

OBJECTIVES

Dialogic reading (DR) is a shared storybook reading intervention previously shown to have a positive effect on both literacy and general language skills. The aim of this study was to examine the effect of DR compared to screen-based intervention on electrophysiological markers supporting narrative comprehension using EEG.

METHODS

Thirty-two typically developing preschoolers, ages 4 to 6 years, were assigned to one of two intervention groups: Dialogic Reading Group (DRG, n = 16) or Screen Story Group (SSG, n = 16). We examined the effect of intervention type using behavioral assessment and a narrative comprehension task with EEG.

RESULTS

The DRG showed improved vocabulary and decreased functional connectivity during the stories-listening task, whereas the SSG group showed no changes in vocabulary or connectivity. Significantly decreased network strength and transitivity and increased network efficiency were observed in the DRG following intervention. Greater network strength and transitivity at follow-up were correlated with increased vocabulary.

CONCLUSIONS

The results suggest the beneficial effect of DR in preschool-age children on vocabulary and EEG-bands related to attention in the ventral stream during narrative comprehension. Decreased functional connectivity may serve as a marker for language gains following reading intervention.

SIGNIFICANCE

DR intervention for preschool-age children may reduce interfering connections related to attention, which is related to better narrative comprehension.

Speed-Accuracy Tradeoffs in Brain and Behavior: Testing the Independence of P300 and N400 Related Processes in Behavioral Responses to Sentence Categorization

Although the N400 was originally discovered in a paradigm designed to elicit a P300 (Kutas and Hillyard, 1980), its relationship with the P300 and how both overlapping event-related potentials (ERPs) determine behavioral profiles is still elusive. Here we conducted an ERP (N = 20) and a multiple-response speed-accuracy tradeoff (SAT) experiment (N = 16) on distinct participant samples using an antonym paradigm (The opposite of black is white/nice/yellow with acceptability judgment). We hypothesized that SAT profiles incorporate processes of task-related decision-making (P300) and stimulus-related expectation violation (N400). We replicated previous ERP results (Roehm et al., 2007): in the correct condition (white), the expected target elicits a P300, while both expectation violations engender an N400 [reduced for related (yellow) vs. unrelated targets (nice)]. Using multivariate Bayesian mixed-effects models, we modeled the P300 and N400 responses simultaneously and found that correlation between residuals and subject-level random effects of each response window was minimal, suggesting that the components are largely independent. For the SAT data, we found that antonyms and unrelated targets had a similar slope (rate of increase in accuracy over time) and an asymptote at ceiling, while related targets showed both a lower slope and a lower asymptote, reaching only approximately 80% accuracy. Using a GLMM-based approach (Davidson and Martin, 2013), we modeled these dynamics using response time and condition as predictors. Replacing the predictor for condition with the averaged P300 and N400 amplitudes from the ERP experiment, we achieved identical model performance. We then examined the piecewise contribution of the P300 and N400 amplitudes with partial effects (see Hohenstein and Kliegl, 2015). Unsurprisingly, the P300 amplitude was the strongest contributor to the SAT-curve in the antonym condition and the N400 was the strongest contributor in the unrelated condition. In brief, this is the first demonstration of how overlapping ERP responses in one sample of participants predict behavioral SAT profiles of another sample. The P300 and N400 reflect two independent but interacting processes and the competition between these processes is reflected differently in behavioral parameters of speed and accuracy.

Toward a Neurobiologically Plausible Model of Language-Related, Negative Event-Related Potentials

Language-related event-related potential (ERP) components such as the N400 have traditionally been associated with linguistic or cognitive functional interpretations. By contrast, it has been considerably more difficult to relate these components to neurobiologically grounded accounts of language. Here, we propose a theoretical framework based on a predictive coding architecture, within which negative language-related ERP components such as the N400 can be accounted for in a neurobiologically plausible manner. Specifically, we posit that the amplitude of negative language-related ERP components reflects precision-weighted prediction error signals, i.e., prediction errors weighted by the relevance of the information source leading to the error. From this perspective, precision has a direct link to cue validity in a particular language and, thereby, to relevance of individual linguistic features for internal model updating. We view components such as the N400 and LAN as members of a family with similar functional characteristics and suggest that latency and topography differences between these components reflect the locus of prediction errors and model updating within a hierarchically organized cortical predictive coding architecture. This account has the potential to unify findings from the full range of the N400 literature, including word-level, sentence-, and discourse-level results as well as cross-linguistic differences.

Hierarchical structure guides rapid linguistic predictions during naturalistic listening

The grammar, or syntax, of human language is typically understood in terms of abstract hierarchical structures. However, theories of language processing that emphasize sequential information, not hierarchy, successfully model diverse phenomena. Recent work probing brain signals has shown mixed evidence for hierarchical information in some tasks. We ask whether sequential or hierarchical information guides the expectations that a human listener forms about a word's part-of-speech when simply listening to every-day language. We compare the predictions of three computational models against electroencephalography signals recorded from human participants who listen passively to an audiobook story. We find that predictions based on hierarchical structure correlate with the human brain response above-and-beyond predictions based only on sequential information. This establishes a link between hierarchical linguistic structure and neural signals that generalizes across the range of syntactic structures found in every-day language.

Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension

Recent decades have ushered in tremendous progress in understanding the neural basis of language. Most of our current knowledge on language and the brain, however, is derived from lab-based experiments that are far removed from everyday language use, and that are inspired by questions originating in linguistic and psycholinguistic contexts. In this paper we argue that in order to make progress, the field needs to shift its focus to understanding the neurobiology of naturalistic language comprehension. We present here a new conceptual framework for understanding the neurobiological organization of language comprehension. This framework is non-language-centered in the computational/neurobiological constructs it identifies, and focuses strongly on context. Our core arguments address three general issues: (i) the difficulty in extending language-centric explanations to discourse; (ii) the necessity of taking context as a serious topic of study, modeling it formally and acknowledging the limitations on external validity when studying language comprehension outside context; and (iii) the tenuous status of the language network as an explanatory construct. We argue that adopting this framework means that neurobiological studies of language will be less focused on identifying correlations between brain activity patterns and mechanisms postulated by psycholinguistic theories. Instead, they will be less self-referential and increasingly more inclined towards integration of language with other cognitive systems, ultimately doing more justice to the neurobiological organization of language and how it supports language as it is used in everyday life.

Congruency of Separable Affix Verb Combinations Is Linearly Indexed by the N400

Separable affix verbs consist of a stem and a derivational affix, which, in some languages can appear together or in discontinuous, distributed form, e.g., German “aufgreifen” and “greifen … auf” [“up-pick(ing)” and “pick … up”]. Certain stems can combine with only certain affixes. However, many such combinations are evaluated not as clearly correct or incorrect, but frequently take an intermediate status with participants rating them ambiguously. Here, we mapped brain responses to combinations of verb stems and affixes realized in short sentences, including more and less common particle verbs, borderline acceptable combinations and clear violations. Event-related potential responses to discontinuous particle verbs were obtained for five affixes re-combined with 10 verb stems, situated within short, German sentences, i.e., “sie <stem>en es <affix>,” English: “they <stem> it <affix>.” The congruity of combinations was assessed both with behavioral ratings of the stimuli and corpus-derived probability measures. The size of a frontal N400 correlated with the degree of incongruency between stem and affix, as assessed by both measures. Behavioral ratings performed better than corpus-derived measures in predicting N400 magnitudes, and a combined model performed best of all. No evidence for a discrete, right/wrong effect was found. We discuss methodological implications and integrate the results into past research on the N400 and neurophysiological studies on separable-affix verbs, generally.