So that's what you meant! Event-related potentials reveal multiple aspects of context use during construction of message-level meaning

Understanding words in context: A naturalistic EEG study of children's lexical processing

When listening to speech, adults rely on context to anticipate upcoming words. Evidence for this comes from studies demonstrating that the N400, an event-related potential (ERP) that indexes ease of lexical-semantic processing, is influenced by the predictability of a word in context. We know far less about the role of context in children's speech comprehension. The present study explored lexical processing in adults and 5-10-year-old children as they listened to a story. ERPs time-locked to the onset of every word were recorded. Each content word was coded for frequency, semantic association, and predictability. In both children and adults, N400s reflect word predictability, even when controlling for frequency and semantic association. These findings suggest that both adults and children use top-down constraints from context to anticipate upcoming words when listening to stories.

Yes or no? A study of ErrPs in the "guess what I am thinking" paradigm with stimuli of different visual content

Error-related potentials (ErrPs) have attracted attention in part because of their practical potential for building brain-computer interface (BCI) paradigms. BCIs, facilitating direct communication between the brain and machines, hold great promise for brain-AI interaction. Therefore, a comprehensive understanding of ErrPs is crucial to ensure reliable BCI outcomes. In this study, we investigated ErrPs in the context of the "guess what I am thinking" paradigm. 23 healthy participants were instructed to imagine an object from a predetermined set, while an algorithm randomly selected another object that was either the same as or different from the imagined object. We recorded and analyzed the participants' EEG activity to capture their mental responses to the algorithm's "predictions". The study identified components distinguishing correct from incorrect responses. It discusses their nature and how they differ from ErrPs extensively studied in other BCI paradigms. We observed pronounced variations in the shape of ErrPs across different stimulus sets, underscoring the significant influence of visual stimulus appearance on ErrP peaks. These findings have implications for designing effective BCI systems, especially considering the less conventional BCI paradigm employed. They emphasize the necessity of accounting for stimulus factors in BCI development.

On the Mathematical Relationship Between Contextual Probability and N400 Amplitude

Accounts of human language comprehension propose different mathematical relationships between the contextual probability of a word and how difficult it is to process, including linear, logarithmic, and super-logarithmic ones. However, the empirical evidence favoring any of these over the others is mixed, appearing to vary depending on the index of processing difficulty used and the approach taken to calculate contextual probability. To help disentangle these results, we focus on the mathematical relationship between corpus-derived contextual probability and the N400, a neural index of processing difficulty. Specifically, we use 37 contemporary transformer language models to calculate the contextual probability of stimuli from 6 experimental studies of the N400, and test whether N400 amplitude is best predicted by a linear, logarithmic, super-logarithmic, or sub-logarithmic transformation of the probabilities calculated using these language models, as well as combinations of these transformed metrics. We replicate the finding that on some datasets, a combination of linearly and logarithmically-transformed probability can predict N400 amplitude better than either metric alone. In addition, we find that overall, the best single predictor of N400 amplitude is sub-logarithmically-transformed probability, which for almost all language models and datasets explains all the variance in N400 amplitude otherwise explained by the linear and logarithmic transformations. This is a novel finding that is not predicted by any current theoretical accounts, and thus one that we argue is likely to play an important role in increasing our understanding of how the statistical regularities of language impact language comprehension.

The electrophysiology of lexical prediction of emoji and text

As emoji often appear naturally alongside text in utterances, they provide a way to study how prediction unfolds in multimodal sentences in direct comparison to unimodal sentences. In this experiment, participants (N = 40) read sentences in which the sentence-final noun appeared in either word form or emoji form, a between-subjects manipulation. The experiment featured both high constraint sentences and low constraint sentences to examine how the lexical processing of emoji interacts with prediction processes in sentence comprehension. Two well-established ERP components linked to lexical processing and prediction - the N400 and the Late Frontal Positivity - are investigated for sentence-final words and emoji to assess whether, to what extent, and in what linguistic contexts emoji are processed like words. Results indicate that the expected effects, namely an N400 effect to an implausible lexical item compared to a plausible one and an LFP effect to an unexpected lexical item compared to an expected one, emerged for both words and emoji. This paper discusses the similarities and differences between the stimulus types and constraint conditions, contextualized within theories of linguistic prediction, ERP components, and a multimodal lexicon.

The Costs (and Benefits?) of Effortful Listening for Older Adults: Insights from Simultaneous Electrophysiology, Pupillometry, and Memory

Although the impact of acoustic challenge on speech processing and memory increases as a person ages, older adults may engage in strategies that help them compensate for these demands. In the current preregistered study, older adults (n = 48) listened to sentences-presented in quiet or in noise-that were high constraint with either expected or unexpected endings or were low constraint with unexpected endings. Pupillometry and EEG were simultaneously recorded, and subsequent sentence recognition and word recall were measured. Like young adults in prior work, we found that noise led to increases in pupil size, delayed and reduced ERP responses, and decreased recall for unexpected words. However, in contrast to prior work in young adults where a larger pupillary response predicted a recovery of the N400 at the cost of poorer memory performance in noise, older adults did not show an associated recovery of the N400 despite decreased memory performance. Instead, we found that in quiet, increases in pupil size were associated with delays in N400 onset latencies and increased recognition memory performance. In conclusion, we found that transient variation in pupil-linked arousal predicted trade-offs between real-time lexical processing and memory that emerged at lower levels of task demand in aging. Moreover, with increased acoustic challenge, older adults still exhibited costs associated with transient increases in arousal without the corresponding benefits.

A predictive coding model of the N400

The N400 event-related component has been widely used to investigate the neural mechanisms underlying real-time language comprehension. However, despite decades of research, there is still no unifying theory that can explain both its temporal dynamics and functional properties. In this work, we show that predictive coding - a biologically plausible algorithm for approximating Bayesian inference - offers a promising framework for characterizing the N400. Using an implemented predictive coding computational model, we demonstrate how the N400 can be formalized as the lexico-semantic prediction error produced as the brain infers meaning from the linguistic form of incoming words. We show that the magnitude of lexico-semantic prediction error mirrors the functional sensitivity of the N400 to various lexical variables, priming, contextual effects, as well as their higher-order interactions. We further show that the dynamics of the predictive coding algorithm provides a natural explanation for the temporal dynamics of the N400, and a biologically plausible link to neural activity. Together, these findings directly situate the N400 within the broader context of predictive coding research. More generally, they raise the possibility that the brain may use the same computational mechanism for inference across linguistic and non-linguistic domains.

Neurophysiological signatures of prediction in language: A critical review of anticipatory negativities

Recent event-related potential (ERP) studies in language comprehension converge in finding anticipatory negativities preceding words or word segments that can be pre-activated based on either sentence contexts or phonological cues. We review these findings from different paradigms in the light of evidence from other cognitive domains in which slow negative potentials have long been associated with anticipatory processes and discuss their potential underlying mechanisms. We propose that this family of anticipatory negativities captures common mechanisms associated with the pre-activation of linguistic information both within words and within sentences. Future studies could utilize these anticipatory negativities in combination with other, well-established ERPs, to simultaneously track prediction-related processes emerging at different time intervals (before and after the perception of pre-activated input) and with distinct time courses (shorter-lived and longer-lived cognitive operations).

Characterizing directional dynamics of semantic prediction based on inter-regional temporal generalization

The event-related potential/field component N400(m) has been widely used as a neural index for semantic prediction. It has long been hypothesized that feedback information from inferior frontal areas plays a critical role in generating the N400. However, due to limitations in causal connectivity estimation, direct testing of this hypothesis has remained difficult. Here, magnetoencephalography (MEG) data was obtained during a classic N400 paradigm where the semantic predictability of a fixed target noun was manipulated in simple German sentences. To estimate causality, we implemented a novel approach based on machine learning and temporal generalization to estimate the effect of inferior frontal gyrus (IFG) on temporal areas. In this method, a support vector machine (SVM) classifier is trained on each time point of the neural activity in IFG to classify less predicted (LP) and highly predicted (HP) nouns and then tested on all time points of superior/middle temporal sub-regions activity (and vice versa, to establish spatio-temporal evidence for or against causality). The decoding accuracy was significantly above chance level when the classifier was trained on IFG activity and tested on future activity in superior and middle temporal gyrus (STG/MTG). The results present new evidence for a model predictive speech comprehension where predictive IFG activity is fed back to shape subsequent activity in STG/MTG, implying a feedback mechanism in N400 generation. In combination with the also observed strong feedforward effect from left STG/MTG to IFG, our findings provide evidence of dynamic feedback and feedforward influences between IFG and temporal areas during N400 generation.

The Impact of Linguistic Prediction Violations on Downstream Recognition Memory and Sentence Recall

Predicting upcoming words during language comprehension not only affects processing in the moment but also has consequences for memory, although the source of these memory effects (e.g., whether driven by lingering pre-activations, re-analysis following prediction violations, or other mechanisms) remains underspecified. Here, we investigated downstream impacts of prediction on memory in two experiments. First, we recorded EEG as participants read strongly and weakly constraining sentences with expected, unexpected but plausible, or semantically anomalous endings ("He made a holster for his gun / father / train") and were tested on their recognition memory for the sentence endings. Participants showed similar rates of false alarms for predicted but never presented sentence endings whether the prediction violation was plausible or anomalous, suggesting that these arise from pre-activation of the expected words during reading. During sentence reading, especially in strongly constraining sentences, plausible prediction violations elicited an anterior positivity; anomalous endings instead elicited a posterior positivity, whose amplitude was predictive of later memory for those anomalous words. ERP patterns at the time of recognition differentiated plausible and anomalous sentence endings: Words that had been plausible prediction violations elicited enhanced late positive complex amplitudes, suggesting greater episodic recollection, whereas anomalous sentence endings elicited greater N1 amplitudes, suggesting attentional tagging. In a follow-up behavioral study, a separate group of participants read the same sentence stimuli and were tested for sentence-level recall. We found that recall of full sentences was impaired when sentences ended with a prediction violation. Taken together, the results suggest that prediction violations draw attention and affect encoding of the violating word, in a manner that depends on plausibility, and that this, in turn, may impair future memory of the gist of the sentence.

Discourse coherence modulates use of predictive processing during sentence comprehension

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze probability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, predictable words were processed more quickly and anomalous words more slowly when they occurred in highly coherent discourse. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating semantic and lexical information less when their expectations are less likely to be reliable.

Graded and ungraded expectation patterns: Prediction dynamics during active comprehension

Language comprehension can be facilitated by the accurate prediction of upcoming words, but prediction effects are not ubiquitous, and comprehenders likely use predictive processing to varying degrees depending on task demands. To ascertain the processing consequences of prioritizing prediction, we here compared ERPs elicited when young adult participants simply read for comprehension with those collected in a subsequent block that required active prediction. We were particularly interested in frontally-distributed post-N400 effects for expected and unexpected words in strongly constraining contexts, which have previously been documented as two distinct patterns: an enhanced positivity ("anterior positivity") observed for prediction violations compared to words that are merely unpredictable (because they occur in weakly constraining sentences) and a distinction between expected endings in more constraining contexts and those same weakly constrained words ("frontal negativity" to the strongly predicted words). We found that the size of the anterior positivity effect was unchanged between passive comprehension and active prediction, suggesting that some processes related to prediction may engage state-like networks. On the other hand, the frontal negativity showed graded patterns from the interaction of task and sentence type. These differing patterns support the hypothesis that there are two separate effects with frontal scalp distributions that occur after the N400 and further suggest that the impact of violating predictions (as long as prediction is engaged at all) is largely stable across varying levels of effort/attention directed toward prediction, whereas other comprehension processes can be modulated by task demands.

Multiple predictions during language comprehension: Friends, foes, or indifferent companions?

To comprehend language, we continually use prior context to pre-activate expected upcoming information, resulting in facilitated processing of incoming words that confirm these predictions. But what are the consequences of disconfirming prior predictions? To address this question, most previous studies have examined unpredictable words appearing in contexts that constrain strongly for a single continuation. However, during natural language processing, it is far more common to encounter contexts that constrain for multiple potential continuations, each with some probability. Here, we ask whether and how pre-activating both higher and lower probability alternatives influences the processing of the lower probability incoming word. One possibility is that, similar to language production, there is continuous pressure to select the higher-probability pre-activated alternative through competitive inhibition. During comprehension, this would result in relative costs in processing the lower probability target. A second possibility is that if the two pre-activated alternatives share semantic features, they mutually enhance each other's pre-activation. This would result in greater facilitation in processing the lower probability target. To distinguish between these accounts, we recorded ERPs as participants read three-sentence scenarios that constrained either for a single word or for two potential continuations - a higher probability expected candidate and a lower probability second-best candidate. We found no evidence that competitive pre-activation between the expected and second-best candidates resulted in costs in processing the second-best target, either during lexico-semantic processing (indexed by the N400) or at later stages of processing (indexed by a later frontal positivity). Instead, we found only benefits of pre-activating multiple alternatives, with evidence of enhanced graded facilitation on lower-probability targets that were semantically related to a higher-probability pre-activated alternative. These findings are consistent with a previous eye-tracking study by Luke and Christianson (2016, Cogn Psychol) using corpus-based materials. They have significant theoretical implications for models of predictive language processing, indicating that routine graded prediction in language comprehension does not operate through the same competitive mechanisms that are engaged in language production. Instead, our results align more closely with hierarchical probabilistic accounts of language comprehension, such as predictive coding.

Ignoring the alternatives: The N400 is sensitive to stimulus preactivation alone

The N400 component of the event-related brain potential is a neural signal of processing difficulty. In the language domain, it is widely believed to be sensitive to the degree to which a given word or its semantic features have been preactivated in the brain based on the preceding context. However, it has also been shown that the brain often preactivates many words in parallel. It is currently unknown whether the N400 is also affected by the preactivations of alternative words other than the stimulus that is actually presented. This leaves a weak link in the derivation chain-how can we use the N400 to understand the mechanisms of preactivation if we do not know what it indexes? This study directly addresses this gap. We estimate the extent to which all words in a lexicon are preactivated in a given context using the predictions of contemporary large language models. We then directly compare two competing possibilities: that the amplitude of the N400 is sensitive only to the extent to which the stimulus is preactivated, and that it is also sensitive to the preactivation states of the alternatives. We find evidence of the former. This result allows for better grounded inferences about the mechanisms underlying the N400, lexical preactivation in the brain, and language processing more generally.

Read carefully, because this is important! How value-driven strategies impact sentence memory

Little is understood about how people strategically process and remember important but complex information, such as sentences. In the current study, we asked whether people can effectively prioritize memory for sentences as a function of their relative importance (operationalized as a reward point value) and whether they do so, in part, by changing their sentence processing strategies when value information is available in advance. We adapted the value-directed remembering paradigm (Castel, Psychol Learn Motiv 48:225-270, 2007) for sentences that varied in constraint and predictability. Each sentence was associated with a high or low value for subsequent free recall (whole sentence) and recognition (sentence-final words) tests. Value information appeared after or before each sentence as a between-subject manipulation. Regardless of condition, we observed that high-value sentences were recalled more often than low-value sentences, showing that people can strategically prioritize their encoding of sentences. However, memory patterns differed depending on when value information was available. Recall for high-value sentences that ended unexpectedly (and therefore violated one's predictions) was reduced in the Before compared to the After condition. Before condition participants also showed a greater tendency to false alarm to lures (words that were the predicted - but not obtained - ending) from strongly constraining sentences. These observations suggest that when people try to prioritize sentence-level information that they know is valuable, the reading strategies they employ may paradoxically lead to worse memory.

Individual Differences in Holistic and Compositional Language Processing

Individual differences in cognitive abilities are ubiquitous across the spectrum of proficient language users. Although speakers differ with regard to their memory capacity, ability for inhibiting distraction, and ability to shift between different processing levels, comprehension is generally successful. However, this does not mean it is identical across individuals; listeners and readers may rely on different processing strategies to exploit distributional information in the service of efficient understanding. In the following psycholinguistic reading experiment, we investigate potential sources of individual differences in the processing of co-occurring words. Participants read modifier-noun bigrams like absolute silence in a self-paced reading task. Backward transition probability (BTP) between the two lexemes was used to quantify the prominence of the bigram as a whole in comparison to the frequency of its parts. Of five individual difference measures (processing speed, verbal working memory, cognitive inhibition, global-local scope shifting, and personality), two proved to be significantly associated with the effect of BTP on reading times. Participants who could inhibit a distracting global environment in order to more efficiently retrieve a single part and those that preferred the local level in the shifting task showed greater effects of the co-occurrence probability of the parts. We conclude that some participants are more likely to retrieve bigrams via their parts and their co-occurrence statistics whereas others more readily retrieve the two words together as a single chunked unit.

Readers scrutinize lexical familiarity only in the absence of expectations: Evidence from lexicality effects on event-related potentials

Readers generate predictions about the meaning of upcoming words while reading constraining sentences. These predictions feed down to predictions about orthographic form. For example, orthographic neighbors of predicted words yield reduced N400 amplitudes compared to non-neighbors regardless of lexical status (Laszlo & Federmeier, 2009). We investigated whether readers are sensitive to lexicality in low constraint sentences when they must scrutinize the perceptual input more closely for word recognition. In a replication and extension of Laszlo and Federmeier (2009), we observed similar patterns as the original study in high constraint sentences, but found a lexicality effect in low constraint sentences that was not present when the sentence was highly constraining. This suggests that, in the absence of strong expectations, readers adopt a different reading strategy to scrutinize the structure of words more in depth to make sense of what they have read compared to when they encounter a supportive sentence context.

Event-related potentials show that parafoveal vision is insufficient for semantic integration

Readers extract information from a word from parafoveal vision prior to looking at it. It has been argued that parafoveal perception allows readers to initiate linguistic processes, but it is unclear which stages of word processing are engaged: the process of extracting letter information to recognize words, or the process of extracting meaning to comprehend them. This study used the event-related brain potential (ERP) technique to investigate how word recognition (indexed by the N400 effect for unexpected or anomalous compared to expected words) and semantic integration (indexed by the Late-positive component; LPC effect for anomalous compared to expected words) are or are not elicited when the word is perceived only in parafoveal vision. Participants read a target word following a sentence that made it expected, unexpected, or anomalous, and read the sentences presented three words at a time in the Rapid Serial Visual Presentation (RSVP) with flankers paradigm so that words were perceived in parafoveal and foveal vision. We orthogonally manipulated whether the target word was masked in parafoveal and/or foveal vision to dissociate the processing associated with perception of the target word from either location. We found that the N400 effect was generated from parafoveally perceived words, and was reduced for foveally perceived words if they were previously perceived parafoveally. In contrast, the LPC effect was only elicited if the word was perceived foveally, suggesting that readers must attend to a word directly in foveal vision in order to attempt to integrate its meaning into the sentence context.

ERPs Reveal How Semantic and Syntactic Processing Unfold across Parafoveal and Foveal Vision during Sentence Comprehension

We examined how readers process content and function words in sentence comprehension with ERPs. Participants read simple declarative sentences using a rapid serial visual presentation (RSVP) with flankers paradigm. Sentences contained either an unexpected semantically anomalous content word, an unexpected syntactically anomalous function word or were well formed with no anomalies. ERPs were examined when target words were in the parafoveal or foveal vision. Unexpected content words elicited a typically distributed N400 when displayed in the parafovea, followed by a longer-lasting, widely distributed positivity starting around 300 ms once foveated. Unexpected function words elicited a left lateralized LAN-like component when presented in the parafovea, followed by a left lateralized, posteriorly distributed P600 when foveated. These results suggested that both semantic and syntactic processing involve two stages-the initial, fast process that can be completed in parafovea, followed by a more in depth attentionally mediated assessment that occurs with direct attention.

Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability

Stimuli are easier to process when context makes them predictable, but does context-based facilitation arise from preactivation of a limited set of relatively probable upcoming stimuli (with facilitation then linearly related to probability) or, instead, because the system maintains and updates a probability distribution across all items (with facilitation logarithmically related to probability)? We measured the N400, an index of semantic access, to words of varying probability, including unpredictable words. Word predictability was measured using both cloze probabilities and a state-of-the-art machine learning language model (GPT-2). We reanalyzed five datasets (n = 138) to demonstrate and then replicate that context-based facilitation on the N400 is graded, even among unpredictable words. Furthermore, we established that the relationship between word predictability and context-based facilitation combines linear and logarithmic functions. We argue that this composite function reveals properties of the mapping between words and semantic features and how feature- and word-related information is activated on-line.

Age-related differences in expectation-based novel word learning

Adult language users can infer the meaning of a previously unfamiliar word from a single exposure to this word in a semantically and thematically constrained context, henceforth, predictive context (Borovsky et al., 2010 Cognition, 116(2), 289-296; Borovsky et al., 2012 Language Learning and Development, 8(3), 278-302). Children use predictive contexts to anticipate upcoming stimuli (Borovsky et al., 2012 Language Learning and Development, 8(3), 278-302; Mani & Huettig, 2012 Journal of Experimental Psychology: Human Perception and Performance, 38(4), 843-847), but the extent to which they rely on prediction to learn novel word forms is unclear (Gambi et al., 2021 Cognition, 211, 104650). Here, we examine children's one-shot learning from predictive contexts using a modified version of the one-shot learning ERP paradigm for children aged 7-13 years. In a first learning phase, we presented audio recordings of expected words and unexpected novel pseudowords in strongly and weakly constraining sentence contexts. In the following priming phase, the same recorded words and pseudowords were used as primes to identical/synonymous, related, and unrelated target words. We measured N400 modulations to the word and pseudoword continuations in the learning phase and to the identical/synonymous, related, or unrelated target words in the priming phase. When initially presented in strongly constraining sentences, novel pseudowords primed synonymous targets equally well as word primes of the same intended meaning. This pattern was particularly pronounced in older children. Our findings suggest that, around early adolescence, children can use single exposures to constraining contexts to infer the meaning of novel words and to integrate these novel words in their lexicons.

Connecting and considering: Electrophysiology provides insights into comprehension

The ability to rapidly and systematically access knowledge stored in long-term memory in response to incoming sensory information-that is, to derive meaning from the world-lies at the core of human cognition. Research using methods that can precisely track brain activity over time has begun to reveal the multiple cognitive and neural mechanisms that make this possible. In this article, I delineate how a process of connecting affords an effortless, continuous infusion of meaning into human perception. In a relatively invariant time window, uncovered through studies using the N400 component of the event-related potential, incoming sensory information naturally induces a graded landscape of activation across long-term semantic memory, creating what might be called "proto-concepts". Connecting can be (but is not always) followed by a process of further considering those activations, wherein a set of more attentionally demanding "active comprehension" mechanisms mediate the selection, augmentation, and transformation of the initial semantic representations. The result is a limited set of more stable bindings that can be arranged in time or space, revised as needed, and brought to awareness. With this research, we are coming closer to understanding how the human brain is able to fluidly link sensation to experience, to appreciate language sequences and event structures, and, sometimes, to even predict what might be coming up next.

Investigating Heritage Language Processing: Meaning Composition in Chinese Classifier-Noun Phrasal Contexts

The aim of the present study was to investigate how Chinese-Malay bilingual speakers with Chinese as heritage language process semantic congruency effects in Chinese and how their brain activities compare to those of monolingual Chinese speakers using electroencephalography (EEG) recordings. To this end, semantic congruencies were manipulated in Chinese classifier-noun phrases, resulting in four conditions: (i) a strongly constraining/high-cloze, plausible (SP) condition, (ii) a weakly constraining/low-cloze, plausible (WP) condition, (iii) a strongly constraining/implausible (SI) condition, and (iv) a weakly constraining/implausible (WI) condition. The analysis of EEG data focused on two event-related potential components, i.e., the N400, which is known for its sensitivity to semantic fit of a target word to its context, and a post-N400 late positive complex (LPC), which is linked to semantic integration after prediction violations and retrospective, evaluative processes. We found similar N400/LPC effects in response to the manipulations of semantic congruency in the mono- and bilingual groups, with a gradient N400 pattern (WI/SI > WP > SP), a larger frontal LPC in response to WP compared to SP, SI, and WI, as well as larger centro-parietal LPCs in response to WP compared to SI and WI, and a larger centro-parietal LPC for SP compared to SI. These results suggest that, in terms of event-related potential (ERP) data, Chinese-Malay early bilingual speakers predict and integrate upcoming semantic information in Chinese classifier-noun phrase to the same extent as monolingual Chinese speakers. However, the global field power (GFP) data showed significant differences between SP and WP in the N400 and LPC time windows in bilinguals, whereas no such effects were observed in monolinguals. This finding was interpreted as showing that bilinguals differ from their monolingual peers in terms of global field power intensity of the brain by processing plausible classifier-noun pairs with different congruency effects.

Dividing attention influences contextual facilitation and revision during language comprehension

Although we often seem to successfully comprehend language in the face of distraction, few studies have examined the role of sustained attention in critical components of sentence processing, such as integrating information over a sentence and revising predictions when unexpected information is encountered. The current study investigated the impact of attention on sentence processing using a novel dual-task paradigm. Participants read weakly and strongly constraining sentences with expected or unexpected endings while also tracking the motion of dots in the background, and their EEG was recorded. Under full attention, the amplitude of the N400 component of the ERP, a measure of semantic access, was reduced (facilitated) in a graded fashion by contextual strength and fit. This context-based facilitation was attenuated when attention was divided, suggesting that sustained attention is important for building up message-level representations. In contrast, the post-N400 frontal positivity that has been observed to prediction violations and associated with revision processes was unaffected by dividing attention. However, under divided attention, participants also elicited posteriorly-distributed effects to these violations. Thus, predictive processes seem to be engaged even when attention is divided, but additional resources may then be required to process unexpected information.

Representational Pattern Similarity of Electrical Brain Activity Reveals Rapid and Specific Prediction during Language Comprehension

Predicting upcoming events is a critical function of the brain, and language provides a fertile testing ground for studying prediction, as comprehenders use context to predict features of upcoming words. Many aspects of the mechanisms of prediction remain elusive, partly due to a lack of methodological tools to probe prediction formation in the moment. To elucidate what features are neurally preactivated and when, we used representational similarity analysis on previously collected sentence reading data. We compared EEG activity patterns elicited by expected and unexpected sentence final words to patterns from the preceding words of the sentence, in both strongly and weakly constraining sentences. Pattern similarity with the final word was increased in an early time window following the presentation of the pre-final word, and this increase was modulated by both expectancy and constraint. This was not seen at earlier words, suggesting that predictions were precisely timed. Additionally, pre-final word activity-the predicted representation-had negative similarity with later final word activity, but only for strongly expected words. These findings shed light on the mechanisms of prediction in the brain: rapid preactivation occurs following certain cues, but the predicted features may receive reduced processing upon confirmation.

The costs (and benefits) of effortful listening on context processing: A simultaneous electrophysiology, pupillometry, and behavioral study

There is an apparent disparity between the fields of cognitive audiology and cognitive electrophysiology as to how linguistic context is used when listening to perceptually challenging speech. To gain a clearer picture of how listening effort impacts context use, we conducted a pre-registered study to simultaneously examine electrophysiological, pupillometric, and behavioral responses when listening to sentences varying in contextual constraint and acoustic challenge in the same sample. Participants (N = 44) listened to sentences that were highly constraining and completed with expected or unexpected sentence-final words ("The prisoners were planning their escape/party") or were low-constraint sentences with unexpected sentence-final words ("All day she thought about the party"). Sentences were presented either in quiet or with +3 dB SNR background noise. Pupillometry and EEG were simultaneously recorded and subsequent sentence recognition and word recall were measured. While the N400 expectancy effect was diminished by noise, suggesting impaired real-time context use, we simultaneously observed a beneficial effect of constraint on subsequent recognition memory for degraded speech. Importantly, analyses of trial-to-trial coupling between pupil dilation and N400 amplitude showed that when participants' showed increased listening effort (i.e., greater pupil dilation), there was a subsequent recovery of the N400 effect, but at the same time, higher effort was related to poorer subsequent sentence recognition and word recall. Collectively, these findings suggest divergent effects of acoustic challenge and listening effort on context use: while noise impairs the rapid use of context to facilitate lexical semantic processing in general, this negative effect is attenuated when listeners show increased effort in response to noise. However, this effort-induced reliance on context for online word processing comes at the cost of poorer subsequent memory.

Rational Adaptation in Lexical Prediction: The Influence of Prediction Strength

Recent studies indicate that the processing of an unexpected word is costly when the initial, disconfirmed prediction was strong. This penalty was suggested to stem from commitment to the strongly predicted word, requiring its inhibition when disconfirmed. Additional studies show that comprehenders rationally adapt their predictions in different situations. In the current study, we hypothesized that since the disconfirmation of strong predictions incurs costs, it would also trigger adaptation mechanisms influencing the processing of subsequent (potentially) strong predictions. In two experiments (in Hebrew and English), participants made speeded congruency judgments on two-word phrases in which the first word was either highly constraining (e.g., “climate,” which strongly predicts “change”) or not (e.g., “vegetable,” which does not have any highly probable completion). We manipulated the proportion of disconfirmed predictions in highly constraining contexts between participants. The results provide additional evidence of the costs associated with the disconfirmation of strong predictions. Moreover, they show a reduction in these costs when participants experience a high proportion of disconfirmed strong predictions throughout the experiment, indicating that participants adjust the strength of their predictions when strong prediction is discouraged. We formulate a Bayesian adaptation model whereby prediction failure cost is weighted by the participant’s belief (updated on each trial) about the likelihood of encountering the expected word, and show that it accounts for the trial-by-trial data.

Characterizing multi-word speech production using event-related potentials

Event-related potentials (ERPs) derived from electroencephalography (EEG) have proven useful for understanding linguistic processes during language perception and production. Words are commonly produced in sequences, yet most ERP studies have used single-word experimental designs. Single-word designs reduce potential ERP overlap in word sequence production. However, word sequence production engages brain mechanisms in different ways than single word production. In particular, speech monitoring and planning mechanisms are more engaged than for single words since several words must be produced in a short period of time. This study evaluates the feasibility of recording ERP components in the context of word sequence production, and whether separate components could be isolated for each word. Scalp EEG data were acquired, while participants recited word sequences from memory at a regular pace, using a tongue-twister paradigm. The results revealed fronto-central error-related negativity, previously associated with speech monitoring, which could be distinguished for each word. Its peak amplitude was sensitive to Cycle and Phonological Similarity. However, an effect of sequential production was also observable on baseline measures, indicating baseline shifts throughout the word sequence due to concurrent sustained medial-frontal EEG activity. We also report a late left anterior negativity (LLAN), associated with verbal response planning and execution, onsetting around 100 ms before the first word in each cycle and sustained throughout the rest of the cycle. This work underlines the importance of considering the contribution of transient and sustained EEG activity on ERPs, and provides evidence that ERPs can be used to study sequential word production.

Graded expectations in visually situated comprehension: Costs and benefits as indexed by the N400

Recently, Ankener et al. (Frontiers in Psychology, 9, 2387, 2018) presented a visual world study which combined both attention and pupillary measures to demonstrate that anticipating a target results in lower effort to integrate that target (noun). However, they found no indication that the anticipatory processes themselves, i.e., the reduction of uncertainty about upcoming referents, results in processing effort (cf. Linzen and Jaeger, Cognitive Science, 40(6), 1382-1411, 2016). In contrast, Maess et al. (Frontiers in Human Neuroscience, 10, 1-11, 2016) found that more constraining verbs elicited a higher N400 amplitude than unconstraining verbs. The aim of the present study was therefore twofold: Firstly, we examined whether the graded ICA effect, which was previously found on the noun as a result of a likelihood manipulation, replicates in ERP measures. Secondly, we set out to investigate whether the processes leading to the generation of expectations (derived during verb and scene processing) induce an N400 modulation. Our results confirm that visual context is combined with the verb's meaning to establish expectations about upcoming nouns and that these expectations affect the retrieval of the upcoming noun (modulated N400 on the noun). Importantly, however, we find no evidence for different costs in generating more or less specific expectations for upcoming nouns. Thus, the benefits of generating expectations are not associated with any costs in situated language comprehension.

From pre-activation to pre-updating: A threshold mechanism for commitment to strong predictions

Ample evidence suggests that during sentence processing comprehenders can "pre-activate" lexical/semantic knowledge stored in long-term memory. A relatively recent development suggests that in some cases a stronger form of prediction is employed, involving "pre-updating" the predicted content into the sentence's representation being built in working memory. The current study argues for an activation threshold mechanism by which pre-updating is initiated, within the routine processing stages of a word in a context. By combining a speeded cloze task with event-related potentials, we were able to analyze electrophysiological data measured prior to when participants were prompted to produce a completion, based on the participant's cloze response, reflecting their strongest prediction at that specific moment in time. A P600 effect reflecting pre-updating was observed in high (relative to low) constraint sentences, even in trials where the participant predicted a low cloze word. The results support a mechanism in which multiple predictions accumulate activations, "racing" toward a retrieval threshold. Once the activation level of a certain word passes the threshold, the word is integrated into the sentence representation in working memory. Pre-updating occurs if a certain prediction passes the retrieval threshold prior to its realization in the input.

Neural representation of linguistic feature hierarchy reflects second-language proficiency

Acquiring a new language requires individuals to simultaneously and gradually learn linguistic attributes on multiple levels. Here, we investigated how this learning process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded from native Mandarin speakers with varied English proficiency and from native English speakers while they listened to audio-stories in English. We measured the temporal response functions (TRFs) for acoustic, phonemic, phonotactic, and semantic features in individual participants and found a main effect of proficiency on linguistic encoding. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of “new” phonemic contrasts (i.e., English contrasts that do not exist in Mandarin) with increasing proficiency. Overall, we found that the nonnative listeners with higher proficiency levels had a linguistic feature representation more similar to that of native listeners, which enabled the accurate decoding of language proficiency. This result advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency.

Diferencias temporales y espaciales en el procesamiento de incongruencias léxicas: un estudio de potenciales relacionados a eventos

Introducción. La comprensión del lenguaje está determinada por diversos procesos, entre los que se encuentra el reconocimiento léxico. Según el modelo conexionista, este reconocimiento se genera por activación mediante el emparejamiento de la información acumulada y por la inhibición de las entradas léxicas que compiten por activación.Objetivo. Determinar las diferencias temporales y espaciales de procesamiento entre las incongruencias léxicas semánticamente relacionadas a un contexto lingüístico oracional y las no relacionadas.Materiales y métodos. Se realizó medición de los potenciales relacionados a eventos en 10 sujetos sanos por medio de un paradigma de 240 oraciones en español agrupadas de la siguiente manera: 80 oraciones congruentes, 80 con incongruencias dentro del campo léxico y 80 con incongruencias fuera del campo léxico.Resultados. Se observó una diferencia estadística en la latencia de aparición del componente N400 entre las dos condiciones. Por su parte, se encontró una mayor activación del precúneo, del giro orbitofrontal, del giro angular y del giro supramarginal en la condición de incongruencia fuera del campo léxico.Conclusión. Se encontraron diferencias temporales y espaciales (activación del precúneo, del giro orbitofrontal, del giro angular y del giro supramarginal) entre el procesamiento de las incongruencias léxicas y no léxicas.

An alternative structure rescues failed semantics? Strong global expectancy reduces local-mismatch N400 in Chinese flexible structures

During comprehension of a hierarchical structure, semantic integration between sequential, mismatched sentential constituents does not proceed when the later word in the sequence (e.g., the noun in the verb + classifier + noun) can be assigned an alternative role in the sentence (e.g., as a modifier of a subsequent object noun) (Zhang et al., 2011). Using electrophysiological measures, we examined how the availability of an alternative structure and whether the reader is motivated to generate an alternative structure affect semantic integration in a hierarchical structure. The same set of sentences was employed in three experiments in which the semantic congruency between the adjective and the object noun in the local structure and the sentential-contextual expectancy towards a noun were manipulated simultaneously. The reader's motivation was manipulated by tasks in which they made semantic acceptability judgment (Experiment 1), passively viewed sentences for comprehension (Experiment 2), or actively predicted the upcoming word and monitored the compatibility between the anticipated and the actual input (Experiment 3). Stronger contextual expectancy reduced N400 responses to the object noun regardless of semantic congruency, indicating that strengthening the constraints of an alternative argument decreased the effort of semantic integration between incompatible local constituents. Relative to weaker expectancy, the incongruent noun preceded by a stronger expectancy context elicited reduced N400 responses when no active prediction was demanded but showed equal amplitudes when such a demand was high. These findings demonstrate that the semantic failure in a flexible structure is likely to be resolved by the comprehender's motivation to generate an alternative structure based on the active use of contextual expectancy information, thus highlighting the "role-dependent semantic processing" during sentence comprehension.

Effects of Normative Aging on Eye Movements during Reading

Substantial progress has been made in understanding the mostly detrimental effects of normative aging on eye movements during reading. This article provides a review of research on aging effects on eye movements during reading for different writing systems (i.e., alphabetic systems like English compared to non-alphabetic systems like Chinese), focused on appraising the importance of visual and cognitive factors, considering key methodological issues, and identifying vital questions that need to be addressed and topics for further investigation.

Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (n = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain's electrophysiological index of semantic processing. A spatio-temporally fine-grained mixed-effect multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatio-temporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Individual Differences in Reading Speed are Linked to Variability in the Processing of Lexical and Contextual Information: Evidence from Single-trial Event-related Brain Potentials

In the current paper, we examined the effects of lexical (e.g. word frequency, orthographic neighborhood density) and contextual (e.g. word predictability in the form of cloze probability) features on single-trial event-related brain potentials in a self-paced reading paradigm. Critically, we examined whether individual differences in reading speed modulated single-trial effects on the N400, an ERP component linked to semantic memory access. Consistent with past work, we found that word frequency effects on the N400 were attenuated with increasing predictability. However, effects of orthographic neighborhood density were robust across all levels of predictability. Importantly, individual differences in reading speed moderated the influence of both frequency and predictability (but not orthographic neighborhood density) on the N400, such that slower readers showed reduced effects compared to faster readers. These data show that different lexical factors influence word processing through dissociable mechanisms. Our findings support a dynamic semantic-memory access model of the N400, in which information at multiple levels (lexical, sentential, individual) simultaneously contributes to the unfolding neural dynamics of comprehension.

Investigating the Comprehension of Negated Sentences Employing World Knowledge: An Event-Related Potential Study

Previous event-related potential (ERP) studies comparing affirmative and negative sentences revealed an N400 for semantically mismatching final words, resulting in a larger N400 for false relative to true affirmative sentences and an opposite effect for negative sentences. Hence, the N400 was independent of the presence of a negation. However, the true negative as well as the false affirmative condition often contained entities or features from different semantic categories and thereby with weak feature overlap, such as e.g., A cat is (not) a saw or Fears are (not) round, which were then compared to true affirmative and false negative sentences containing entities with stronger feature overlap and partially even hyponomy relations, e.g., A cat is (not) an animal or Planets are (not) round. Employing world-knowledge variations, in the current study, we investigate whether increasing the feature overlap between the entities of all conditions leads to similar ERP-patterns as in the previous studies. For this purpose, we use sentences of the following type: George Clooney is (not) an actor vs. George Clooney is (not) a singer where both target words describe a similar profession and thereby function as alternatives to each other. However, in line with the previous studies, we find a truth by polarity interaction, namely, the N400 ERPs are significantly larger for false compared to true affirmative sentences, whereas the effect for negative sentences shows a reversed, though not significant, trend. Overall, the ERP-data suggest that the integration of a negation with the information in its scope is neither fully incremental nor fully delayed, which might be linked to the use of cohyponyms and to the increased feature overlap between alternatives (e.g., actor, singer). Additionally, questionnaire-based rating data show that affirmative sentences are perceived as more natural than negative sentences, and, moreover, that true sentences are perceived as more natural than false sentences, independent of their polarity.

To believe or not to believe? How voice and accent information in speech alter listener impressions of trust

Our decision to believe what another person says can be influenced by vocally expressed confidence in speech and by whether the speaker–listener are members of the same social group. The dynamic effects of these two information sources on neurocognitive processes that promote believability impressions from vocal cues are unclear. Here, English Canadian listeners were presented personal statements ( She has access to the building) produced in a confident or doubtful voice by speakers of their own dialect (in-group) or speakers from two different “out-groups” (regional or foreign-accented English). Participants rated how believable the speaker is for each statement and event-related potentials (ERPs) were analysed from utterance onset. Believability decisions were modulated by both the speaker’s vocal confidence level and their perceived in-group status. For in-group speakers, ERP effects revealed an early differentiation of vocally expressed confidence (i.e., N100, P200), highlighting the motivational significance of doubtful voices for drawing believability inferences. These early effects on vocal confidence perception were qualitatively different or absent when speakers had an accent; evaluating out-group voices was associated with increased demands on contextual integration and re-analysis of a non-native representation of believability (i.e., increased N400, late negativity response). Accent intelligibility and experience with particular out-group accents each influenced how vocal confidence was processed for out-group speakers. The N100 amplitude was sensitive to out-group attitudes and predicted actual believability decisions for certain out-group speakers. We propose a neurocognitive model in which vocal identity information (social categorization) dynamically influences how vocal expressions are decoded and used to derive social inferences during person perception.

Event-related brain potentials reveal how multiple aspects of semantic processing unfold across parafoveal and foveal vision during sentence reading

Recent event-related brain potential (ERP) experiments have demonstrated parafoveal N400 expectancy and congruity effects, showing that semantic information can be accessed from words in parafoveal vision (a conclusion also supported by some eye-tracking work). At the same time, it is unclear how higher-order integrative aspects of language comprehension unfold across the visual field during reading. In the current study, we recorded ERPs in a parafoveal flanker paradigm, while readers were instructed to read passively for comprehension or to judge the plausibility of sentences in which target words varied in their semantic expectancy and congruity. We directly replicated prior work showing graded N400 effects for parafoveal viewing, which are then not duplicated when the target words are processed foveally. Critically, although N400 effects were not modulated by task goals, a posteriorly distributed late positive component thought to reflect semantic integration processes was observed to semantic incongruities only in the plausibility judgment task. However, this effect was observed at a considerable delay, appearing only after words had moved into foveal vision. Our findings thus suggest that semantic access can be initiated in parafoveal vision, whereas central foveal vision may be necessary to enact higher-order (and task-dependent) integrative processing.

Neural evidence for Bayesian trial-by-trial adaptation on the N400 during semantic priming

When semantic information is activated by a context prior to new bottom-up input (i.e. when a word is predicted), semantic processing of that incoming word is typically facilitated, attenuating the amplitude of the N400 event related potential (ERP) - a direct neural measure of semantic processing. N400 modulation is observed even when the context is a single semantically related "prime" word. This so-called "N400 semantic priming effect" is sensitive to the probability of encountering a related prime-target pair within an experimental block, suggesting that participants may be adapting the strength of their predictions to the predictive validity of their broader experimental environment. We formalize this adaptation using a Bayesian learning model that estimates and updates the probability of encountering a related versus an unrelated prime-target pair on each successive trial. We found that our model's trial-by-trial estimates of target word probability accounted for significant variance in trial-by-trial N400 amplitude. These findings suggest that Bayesian principles contribute to how comprehenders adapt their semantic predictions to the statistical structure of their broader environment, with implications for the functional significance of the N400 component and the predictive nature of language processing.

Oscillatory dynamics of cortical functional connections in semantic prediction

An event related potential, known as the N400, has been particularly useful in investigating language processing as it serves as a neural index for semantic prediction. There are numerous studies on the functional segregation of N400 neural sources; however, the oscillatory dynamics of functional connections among the relevant sources has remained elusive. In this study we acquired magnetoencephalography data during a classic N400 paradigm, where the semantic predictability of a fixed target noun was manipulated in simple German sentences. We conducted inter-regional functional connectivity (FC) and time-frequency analysis on known regions of the semantic network, encompassing bilateral temporal, and prefrontal cortices. Increased FC was found in less predicted (LP) nouns compared with highly predicted (HP) nouns in three connections: (a) right inferior frontal gyrus (IFG) and right middle temporal gyrus (MTG) from 0 to 300 ms mainly within the alpha band, (b) left lateral orbitofrontal (LOF) and right IFG around 400 ms within the beta band, and (c) left superior temporal gyrus (STG) and left LOF from 300 to 700 ms in the beta and low gamma bands. Furthermore, gamma spectral power (31-70 Hz) was stronger in HP nouns than in LP nouns in left anterior temporal cortices in earlier time windows (0-200 ms). Our findings support recent theories in language comprehension, suggesting fronto-temporal top-down connections are mainly mediated through beta oscillations while gamma band frequencies are involved in matching between prediction and input.

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Visual narratives like comics involve a range of complex cognitive operations in order to be understood. The Parallel Interfacing Narrative‐Semantics (PINS) Model integrates an emerging literature showing that comprehension of wordless image sequences balances two representational levels of semantic and narrative structure. The neurocognitive mechanisms that guide these processes are argued to overlap with other domains, such as language and music.