What do we mean by prediction in language comprehension?

Looking for immediate and downstream evidence of lexical prediction in eye movements during reading

Previous investigations of whether readers make predictions about the full identity of upcoming words have focused on the extent to which there are processing consequences when readers encounter linguistic input that is incompatible with their expectations. To date, eye-movement studies have revealed inconsistent evidence of the processing costs that would be expected to accompany lexical prediction. This study investigated whether readers' lexical predictions were observable during or downstream from their initial point of activation. Three experiments assessed readers' eye movements to predictable and unpredictable words, and then to subsequent downstream words, which probed the lingering activation of previously expected words. The results showed novel evidence of processing costs for unexpected input but only when supported by a plausible linguistic environment, suggesting that readers could strategically modulate their predictive processing. However, there was limited evidence that their lexical predictions affected downstream processing. The implications of these findings for understanding the role of prediction in language processing are discussed.

Language models outperform cloze predictability in a cognitive model of reading

Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear. This study investigates whether large language models (LLMs) can fill this gap. Contextual predictions are implemented via a novel parallel-graded mechanism, where all predicted words at a given position are pre-activated as a function of contextual certainty, which varies dynamically as text processing unfolds. Through reading simulations with OB1-reader, a cognitive model of word recognition and eye-movement control in reading, we compare the model's fit to eye-movement data when using predictability values derived from a cloze task against those derived from LLMs (GPT-2 and LLaMA). Root Mean Square Error between simulated and human eye movements indicates that LLM predictability provides a better fit than cloze. This is the first study to use LLMs to augment a cognitive model of reading with higher-order language processing while proposing a mechanism on the interplay between word predictability and eye movements.

Is there a cost when predictions are not met? A VWP study investigating L1 and L2 speakers

Research has found that both first language (L1) and second language (L2) speakers make predictions about upcoming linguistic information, with predictive behaviour being impacted by individual differences and methodological factors. However, it is not clear whether a cost is incurred when a prediction is made, but not met. L2 speakers have less experience with their L2 and parsing can be cognitively demanding, which together may lead L2 speakers to incur prediction costs differently relative to L1 speakers. In this study using the visual world paradigm, we test whether L1 and L2 speakers predict in the same way, within the same time frame, and incur the same costs if predictions are not met. We also explore the role of proficiency and speech rate. We found that both groups predict in a similar way and within a similar time frame. In addition, neither group incurred a prediction cost when the target was the most likely alternative, though L2 speakers take longer to shift their attention to the target object when predictions are not met. We argue that this reflects a slowing of lexical access rather than a specific cost of prediction. We only found prediction differences when speech rate was included in the analysis, highlighting the importance of attending to speech rate in studies using the visual world paradigm. Overall, this study supports research showing that both L1 and L2 speakers may make multiple partial predictions about upcoming information rather than predicting one specific lexical candidate while inhibiting less likely lexical candidates.

Uses of Linguistic Context in Speech Listening: Does Acquired Hearing Loss Lead to Reduced Engagement of Prediction?

Research investigating the complex interplay of cognitive mechanisms involved in speech listening for people with hearing loss has been gaining prominence. In particular, linguistic context allows the use of several cognitive mechanisms that are not well distinguished in hearing science, namely those relating to "postdiction", "integration", and "prediction". We offer the perspective that an unacknowledged impact of hearing loss is the differential use of predictive mechanisms relative to age-matched individuals with normal hearing. As evidence, we first review how degraded auditory input leads to reduced prediction in people with normal hearing, then consider the literature exploring context use in people with acquired postlingual hearing loss. We argue that no research on hearing loss has directly assessed prediction. Because current interventions for hearing do not fully alleviate difficulty in conversation, and avoidance of spoken social interaction may be a mediator between hearing loss and cognitive decline, this perspective could lead to greater understanding of cognitive effects of hearing loss and provide insight regarding new targets for intervention.

Prior Context and Individual Alpha Frequency Influence Predictive Processing during Language Comprehension

The extent to which the brain predicts upcoming information during language processing remains controversial. To shed light on this debate, the present study reanalyzed Nieuwland and colleagues' (2018) [Nieuwland, M. S., Politzer-Ahles, S., Heyselaar, E., Segaert, K., Darley, E., Kazanina, N., et al. Large-scale replication study reveals a limit on probabilistic prediction in language comprehension. eLife, 7, e33468, 2018] replication of DeLong and colleagues (2015) [DeLong, K. A., Urbach, T. P., & Kutas, M. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience, 8, 1117-1121, 2005]. Participants (n = 356) viewed sentences containing articles and nouns of varying predictability, while their EEG was recorded. We measured ERPs preceding the critical words (namely, the semantic prediction potential), in conjunction with postword N400 patterns and individual neural metrics. ERP activity was compared with two measures of word predictability: cloze probability and lexical surprisal. In contrast to prior literature, semantic prediction potential amplitudes did not increase as cloze probability increased, suggesting that the component may not reflect prediction during natural language processing. Initial N400 results at the article provided evidence against phonological prediction in language, in line with Nieuwland and colleagues' findings. Strikingly, however, when the surprisal of the prior words in the sentence was included in the analysis, increases in article surprisal were associated with increased N400 amplitudes, consistent with prediction accounts. This relationship between surprisal and N400 amplitude was not observed when the surprisal of the two prior words was low, suggesting that expectation violations at the article may be overlooked under highly predictable conditions. Individual alpha frequency also modulated the relationship between article surprisal and the N400, emphasizing the importance of individual neural factors for prediction. The present study extends upon existing neurocognitive models of language and prediction more generally, by illuminating the flexible and subject-specific nature of predictive processing.

From Infancy to Childhood: A Comprehensive Review of Event- and Task-Related Brain Oscillations

Brain development from infancy through childhood involves complex structural and functional changes influenced by both internal and external factors. This review provides a comprehensive analysis of event and task-related brain oscillations, focusing on developmental changes across different frequency bands, including delta, theta, alpha, beta, and gamma. Electroencephalography (EEG) studies highlight that these oscillations serve as functional building blocks for sensory and cognitive processes, with significant variations observed across different developmental stages. Delta oscillations, primarily associated with deep sleep and early cognitive demands, gradually diminish as children age. Theta rhythms, crucial for attention and memory, display a distinct pattern in early childhood, evolving with cognitive maturation. Alpha oscillations, reflecting thalamocortical interactions and cognitive performance, increase in complexity with age. Beta rhythms, linked to active thinking and problem-solving, show developmental differences in motor and cognitive tasks. Gamma oscillations, associated with higher cognitive functions, exhibit notable changes in response to sensory stimuli and cognitive tasks. This review underscores the importance of understanding oscillatory dynamics to elucidate brain development and its implications for sensory and cognitive processing in childhood. The findings provide a foundation for future research on developmental neuroscience and potential clinical applications.

Exploring Pragmatic Factors on the Logical Relationships of Conditional Reasoning: A Study of Counterfactual and Hypothetical Conditionals

Previous theories have established the mental model activation of processing different types of conditionals, stating that counterfactual conditionals expressing events that contradict known facts (e.g., "If it had rained, then they would not go to the park.") are considered to trigger two mental models: (1) a hypothetical but factually wrong model (e.g., "rain" and "did not go to the park") and (2) a corresponding real-world model (e.g., "did not rain" and "went to the park"). This study aimed to investigate whether pragmatic factors differentially influence readers' comprehension and distinction between counterfactual and hypothetical conditional sentences in Mandarin Chinese. Participants were required to read and judge the comprehensibility of Chinese hypothetical and counterfactual conditionals, which were different in temporal indicators (past vs. future temporal indicators) in the antecedent. Different polarities (with vs. without negators) and different moving directions (different directional verbs: lai2 [come] vs. qu4 [go]) in the consequent were also manipulated. Linear mixed-effects models (LMEM) revealed that hypothetical conditionals (with future temporal indicators) were more comprehensible than counterfactual conditionals (with past temporal indicators). The semantic similarities within the subordinate clause revealed future temporal indicators had higher lexical-semantic co-occurrence than past indicators, suggesting that temporal indicators impact comprehension partly through lexical semantics in the premise, and hypothetical conditionals are more easily processed. However, the semantic similarity analysis of the main and the subordinate clauses showed no effect of temporal indicators, suggesting that lexical-semantic co-occurrence across clauses may not substantially contribute to the distinction between hypothetical conditionals and counterfactual conditionals. In conclusion, this study offers insights into the comprehension of Chinese conditional sentences by shedding light on the pragmatic factors influencing the activation of different mental models.

Language in Brains, Minds, and Machines

It has long been argued that only humans could produce and understand language. But now, for the first time, artificial language models (LMs) achieve this feat. Here we survey the new purchase LMs are providing on the question of how language is implemented in the brain. We discuss why, a priori, LMs might be expected to share similarities with the human language system. We then summarize evidence that LMs represent linguistic information similarly enough to humans to enable relatively accurate brain encoding and decoding during language processing. Finally, we examine which LM properties-their architecture, task performance, or training-are critical for capturing human neural responses to language and review studies using LMs as in silico model organisms for testing hypotheses about language. These ongoing investigations bring us closer to understanding the representations and processes that underlie our ability to comprehend sentences and express thoughts in language.

Understanding words in context: A naturalistic EEG study of children's lexical processing

When listening to speech, adults rely on context to anticipate upcoming words. Evidence for this comes from studies demonstrating that the N400, an event-related potential (ERP) that indexes ease of lexical-semantic processing, is influenced by the predictability of a word in context. We know far less about the role of context in children's speech comprehension. The present study explored lexical processing in adults and 5-10-year-old children as they listened to a story. ERPs time-locked to the onset of every word were recorded. Each content word was coded for frequency, semantic association, and predictability. In both children and adults, N400s reflect word predictability, even when controlling for frequency and semantic association. These findings suggest that both adults and children use top-down constraints from context to anticipate upcoming words when listening to stories.

Individual differences in visual pattern completion predict adaptation to degraded speech

Recognizing acoustically degraded speech relies on predictive processing whereby incomplete auditory cues are mapped to stored linguistic representations via pattern recognition processes. While listeners vary in their ability to recognize degraded speech, performance improves when a written transcription is presented, allowing completion of the partial sensory pattern to preexisting representations. Building on work characterizing predictive processing as pattern completion, we examined the relationship between domain-general pattern recognition and individual variation in degraded speech learning. Participants completed a visual pattern recognition task to measure individual-level tendency towards pattern completion. Participants were also trained to recognize noise-vocoded speech with written transcriptions and tested on speech recognition pre- and post-training using a retrieval-based transcription task. Listeners significantly improved in recognizing speech after training, and pattern completion on the visual task predicted improvement for novel items. The results implicate pattern completion as a domain-general learning mechanism that can facilitate speech adaptation in challenging contexts.

The prediction potential indexes the meaning and communicative function of upcoming utterances

Prediction has a fundamental role in language processing. However, predictions can be made at different levels, and it is not always clear whether speech sounds, morphemes, words, meanings, or communicative functions are anticipated during dialogues. Previous studies reported specific brain signatures of communicative pragmatic function, in particular enhanced brain responses immediately after encountering an utterance used to request an object from a partner, but relatively smaller ones when the same utterance was used for naming the object. The present experiment now investigates whether similar neuropragmatic signatures emerge in recipients before the onset of upcoming utterances carrying different predictable communicative functions. Trials started with a context question and object pictures displayed on the screen, raising the participant's expectation that words from a specific semantic category (food or tool) would subsequently be used to either name or request one of the objects. Already 600 msec before utterance onset, a larger prediction potential was observed when a request was anticipated relative to naming expectation. As this result is congruent with the neurophysiological difference previously observed right after the critical utterance, the anticipatory brain activity may index predictions about the social-communicative function of upcoming utterances. In addition, we also found that the predictable semantic category of the upcoming word was likewise reflected in the anticipatory brain potential. Thus, the neurophysiological characteristics of the prediction potential can capture different types of upcoming linguistic information, including semantic and pragmatic aspects of an upcoming utterance and communicative action.

Cross-linguistic syntactic priming as rational expectation for syntactic repetition in the bilingual environment

Recent research suggests that syntactic priming in language comprehension-the facilitated processing of repeated syntactic structures-arises from the expectation for syntactic repetition due to rational adaptation to the linguistic environment. To further evaluate the generalizability of this expectation adaptation account in cross-linguistic syntactic priming and explore the influence of second language (L2) proficiency, we conducted a self-paced reading study with Chinese L2 learners of English by utilizing the sentential complement-direct object (SC-DO) ambiguity. The results showed that participants exposed to clusters of SC structures subsequently processed repetitions of this structure more rapidly (i.e., larger priming effects) than those exposed to the same number of SC structures but spaced in time, despite the prime and target being in two different languages (Chinese and English). Furthermore, this difference in priming strength was more pronounced for participants with higher L2 (English) proficiency. These findings demonstrate that cross-linguistic syntactic priming is consistent with the expectation for syntactic repetition that rationally adapts to syntactic clustering properties in surrounding bilingual environments, and such adaptation is enhanced as L2 proficiency increases. Taken together, our study extends the expectation adaptation account to cross-linguistic syntactic priming and integrates the role of L2 proficiency, which can shed new light on the mechanisms underlying syntactic priming, bilingual shared syntactic representations and expectation-based sentence processing.

Reward-related regions play a role in natural story comprehension

The reward system was shown to be involved in a wide array of processes. Nevertheless, the exploration of the involvement of the reward system during language processing has not yet been directly tested. We investigated the role of reward-processing regions while listening to a natural story. We utilized a published dataset in which half of the participants listened to a natural story and the others listened to a scrambled version of it to compare the functional MRI signals in the reward system between these conditions and discovered a distinct pattern between conditions. This suggests that the reward system is activated during the comprehension of natural stories. We also show evidence that the fMRI signals in reward-related areas might potentially correlate with the predictability level of processed sentences. Further research is needed to determine the nature of the involvement and the way the activity interacts with various aspects of the sentences.

Linguistic inputs must be syntactically parsable to fully engage the language network

Human language comprehension is remarkably robust to ill-formed inputs (e.g., word transpositions). This robustness has led some to argue that syntactic parsing is largely an illusion, and that incremental comprehension is more heuristic, shallow, and semantics-based than is often assumed. However, the available data are also consistent with the possibility that humans always perform rule-like symbolic parsing and simply deploy error correction mechanisms to reconstruct ill-formed inputs when needed. We put these hypotheses to a new stringent test by examining brain responses to a) stimuli that should pose a challenge for syntactic reconstruction but allow for complex meanings to be built within local contexts through associative/shallow processing (sentences presented in a backward word order), and b) grammatically well-formed but semantically implausible sentences that should impede semantics-based heuristic processing. Using a novel behavioral syntactic reconstruction paradigm, we demonstrate that backward-presented sentences indeed impede the recovery of grammatical structure during incremental comprehension. Critically, these backward-presented stimuli elicit a relatively low response in the language areas, as measured with fMRI. In contrast, semantically implausible but grammatically well-formed sentences elicit a response in the language areas similar in magnitude to naturalistic (plausible) sentences. In other words, the ability to build syntactic structures during incremental language processing is both necessary and sufficient to fully engage the language network. Taken together, these results provide strongest to date support for a generalized reliance of human language comprehension on syntactic parsing.

Exploring neural tracking of acoustic and linguistic speech representations in individuals with post-stroke aphasia

Aphasia is a communication disorder that affects processing of language at different levels (e.g., acoustic, phonological, semantic). Recording brain activity via Electroencephalography while people listen to a continuous story allows to analyze brain responses to acoustic and linguistic properties of speech. When the neural activity aligns with these speech properties, it is referred to as neural tracking. Even though measuring neural tracking of speech may present an interesting approach to studying aphasia in an ecologically valid way, it has not yet been investigated in individuals with stroke-induced aphasia. Here, we explored processing of acoustic and linguistic speech representations in individuals with aphasia in the chronic phase after stroke and age-matched healthy controls. We found decreased neural tracking of acoustic speech representations (envelope and envelope onsets) in individuals with aphasia. In addition, word surprisal displayed decreased amplitudes in individuals with aphasia around 195 ms over frontal electrodes, although this effect was not corrected for multiple comparisons. These results show that there is potential to capture language processing impairments in individuals with aphasia by measuring neural tracking of continuous speech. However, more research is needed to validate these results. Nonetheless, this exploratory study shows that neural tracking of naturalistic, continuous speech presents a powerful approach to studying aphasia.

What eye and hand movements tell us about expectations towards argument order: An eye- and mouse-tracking study in German

Previous research on real-time sentence processing in German has shown that listeners use the morphological marking of accusative case on a sentence-initial noun phrase to not only interpret the current argument as the object and patient, but also to predict a plausible agent. So far, less is known about the use of case marking to predict the semantic role of upcoming arguments after the subject/agent has been encountered. In the present study, we examined the use of case marking for argument interpretation in transitive as well as ditransitive structures. We aimed to control for multiple factors that could have influenced processing in previous studies, including the animacy of arguments, world knowledge, and the perceptibility of the case cue. Our results from eye- and mouse-tracking indicate that the exploitation of the first case cue that enables the interpretation of the unfolding sentence is influenced by (i) the strength of argument order expectation and (ii) the perceptual salience of the case cue. PsycINFO code: 2720 Linguistics & Language & Speech.

Predictive processing of music and language in autism: Evidence from Mandarin and English speakers

Atypical predictive processing has been associated with autism across multiple domains, based mainly on artificial antecedents and consequents. As structured sequences where expectations derive from implicit learning of combinatorial principles, language and music provide naturalistic stimuli for investigating predictive processing. In this study, we matched melodic and sentence stimuli in cloze probabilities and examined musical and linguistic prediction in Mandarin- (Experiment 1) and English-speaking (Experiment 2) autistic and non-autistic individuals using both production and perception tasks. In the production tasks, participants listened to unfinished melodies/sentences and then produced the final notes/words to complete these items. In the perception tasks, participants provided expectedness ratings of the completed melodies/sentences based on the most frequent notes/words in the norms. While Experiment 1 showed intact musical prediction but atypical linguistic prediction in autism in the Mandarin sample that demonstrated imbalanced musical training experience and receptive vocabulary skills between groups, the group difference disappeared in a more closely matched sample of English speakers in Experiment 2. These findings suggest the importance of taking an individual differences approach when investigating predictive processing in music and language in autism, as the difficulty in prediction in autism may not be due to generalized problems with prediction in any type of complex sequence processing.

Predictive language processing: integrating comprehension and production, and what atypical populations can tell us

Predictive processing, a crucial aspect of human cognition, is also relevant for language comprehension. In everyday situations, we exploit various sources of information to anticipate and therefore facilitate processing of upcoming linguistic input. In the literature, there are a variety of models that aim at accounting for such ability. One group of models propose a strict relationship between prediction and language production mechanisms. In this review, we first introduce very briefly the concept of predictive processing during language comprehension. Secondly, we focus on models that attribute a prominent role to language production and sensorimotor processing in language prediction ("prediction-by-production" models). Contextually, we provide a summary of studies that investigated the role of speech production and auditory perception on language comprehension/prediction tasks in healthy, typical participants. Then, we provide an overview of the limited existing literature on specific atypical/clinical populations that may represent suitable testing ground for such models-i.e., populations with impaired speech production and auditory perception mechanisms. Ultimately, we suggest a more widely and in-depth testing of prediction-by-production accounts, and the involvement of atypical populations both for model testing and as targets for possible novel speech/language treatment approaches.

A predictive coding model of the N400

The N400 event-related component has been widely used to investigate the neural mechanisms underlying real-time language comprehension. However, despite decades of research, there is still no unifying theory that can explain both its temporal dynamics and functional properties. In this work, we show that predictive coding - a biologically plausible algorithm for approximating Bayesian inference - offers a promising framework for characterizing the N400. Using an implemented predictive coding computational model, we demonstrate how the N400 can be formalized as the lexico-semantic prediction error produced as the brain infers meaning from the linguistic form of incoming words. We show that the magnitude of lexico-semantic prediction error mirrors the functional sensitivity of the N400 to various lexical variables, priming, contextual effects, as well as their higher-order interactions. We further show that the dynamics of the predictive coding algorithm provides a natural explanation for the temporal dynamics of the N400, and a biologically plausible link to neural activity. Together, these findings directly situate the N400 within the broader context of predictive coding research. More generally, they raise the possibility that the brain may use the same computational mechanism for inference across linguistic and non-linguistic domains.

Neurophysiological signatures of prediction in language: A critical review of anticipatory negativities

Recent event-related potential (ERP) studies in language comprehension converge in finding anticipatory negativities preceding words or word segments that can be pre-activated based on either sentence contexts or phonological cues. We review these findings from different paradigms in the light of evidence from other cognitive domains in which slow negative potentials have long been associated with anticipatory processes and discuss their potential underlying mechanisms. We propose that this family of anticipatory negativities captures common mechanisms associated with the pre-activation of linguistic information both within words and within sentences. Future studies could utilize these anticipatory negativities in combination with other, well-established ERPs, to simultaneously track prediction-related processes emerging at different time intervals (before and after the perception of pre-activated input) and with distinct time courses (shorter-lived and longer-lived cognitive operations).

A neuro-cognitive model of comprehension based on prediction and unification

Most architectures and models of language processing have been built upon a restricted view of language, which is limited to sentence processing. These approaches fail to capture one primordial characteristic: efficiency. Many facilitation effects are known to be at play in natural situations such as conversation (shallow processing, no real access to the lexicon, etc.) without any impact on the comprehension. In this study, on the basis of a new model integrating into a unique architecture, we present these facilitation effects for accessing the meaning into the classical compositional architecture. This model relies on two mechanisms, prediction and unification, and provides a unique architecture for the description of language processing in its natural environment.

Used to Be a Dime, Now It's a Dollar: Revised Speech Perception in Noise Key Word Predictability Revisited 40 Years On

PURPOSE

Almost 40 years after its development, in this article, we reexamine the relevance and validity of the ubiquitously used Revised Speech Perception in Noise (R-SPiN) sentence corpus. The R-SPiN corpus includes "high-context" and "low-context" sentences and has been widely used in the field of hearing research to examine the benefit derived from semantic context across English-speaking listeners, but research investigating age differences has yielded somewhat inconsistent findings. We assess the appropriateness of the corpus for use today in different English-language cultures (i.e., British and American) as well as for older and younger adults.

METHOD

Two hundred forty participants, including older (60-80 years) and younger (19-31 years) adult groups in the the United Kingdom and United States, completed a cloze task consisting of R-SPiN sentences with the final word removed. Cloze, as a measure of predictability, and entropy, as a measure of response uncertainty, were compared between culture and age groups.

RESULTS

Most critically, of the 200 "high-context" stimuli, only around half were assessed as highly predictable for older adults (United Kingdom: 109; United States: 107); and fewer still, for younger adults (United Kingdom: 75; United States: 81). We also found dominant responses to these "high-context" stimuli varied between cultures, with U.S. responses being more likely to match the original R-SPiN target.

CONCLUSIONS

Our findings highlight the issue of incomplete transferability of corpus items across English-language cultures as well as diminished equivalency for older and younger adults. By identifying relevant items for each population, this work could facilitate the interpretation of inconsistent findings in the literature, particularly relating to age effects.

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Online eye tracking and real-time sentence processing: On opportunities and efficacy for capturing psycholinguistic effects of different magnitudes and diversity

Online research methods have the potential to facilitate equitable accessibility to otherwise-expensive research resources, as well as to more diverse populations and language combinations than currently populate our studies. In psycholinguistics specifically, webcam-based eye tracking is emerging as a powerful online tool capable of capturing sentence processing effects in real time. The present paper asks whether webcam-based eye tracking provides the necessary granularity to replicate effects-crucially both large and small-that tracker-based eye tracking has shown. Using the Gorilla Experiment Builder platform, this study set out to replicate two psycholinguistic effects: a robust one, the verb semantic constraint effect, first reported in Altmann and Kamide,  Cognition 73(3), 247-264 (1999), and a smaller one, the lexical interference effect, first examined by Kukona et al. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(2), 326 (2014). Webcam-based eye tracking was able to replicate both effects, thus showing that its functionality is not limited to large effects. Moreover, the paper also reports two approaches to computing statistical power and discusses the differences in their outputs. Beyond discussing several important methodological, theoretical, and practical implications, we offer some further technical details and advice on how to implement webcam-based eye-tracking studies. We believe that the advent of webcam-based eye tracking, at least in respect of the visual world paradigm, will kickstart a new wave of more diverse studies with more diverse populations.

Dissociating the pre-activation of word meaning and form during sentence comprehension: Evidence from EEG representational similarity analysis

During language comprehension, the processing of each incoming word is facilitated in proportion to its predictability. Here, we asked whether anticipated upcoming linguistic information is actually pre-activated before new bottom-up input becomes available, and if so, whether this pre-activation is limited to the level of semantic features, or whether extends to representations of individual word-forms (orthography/phonology). We carried out Representational Similarity Analysis on EEG data while participants read highly constraining sentences. Prior to the onset of the expected target words, sentence pairs predicting semantically related words (financial "bank" - "loan") and form-related words (financial "bank" - river "bank") produced more similar neural patterns than pairs predicting unrelated words ("bank" - "lesson"). This provides direct neural evidence for item-specific semantic and form predictive pre-activation. Moreover, the semantic pre-activation effect preceded the form pre-activation effect, suggesting that top-down pre-activation is propagated from higher to lower levels of the linguistic hierarchy over time.

Filling the gap: Cloze probability and sentence constraint norms for 807 European Portuguese sentences

Sentence processing is affected by the sentence context and word expectancy. To investigate sentence comprehension experimentally, it is useful to have sentence completion norms with both context constraint and word expectancy measures. In this study, two experiments were conducted to collect norms for completion of 807 European Portuguese sentences. Context constraint was measured through type-token ratio and proportion idiosyncratic responses, while word expectancy was assessed by cloze probability. Besides establishing norms for a large sample of sentences, the study investigated the impact of the production procedure and method of analysis. In Experiment 1, a single-production procedure was used, i.e., participants completed each sentence fragment with only a single response, whereas in Experiment 2, a multiple-production procedure was used, i.e., participants have to provide up to three completion words for each sentence fragment. In Experiment 2, the analyses were obtained using two distinct methods: first-response analysis and combined-response analysis. The results showed that cloze and context measures are comparable between production paradigms and that the results from both analysis methods were correlated. The advantages of each production procedure and analysis method are discussed.

Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data

Many recent studies have shown that representations drawn from neural network language models are extremely effective at predicting brain responses to natural language. But why do these models work so well? One proposed explanation is that language models and brains are similar because they have the same objective: to predict upcoming words before they are perceived. This explanation is attractive because it lends support to the popular theory of predictive coding. We provide several analyses that cast doubt on this claim. First, we show that the ability to predict future words does not uniquely (or even best) explain why some representations are a better match to the brain than others. Second, we show that within a language model, representations that are best at predicting future words are strictly worse brain models than other representations. Finally, we argue in favor of an alternative explanation for the success of language models in neuroscience: These models are effective at predicting brain responses because they generally capture a wide variety of linguistic phenomena.

Surprisal From Language Models Can Predict ERPs in Processing Predicate-Argument Structures Only if Enriched by an Agent Preference Principle

Language models based on artificial neural networks increasingly capture key aspects of how humans process sentences. Most notably, model-based surprisals predict event-related potentials such as N400 amplitudes during parsing. Assuming that these models represent realistic estimates of human linguistic experience, their success in modeling language processing raises the possibility that the human processing system relies on no other principles than the general architecture of language models and on sufficient linguistic input. Here, we test this hypothesis on N400 effects observed during the processing of verb-final sentences in German, Basque, and Hindi. By stacking Bayesian generalised additive models, we show that, in each language, N400 amplitudes and topographies in the region of the verb are best predicted when model-based surprisals are complemented by an Agent Preference principle that transiently interprets initial role-ambiguous noun phrases as agents, leading to reanalysis when this interpretation fails. Our findings demonstrate the need for this principle independently of usage frequencies and structural differences between languages. The principle has an unequal force, however. Compared to surprisal, its effect is weakest in German, stronger in Hindi, and still stronger in Basque. This gradient is correlated with the extent to which grammars allow unmarked NPs to be patients, a structural feature that boosts reanalysis effects. We conclude that language models gain more neurobiological plausibility by incorporating an Agent Preference. Conversely, theories of human processing profit from incorporating surprisal estimates in addition to principles like the Agent Preference, which arguably have distinct evolutionary roots.

I know how you'll say it: evidence of speaker-specific speech prediction

Most models of language comprehension assume that the linguistic system is able to pre-activate phonological information. However, the evidence for phonological prediction is mixed and controversial. In this study, we implement a paradigm that capitalizes on the fact that foreign speakers usually make phonological errors. We investigate whether speaker identity (native vs. foreign) is used to make specific phonological predictions. Fifty-two participants were recruited to read sentence frames followed by a last spoken word which was uttered by either a native or a foreign speaker. They were required to perform a lexical decision on the last spoken word, which could be either semantically predictable or not. Speaker identity (native vs. foreign) may or may not be cued by the face of the speaker. We observed that the face cue is effective in speeding up the lexical decision when the word is predictable, but it is not effective when the word is not predictable. This result shows that speech prediction takes into account the phonological variability between speakers, suggesting that it is possible to pre-activate in a detailed and specific way the phonological representation of a predictable word.

Language prediction in monolingual and bilingual speakers: an EEG study

Prediction of upcoming words is thought to be crucial for language comprehension. Here, we are asking whether bilingualism entails changes to the electrophysiological substrates of prediction. Prior findings leave it open whether monolingual and bilingual speakers predict upcoming words to the same extent and in the same manner. We address this issue with a naturalistic approach, employing an information-theoretic metric, surprisal, to predict and contrast the N400 brain potential in monolingual and bilingual speakers. We recruited 18 Iranian Azeri-Persian bilingual speakers and 22 Persian monolingual speakers. Subjects listened to a story in Persian while their electroencephalogram (EEG) was recorded. Bayesian item-level analysis was used. While in monolingual speakers N400 was sensitive to information-theoretic properties of both the current and previous words, in bilingual speakers N400 reflected the properties of the previous word only. Our findings show evidence for a processing delay in bilingual speakers which is consistent with prior research.

Large-scale evidence for logarithmic effects of word predictability on reading time

During real-time language comprehension, our minds rapidly decode complex meanings from sequences of words. The difficulty of doing so is known to be related to words' contextual predictability, but what cognitive processes do these predictability effects reflect? In one view, predictability effects reflect facilitation due to anticipatory processing of words that are predictable from context. This view predicts a linear effect of predictability on processing demand. In another view, predictability effects reflect the costs of probabilistic inference over sentence interpretations. This view predicts either a logarithmic or a superlogarithmic effect of predictability on processing demand, depending on whether it assumes pressures toward a uniform distribution of information over time. The empirical record is currently mixed. Here, we revisit this question at scale: We analyze six reading datasets, estimate next-word probabilities with diverse statistical language models, and model reading times using recent advances in nonlinear regression. Results support a logarithmic effect of word predictability on processing difficulty, which favors probabilistic inference as a key component of human language processing.

Sentence-final completion norms for 2925 Mexican Spanish sentence contexts

Sentence-final completion tasks serve as valuable tools in studying language processing and the associated predictive mechanisms. There are several established sentence-completion norms for languages like English, Portuguese, French, and Spanish, each tailored to the language it was designed for and evaluated in. Yet, cultural variations among native speakers of the same language complicate the claim of a universal application of these norms. In this study, we developed a corpus of 2925 sentence-completion norms specifically for Mexican Spanish. This corpus is distinctive for several reasons: Firstly, it is the most comprehensive set of sentence-completion norms for Mexican Spanish to date. Secondly, it offers a substantial range of experimental stimuli with considerable variability in terms of the predictability of word sentence completion (cloze probability/surprisal) and the level of uncertainty inherent in the sentence context (entropy). Thirdly, the syntactic complexity of the sentences in the corpus is varied, as are the characteristics of the final word nouns (including aspects of concreteness/abstractness, length, and frequency). This paper details the generation of the sentence contexts, explains the methodology employed for data collection from a total of 1470 participants, and outlines the approach to data analysis for the establishment of sentence-completion norms. These norms provide a significant contribution to fields such as linguistics, cognitive science, and machine learning, among others, by enhancing our understanding of language, predictive mechanisms, knowledge representation, and context representation. The collected data is accessible through the Open Science Framework (OSF) at the following link: https://osf.io/js359/?view_only=bb1b328d37d643df903ed69bb2405ac0 .

Information entropy facilitates (not impedes) lexical processing during language comprehension

It is well known that contextual predictability facilitates word identification, but it is less clear whether the uncertainty associated with the current context (i.e., its lexical entropy) influences sentence processing. On the one hand, high entropy contexts may lead to interference due to greater number of lexical competitors. On the other hand, predicting multiple lexical competitors may facilitate processing through the preactivation of shared semantic features. In this study, we examined whether entropy measured at the trial level (i.e., for each participant, for each item) corresponds to facilitatory or inhibitory effects. Trial-level entropy captures each individual's knowledge about specific contexts and is therefore a more valid and sensitive measure of entropy (relative to the commonly employed item-level entropy). Participants (N = 112) completed two experimental sessions (with counterbalanced orders) that were separated by a 3- to 14-day interval. In one session, they produced up to 10 completions for sentence fragments (N = 647). In another session, they read the same sentences including a target word (whose entropy value was calculated based on the produced completions) while reading times were measured. We observed a facilitatory (not inhibitory) effect of trial-level entropy on lexical processing over and above item-level measures of lexical predictability (including cloze probability, surprisal, and semantic constraint). Extra analyses revealed that greater semantic overlap between the target and the produced responses facilitated target processing. Thus, the results lend support to theories of lexical prediction maintaining that prediction involves broad activation of semantic features rather than activation of full lexical forms.

Effects of linguistic context and noise type on speech comprehension

Introduction

Understanding speech in background noise is an effortful endeavor. When acoustic challenges arise, linguistic context may help us fill in perceptual gaps. However, more knowledge is needed regarding how different types of background noise affect our ability to construct meaning from perceptually complex speech input. Additionally, there is limited evidence regarding whether perceptual complexity (e.g., informational masking) and linguistic complexity (e.g., occurrence of contextually incongruous words) interact during processing of speech material that is longer and more complex than a single sentence. Our first research objective was to determine whether comprehension of spoken sentence pairs is impacted by the informational masking from a speech masker. Our second objective was to identify whether there is an interaction between perceptual and linguistic complexity during speech processing.

Methods

We used multiple measures including comprehension accuracy, reaction time, and processing effort (as indicated by task-evoked pupil response), making comparisons across three different levels of linguistic complexity in two different noise conditions. Context conditions varied by final word, with each sentence pair ending with an expected exemplar (EE), within-category violation (WV), or between-category violation (BV). Forty young adults with typical hearing performed a speech comprehension in noise task over three visits. Each participant heard sentence pairs presented in either multi-talker babble or spectrally shaped steady-state noise (SSN), with the same noise condition across all three visits.

Results

We observed an effect of context but not noise on accuracy. Further, we observed an interaction of noise and context in peak pupil dilation data. Specifically, the context effect was modulated by noise type: context facilitated processing only in the more perceptually complex babble noise condition.

Discussion

These findings suggest that when perceptual complexity arises, listeners make use of the linguistic context to facilitate comprehension of speech obscured by background noise. Our results extend existing accounts of speech processing in noise by demonstrating how perceptual and linguistic complexity affect our ability to engage in higher-level processes, such as construction of meaning from speech segments that are longer than a single sentence.

Individual word representations dissociate from linguistic context along a cortical unimodal to heteromodal gradient

Language comprehension involves multiple hierarchical processing stages across time, space, and levels of representation. When processing a word, the sensory input is transformed into increasingly abstract representations that need to be integrated with the linguistic context. Thus, language comprehension involves both input-driven as well as context-dependent processes. While neuroimaging research has traditionally focused on mapping individual brain regions to the distinct underlying processes, recent studies indicate that whole-brain distributed patterns of cortical activation might be highly relevant for cognitive functions, including language. One such pattern, based on resting-state connectivity, is the 'principal cortical gradient', which dissociates sensory from heteromodal brain regions. The present study investigated the extent to which this gradient provides an organizational principle underlying language function, using a multimodal neuroimaging dataset of functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) recordings from 102 participants during sentence reading. We found that the brain response to individual representations of a word (word length, orthographic distance, and word frequency), which reflect visual; orthographic; and lexical properties, gradually increases towards the sensory end of the gradient. Although these properties showed opposite effect directions in fMRI and MEG, their association with the sensory end of the gradient was consistent across both neuroimaging modalities. In contrast, MEG revealed that properties reflecting a word's relation to its linguistic context (semantic similarity and position within the sentence) involve the heteromodal end of the gradient to a stronger extent. This dissociation between individual word and contextual properties was stable across earlier and later time windows during word presentation, indicating interactive processing of word representations and linguistic context at opposing ends of the principal gradient. To conclude, our findings indicate that the principal gradient underlies the organization of a range of linguistic representations while supporting a gradual distinction between context-independent and context-dependent representations. Furthermore, the gradient reveals convergent patterns across neuroimaging modalities (similar location along the gradient) in the presence of divergent responses (opposite effect directions).

Syntactic adaptation leads to updated knowledge for local structural frequencies

Syntactic adaptation has been shown to occur for various temporarily ambiguous structures, wherein an initially unexpected resolution becomes easier to process after repeated exposure. More controversial and less replicated is the claim that this adaptation towards a locally frequent structure occurs due to a strategic shifting of expectations to match short-term statistical regularities such that readers adapt away from the a priori more frequent structure. Experiment 1 replicates the initial adaptation towards a coordination garden path structure using self-paced reading; however, this paradigm has been criticised for its low reliability for detecting such small effects. To this end, Experiments 2 and 3 use a combination of self-paced reading and sentence completion tasks to replicate initial adaptation towards both coordination and reduced relative garden path structures and show evidence for a preference for these structures over their a priori more frequent alternatives. Together, these data reveal that participants may be tracking local structural statistics in real time; however, they may not be able to rapidly use that information to update processing behaviours.

The Interaction of Context Constraints and Predictive Validity during Sentence Reading

Words are not processed in isolation; instead, they are commonly embedded in phrases and sentences. The sentential context influences the perception and processing of a word. However, how this is achieved by brain processes and whether predictive mechanisms underlie this process remain a debated topic. Here, we employed an experimental paradigm in which we orthogonalized sentence context constraints and predictive validity, which was defined as the ratio of congruent to incongruent sentence endings within the experiment. While recording electroencephalography, participants read sentences with three levels of sentential context constraints (high, medium, and low). Participants were also separated into two groups that differed in their ratio of valid congruent to incongruent target words that could be predicted from the sentential context. For both groups, we investigated modulations of alpha power before, and N400 amplitude modulations after target word onset. The results reveal that the N400 amplitude gradually decreased with higher context constraints and cloze probability. In contrast, alpha power was not significantly affected by context constraint. Neither the N400 nor alpha power were significantly affected by changes in predictive validity.

Emergence of syntax and word prediction in an artificial neural circuit of the cerebellum

The cerebellum, interconnected with the cerebral neocortex, plays a vital role in human-characteristic cognition such as language processing, however, knowledge about the underlying circuit computation of the cerebellum remains very limited. To gain a better understanding of the computation underlying cerebellar language processing, we developed a biologically constrained cerebellar artificial neural network (cANN) model, which implements the recently identified cerebello-cerebellar recurrent pathway. We found that while cANN acquires prediction of future words, another function of syntactic recognition emerges in the middle layer of the prediction circuit. The recurrent pathway of the cANN was essential for the two language functions, whereas cANN variants with further biological constraints preserved these functions. Considering the uniform structure of cerebellar circuitry across all functional domains, the single-circuit computation, which is the common basis of the two language functions, can be generalized to fundamental cerebellar functions of prediction and grammar-like rule extraction from sequences, that underpin a wide range of cerebellar motor and cognitive functions. This is a pioneering study to understand the circuit computation of human-characteristic cognition using biologically-constrained ANNs.

Multiple Concurrent Predictions Inform Prediction Error in the Human Auditory Pathway

The key assumption of the predictive coding framework is that internal representations are used to generate predictions on how the sensory input will look like in the immediate future. These predictions are tested against the actual input by the so-called prediction error units, which encode the residuals of the predictions. What happens to prediction errors, however, if predictions drawn by different stages of the sensory hierarchy contradict each other? To answer this question, we conducted two fMRI experiments while female and male human participants listened to sequences of sounds: pure tones in the first experiment and frequency-modulated sweeps in the second experiment. In both experiments, we used repetition to induce predictions based on stimulus statistics (stats-informed predictions) and abstract rules disclosed in the task instructions to induce an orthogonal set of (task-informed) predictions. We tested three alternative scenarios: neural responses in the auditory sensory pathway encode prediction error with respect to (1) the stats-informed predictions, (2) the task-informed predictions, or (3) a combination of both. Results showed that neural populations in all recorded regions (bilateral inferior colliculus, medial geniculate body, and primary and secondary auditory cortices) encode prediction error with respect to a combination of the two orthogonal sets of predictions. The findings suggest that predictive coding exploits the non-linear architecture of the auditory pathway for the transmission of predictions. Such non-linear transmission of predictions might be crucial for the predictive coding of complex auditory signals like speech.Significance Statement Sensory systems exploit our subjective expectations to make sense of an overwhelming influx of sensory signals. It is still unclear how expectations at each stage of the processing pipeline are used to predict the representations at the other stages. The current view is that this transmission is hierarchical and linear. Here we measured fMRI responses in auditory cortex, sensory thalamus, and midbrain while we induced two sets of mutually inconsistent expectations on the sensory input, each putatively encoded at a different stage. We show that responses at all stages are concurrently shaped by both sets of expectations. The results challenge the hypothesis that expectations are transmitted linearly and provide for a normative explanation of the non-linear physiology of the corticofugal sensory system.

Nuances of knowing: Brain potentials reveal implicit effects of domain knowledge on word processing in the absence of sentence-level knowledge

In previous work investigating the relationship between domain knowledge (of the fictional world of Harry Potter) and sentence comprehension, domain knowledge had a greater impact on electrical brain potentials to words which completed sentences about fictional "facts" participants reported they did not know compared to facts they did. This suggests that individuals use domain knowledge continuously to activate relevant/related concepts as they process sentences, even with only partial knowledge. As that study relied on subjective reports, it may have resulted in response bias related to an individual's overall domain knowledge. In the present study, we therefore asked participants with varying degrees of domain knowledge to complete sentences describing fictional "facts" as an objective measure of sentence-level knowledge. We then recorded EEG as the same individuals (re-)read the same sentences, including their appropriate final words, and sorted these according to their objective knowledge scores. Replicating and extending Troyer et al., domain knowledge immediately facilitated access to meaning for unknown words; greater domain knowledge was associated with reduced N400 amplitudes for unknown words. These findings constitute novel evidence for graded preactivation of conceptual knowledge (e.g., at the level of semantic features and/or relations) in the absence of lexical prediction. Knowledge also influenced post-N400 memory/integration processes for these same unknown words; greater domain knowledge was associated with enhanced late positive components (LPCs), suggesting that deeper encoding during language processing may be engendered when knowledgeable individuals encounter an apparent gap in their knowledge.

The Impact of Linguistic Prediction Violations on Downstream Recognition Memory and Sentence Recall

Predicting upcoming words during language comprehension not only affects processing in the moment but also has consequences for memory, although the source of these memory effects (e.g., whether driven by lingering pre-activations, re-analysis following prediction violations, or other mechanisms) remains underspecified. Here, we investigated downstream impacts of prediction on memory in two experiments. First, we recorded EEG as participants read strongly and weakly constraining sentences with expected, unexpected but plausible, or semantically anomalous endings ("He made a holster for his gun / father / train") and were tested on their recognition memory for the sentence endings. Participants showed similar rates of false alarms for predicted but never presented sentence endings whether the prediction violation was plausible or anomalous, suggesting that these arise from pre-activation of the expected words during reading. During sentence reading, especially in strongly constraining sentences, plausible prediction violations elicited an anterior positivity; anomalous endings instead elicited a posterior positivity, whose amplitude was predictive of later memory for those anomalous words. ERP patterns at the time of recognition differentiated plausible and anomalous sentence endings: Words that had been plausible prediction violations elicited enhanced late positive complex amplitudes, suggesting greater episodic recollection, whereas anomalous sentence endings elicited greater N1 amplitudes, suggesting attentional tagging. In a follow-up behavioral study, a separate group of participants read the same sentence stimuli and were tested for sentence-level recall. We found that recall of full sentences was impaired when sentences ended with a prediction violation. Taken together, the results suggest that prediction violations draw attention and affect encoding of the violating word, in a manner that depends on plausibility, and that this, in turn, may impair future memory of the gist of the sentence.

Discourse coherence modulates use of predictive processing during sentence comprehension

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze probability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, predictable words were processed more quickly and anomalous words more slowly when they occurred in highly coherent discourse. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating semantic and lexical information less when their expectations are less likely to be reliable.

Does negation influence the choice of sentence continuations? Evidence from a four-choice cloze task

Event plausibility facilitates the processing of affirmative sentences, but little is known about how it affects negative sentences. In six behavioural experiments, we investigated negation's impact on the choice of sentence continuations that differ with respect to event plausibility. In a four-choice cloze task, participants saw affirmative and negative sentence fragments (The child will [not] eat the . . .) in combination with four potential continuations: yoghurt (a plausible word), shellfish (a weak world knowledge violating word), branch (a severe world knowledge violating word), and minivan (a word resulting in a semantic violation). Across all experiments the plausible word was highly preferred in both affirmative and negative sentences. Experiment 2 replicated Experiment 1 while ruling out the possibility that the lack of effect of negation in Experiment 1 stemmed from participants not fully processing the negation. Experiment 3 showed that the observed plausibility effects can be generalised to other aspectual forms (The child has [not] eaten the yoghurt). Experiment 4 ruled out the possibility that the choices were mainly driven by lexical associations and additionally suggested a role for informativity. Experiment 5 replicated Experiment 4 and reinforced the general pattern according to which negative sentences express the denial of plausible positive events. Experiment 6 provided evidence that informativity might be driving patterns of choices in the negative sentences. All in all, these findings suggest that upcoming continuations are chosen to maximise the plausibility of the event in the affirmative sentences and to deny that event in the negative sentences. The observed plausibility effects do not seem to be modulated by the internal representation of events, but they can be modulated by changes to the expected informativity of the sentence.

Hand Gestures Have Predictive Potential During Conversation: An Investigation of the Timing of Gestures in Relation to Speech

During face-to-face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face-to-face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co-speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face-to-face conversation.

Context-specific effects of violated expectations: ERP evidence

A complete understanding of the predictive processing effect in sentence comprehension needs to understand both the facilitation effect of successful prediction and the cost associated with disconfirmed predictions. The current study compares the predictive processing effect across two types of contexts in Mandarin Chinese: the classifier-noun vs. verb-noun phrases, when controlling for the degree of contextual constraints and cloze probability of the target nouns across the two contexts. The two contexts showed similar N400 patterns for expected target nouns, indicative of an identical facilitation effect of confirmed contextual expectation. But in the post-N400 time window, the processing cost associated with the unexpected words differed between the two contexts. Additional differences between the two contexts were also revealed by the neural oscillation patterns obtained prior to the target noun. The differences between the classifier vs. verb contexts shed new light on the revision mechanism that deals with disconfirmed expectations.

Predictions about prosody facilitate lexical access: Evidence from P50/N100 and MMN components

Research into the neural foundation of perception asserts a model where top-down predictions modulate the bottom-up processing of sensory input. Despite becoming increasingly influential in cognitive neuroscience, the precise account of this predictive coding framework remains debated. In this study, we aim to contribute to this debate by investigating how predictions about prosody facilitate speech perception, and to shed light especially on lexical access influenced by simultaneous predictions in different domains, inter alia, prosodic and semantic. Using a passive auditory oddball paradigm, we examined neural responses to prosodic changes, leading to a semantic change as in Dutch nouns canon ['kaːnɔn] 'canon' vs kanon [kaː'nɔn] 'cannon', and used acoustically identical pseudowords as controls. Results from twenty-eight native speakers of Dutch (age range 18-32 years) indicated an enhanced P50/N100 complex to prosodic change in pseudowords as well as an MMN response to both words and pseudowords. The enhanced P50/N100 response to pseudowords is claimed to indicate that all relevant auditory information is still processed by the brain, whereas the reduced response to words might reflect the suppression of information that has already been encoded. The MMN response to pseudowords and words, on the other hand, is best justified by the unification of previously established prosodic representations with sensory and semantic input respectively. This pattern of results is in line with the predictive coding framework acting on multiple levels and is of crucial importance to indicate that predictions about linguistic prosodic information are utilized by the brain as early as 50 ms.

Multiple predictions during language comprehension: Friends, foes, or indifferent companions?

To comprehend language, we continually use prior context to pre-activate expected upcoming information, resulting in facilitated processing of incoming words that confirm these predictions. But what are the consequences of disconfirming prior predictions? To address this question, most previous studies have examined unpredictable words appearing in contexts that constrain strongly for a single continuation. However, during natural language processing, it is far more common to encounter contexts that constrain for multiple potential continuations, each with some probability. Here, we ask whether and how pre-activating both higher and lower probability alternatives influences the processing of the lower probability incoming word. One possibility is that, similar to language production, there is continuous pressure to select the higher-probability pre-activated alternative through competitive inhibition. During comprehension, this would result in relative costs in processing the lower probability target. A second possibility is that if the two pre-activated alternatives share semantic features, they mutually enhance each other's pre-activation. This would result in greater facilitation in processing the lower probability target. To distinguish between these accounts, we recorded ERPs as participants read three-sentence scenarios that constrained either for a single word or for two potential continuations - a higher probability expected candidate and a lower probability second-best candidate. We found no evidence that competitive pre-activation between the expected and second-best candidates resulted in costs in processing the second-best target, either during lexico-semantic processing (indexed by the N400) or at later stages of processing (indexed by a later frontal positivity). Instead, we found only benefits of pre-activating multiple alternatives, with evidence of enhanced graded facilitation on lower-probability targets that were semantically related to a higher-probability pre-activated alternative. These findings are consistent with a previous eye-tracking study by Luke and Christianson (2016, Cogn Psychol) using corpus-based materials. They have significant theoretical implications for models of predictive language processing, indicating that routine graded prediction in language comprehension does not operate through the same competitive mechanisms that are engaged in language production. Instead, our results align more closely with hierarchical probabilistic accounts of language comprehension, such as predictive coding.

The role of multimodal cues in second language comprehension

In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues - meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal - while using multimodal cues to a lesser extent than L1 comprehenders overall.

Neurocomputations on dual-brain signals underlie interpersonal prediction during a natural conversation

Prediction on the partner's speech plays a key role in a smooth conversation. However, previous studies on this issue have been majorly conducted at the single-brain rather than dual-brain level, leaving the interpersonal prediction hypothesis untested. To fill this gap, this study combined a neurocomputational modeling approach with a natural conversation paradigm in which two salespersons persuaded a customer to buy their product with their haemodynamic signals being collected using functional near-infrared spectroscopy hyperscanning. First, the results showed a cognitive hierarchy in a natural conversation, with the lower-level process (i.e., pragmatic representation of the persuasion) in the salesperson interacting with the higher-level process (i.e., value representation of the product) in the customer. Next, we found that the right dorsal lateral prefrontal cortex (rdlPFC) and temporoparietal junction (rTPJ) were associated with the representation of the product's value in the customer, while the right inferior frontal cortex (rIFC) was associated with the representation of the pragmatic processes in the salesperson. Finally, neurocomputational modeling results supported the prediction of the salesperson's lower-level brain activity based on the customer's higher-level brain activity. Moreover, the updating weight of the prediction model based on the neural computation between the rIFC of the salesperson and the rTPJ of the customer was closely associated with the interaction context, whereas that based on the rIFC-rdlPFC was not. In summary, these findings provide initial support for the interpersonal prediction hypothesis at the dual-brain level and reveal a hierarchy for the interpersonal prediction process.

An eye on semantics: a study on the influence of concreteness and predictability on early fixation durations

We used eye-tracking during natural reading to study how semantic control and representation mechanisms interact for the successful comprehension of sentences, by manipulating sentence context and single-word meaning. Specifically, we examined whether a word's semantic characteristic (concreteness) affects first fixation and gaze durations (FFDs and GDs) and whether it interacts with the predictability of a word. We used a linear mixed effects model including several possible psycholinguistic covariates. We found a small but reliable main effect of concreteness and replicated a predictability effect on FFDs, but we found no interaction between the two. The results parallel previous findings of additive effects of predictability (context) and frequency (lexical level) in fixation times. Our findings suggest that the semantics of a word and the context created by the preceding words additively influence early stages of word processing in natural sentence reading.