Surprisal From Language Models Can Predict ERPs in Processing Predicate-Argument Structures Only if Enriched by an Agent Preference Principle

Language models based on artificial neural networks increasingly capture key aspects of how humans process sentences. Most notably, model-based surprisals predict event-related potentials such as N400 amplitudes during parsing. Assuming that these models represent realistic estimates of human linguistic experience, their success in modeling language processing raises the possibility that the human processing system relies on no other principles than the general architecture of language models and on sufficient linguistic input. Here, we test this hypothesis on N400 effects observed during the processing of verb-final sentences in German, Basque, and Hindi. By stacking Bayesian generalised additive models, we show that, in each language, N400 amplitudes and topographies in the region of the verb are best predicted when model-based surprisals are complemented by an Agent Preference principle that transiently interprets initial role-ambiguous noun phrases as agents, leading to reanalysis when this interpretation fails. Our findings demonstrate the need for this principle independently of usage frequencies and structural differences between languages. The principle has an unequal force, however. Compared to surprisal, its effect is weakest in German, stronger in Hindi, and still stronger in Basque. This gradient is correlated with the extent to which grammars allow unmarked NPs to be patients, a structural feature that boosts reanalysis effects. We conclude that language models gain more neurobiological plausibility by incorporating an Agent Preference. Conversely, theories of human processing profit from incorporating surprisal estimates in addition to principles like the Agent Preference, which arguably have distinct evolutionary roots.

Event Related Brain Responses Reveal the Impact of Spatial Augmented Reality Predictive Cues on Mental Effort

This article presents the results from a Spatial Augmented Reality (SAR) study which evaluated the cognitive cost of several predictive cues. Participants performed a validated procedural button pressing task, where the predictive cue annotations guided them to the upcoming task. While existing research has evaluated predictive cues based on their performance and self-rated mental effort, actual cognitive cost has yet to be investigated. To measure the user's brain activity, this study utilized electroencephalogram (EEG) recordings. Cognitive load was evaluated by measuring brain responses for a secondary auditory oddball task, with reduced brain responses to oddball tones expected when cognitive load in the primary task is highest. A simple monitor n-back task and procedural task comparing monitor versus SAR were conducted, followed by a version of the procedural task comparing the SAR predictive cues. Results from the brain responses were able to distinguish between performance enhancing cues with a high and low cognitive load. Electrical brain responses also revealed that having an arc or arrow guide towards the upcoming task required the least amount of mental effort.

Multi-Level Precues for Guiding Tasks Within and Between Workspaces in Spatial Augmented Reality

We explore Spatial Augmented Reality (SAR) precues (predictive cues) for procedural tasks within and between workspaces and for visualizing multiple upcoming steps in advance. We designed precues based on several factors: cue type, color transparency, and multi-level (number of precues). Precues were evaluated in a procedural task requiring the user to press buttons in three surrounding workspaces. Participants performed fastest in conditions where tasks were linked with line cues with different levels of color transparency. Precue performance was also affected by whether the next task was in the same workspace or a different one.

Individual differences in information processing during sleep and wake predict sleep-based memory consolidation of complex rules

Memory is critical for many cognitive functions, from remembering facts, to learning complex environmental rules. While memory encoding occurs during wake, memory consolidation is associated with sleep-related neural activity. Further, research suggests that individual differences in alpha frequency during wake (∼7 - 13 Hz) modulate memory processes, with higher individual alpha frequency (IAF) associated with greater memory performance. However, the relationship between wake-related EEG individual differences, such as IAF, and sleep-related neural correlates of memory consolidation has been largely unexplored, particularly in a complex rule-based memory context. Here, we aimed to investigate whether wake-derived IAF and sleep neurophysiology interact to influence rule learning in a sample of 35 healthy adults (16 males; mean age = 25.4, range: 18 - 40). Participants learned rules of a modified miniature language prior to either 8hrs of sleep or wake, after which they were tested on their knowledge of the rules in a grammaticality judgement task. Results indicate that sleep neurophysiology and wake-derived IAF do not interact but modulate memory for complex linguistic rules separately. Phase-amplitude coupling between slow oscillations and spindles during non-rapid eye-movement (NREM) sleep also promoted memory for rules that were analogous to the canonical English word order. As an exploratory analysis, we found that rapid eye-movement (REM) sleep theta power at posterior regions interacts with IAF to predict rule learning and proportion of time in REM sleep predicts rule learning differentially depending on grammatical rule type. Taken together, the current study provides behavioural and electrophysiological evidence for a complex role of NREM and REM sleep neurophysiology and wake-derived IAF in the consolidation of rule-based information.

An Agent-First Preference in a Patient-First Language During Sentence Comprehension

The language comprehension system preferentially assumes that agents come first during incremental processing. While this might reflect a biologically fixed bias, shared with other domains and other species, the evidence is limited to languages that place agents first, and so the bias could also be learned from usage frequency. Here, we probe the bias with electroencephalography (EEG) in Äiwoo, a language that by default places patients first, but where sentence-initial nouns are still locally ambiguous between patient or agent roles. Comprehenders transiently interpreted nonhuman nouns as patients, eliciting a negativity when disambiguation was toward the less common agent-initial order. By contrast and against frequencies, human nouns were transiently interpreted as agents, eliciting an N400-like negativity when the disambiguation was toward patient-initial order. Consistent with the notion of a fixed property, the agent bias is robust against usage frequency for human referents. However, this bias can be reversed by frequency experience for nonhuman referents.

Neural and cognitive correlates of performance in dynamic multi-modal settings

The endeavour to understand human cognition has largely relied upon investigation of task-related brain activity. However, resting-state brain activity can also offer insights into individual information processing and performance capabilities. Previous research has identified electroencephalographic resting-state characteristics (most prominently: the individual alpha frequency; IAF) that predict cognitive function. However, it has largely overlooked a second component of electrophysiological signals: aperiodic 1/ƒ activity. The current study examined how both oscillatory and aperiodic resting-state EEG measures, alongside traditional cognitive tests, can predict performance in a dynamic and complex, semi-naturalistic cognitive task. Participants' resting-state EEG was recorded prior to engaging in a Target Motion Analysis (TMA) task in a simulated submarine control room environment (CRUSE), which required participants to integrate dynamically changing information over time. We demonstrated that the relationship between IAF and cognitive performance extends from simple cognitive tasks (e.g., digit span) to complex, dynamic measures of information processing. Further, our results showed that individual 1/ƒ parameters (slope and intercept) differentially predicted performance across practice and testing sessions, whereby flatter slopes and higher intercepts were associated with improved performance during learning. In addition to the EEG predictors, we demonstrate a link between cognitive skills most closely related to the TMA task (i.e., spatial imagery) and subsequent performance. Overall, the current study highlights (1) how resting-state metrics - both oscillatory and aperiodic - have the potential to index higher-order cognitive capacity, while (2) emphasising the importance of examining these electrophysiological components within more dynamic settings and over time.

Task-related, intrinsic oscillatory and aperiodic neural activity predict performance in naturalistic team-based training scenarios

Effective teams are essential for optimally functioning societies. However, little is known regarding the neural basis of two or more individuals engaging cooperatively in real-world tasks, such as in operational training environments. In this exploratory study, we recruited forty individuals paired as twenty dyads and recorded dual-EEG at rest and during realistic training scenarios of increasing complexity using virtual simulation systems. We estimated markers of intrinsic brain activity (i.e., individual alpha frequency and aperiodic activity), as well as task-related theta and alpha oscillations. Using nonlinear modelling and a logistic regression machine learning model, we found that resting-state EEG predicts performance and can also reliably differentiate between members within a dyad. Task-related theta and alpha activity during easy training tasks predicted later performance on complex training to a greater extent than prior behaviour. These findings complement laboratory-based research on both oscillatory and aperiodic activity in higher-order cognition and provide evidence that theta and alpha activity play a critical role in complex task performance in team environments.

Rapid adaptation of predictive models during language comprehension: Aperiodic EEG slope, individual alpha frequency and idea density modulate individual differences in real-time model updating

Predictive coding provides a compelling, unified theory of neural information processing, including for language. However, there is insufficient understanding of how predictive models adapt to changing contextual and environmental demands and the extent to which such adaptive processes differ between individuals. Here, we used electroencephalography (EEG) to track prediction error responses during a naturalistic language processing paradigm. In Experiment 1, 45 native speakers of English listened to a series of short passages. Via a speaker manipulation, we introduced changing intra-experimental adjective order probabilities for two-adjective noun phrases embedded within the passages and investigated whether prediction error responses adapt to reflect these intra-experimental predictive contingencies. To this end, we calculated a novel measure of speaker-based, intra-experimental surprisal (“speaker-based surprisal”) as defined on a trial-by-trial basis and by clustering together adjectives with a similar meaning. N400 amplitude at the position of the critical second adjective was used as an outcome measure of prediction error. Results showed that N400 responses attuned to speaker-based surprisal over the course of the experiment, thus indicating that listeners rapidly adapt their predictive models to reflect local environmental contingencies (here: the probability of one type of adjective following another when uttered by a particular speaker). Strikingly, this occurs in spite of the wealth of prior linguistic experience that participants bring to the laboratory. Model adaptation effects were strongest for participants with a steep aperiodic (1/f) slope in resting EEG and low individual alpha frequency (IAF), with idea density (ID) showing a more complex pattern. These results were replicated in a separate sample of 40 participants in Experiment 2, which employed a highly similar design to Experiment 1. Overall, our results suggest that individuals with a steep aperiodic slope adapt their predictive models most strongly to context-specific probabilistic information. Steep aperiodic slope is thought to reflect low neural noise, which in turn may be associated with higher neural gain control and better cognitive control. Individuals with a steep aperiodic slope may thus be able to more effectively and dynamically reconfigure their prediction-related neural networks to meet current task demands. We conclude that predictive mechanisms in language are highly malleable and dynamic, reflecting both the affordances of the present environment as well as intrinsic information processing capabilities of the individual.

Cross-linguistic differences in case marking shape neural power dynamics and gaze behavior during sentence planning

Languages differ in how they mark the dependencies between verbs and arguments, e.g., by case. An eye tracking and EEG picture description study examined the influence of case marking on the time course of sentence planning in Basque and Swiss German. While German assigns an unmarked (nominative) case to subjects, Basque specifically marks agent arguments through ergative case. Fixations to agents and event-related synchronization (ERS) in the theta and alpha frequency bands, as well as desynchronization (ERD) in the alpha and beta bands revealed multiple effects of case marking on the time course of early sentence planning. Speakers decided on case marking under planning early when preparing sentences with ergative-marked agents in Basque, whereas sentences with unmarked agents allowed delaying structural commitment across languages. These findings support hierarchically incremental accounts of sentence planning and highlight how cross-linguistic differences shape the neural dynamics underpinning language use.

Oscillatory and Aperiodic Neural Activity Jointly Predict Language Learning

Memory formation involves the synchronous firing of neurons in task-relevant networks, with recent models postulating that a decrease in low-frequency oscillatory activity underlies successful memory encoding and retrieval. However, to date, this relationship has been investigated primarily with face and image stimuli; considerably less is known about the oscillatory correlates of complex rule learning, as in language. Furthermore, recent work has shown that nonoscillatory (1/ƒ) activity is functionally relevant to cognition, yet its interaction with oscillatory activity during complex rule learning remains unknown. Using spectral decomposition and power-law exponent estimation of human EEG data (17 females, 18 males), we show for the first time that 1/ƒ and oscillatory activity jointly influence the learning of word order rules of a miniature artificial language system. Flexible word-order rules were associated with a steeper 1/ƒ slope, whereas fixed word-order rules were associated with a shallower slope. We also show that increased theta and alpha power predicts fixed relative to flexible word-order rule learning and behavioral performance. Together, these results suggest that 1/ƒ activity plays an important role in higher-order cognition, including language processing, and that grammar learning is modulated by different word-order permutations, which manifest in distinct oscillatory profiles.

EEG and behavioral correlates of attentional processing while walking and navigating naturalistic environments

The capacity to regulate one's attention in accordance with fluctuating task demands and environmental contexts is an essential feature of adaptive behavior. Although the electrophysiological correlates of attentional processing have been extensively studied in the laboratory, relatively little is known about the way they unfold under more variable, ecologically-valid conditions. Accordingly, this study employed a 'real-world' EEG design to investigate how attentional processing varies under increasing cognitive, motor, and environmental demands. Forty-four participants were exposed to an auditory oddball task while (1) sitting in a quiet room inside the lab, (2) walking around a sports field, and (3) wayfinding across a university campus. In each condition, participants were instructed to either count or ignore oddball stimuli. While behavioral performance was similar across the lab and field conditions, oddball count accuracy was significantly reduced in the campus condition. Moreover, event-related potential components (mismatch negativity and P3) elicited in both 'real-world' settings differed significantly from those obtained under laboratory conditions. These findings demonstrate the impact of environmental factors on attentional processing during simultaneously-performed motor and cognitive tasks, highlighting the value of incorporating dynamic and unpredictable contexts within naturalistic designs.

Resting-state aperiodic neural dynamics predict individual differences in visuomotor performance and learning

An emerging body of work has demonstrated that resting-state non-oscillatory, or aperiodic, 1/f neural activity is a functional and behaviorally relevant marker of cognitive function capacity. In the motor domain, previous work has only applied 1/f analyses to investigations of motor coordination and performance measures. The value of aperiodic resting-state neural dynamics as a marker of individual visuomotor performance capacity remains unknown. Accordingly, the aim of this work was to investigate if individual 1/f intercept and slope parameters of aperiodic resting-state neural activity predict reaction time and perceptual sensitivity in an immersive virtual reality marksmanship task. The marksmanship task required speeded selection of target stimuli and avoidance of selecting non-target stimuli. Motor and perceptual demands were incrementally increased across task blocks and participants performed the task across three training sessions spanning one week. When motor demands were high, steeper individual 1/f slope predicted shorter reaction time. This relationship did not change with practice. Increased 1/f intercept and a steeper 1/f slope were associated with higher perceptual sensitivity, measured as d'. However, this association was only observed under the highest levels of perceptual demand and only in the initial exposure to these conditions. Individuals with a lower 1/f intercept and a shallower 1/f slope demonstrated the greatest gains in perceptual sensitivity from task practice. These findings demonstrate that individual differences in motor and perceptual performance can be accounted for with resting-state aperiodic neural dynamics. The 1/f aperiodic parameters are most informative in predicting visuomotor performance under complex and demanding task conditions. In addition to predicting capacity for high visuomotor performance with a novel task, 1/f aperiodic parameters might also be useful in predicting which individuals might derive the most improvements from practice.

Individual Differences in Peripheral Hearing and Cognition Reveal Sentence Processing Differences in Healthy Older Adults

When viewed cross-sectionally, aging seems to negatively affect speech comprehension. However, aging is a heterogeneous process, and variability among older adults is typically large. In this study, we investigated language comprehension as a function of individual differences in older adults. Specifically, we tested whether hearing thresholds, working memory, inhibition, and individual alpha frequency would predict event-related potential amplitudes in response to classic psycholinguistic manipulations at the sentence level. Twenty-nine healthy older adults (age range 61-76 years) listened to English sentences containing reduced relative clauses and object-relative clauses while their electroencephalogram was recorded. We found that hearing thresholds and working memory predicted P600 amplitudes early during reduced relative clause processing, while individual alpha frequency predicted P600 amplitudes at a later point in time. The results suggest that participants with better hearing and larger working memory capacity simultaneously activated both the preferred and the dispreferred interpretation of reduced relative clauses, while participants with worse hearing and smaller working memory capacity only activated the preferred interpretation. They also suggest that participants with a higher individual alpha frequency had a higher likelihood of successfully reanalysing the sentence toward the reduced relative clause reading than participants with a lower individual alpha frequency. By contrast, we found no relationship between object-relative clause processing and working memory or hearing thresholds. Taken together, the results support the view that older adults employ different strategies during auditory sentence processing dependent on their hearing and cognitive abilities and that there is no single ability that uniformly predicts sentence processing outcomes.

Semantic reversal anomalies under the microscope: Task and modality influences on language-associated event-related potentials

Semantic reversal anomalies (SRAs)-sentences where an implausibility is created by reversing participant roles-have attracted much attention in the literature on the electrophysiology of language. In spite of being syntactically well formed but semantically implausible, these sentences unexpectedly elicited a monophasic P600 effect in English and Dutch rather than an N400 effect. Subsequent research revealed variability in the presence/absence of an N400 effect to SRAs depending on the language examined and the choice of verb type in English. However, most previous studies employed the same presentation modality (visual) and task (acceptability judgement). Here, we conducted two experiments and three statistical analyses to investigate the influence of stimulus modality, task demand and statistical choices on event-related potential (ERP) response patterns to SRAs in English. We reproduced a previous study's procedure and analysis (N. Bourguignon et al. (2012) Brain and Language, 122, 179-189) and further introduced between-subjects factors of task type and modality, using mixed-effects modelling to analyse the data. We observed an N400 effect to typical English SRAs (agent subject verbs, e.g. "the fries will eat the boys"), which contrasts existing literature and was not predicted by existing theories that account for SRA processing. Task demand modulated the ERPs elicited by SRAs, while auditory presentation led to increased comprehension accuracy and a more broadly distributed ERP. Finally, the statistical methods used influenced the presence/absence of ERP effects. Our results suggest a sensitivity of language-related ERP patterns to methodological parameters, and we conclude that future experiments should take this into careful consideration.

Language Processing as a Precursor to Language Change: Evidence From Icelandic

One of the main characteristics of human languages is that they are subject to fundamental changes over time. However, because of the long transitional periods involved, the internal dynamics of such changes are typically inaccessible. Here, we present a new approach to examining language change via its connection to language comprehension. By means of an EEG experiment on Icelandic, a prominent current example of a language in transition, we show that the neurophysiological responses of native speakers already reflect projected changes that are not yet apparent in their overt behavior. Neurocognitive measures thus offer a means of predicting, rather than only retracing, language change.

Toward a Neurobiologically Plausible Model of Language-Related, Negative Event-Related Potentials

Language-related event-related potential (ERP) components such as the N400 have traditionally been associated with linguistic or cognitive functional interpretations. By contrast, it has been considerably more difficult to relate these components to neurobiologically grounded accounts of language. Here, we propose a theoretical framework based on a predictive coding architecture, within which negative language-related ERP components such as the N400 can be accounted for in a neurobiologically plausible manner. Specifically, we posit that the amplitude of negative language-related ERP components reflects precision-weighted prediction error signals, i.e., prediction errors weighted by the relevance of the information source leading to the error. From this perspective, precision has a direct link to cue validity in a particular language and, thereby, to relevance of individual linguistic features for internal model updating. We view components such as the N400 and LAN as members of a family with similar functional characteristics and suggest that latency and topography differences between these components reflect the locus of prediction errors and model updating within a hierarchically organized cortical predictive coding architecture. This account has the potential to unify findings from the full range of the N400 literature, including word-level, sentence-, and discourse-level results as well as cross-linguistic differences.

A Comparison of Predictive Spatial Augmented Reality Cues for Procedural Tasks

Previous research has demonstrated that Augmented Reality can reduce a user's task response time and mental effort when completing a procedural task. This paper investigates techniques to improve user performance and reduce mental effort by providing projector-based Spatial Augmented Reality predictive cues for future responses. The objective of the two experiments conducted in this study was to isolate the performance and mental effort differences from several different annotation cueing techniques for simple (Experiment 1) and complex (Experiment 2) button-pressing tasks. Comporting with existing cognitive neuroscience literature on prediction, attentional orienting, and interference, we hypothesized that for both simple procedural tasks and complex search-based tasks, having a visual cue guiding to the next task's location would positively impact performance relative to a baseline, no-cue condition. Additionally, we predicted that direction-based cues would provide a more significant positive impact than target-based cues. The results indicated that providing a line to the next task was the most effective technique for improving the users' task time and mental effort in both the simple and complex tasks.

Sleep-Dependent Memory Consolidation and Incremental Sentence Comprehension: Computational Dependencies during Language Learning as Revealed by Neuronal Oscillations

We hypothesize a beneficial influence of sleep on the consolidation of the combinatorial mechanisms underlying incremental sentence comprehension. These predictions are grounded in recent work examining the effect of sleep on the consolidation of linguistic information, which demonstrate that sleep-dependent neurophysiological activity consolidates the meaning of novel words and simple grammatical rules. However, the sleep-dependent consolidation of sentence-level combinatorics has not been studied to date. Here, we propose that dissociable aspects of sleep neurophysiology consolidate two different types of combinatory mechanisms in human language: sequence-based (order-sensitive) and dependency-based (order-insensitive) combinatorics. The distinction between the two types of combinatorics is motivated both by cross-linguistic considerations and the neurobiological underpinnings of human language. Unifying this perspective with principles of sleep-dependent memory consolidation, we posit that a function of sleep is to optimize the consolidation of sequence-based knowledge (the when) and the establishment of semantic schemas of unordered items (the what) that underpin cross-linguistic variations in sentence comprehension. This hypothesis builds on the proposal that sleep is involved in the construction of predictive codes, a unified principle of brain function that supports incremental sentence comprehension. Finally, we discuss neurophysiological measures (EEG/MEG) that could be used to test these claims, such as the quantification of neuronal oscillations, which reflect basic mechanisms of information processing in the brain.

Toward a reliable, automated method of individual alpha frequency (IAF) quantification

Individual alpha frequency (IAF) is a promising electrophysiological marker of interindividual differences in cognitive function. IAF has been linked with trait-like differences in information processing and general intelligence, and provides an empirical basis for the definition of individualized frequency bands. Despite its widespread application, however, there is little consensus on the optimal method for estimating IAF, and many common approaches are prone to bias and inconsistency. Here, we describe an automated strategy for deriving two of the most prevalent IAF estimators in the literature: peak alpha frequency (PAF) and center of gravity (CoG). These indices are calculated from resting-state power spectra that have been smoothed using a Savitzky-Golay filter (SGF). We evaluate the performance characteristics of this analysis procedure in both empirical and simulated EEG data sets. Applying the SGF technique to resting-state data from n = 63 healthy adults furnished 61 PAF and 62 CoG estimates. The statistical properties of these estimates were consistent with previous reports. Simulation analyses revealed that the SGF routine was able to reliably extract target alpha components, even under relatively noisy spectral conditions. The routine consistently outperformed a simpler method of automated peak detection that did not involve spectral smoothing. The SGF technique is fast, open source, and available in two popular programming languages (MATLAB, Python), and thus can easily be integrated within the most popular M/EEG toolsets (EEGLAB, FieldTrip, MNE-Python). As such, it affords a convenient tool for improving the reliability and replicability of future IAF-related research.

Electrophysiology Reveals the Neural Dynamics of Naturalistic Auditory Language Processing: Event-Related Potentials Reflect Continuous Model Updates

The recent trend away from ANOVA-based analyses places experimental investigations into the neurobiology of cognition in more naturalistic and ecologically valid designs within reach. Using mixed-effects models for epoch-based regression, we demonstrate the feasibility of examining event-related potentials (ERPs), and in particular the N400, to study the neural dynamics of human auditory language processing in a naturalistic setting. Despite the large variability between trials during naturalistic stimulation, we replicated previous findings from the literature: the effects of frequency, animacy, and word order and find previously unexplored interaction effects. This suggests a new perspective on ERPs, namely, as a continuous modulation reflecting continuous stimulation instead of a series of discrete and essentially sequential processes locked to discrete events.

Comprehension demands modulate re-reading, but not first pass reading behavior

Several studies have examined effects of explicit task demands on eye movements in reading. However, there is relatively little prior research investigating the influence of implicit processing demands. In the present study, processing demands were manipulated by means of a between-subject manipulation of comprehension question difficulty. Consistent with previous results from Wotschack and Kliegl (2013), the question difficulty manipulation influenced the probability of regressing from late in sentences and re-reading earlier regions; readers who expected difficult comprehension questions were more likely to re-read. However, this manipulation had no reliable influence on eye movements during first pass reading of earlier sentence regions. Moreover, for the subset of sentences that contained a plausibility manipulation, the disruption induced by implausibility was not modulated by the question manipulation. We interpret these results as suggesting that comprehension demands influence reading behavior primarily by modulating a criterion for comprehension that readers apply after completing first-pass processing.

Where Is the Beat? The Neural Correlates of Lexical Stress and Rhythmical Well-formedness in Auditory Story Comprehension

While listening to continuous speech, humans process beat information to correctly identify word boundaries. The beats of language are stress patterns that are created by combining lexical (word-specific) stress patterns and the rhythm of a specific language. Sometimes, the lexical stress pattern needs to be altered to obey the rhythm of the language. This study investigated the interplay of lexical stress patterns and rhythmical well-formedness in natural speech with fMRI. Previous electrophysiological studies on cases in which a regular lexical stress pattern may be altered to obtain rhythmical well-formedness showed that even subtle rhythmic deviations are detected by the brain if attention is directed toward prosody. Here, we present a new approach to this phenomenon by having participants listen to contextually rich stories in the absence of a task targeting the manipulation. For the interaction of lexical stress and rhythmical well-formedness, we found one suprathreshold cluster localized between the cerebellum and the brain stem. For the main effect of lexical stress, we found higher BOLD responses to the retained lexical stress pattern in the bilateral SMA, bilateral postcentral gyrus, bilateral middle fontal gyrus, bilateral inferior and right superior parietal lobule, and right precuneus. These results support the view that lexical stress is processed as part of a sensorimotor network of speech comprehension. Moreover, our results connect beat processing in language to domain-independent timing perception.

Sentence understanding depends on contextual use of semantic and real world knowledge

Human language allows us to express our thoughts and ideas by combining entities, concepts and actions into multi-event episodes. Yet, the functional neuroanatomy engaged in interpretation of such high-level linguistic input remains poorly understood. Here, we used easy to detect and more subtle "borderline" anomalies to investigate the brain regions and mechanistic principles involved in the use of real-world event knowledge in language comprehension. Overall, the results showed that the processing of sentences in context engages a complex set of bilateral brain regions in the frontal, temporal and inferior parietal lobes. Easy anomalies preferentially engaged lower-order cortical areas adjacent to the primary auditory cortex. In addition, the left supramarginal gyrus and anterior temporal sulcus as well as the right posterior middle temporal gyrus contributed to the processing of easy and borderline anomalies. The observed pattern of results is explained in terms of (i) hierarchical processing along a dorsal-ventral axis and (ii) the assumption of high-order association areas serving as cortical hubs in the convergence of information in a distributed network. Finally, the observed modulation of BOLD signal in prefrontal areas provides support for their role in the implementation of executive control processes.

Age-Related Changes in Predictive Capacity Versus Internal Model Adaptability: Electrophysiological Evidence that Individual Differences Outweigh Effects of Age

Hierarchical predictive coding has been identified as a possible unifying principle of brain function, and recent work in cognitive neuroscience has examined how it may be affected by age–related changes. Using language comprehension as a test case, the present study aimed to dissociate age-related changes in prediction generation versus internal model adaptation following a prediction error. Event-related brain potentials (ERPs) were measured in a group of older adults (60–81 years; n = 40) as they read sentences of the form “The opposite of black is white/yellow/nice.” Replicating previous work in young adults, results showed a target-related P300 for the expected antonym (“white”; an effect assumed to reflect a prediction match), and a graded N400 effect for the two incongruous conditions (i.e. a larger N400 amplitude for the incongruous continuation not related to the expected antonym, “nice,” versus the incongruous associated condition, “yellow”). These effects were followed by a late positivity, again with a larger amplitude in the incongruous non-associated versus incongruous associated condition. Analyses using linear mixed-effects models showed that the target-related P300 effect and the N400 effect for the incongruous non-associated condition were both modulated by age, thus suggesting that age-related changes affect both prediction generation and model adaptation. However, effects of age were outweighed by the interindividual variability of ERP responses, as reflected in the high proportion of variance captured by the inclusion of by-condition random slopes for participants and items. We thus argue that – at both a neurophysiological and a functional level – the notion of general differences between language processing in young and older adults may only be of limited use, and that future research should seek to better understand the causes of interindividual variability in the ERP responses of older adults and its relation to cognitive performance.

Processing of false belief passages during natural story comprehension: An fMRI study

The neural correlates of theory of mind (ToM) are typically studied using paradigms which require participants to draw explicit, task-related inferences (e.g., in the false belief task). In a natural setup, such as listening to stories, false belief mentalizing occurs incidentally as part of narrative processing. In our experiment, participants listened to auditorily presented stories with false belief passages (implicit false belief processing) and immediately after each story answered comprehension questions (explicit false belief processing), while neural responses were measured with functional magnetic resonance imaging (fMRI). All stories included (among other situations) one false belief condition and one closely matched control condition. For the implicit ToM processing, we modeled the hemodynamic response during the false belief passages in the story and compared it to the hemodynamic response during the closely matched control passages. For implicit mentalizing, we found activation in typical ToM processing regions, that is the angular gyrus (AG), superior medial frontal gyrus (SmFG), precuneus (PCUN), middle temporal gyrus (MTG) as well as in the inferior frontal gyrus (IFG) billaterally. For explicit ToM, we only found AG activation. The conjunction analysis highlighted the left AG and MTG as well as the bilateral IFG as overlapping ToM processing regions for both implicit and explicit modes. Implicit ToM processing during listening to false belief passages, recruits the left SmFG and billateral PCUN in addition to the "mentalizing network" known form explicit processing tasks.

The neurophysiology of language processing shapes the evolution of grammar: evidence from case marking

Do principles of language processing in the brain affect the way grammar evolves over time or is language change just a matter of socio-historical contingency? While the balance of evidence has been ambiguous and controversial, we identify here a neurophysiological constraint on the processing of language that has a systematic effect on the evolution of how noun phrases are marked by case (i.e. by such contrasts as between the English base form she and the object form her). In neurophysiological experiments across diverse languages we found that during processing, participants initially interpret the first base-form noun phrase they hear (e.g. she…) as an agent (which would fit a continuation like … greeted him), even when the sentence later requires the interpretation of a patient role (as in … was greeted). We show that this processing principle is also operative in Hindi, a language where initial base-form noun phrases most commonly denote patients because many agents receive a special case marker ("ergative") and are often left out in discourse. This finding suggests that the principle is species-wide and independent of the structural affordances of specific languages. As such, the principle favors the development and maintenance of case-marking systems that equate base-form cases with agents rather than with patients. We confirm this evolutionary bias by statistical analyses of phylogenetic signals in over 600 languages worldwide, controlling for confounding effects from language contact. Our findings suggest that at least one core property of grammar systematically adapts in its evolution to the neurophysiological conditions of the brain, independently of socio-historical factors. This opens up new avenues for understanding how specific properties of grammar have developed in tight interaction with the biological evolution of our species.

Neurobiological roots of language in primate audition: common computational properties

Here, we present a new perspective on an old question: how does the neurobiology of human language relate to brain systems in nonhuman primates? We argue that higher-order language combinatorics, including sentence and discourse processing, can be situated in a unified, cross-species dorsal-ventral streams architecture for higher auditory processing, and that the functions of the dorsal and ventral streams in higher-order language processing can be grounded in their respective computational properties in primate audition. This view challenges an assumption, common in the cognitive sciences, that a nonhuman primate model forms an inherently inadequate basis for modeling higher-level language functions.

Two routes to actorhood: lexicalized potency to act and identification of the actor role

The inference of causality is a crucial cognitive ability and language processing is no exception: recent research suggests that, across different languages, the human language comprehension system attempts to identify the primary causer of the state of affairs described (the "actor") quickly and unambiguously (Bornkessel-Schlesewsky and Schlesewsky, 2009). This identification can take place verb-independently based on certain prominence cues (e.g., case, word order, animacy). Here, we present two experiments demonstrating that actor potential is also encoded at the level of individual nouns (a king is a better actor than a beggar). Experiment 1 collected ratings for 180 German nouns on 12 scales defined by adjective oppositions and deemed relevant for actorhood potential. By means of structural equation modeling, an actor potential (ACT) value was calculated for each noun. Experiment 2, an event-related potential study, embedded nouns from Experiment 1 in verb-final sentences, in which they were either actors or non-actors. N400 amplitude increased with decreasing ACT values and this modulation was larger for highly frequent nouns and for actor versus non-actor nouns. We argue that potency to act is lexically encoded for individual nouns and, since it modulates the N400 even for non-actor participants, it should be viewed as a property that modulates ease of lexical access (akin, for example, to lexical frequency). We conclude that two separate dimensions of actorhood computation are crucial to language comprehension: an experience-based, lexically encoded (bottom-up) representation of actorhood potential, and a prominence-based, computational mechanism for calculating goodness-of-fit to the actor role in a particular (top-down) sentence context.

The P600-as-P3 hypothesis revisited: single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned

The P600, a late positive ERP component following linguistically deviant stimuli, is commonly seen as indexing structural, high-level processes, e.g. of linguistic (re)analysis. It has also been identified with the P3 (P600-as-P3 hypothesis), which is thought to reflect a systemic neuromodulator release facilitating behavioural shifts and is usually response time aligned. We investigated single-trial alignment of the P600 to response, a critical prediction of the P600-as-P3 hypothesis. Participants heard sentences containing morphosyntactic and semantic violations and responded via a button press. The elicited P600 was perfectly response aligned, while an N400 following semantic deviations was stimulus aligned. This is, to our knowledge, the first single-trial analysis of language processing data using within-sentence behavioural responses as temporal covariates. Results support the P600-as-P3 perspective and thus constitute a step towards a neurophysiological grounding of language-related ERPs.

Cross-linguistic variation in the neurophysiological response to semantic processing: evidence from anomalies at the borderline of awareness

The N400 event-related brain potential (ERP) has played a major role in the examination of how the human brain processes meaning. For current theories of the N400, classes of semantic inconsistencies which do not elicit N400 effects have proven particularly influential. Semantic anomalies that are difficult to detect are a case in point ("borderline anomalies", e.g. "After an air crash, where should the survivors be buried?"), engendering a late positive ERP response but no N400 effect in English (Sanford, Leuthold, Bohan, & Sanford, 2011). In three auditory ERP experiments, we demonstrate that this result is subject to cross-linguistic variation. In a German version of Sanford and colleagues' experiment (Experiment 1), detected borderline anomalies elicited both N400 and late positivity effects compared to control stimuli or to missed borderline anomalies. Classic easy-to-detect semantic (non-borderline) anomalies showed the same pattern as in English (N400 plus late positivity). The cross-linguistic difference in the response to borderline anomalies was replicated in two additional studies with a slightly modified task (Experiment 2a: German; Experiment 2b: English), with a reliable LANGUAGE×ANOMALY interaction for the borderline anomalies confirming that the N400 effect is subject to systematic cross-linguistic variation. We argue that this variation results from differences in the language-specific default weighting of top-down and bottom-up information, concluding that N400 amplitude reflects the interaction between the two information sources in the form-to-meaning mapping.

Action and language mechanisms in the brain: data, models and neuroinformatics

We assess the challenges of studying action and language mechanisms in the brain, both singly and in relation to each other to provide a novel perspective on neuroinformatics, integrating the development of databases for encoding – separately or together – neurocomputational models and empirical data that serve systems and cognitive neuroscience.

Yes, you can? A speaker's potency to act upon his words orchestrates early neural responses to message-level meaning

Evidence is accruing that, in comprehending language, the human brain rapidly integrates a wealth of information sources-including the reader or hearer's knowledge about the world and even his/her current mood. However, little is known to date about how language processing in the brain is affected by the hearer's knowledge about the speaker. Here, we investigated the impact of social attributions to the speaker by measuring event-related brain potentials while participants watched videos of three speakers uttering true or false statements pertaining to politics or general knowledge: a top political decision maker (the German Federal Minister of Finance at the time of the experiment), a well-known media personality and an unidentifiable control speaker. False versus true statements engendered an N400 - late positivity response, with the N400 (150-450 ms) constituting the earliest observable response to message-level meaning. Crucially, however, the N400 was modulated by the combination of speaker and message: for false versus true political statements, an N400 effect was only observable for the politician, but not for either of the other two speakers; for false versus true general knowledge statements, an N400 was engendered by all three speakers. We interpret this result as demonstrating that the neurophysiological response to message-level meaning is immediately influenced by the social status of the speaker and whether he/she has the power to bring about the state of affairs described.

Reconciling time, space and function: a new dorsal-ventral stream model of sentence comprehension

We present a new dorsal-ventral stream framework for language comprehension which unifies basic neurobiological assumptions (Rauschecker & Scott, 2009) with a cross-linguistic neurocognitive sentence comprehension model (eADM; Bornkessel & Schlesewsky, 2006). The dissociation between (time-dependent) syntactic structure-building and (time-independent) sentence interpretation assumed within the eADM provides a basis for the division of labour between the dorsal and ventral streams in comprehension. We posit that the ventral stream performs time-independent unifications of conceptual schemata, serving to create auditory objects of increasing complexity. The dorsal stream engages in the time-dependent combination of elements, subserving both syntactic structuring and a linkage to action. Furthermore, frontal regions accomplish general aspects of cognitive control in the service of action planning and execution rather than linguistic processing. This architecture is supported by a range of existing empirical findings and helps to resolve a number of theoretical and empirical puzzles within the existing dorsal-ventral streams literature.

Subjective impressions do not mirror online reading effort: concurrent EEG-eyetracking evidence from the reading of books and digital media

In the rapidly changing circumstances of our increasingly digital world, reading is also becoming an increasingly digital experience: electronic books (e-books) are now outselling print books in the United States and the United Kingdom. Nevertheless, many readers still view e-books as less readable than print books. The present study thus used combined EEG and eyetracking measures in order to test whether reading from digital media requires higher cognitive effort than reading conventional books. Young and elderly adults read short texts on three different reading devices: a paper page, an e-reader and a tablet computer and answered comprehension questions about them while their eye movements and EEG were recorded. The results of a debriefing questionnaire replicated previous findings in that participants overwhelmingly chose the paper page over the two electronic devices as their preferred reading medium. Online measures, by contrast, showed shorter mean fixation durations and lower EEG theta band voltage density – known to covary with memory encoding and retrieval – for the older adults when reading from a tablet computer in comparison to the other two devices. Young adults showed comparable fixation durations and theta activity for all three devices. Comprehension accuracy did not differ across the three media for either group. We argue that these results can be explained in terms of the better text discriminability (higher contrast) produced by the backlit display of the tablet computer. Contrast sensitivity decreases with age and degraded contrast conditions lead to longer reading times, thus supporting the conclusion that older readers may benefit particularly from the enhanced contrast of the tablet. Our findings thus indicate that people's subjective evaluation of digital reading media must be dissociated from the cognitive and neural effort expended in online information processing while reading from such devices.

Prominence vs. aboutness in sequencing: a functional distinction within the left inferior frontal gyrus

Prior research on the neural bases of syntactic comprehension suggests that activation in the left inferior frontal gyrus (lIFG) correlates with the processing of word order variations. However, there are inconsistencies with respect to the specific subregion within the IFG that is implicated by these findings: the pars opercularis or the pars triangularis. Here, we examined the hypothesis that the dissociation between pars opercularis and pars triangularis activation may reflect functional differences between clause-medial and clause-initial word order permutations, respectively. To this end, we directly compared clause-medial and clause-initial object-before-subject orders in German in a within-participants, event-related fMRI design. Our results showed increased activation for object-initial sentences in a bilateral network of frontal, temporal and subcortical regions. Within the lIFG, posterior and inferior subregions showed only a main effect of word order, whereas more anterior and superior subregions showed effects of word order and sentence type, with higher activation for sentences with an argument in the clause-initial position. These findings are interpreted as evidence for a functional gradation of sequence processing within the left IFG: posterior subportions correlate with argument prominence-based (local) aspects of sequencing, while anterior subportions correlate with aboutness-based aspects of sequencing, which are crucial in linking the current sentence to the wider discourse. This proposal appears compatible with more general hypotheses about information processing gradients in prefrontal cortex (Koechlin & Summerfield, 2007).

Meaningful physical changes mediate lexical-semantic integration: top-down and form-based bottom-up information sources interact in the N400

Models of how the human brain reconstructs an intended meaning from a linguistic input often draw upon the N400 event-related potential (ERP) component as evidence. Current accounts of the N400 emphasise either the role of contextually induced lexical preactivation of a critical word (Lau, Phillips, & Poeppel, 2008) or the ease of integration into the overall discourse context including a wide variety of influencing factors (Hagoort & van Berkum, 2007). The present ERP study challenges both types of accounts by demonstrating a contextually independent and purely form-based bottom-up influence on the N400: the N400 effect for implausible sentence-endings was attenuated when the critical sentence-final word was capitalised (following a lowercase sentence context). By contrast, no N400 modulation occurred when the critical word involved a change from uppercase (sentence context) to lowercase. Thus, the N400 was only affected by a change to uppercase letters, as is often employed in computer-mediated communication as a sign of emphasis. This result indicates that N400 amplitude is reduced for unexpected words when a bottom-up (orthographic) cue signals that the word is likely to be highly informative. The lexical-semantic N400 thereby reflects the degree to which the semantic informativity of a critical word matches expectations, as determined by an interplay between top-down and bottom-up information sources, including purely form-based bottom-up information.

Think globally: cross-linguistic variation in electrophysiological activity during sentence comprehension

This paper demonstrates systematic cross-linguistic differences in the electrophysiological correlates of conflicts between form and meaning ("semantic reversal anomalies"). These engender P600 effects in English and Dutch (e.g. Kolk et al., 2003; Kuperberg et al., 2003), but a biphasic N400 - late positivity pattern in German (Schlesewsky and Bornkessel-Schlesewsky, 2009), and monophasic N400 effects in Turkish (Experiment 1) and Mandarin Chinese (Experiment 2). Experiment 3 revealed that, in Icelandic, semantic reversal anomalies show the English pattern with verbs requiring a position-based identification of argument roles, but the German pattern with verbs requiring a case-based identification of argument roles. The overall pattern of results reveals two separate dimensions of cross-linguistic variation: (i) the presence vs. absence of an N400, which we attribute to cross-linguistic differences with regard to the sequence-dependence of the form-to-meaning mapping and (ii) the presence vs. absence of a late positivity, which we interpret as an instance of a categorisation-related late P300, and which is observable when the language under consideration allows for a binary well-formedness categorisation of reversal anomalies. We conclude that, rather than reflecting linguistic domains such as syntax and semantics, the late positivity vs. N400 distinction is better understood in terms of the strategies that serve to optimise the form-to-meaning mapping in a given language.

Word order and Broca's region: evidence for a supra-syntactic perspective

It has often been suggested that the role of Broca's region in sentence comprehension can be explained with reference to general cognitive mechanisms (e.g. working memory, cognitive control). However, the (language-related) basis for such proposals is often restricted to findings on English. Here, we argue that an extension of the database to other languages can shed new light on the types of mechanisms that an adequate account of Broca's region should be equipped to deal with. This becomes most readily apparent in the domain of word order variations, which we examined in German verb-final sentences using event-related fMRI. Our results showed that activation in the pars opercularis--a core subregion of Broca's area--was not only modulated by the relative ordering of subject and object, but also by a further factor known to affect word order in a number of languages, namely referentiality. Notably, the finding provides the first demonstration of a word order-related activation difference within subject-initial sentences in this region. Additional parametric analyses using individual behavioral data as predictors further attest to the independence of the pars opercularis activation from: (a) sentence acceptability, and (b) difficulty in performing the experimental (judgment) task. We argue that these and related findings attest to the need for a processing mechanism that can manipulate predicate-independent, interacting and hierarchically structured relational representations during real time comprehension. These properties pose a challenge to existing accounts of pars opercularis function.

Setting the frame: the human brain activates a basic low-frequency network for language processing

Low-frequency fluctuations (LFFs) are a major source of variation in fMRI data. This has been established in numerous experiments-particularly in the resting state. Here we investigate LFFs in a task-dependent setting. We hypothesized that LFFs may contain information about cognitive networks that are specific to the overall task domain without being time locked to stimulus onsets. We analyzed data of 6 fMRI experiments, 4 of which belonged to the language domain. After regressing out specifics of the experimental design and low-pass filtering (<0.1 Hz), we found that the 4 language experiments produced a correlational pattern that was not present in the 2 nonlanguage studies. Specifically, a region in the posterior part of the left superior temporal sulcus/gyrus was consistently correlated with both the left Brodmann's area 44 and the left frontal operculum in all 4 language studies, whereas this correlation was not found in the 2 other experiments. This finding indicates the existence of a basic network that acts as a general framework for language processing. In contrast to networks obtained by a conventional conjunction analysis of activation maps, this network is independent of experimental specifics such as stimulus onsets and exists in the low-frequency range.