Correlated Brain Indexes of Semantic Prediction and Prediction Error: Brain Localization and Category Specificity

Speech Prosody Serves Temporal Prediction of Language via Contextual Entrainment

Temporal prediction assists language comprehension. In a series of recent behavioral studies, we have shown that listeners specifically employ rhythmic modulations of prosody to estimate the duration of upcoming sentences, thereby speeding up comprehension. In the current human magnetoencephalography (MEG) study on participants of either sex, we show that the human brain achieves this function through a mechanism termed entrainment. Through entrainment, electrophysiological brain activity maintains and continues contextual rhythms beyond their offset. Our experiment combined exposure to repetitive prosodic contours with the subsequent presentation of visual sentences that either matched or mismatched the duration of the preceding contour. During exposure to prosodic contours, we observed MEG coherence with the contours, which was source-localized to right-hemispheric auditory areas. During the processing of the visual targets, activity at the frequency of the preceding contour was still detectable in the MEG; yet sources shifted to the (left) frontal cortex, in line with a functional inheritance of the rhythmic acoustic context for prediction. Strikingly, when the target sentence was shorter than expected from the preceding contour, an omission response appeared in the evoked potential record. We conclude that prosodic entrainment is a functional mechanism of temporal prediction in language comprehension. In general, acoustic rhythms appear to endow language for employing the brain's electrophysiological mechanisms of temporal prediction.

The prediction potential indexes the meaning and communicative function of upcoming utterances

Prediction has a fundamental role in language processing. However, predictions can be made at different levels, and it is not always clear whether speech sounds, morphemes, words, meanings, or communicative functions are anticipated during dialogues. Previous studies reported specific brain signatures of communicative pragmatic function, in particular enhanced brain responses immediately after encountering an utterance used to request an object from a partner, but relatively smaller ones when the same utterance was used for naming the object. The present experiment now investigates whether similar neuropragmatic signatures emerge in recipients before the onset of upcoming utterances carrying different predictable communicative functions. Trials started with a context question and object pictures displayed on the screen, raising the participant's expectation that words from a specific semantic category (food or tool) would subsequently be used to either name or request one of the objects. Already 600 msec before utterance onset, a larger prediction potential was observed when a request was anticipated relative to naming expectation. As this result is congruent with the neurophysiological difference previously observed right after the critical utterance, the anticipatory brain activity may index predictions about the social-communicative function of upcoming utterances. In addition, we also found that the predictable semantic category of the upcoming word was likewise reflected in the anticipatory brain potential. Thus, the neurophysiological characteristics of the prediction potential can capture different types of upcoming linguistic information, including semantic and pragmatic aspects of an upcoming utterance and communicative action.

Neurophysiological signatures of prediction in language: A critical review of anticipatory negativities

Recent event-related potential (ERP) studies in language comprehension converge in finding anticipatory negativities preceding words or word segments that can be pre-activated based on either sentence contexts or phonological cues. We review these findings from different paradigms in the light of evidence from other cognitive domains in which slow negative potentials have long been associated with anticipatory processes and discuss their potential underlying mechanisms. We propose that this family of anticipatory negativities captures common mechanisms associated with the pre-activation of linguistic information both within words and within sentences. Future studies could utilize these anticipatory negativities in combination with other, well-established ERPs, to simultaneously track prediction-related processes emerging at different time intervals (before and after the perception of pre-activated input) and with distinct time courses (shorter-lived and longer-lived cognitive operations).

The impact of early and late blindness on language and verbal working memory: A brain-constrained neural model

Neural circuits related to language exhibit a remarkable ability to reorganize and adapt in response to visual deprivation. Particularly, early and late blindness induce distinct neuroplastic changes in the visual cortex, repurposing it for language and semantic processing. Interestingly, these functional changes provoke a unique cognitive advantage - enhanced verbal working memory, particularly in early blindness. Yet, the underlying neuromechanisms and the impact on language and memory-related circuits remain not fully understood. Here, we applied a brain-constrained neural network mimicking the structural and functional features of the frontotemporal-occipital cortices, to model conceptual acquisition in early and late blindness. The results revealed differential expansion of conceptual-related neural circuits into deprived visual areas depending on the timing of visual loss, which is most prominent in early blindness. This neural recruitment is fundamentally governed by the biological principles of neural circuit expansion and the absence of uncorrelated sensory input. Critically, the degree of these changes is constrained by the availability of neural matter previously allocated to visual experiences, as in the case of late blindness. Moreover, we shed light on the implication of visual deprivation on the neural underpinnings of verbal working memory, revealing longer reverberatory neural activity in 'blind models' as compared to the sighted ones. These findings provide a better understanding of the interplay between visual deprivations, neuroplasticity, language processing and verbal working memory.

Predictive Brain Activity Shows Congruent Semantic Specificity in Language Comprehension and Production

Sentence fragments strongly predicting a specific subsequent meaningful word elicit larger preword slow waves, prediction potentials (PPs), than unpredictive contexts. To test the current predictive processing models, 128-channel EEG data were collected from both sexes to examine whether (1) different semantic PPs are elicited in language comprehension and production and (2) whether these PPs originate from the same specific "prediction area(s)" or rather from widely distributed category-specific neuronal circuits reflecting the meaning of the predicted item. Slow waves larger after predictable than unpredictable contexts were present both before subjects heard the sentence-final word in the comprehension experiment and before they pronounced the sentence-final word in the production experiment. Crucially, cortical sources underlying the semantic PP were distributed across several cortical areas and differed between the semantic categories of the expected words. In both production and comprehension, the anticipation of animal words was reflected by sources in posterior visual areas, whereas predictable tool words were preceded by sources in the frontocentral sensorimotor cortex. For both modalities, PP size increased with higher cloze probability, thus further confirming that it reflects semantic prediction, and with shorter latencies with which participants completed sentence fragments. These results sit well with theories viewing distributed semantic category-specific circuits as the mechanistic basis of semantic prediction in the two modalities.

Predicting and coding sound into action translation in spinal cord injured people

Motor activation in response to perception of action-related stimuli may depend on a resonance mechanism subserving action understanding. The extent to which this mechanism is innate or learned from sensorimotor experience is still unclear. Here, we recorded EEG while people with paraplegia or tetraplegia consequent to spinal cord injury (SCI) and healthy control participants were presented with action sounds produced by body parts (mouth, hands or feet) that were or were not affected by SCI. Non-action sounds were used as further control. We observed reduced brain activation in subjects affected by SCI at both pre- and post-stimulus latencies specifically for those actions whose effector was disconnected by the spinal lesion (i.e., hand sound for tetraplegia and leg sound for both paraplegia and tetraplegia). Correlation analyses showed that these modulations were functionally linked with the chronicity of the lesion, indicating that the longer the time the lesion- EEG data acquisition interval and/or the more the lesion occurred at a young age, the weaker was the cortical activity in response to these action sounds. Tellingly, source estimations confirmed that these modulations originated from a deficit in the motor resonance mechanism, by showing diminished activity in premotor (during prediction and perception) and near the primary motor (during perception) areas. Such dissociation along the cortical hierarchy is consistent with both previous reports in healthy subjects and with hierarchical predictive coding accounts. Overall, these data expand on the notion that sensorimotor experience maintains the cortical representations relevant to anticipate and perceive action-related stimuli.

Swift Prosodic Modulation of Lexical Access: Brain Potentials From Three North Germanic Language Varieties

PURPOSE

According to most models of spoken word recognition, listeners probabilistically activate a set of lexical candidates, which is incrementally updated as the speech signal unfolds. Speech carries segmental (speech sound) as well as suprasegmental (prosodic) information. The role of the latter in spoken word recognition is less clear. We investigated how suprasegments (tone and voice quality) in three North Germanic language varieties affected lexical access by scrutinizing temporally fine-grained neurophysiological effects of lexical uncertainty and information gain.

METHOD

Three event-related potential (ERP) studies were reanalyzed. In all varieties investigated, suprasegments are associated with specific word endings. Swedish has two lexical "word accents" realized as pitch falls with different timings across dialects. In Danish, the distinction is in voice quality. We combined pronunciation lexica and frequency lists to calculate estimates of lexical uncertainty about an unfolding word and information gain upon hearing a suprasegmental cue and the segment upon which it manifests. We used single-trial mixed-effects regression models run every 4 ms.

RESULTS

Only lexical uncertainty showed solid results: a frontal effect at 150-400 ms after suprasegmental cue onset and a later posterior effect after 200 ms. While a model including only segmental information mostly performed better, it was outperformed by the suprasegmental model at 200-330 ms at frontal sites.

CONCLUSIONS

The study points to suprasegmental cues contributing to lexical access over and beyond segments after around 200 ms in the North Germanic varieties investigated. Furthermore, the findings indicate that a previously reported "pre-activation negativity" predominantly reflects forward-looking processing.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.25016486.

Conditionals in context: Brain signatures of prediction in discourse processing

Comprehenders are known to generate expectations about upcoming linguistic input at the sentence and discourse level. However, most previous studies on prediction focused mainly on word-induced brain activity rather than examining neural activity preceding a critical stimulus in discourse processing, where prediction actually takes place. In this EEG study, participants were presented with multiple sentences resembling a discourse including conditional sentences with either only if or if, which are characterized by different semantics, triggering stronger or weaker predictions about the possible continuation of the presented discourses, respectively. Results revealed that discourses including only if, as compared to discourses with bare if, triggered an increased predictive neural activity before the expected critical word, resembling the readiness potential. Moreover, word-induced P300 brain responses were found to be enhanced by unpredictable discourse continuations and reduced in predictable discourse continuations. Intriguingly, brain responses preceding and following the critical word were found to be correlated, which yields evidence for predictive activity modulating word-induced processing on the discourse level. These findings shed light on the predictive nature of neural processes at the discourse level, critically advancing our understanding of the functional interconnection between discourse understanding and prediction processes in brain and mind.

Leading and following: Noise differently affects semantic and acoustic processing during naturalistic speech comprehension

Despite the distortion of speech signals caused by unavoidable noise in daily life, our ability to comprehend speech in noisy environments is relatively stable. However, the neural mechanisms underlying reliable speech-in-noise comprehension remain to be elucidated. The present study investigated the neural tracking of acoustic and semantic speech information during noisy naturalistic speech comprehension. Participants listened to narrative audio recordings mixed with spectrally matched stationary noise at three signal-to-ratio (SNR) levels (no noise, 3 dB, -3 dB), and 60-channel electroencephalography (EEG) signals were recorded. A temporal response function (TRF) method was employed to derive event-related-like responses to the continuous speech stream at both the acoustic and the semantic levels. Whereas the amplitude envelope of the naturalistic speech was taken as the acoustic feature, word entropy and word surprisal were extracted via the natural language processing method as two semantic features. Theta-band frontocentral TRF responses to the acoustic feature were observed at around 400 ms following speech fluctuation onset over all three SNR levels, and the response latencies were more delayed with increasing noise. Delta-band frontal TRF responses to the semantic feature of word entropy were observed at around 200 to 600 ms leading to speech fluctuation onset over all three SNR levels. The response latencies became more leading with increasing noise and decreasing speech comprehension and intelligibility. While the following responses to speech acoustics were consistent with previous studies, our study revealed the robustness of leading responses to speech semantics, which suggests a possible predictive mechanism at the semantic level for maintaining reliable speech comprehension in noisy environments.

Unveiling the need of interactions for social N400s and supporting the N400 inhibition hypothesis

When participants (Pps) are presented with stimuli in the presence of another person, they may consider that person's perspective. Indeed, five recent ERP studies show that the amplitudes of their N400s are increased. The two most recent ones reveal that these social-N400 increases occur even when instructions do not require a focus on the other's perspective. These increases also happen when Pps know that this other person has the same stimulus information as they have. However, in all these works, Pps could see the other person. Here, we tested whether the interaction occurring with this sight is important or whether these social N400 increases also occur when the other person is seated a bit behind Pps, who are aware of it. All had to decide whether the word ending short stories was coherent, incoherent, or equivocal. No social N400 increase was observed: N400s elicited by those words in Pps who were with a confederate (n = 50) were similar to those of Pps who were alone (n = 51). On the other hand, equivocal endings did not elicit larger N400s than coherent ones but triggered larger late posterior positivities (LPPs), like in previous studies. The discussion focuses on the circumstances in which perspective-taking occurs and on the functional significance of the N400 and the LPP.

Predicting coarse-grained semantic features in language comprehension: evidence from ERP representational similarity analysis and Chinese classifier

Existing studies demonstrate that comprehenders can predict semantic information during language comprehension. Most evidence comes from a highly constraining context, in which a specific word is likely to be predicted. One question that has been investigated less is whether prediction can occur when prior context is less constraining for predicting specific words. Here, we aim to address this issue by examining the prediction of animacy features in low-constraining context, using electroencephalography (EEG), in combination with representational similarity analysis (RSA). In Chinese, a classifier follows a numeral and precedes a noun, and classifiers constrain animacy features of upcoming nouns. In the task, native Chinese Mandarin speakers were presented with either animate-constraining or inanimate-constraining classifiers followed by congruent or incongruent nouns. EEG amplitude analysis revealed an N400 effect for incongruent conditions, reflecting the difficulty of semantic integration when an incompatible noun is encountered. Critically, we quantified the similarity between patterns of neural activity following the classifiers. RSA results revealed that the similarity between patterns of neural activity following animate-constraining classifiers was greater than following inanimate-constraining classifiers, before the presentation of the nouns, reflecting pre-activation of animacy features of nouns. These findings provide evidence for the prediction of coarse-grained semantic feature of upcoming words.

Multiple functions of the angular gyrus at high temporal resolution

Here, the functions of the angular gyrus (AG) are evaluated in the light of current evidence from transcranial magnetic/electric stimulation (TMS/TES) and EEG/MEG studies. 65 TMS/TES and 52 EEG/MEG studies were examined in this review. TMS/TES literature points to a causal role in semantic processing, word and number processing, attention and visual search, self-guided movement, memory, and self-processing. EEG/MEG studies reported AG effects at latencies varying between 32 and 800 ms in a wide range of domains, with a high probability to detect an effect at 300-350 ms post-stimulus onset. A three-phase unifying model revolving around the process of sensemaking is then suggested: (1) early AG involvement in defining the current context, within the first 200 ms, with a bias toward the right hemisphere; (2) attention re-orientation and retrieval of relevant information within 200-500 ms; and (3) cross-modal integration at late latencies with a bias toward the left hemisphere. This sensemaking process can favour accuracy (e.g. for word and number processing) or plausibility (e.g. for comprehension and social cognition). Such functions of the AG depend on the status of other connected regions. The much-debated semantic role is also discussed as follows: (1) there is a strong TMS/TES evidence for a causal semantic role, (2) current EEG/MEG evidence is however weak, but (3) the existing arguments against a semantic role for the AG are not strong. Some outstanding questions for future research are proposed. This review recognizes that cracking the role(s) of the AG in cognition is possible only when its exact contributions within the default mode network are teased apart.

Linguistic signs in action: The neuropragmatics of speech acts

What makes human communication exceptional is the ability to grasp speaker's intentions beyond what is said verbally. How the brain processes communicative functions is one of the central concerns of the neurobiology of language and pragmatics. Linguistic-pragmatic theories define these functions as speech acts, and various pragmatic traits characterise them at the levels of propositional content, action sequence structure, related commitments and social aspects. Here I discuss recent neurocognitive studies, which have shown that the use of identical linguistic signs in conveying different communicative functions elicits distinct and ultra-rapid neural responses. Interestingly, cortical areas show differential involvement underlying various pragmatic features related to theory-of-mind, emotion and action for specific speech acts expressed with the same utterances. Drawing on a neurocognitive model, I posit that understanding speech acts involves the expectation of typical partner follow-up actions and that this predictive knowledge is immediately reflected in mind and brain.

Predictive language comprehension in Parkinson's disease

Verb and action knowledge deficits are reported in persons with Parkinson's disease (PD), even in the absence of dementia or mild cognitive impairment. However, the impact of these deficits on combinatorial semantic processing is less well understood. Following on previous verb and action knowledge findings, we tested the hypothesis that PD impairs the ability to integrate event-based thematic fit information during online sentence processing. Specifically, we anticipated persons with PD with age-typical cognitive abilities would perform more poorly than healthy controls during a visual world paradigm task requiring participants to predict a target object constrained by the thematic fit of the agent-verb combination. Twenty-four PD and 24 healthy age-matched participants completed comprehensive neuropsychological assessments. We recorded participants' eye movements as they heard predictive sentences (The fisherman rocks the boat) alongside target, agent-related, verb-related, and unrelated images. We tested effects of group (PD/control) on gaze using growth curve models. There were no significant differences between PD and control participants, suggesting that PD participants successfully and rapidly use combinatory thematic fit information to predict upcoming language. Baseline sentences with no predictive information (e.g., Look at the drum) confirmed that groups showed equivalent sentence processing and eye movement patterns. Additionally, we conducted an exploratory analysis contrasting PD and controls' performance on low-motion-content versus high-motion-content verbs. This analysis revealed fewer predictive fixations in high-motion sentences only for healthy older adults. PD participants may adapt to their disease by relying on spared, non-action-simulation-based language processing mechanisms, although this conclusion is speculative, as the analyses of high- vs. low-motion items was highly limited by the study design. These findings provide novel evidence that individuals with PD match healthy adults in their ability to use verb meaning to predict upcoming nouns despite previous findings of verb semantic impairment in PD across a variety of tasks.

OUP accepted manuscript

During conversations, speech prosody provides important clues about the speaker’s communicative intentions. In many languages, a rising vocal pitch at the end of a sentence typically expresses a question function, whereas a falling pitch suggests a statement. Here, the neurophysiological basis of intonation and speech act understanding were investigated with high-density electroencephalography (EEG) to determine whether prosodic features are reflected at the neurophysiological level. Already approximately 100 ms after the sentence-final word differing in prosody, questions, and statements expressed with the same sentences led to different neurophysiological activity recorded in the event-related potential. Interestingly, low-pass filtered sentences and acoustically matched nonvocal musical signals failed to show any neurophysiological dissociations, thus suggesting that the physical intonation alone cannot explain this modulation. Our results show rapid neurophysiological indexes of prosodic communicative information processing that emerge only when pragmatic and lexico-semantic information are fully expressed. The early enhancement of question-related activity compared with statements was due to sources in the articulatory-motor region, which may reflect the richer action knowledge immanent to questions, namely the expectation of the partner action of answering the question. The present findings demonstrate a neurophysiological correlate of prosodic communicative information processing, which enables humans to rapidly detect and understand speaker intentions in linguistic interactions.

The Neural Basis of Semantic Prediction in Sentence Comprehension

Although prediction plays an important role in language comprehension, its precise neural basis remains unclear. This fMRI study investigated whether and how semantic-category-specific and common cerebral areas are recruited in predictive semantic processing during sentence comprehension. We manipulated the semantic constraint of sentence contexts, upon which a tool-related, a building-related, or no specific category of noun is highly predictable. This noun-predictability effect was measured not only over the target nouns but also over their preceding transitive verbs. Both before and after the appearance of target nouns, left anterior supramarginal gyrus was specifically activated for tool-related nouns and left parahippocampal place area was activated specifically for building-related nouns. The semantic-category common areas included a subset of left inferior frontal gyrus during the anticipation of incoming target nouns (activity enhancement for high predictability) and included a wide spread of areas (bilateral inferior frontal gyrus, left superior/middle temporal gyrus, left medial pFC, and left TPJ) during the integration of actually perceived nouns (activity reduction for high predictability). These results indicated that the human brain recruits fine divisions of cortical areas to distinguish different semantic categories of predicted words, and anticipatory semantic processing relies, at least partially, on top-down prediction conducted in higher-level cortical areas.

Biological constraints on neural network models of cognitive function

Neural network models are potential tools for improving our understanding of complex brain functions. To address this goal, these models need to be neurobiologically realistic. However, although neural networks have advanced dramatically in recent years and even achieve human-like performance on complex perceptual and cognitive tasks, their similarity to aspects of brain anatomy and physiology is imperfect. Here, we discuss different types of neural models, including localist, auto-associative, hetero-associative, deep and whole-brain networks, and identify aspects under which their biological plausibility can be improved. These aspects range from the choice of model neurons and of mechanisms of synaptic plasticity and learning to implementation of inhibition and control, along with neuroanatomical properties including areal structure and local and long-range connectivity. We highlight recent advances in developing biologically grounded cognitive theories and in mechanistically explaining, on the basis of these brain-constrained neural models, hitherto unaddressed issues regarding the nature, localization and ontogenetic and phylogenetic development of higher brain functions. In closing, we point to possible future clinical applications of brain-constrained modelling.

An ERP study of hemispheric differences in perceptual representations of language reveals meaning attribution in the right hemisphere and constituents of the N400-effect

Recent findings have revealed that the right hemisphere (RH) is uniquely involved in integrating perceptual information from linguistic input to simulate a mental model of that input. We extend on these findings by testing whether meaning is generated from such models. Participants (N = 37) heard auditory passages describing the visuospatial arrangement of elements into a perceptual representation of a familiar object, then judged whether a laterally-presented target word matched the object. We found a central N400-effect for left visual-field targets, suggesting that meaning was also accessible to the RH. There was no statistical difference for right visual-field targets. Principle component analysis of the data revealed that the N400-effect was driven by positive components. Consequently, the results suggest that i) RH contributions to language comprehension include integrative and perceptual processes that enable overall meaning to be generated from representations of discourse, and ii) positive ERP components may produce N400-effects.

The encoding of stochastic regularities is facilitated by action-effect predictions

Our brains continuously build and update predictive models of the world, sources of prediction being drawn for example from sensory regularities and/or our own actions. Yet, recent results in the auditory system indicate that stochastic regularities may not be easily encoded when a rare medium pitch deviant is presented between frequent high and low pitch standard sounds in random order, as reflected in the lack of sensory prediction error event-related potentials [i.e., mismatch negativity (MMN)]. We wanted to test the implication of the predictive coding theory that predictions based on higher-order generative models-here, based on action intention, are fed top-down in the hierarchy to sensory levels. Participants produced random sequences of high and low pitch sounds by button presses in two conditions: In a "specific" condition, one button produced high and the other low pitch sounds; in an "unspecific" condition, both buttons randomly produced high or low-pitch sounds. Rare medium pitch deviants elicited larger MMN and N2 responses in the "specific" compared to the "unspecific" condition, despite equal sound probabilities. These results thus demonstrate that action-effect predictions can boost stochastic regularity-based predictions and engage higher-order deviance detection processes, extending previous notions on the role of action predictions at sensory levels.