Prediction in the service of comprehension: modulated early brain responses to omitted speech segments

Morphosyntactic prediction in automatic neural processing of spoken language: EEG evidence

Automatic parsing of syntactic information by the human brain is a well-established phenomenon, but its mechanisms remain poorly understood. Its best-known neurophysiological reflection is the so-called early left-anterior negativity (ELAN) component of event-related potentials (ERPs), with two alternative hypotheses for its origin: (1) error detection, or (2) morphosyntactic prediction/priming. To test these alternatives, we conducted two experiments using a non-attend passive design with visual distraction and recorded ERPs to spoken pronoun-verb phrases with/without agreement violations and to the same critical verbs presented in isolation without preceding pronouns. The results revealed an ELAN at ∼130-220 ms for pronoun-verb gender agreement violations, confirming a high degree of automaticity in early morphosyntactic parsing. Critically, the strongest ELAN was elicited by verbs outside phrasal context, which suggests that the typical ELAN pattern is underpinned by a reduction of ERP amplitudes for felicitous combinations, reflecting syntactic priming/predictability between related words/morphemes (potentially mediated by associative links formed during previous linguistic experience) rather than specialised error-detection processes.

Decoding predicted musical notes from omitted stimulus potentials

Electrophysiological studies have investigated predictive processing in music by examining event-related potentials (ERPs) elicited by the violation of musical expectations. While several studies have reported that the predictability of stimuli can modulate the amplitude of ERPs, it is unclear how specific the representation of the expected note is. The present study addressed this issue by recording the omitted stimulus potentials (OSPs) to avoid contamination of bottom-up sensory processing with top-down predictive processing. Decoding of the omitted content was attempted using a support vector machine, which is a type of machine learning. ERP responses to the omission of four target notes (E, F, A, and C) at the same position in familiar and unfamiliar melodies were recorded from 25 participants. The results showed that the omission N1 were larger in the familiar melody condition than in the unfamiliar melody condition. The decoding accuracy of the four omitted notes was significantly higher in the familiar melody condition than in the unfamiliar melody condition. These results suggest that the OSPs contain discriminable predictive information, and the higher the predictability, the more the specific representation of the expected note is generated.

Processing of auditory feedback in perisylvian and insular cortex

When we speak, we not only make movements with our mouth, lips, and tongue, but we also hear the sound of our own voice. Thus, speech production in the brain involves not only controlling the movements we make, but also auditory and sensory feedback. Auditory responses are typically suppressed during speech production compared to perception, but how this manifests across space and time is unclear. Here we recorded intracranial EEG in seventeen pediatric, adolescent, and adult patients with medication-resistant epilepsy who performed a reading/listening task to investigate how other auditory responses are modulated during speech production. We identified onset and sustained responses to speech in bilateral auditory cortex, with a selective suppression of onset responses during speech production. Onset responses provide a temporal landmark during speech perception that is redundant with forward prediction during speech production. Phonological feature tuning in these "onset suppression" electrodes remained stable between perception and production. Notably, the posterior insula responded at sentence onset for both perception and production, suggesting a role in multisensory integration during feedback control.

Visual omitted stimulus potentials are not retinotopic

Omitted stimulus potentials (OSPs) are elicited in response to the omission of expected stimuli and are thought to reflect prediction errors. If prediction errors are signaled in the sensory cortex, OSPs are expected to be generated in the sensory cortex. The present study investigated the involvement of the early visual cortex in the generation of OSPs by testing whether omitted visual stimuli elicit brain responses in a spatially specific manner. Checkerboard pattern stimuli were presented alternately in the upper and lower visual fields, and the stimuli were omitted in 10 % of the trials. Event-related potentials were recorded from 33 participants. While a retinotopic C1 component was evoked by real visual stimuli, omitted stimuli did not produce any response reflecting retinotopy but did elicit a visual mismatch negativity, which was larger for omitted stimuli expected in the lower visual field than for those in the upper visual field. These results suggest that omitted visual stimuli are processed in a different pathway than actual stimuli.

Multivariate pattern analysis of EEG reveals nuanced impact of negation on sentence processing in the N400 and later time windows

The neurocognitive mechanism underlying negation processing remains controversial. While negation is suggested to modulate the access of word meaning, no such evidence has been observed in the event-related potential (ERP) literature on sentence processing. In the current study, we applied both univariate ERP and multivariate pattern analysis (MVPA) methods to examine the processing of sentence negation. We investigated two types of negative congruent/incongruent sentence pairs with truth-value evaluation (e.g., "A robin is a/not a bird") and without (e.g., "The woman reads a/no book"). In the N400 time window, ERPs consistently showed increased negativity for negative and incongruent conditions. MVPA, on the other hand, revealed nuanced interactions between polarity and congruency. In the later P600 time window, MVPA but not the ERPs revealed an effect of congruency, which may be functionally distinct from the N400 window. We further used cross-decoding to show that the cognitive processes underlying the N400 window for both affirmative and negative sentences are comparable, whereas in the P600 window, only for the truth sentences, negative sentences showed a distinct pattern from their affirmative counterparts. Our results thus speak for a more interactive, but nevertheless serial and biphasic, and potentially construction-specific processing account of negation. We also discuss the advantage of applying MVPA in addition to the classical univariate methods for a better understanding of the neurobiology of negation processing and language comprehension alike.

Parallelisms and deviations: two fundamentals of an aesthetics of poetic diction

Poetic diction routinely involves two complementary classes of features: (i) parallelisms, i.e. repetitive patterns (rhyme, metre, alliteration, etc.) that enhance the predictability of upcoming words, and (ii) poetic deviations that challenge standard expectations/predictions regarding regular word form and order. The present study investigated how these two prediction-modulating fundamentals of poetic diction affect the cognitive processing and aesthetic evaluation of poems, humoristic couplets and proverbs. We developed quantitative measures of these two groups of text features. Across the three text genres, higher deviation scores reduced both comprehensibility and aesthetic liking whereas higher parallelism scores enhanced these. The positive effects of parallelism are significantly stronger than the concurrent negative effects of the features of deviation. These results are in accord with the hypothesis that art reception involves an interplay of prediction errors and prediction error minimization, with the latter paving the way for processing fluency and aesthetic liking. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectives'.

Speaker-induced Suppression in EEG during a Naturalistic Reading and Listening Task

Speaking elicits a suppressed neural response when compared with listening to others' speech, a phenomenon known as speaker-induced suppression (SIS). Previous research has focused on investigating SIS at constrained levels of linguistic representation, such as the individual phoneme and word level. Here, we present scalp EEG data from a dual speech perception and production task where participants read sentences aloud then listened to playback of themselves reading those sentences. Playback was separated into immediate repetition of the previous trial and randomized repetition of a former trial to investigate if forward modeling of responses during passive listening suppresses the neural response. Concurrent EMG was recorded to control for movement artifact during speech production. In line with previous research, ERP analyses at the sentence level demonstrated suppression of early auditory components of the EEG for production compared with perception. To evaluate whether linguistic abstractions (in the form of phonological feature tuning) are suppressed during speech production alongside lower-level acoustic information, we fit linear encoding models that predicted scalp EEG based on phonological features, EMG activity, and task condition. We found that phonological features were encoded similarly between production and perception. However, this similarity was only observed when controlling for movement by using the EMG response as an additional regressor. Our results suggest that SIS operates at a sensory representational level and is dissociated from higher order cognitive and linguistic processing that takes place during speech perception and production. We also detail some important considerations when analyzing EEG during continuous speech production.

Auditory representations for long lasting sounds: Insights from event-related brain potentials and neural oscillations

The basic features of short sounds, such as frequency and intensity including their temporal dynamics, are integrated in a unitary representation. Knowledge on how our brain processes long lasting sounds is scarce. We review research utilizing the Mismatch Negativity event-related potential and neural oscillatory activity for studying representations for long lasting simple versus complex sounds such as sinusoidal tones versus speech. There is evidence for a temporal constraint in the formation of auditory representations: Auditory edges like sound onsets within long lasting sounds open a temporal window of about 350 ms in which the sounds' dynamics are integrated into a representation, while information beyond that window contributes less to that representation. This integration window segments the auditory input into short chunks. We argue that the representations established in adjacent integration windows can be concatenated into an auditory representation of a long sound, thus, overcoming the temporal constraint.

Multiple functions of the angular gyrus at high temporal resolution

Here, the functions of the angular gyrus (AG) are evaluated in the light of current evidence from transcranial magnetic/electric stimulation (TMS/TES) and EEG/MEG studies. 65 TMS/TES and 52 EEG/MEG studies were examined in this review. TMS/TES literature points to a causal role in semantic processing, word and number processing, attention and visual search, self-guided movement, memory, and self-processing. EEG/MEG studies reported AG effects at latencies varying between 32 and 800 ms in a wide range of domains, with a high probability to detect an effect at 300-350 ms post-stimulus onset. A three-phase unifying model revolving around the process of sensemaking is then suggested: (1) early AG involvement in defining the current context, within the first 200 ms, with a bias toward the right hemisphere; (2) attention re-orientation and retrieval of relevant information within 200-500 ms; and (3) cross-modal integration at late latencies with a bias toward the left hemisphere. This sensemaking process can favour accuracy (e.g. for word and number processing) or plausibility (e.g. for comprehension and social cognition). Such functions of the AG depend on the status of other connected regions. The much-debated semantic role is also discussed as follows: (1) there is a strong TMS/TES evidence for a causal semantic role, (2) current EEG/MEG evidence is however weak, but (3) the existing arguments against a semantic role for the AG are not strong. Some outstanding questions for future research are proposed. This review recognizes that cracking the role(s) of the AG in cognition is possible only when its exact contributions within the default mode network are teased apart.

Evidence for predictions established by phantom sound

Predictions, the bridge between the internal and external worlds, are established by prior experience and updated by sensory stimuli. Responses to omitted but unexpected stimuli, known as omission responses, can break the one-to-one mapping of stimulus-response and can expose predictions established by the preceding stimulus built up. While research into exogenous predictions (driven by external stimuli) is often reported, that into endogenous predictions (driven by internal percepts) is rarely available in the literature. Here, we report evidence for endogenous predictions established by the Zwicker tone illusion, a phantom pure-tone-like auditory percept following notch noises. We found that MMN, P300, and theta oscillations could be recorded using an omission paradigm in subjects who can perceive Zwicker tone illusions, but could not in those who cannot. The MMN and P300 responses relied on attention, but theta oscillations did not. In-depth analysis shows that an increase in single-trial theta power, including total and induced theta, with the endogenous prediction, is lateralized to the left frontal brain areas. Our study depicts that the brain automatically analyzes internal perception, progressively establishes predictions and yields prediction errors in the left frontal region when a violation occurs.

Neural dynamics of phoneme sequences reveal position-invariant code for content and order

Speech consists of a continuously-varying acoustic signal. Yet human listeners experience it as sequences of discrete speech sounds, which are used to recognise discrete words. To examine how the human brain appropriately sequences the speech signal, we recorded two-hour magnetoencephalograms from 21 participants listening to short narratives. Our analyses show that the brain continuously encodes the three most recently heard speech sounds in parallel, and maintains this information long past its dissipation from the sensory input. Each speech sound representation evolves over time, jointly encoding both its phonetic features and the amount of time elapsed since onset. As a result, this dynamic neural pattern encodes both the relative order and phonetic content of the speech sequence. These representations are active earlier when phonemes are more predictable, and are sustained longer when lexical identity is uncertain. Our results show how phonetic sequences in natural speech are represented at the level of populations of neurons, providing insight into what intermediary representations exist between the sensory input and sub-lexical units. The flexibility in the dynamics of these representations paves the way for further understanding of how such sequences may be used to interface with higher order structure such as lexical identity.

Distraction by violation of sensory predictions: Functional distinction between deviant sounds and unexpected silences

It has been established that participants performing a continuous categorization task respond significantly slower following the presentation of unexpected, task-irrelevant, auditory stimuli, compared to a repetitive (standard) sound. Evidence indicates that such distraction emerges because of the violation of sensory predictions. This has typically been studied by measuring the impact of replacing the repeated sound by a different sound on rare and unpredictable trials. Here, we examine the impact of a different type of violation: the mere omission of the standard sound. Capitalizing upon the recent finding that deviant sounds exert distinct effects on response times as a function of whether participants produced or withheld a response on the previous trial, we present the results of an experiment seeking to disentangle two potential effects of sound omission: deviance distraction and the removal of an unspecific warning signal. The results indicate that deviant sound and the unexpected omission of the standard sound impact response times through, at least partially, distinct mechanisms. Deviant sounds affect performance by triggering the orienting of attention towards a new sensory input. Sound omissions, in contrast, appear to affect performance in part because responses no longer benefit from an unspecific warning signal to prepare for action.

Event-related responses reflect chunk boundaries in natural speech

Chunking language has been proposed to be vital for comprehension enabling the extraction of meaning from a continuous stream of speech. However, neurocognitive mechanisms of chunking are poorly understood. The present study investigated neural correlates of chunk boundaries intuitively identified by listeners in natural speech drawn from linguistic corpora using magneto- and electroencephalography (MEEG). In a behavioral experiment, subjects marked chunk boundaries in the excerpts intuitively, which revealed highly consistent chunk boundary markings across the subjects. We next recorded brain activity to investigate whether chunk boundaries with high and medium agreement rates elicit distinct evoked responses compared to non-boundaries. Pauses placed at chunk boundaries elicited a closure positive shift with the sources over bilateral auditory cortices. In contrast, pauses placed within a chunk were perceived as interruptions and elicited a biphasic emitted potential with sources located in the bilateral primary and non-primary auditory areas with right-hemispheric dominance, and in the right inferior frontal cortex. Furthermore, pauses placed at stronger boundaries elicited earlier and more prominent activation over the left hemisphere suggesting that brain responses to chunk boundaries of natural speech can be modulated by the relative strength of different linguistic cues, such as syntactic structure and prosody.

The sound of silence: Predictive error responses to unexpected sound omission in adults

The human auditory system excels at detecting patterns needed for processing speech and music. According to predictive coding, the brain predicts incoming sounds, compares predictions to sensory input, and generates a prediction error whenever a mismatch between the prediction and sensory input occurs. Predictive coding can be indexed in EEG with the mismatch negativity (MMN) and P3a components, two ERP components that are elicited by infrequent deviant sounds (e.g., differing in pitch, duration, loudness) in a stream of frequent sounds. If these components reflect prediction error, they should also be elicited by omitting an expected sound, but few studies have examined this. We compared ERPs elicited by infrequent randomly occurring omissions (unexpected silences) in tone sequences presented at 2 tones/sec to ERPs elicited by frequent, regularly occurring omissions (expected silences) within a sequence of tones and resting state EEG (a constant silence). We found that unexpected silences elicited significant MMN and P3a, although the magnitude of these components was quite small and variable. These results provide evidence for hierarchical predictive coding, indicating that the brain predicts silences as well as sounds.

Neural Substrates and Models of Omission Responses and Predictive Processes

Predictive coding theories argue that deviance detection phenomena, such as mismatch responses and omission responses, are generated by predictive processes with possibly overlapping neural substrates. Molecular imaging and electrophysiology studies of mismatch responses and corollary discharge in the rodent model allowed the development of mechanistic and computational models of these phenomena. These models enable translation between human and non-human animal research and help to uncover fundamental features of change-processing microcircuitry in the neocortex. This microcircuitry is characterized by stimulus-specific adaptation and feedforward inhibition of stimulus-selective populations of pyramidal neurons and interneurons, with specific contributions from different interneuron types. The overlap of the substrates of different types of responses to deviant stimuli remains to be understood. Omission responses, which are observed both in corollary discharge and mismatch response protocols in humans, are underutilized in animal research and may be pivotal in uncovering the substrates of predictive processes. Omission studies comprise a range of methods centered on the withholding of an expected stimulus. This review aims to provide an overview of omission protocols and showcase their potential to integrate and complement the different models and procedures employed to study prediction and deviance detection.This approach may reveal the biological foundations of core concepts of predictive coding, and allow an empirical test of the framework’s promise to unify theoretical models of attention and perception.

Predictive Processing in Poetic Language: Event-Related Potentials Data on Rhythmic Omissions in Metered Speech

Predictions during language comprehension are currently discussed from many points of view. One area where predictive processing may play a particular role concerns poetic language that is regularized by meter and rhyme, thus allowing strong predictions regarding the timing and stress of individual syllables. While there is growing evidence that these prosodic regularities influence language processing, less is known about the potential influence of prosodic preferences (binary, strong-weak patterns) on neurophysiological processes. To this end, the present electroencephalogram (EEG) study examined whether the predictability of strong and weak syllables within metered speech would differ as a function of meter (trochee vs. iamb). Strong, i.e., accented positions within a foot should be more predictable than weak, i.e., unaccented positions. Our focus was on disyllabic pseudowords that solely differed between trochaic and iambic structure, with trochees providing the preferred foot in German. Methodologically, we focused on the omission Mismatch Negativity (oMMN) that is elicited when an anticipated auditory stimulus is omitted. The resulting electrophysiological brain response is particularly interesting because its elicitation does not depend on a physical stimulus. Omissions in deviant position of a passive oddball paradigm occurred at either first- or second-syllable position of the aforementioned pseudowords, resulting in a 2-by-2 design with the factors foot type and omission position. Analyses focused on the mean oMMN amplitude and latency differences across the four conditions. The result pattern was characterized by an interaction of the effects of foot type and omission position for both amplitudes and latencies. In first position, omissions resulted in larger and earlier oMMNs for trochees than for iambs. In second position, omissions resulted in larger oMMNs for iambs than for trochees, but the oMMN latency did not differ. The results suggest that omissions, particularly in initial position, are modulated by a trochaic preference in German. The preferred strong-weak pattern may have strengthened the prosodic prediction, especially for matching, trochaic stimuli, such that the violation of this prediction led to an earlier and stronger prediction error. Altogether, predictive processing seems to play a particular role in metered speech, especially if the meter is based on the preferred foot type.

Auditory Pattern Representations Under Conditions of Uncertainty-An ERP Study

The auditory system is able to recognize auditory objects and is thought to form predictive models of them even though the acoustic information arriving at our ears is often imperfect, intermixed, or distorted. We investigated implicit regularity extraction for acoustically intact versus disrupted six-tone sound patterns via event-related potentials (ERPs). In an exact-repetition condition, identical patterns were repeated; in two distorted-repetition conditions, one randomly chosen segment in each sound pattern was replaced either by white noise or by a wrong pitch. In a roving-standard paradigm, sound patterns were repeated 1-12 times (standards) in a row before a new pattern (deviant) occurred. The participants were not informed about the roving rule and had to detect rarely occurring loudness changes. Behavioral detectability of pattern changes was assessed in a subsequent behavioral task. Pattern changes (standard vs. deviant) elicited mismatch negativity (MMN) and P3a, and were behaviorally detected above the chance level in all conditions, suggesting that the auditory system extracts regularities despite distortions in the acoustic input. However, MMN and P3a amplitude were decreased by distortions. At the level of MMN, both types of distortions caused similar impairments, suggesting that auditory regularity extraction is largely determined by the stimulus statistics of matching information. At the level of P3a, wrong-pitch distortions caused larger decreases than white-noise distortions. Wrong-pitch distortions likely prevented the engagement of restoration mechanisms and the segregation of disrupted from true pattern segments, causing stronger informational interference with the relevant pattern information.

Direct Evidence for Prediction Signals in Frontal Cortex Independent of Prediction Error

Predictive coding (PC) has been suggested as one of the main mechanisms used by brains to interact with complex environments. PC theories posit top-down prediction signals, which are compared with actual outcomes, yielding in turn prediction error (PE) signals, which are used, bottom-up, to modify the ensuing predictions. However, disentangling prediction from PE signals has been challenging. Critically, while many studies found indirect evidence for PC in the form of PE signals, direct evidence for the prediction signal is mostly lacking. Here, we provide clear evidence, obtained from intracranial cortical recordings in human surgical patients, that the human lateral prefrontal cortex evinces prediction signals while anticipating an event. Patients listened to task-irrelevant sequences of repetitive tones including infrequent predictable or unpredictable pitch deviants. The broadband high-frequency amplitude (HFA) was decreased prior to the onset of expected relative to unexpected deviants in the frontal cortex only, and its amplitude was sensitive to the increasing likelihood of deviants following longer trains of standards in the unpredictable condition. Single-trial HFA predicted deviations and correlated with poststimulus response to deviations. These results provide direct evidence for frontal cortex prediction signals independent of PE signals.

(Not so) Great Expectations: Listening to Foreign-Accented Speech Reduces the Brain's Anticipatory Processes

This study examines the effect of foreign-accented speech on the predictive ability of our brain. Listeners actively anticipate upcoming linguistic information in the speech signal so as to facilitate and reduce processing load. However, it is unclear whether or not listeners also do this when they are exposed to speech from non-native speakers. In the present study, we exposed native Dutch listeners to sentences produced by native and non-native speakers while measuring their brain activity using electroencephalography. We found that listeners’ brain activity differed depending on whether they listened to native or non-native speech. However, participants’ overall performance as measured by word recall rate was unaffected. We discussed the results in relation to previous findings as well as the automaticity of anticipation.

The Role of Surface and Underlying Forms When Processing Tonal Alternations in Mandarin Chinese: A Mismatch Negativity Study

Phonological alternation (sound change depending on the phonological environment) poses challenges to spoken word recognition models. Mandarin Chinese T3 sandhi is such a phenomenon in which a tone 3 (T3) changes into a tone 2 (T2) when followed by another T3. In a mismatch negativity (MMN) study examining Mandarin Chinese T3 sandhi, participants passively listened to either a T2 word [tʂu2 je4] /tʂu2 je4/, a T3 word [tʂu3 je4] /tʂu3 je4/, a sandhi word [tʂu2 jen3] /tʂu3 jen3/, or a mix of T3 and sandhi word standards. The deviant in each condition was a T2 word [tʂu2]. Results showed an MMN only in the T2 and T3 conditions but not in the Sandhi or Mix conditions. All conditions also yielded omission MMNs. This pattern cannot be explained based on the surface forms of standards and deviants; rather these data suggest an underspecified or underlying T3 stored linguistic representation used in spoken word processing.

Neural Signal to Violations of Abstract Rules Using Speech-Like Stimuli

As the evidence of predictive processes playing a role in a wide variety of cognitive domains increases, the brain as a predictive machine becomes a central idea in neuroscience. In auditory processing, a considerable amount of progress has been made using variations of the Oddball design, but most of the existing work seems restricted to predictions based on physical features or conditional rules linking successive stimuli. To characterize the predictive capacity of the brain to abstract rules, we present here two experiments that use speech-like stimuli to overcome limitations and avoid common confounds. Pseudowords were presented in isolation, intermixed with infrequent deviants that contained unexpected phoneme sequences. As hypothesized, the occurrence of unexpected sequences of phonemes reliably elicited an early prediction error signal. These prediction error signals do not seemed to be modulated by attentional manipulations due to different task instructions, suggesting that the predictions are deployed even when the task at hand does not volitionally involve error detection. In contrast, the amount of syllables congruent with a standard pseudoword presented before the point of deviance exerted a strong modulation. Prediction error's amplitude doubled when two congruent syllables were presented instead of one, despite keeping local transitional probabilities constant. This suggests that auditory predictions can be built integrating information beyond the immediate past. In sum, the results presented here further contribute to the understanding of the predictive capabilities of the human auditory system when facing complex stimuli and abstract rules.

Automatic and feature-specific prediction-related neural activity in the human auditory system

Prior experience enables the formation of expectations of upcoming sensory events. However, in the auditory modality, it is not known whether prediction-related neural signals carry feature-specific information. Here, using magnetoencephalography (MEG), we examined whether predictions of future auditory stimuli carry tonotopic specific information. Participants passively listened to sound sequences of four carrier frequencies (tones) with a fixed presentation rate, ensuring strong temporal expectations of when the next stimulus would occur. Expectation of which frequency would occur was parametrically modulated across the sequences, and sounds were occasionally omitted. We show that increasing the regularity of the sequence boosts carrier-frequency-specific neural activity patterns during both the anticipatory and omission periods, indicating that prediction-related neural activity is indeed feature-specific. Our results illustrate that even without bottom-up input, auditory predictions can activate tonotopically specific templates.

After listening to a predictable sequence of sounds, we can anticipate and predict the next sound in the sequence. Here, the authors show that during expectation of a sound, the brain generates neural activity matching that which is produced by actually hearing the same sound.

Restoration and Efficiency of the Neural Processing of Continuous Speech Are Promoted by Prior Knowledge

Sufficiently noisy listening conditions can completely mask the acoustic signal of significant parts of a sentence, and yet listeners may still report the perception of hearing the masked speech. This occurs even when the speech signal is removed entirely, if the gap is filled with stationary noise, a phenomenon known as perceptual restoration. At the neural level, however, it is unclear the extent to which the neural representation of missing extended speech sequences is similar to the dynamic neural representation of ordinary continuous speech. Using auditory magnetoencephalography (MEG), we show that stimulus reconstruction, a technique developed for use with neural representations of ordinary speech, works also for the missing speech segments replaced by noise, even when spanning several phonemes and words. The reconstruction fidelity of the missing speech, up to 25% of what would be attained if present, depends however on listeners' familiarity with the missing segment. This same familiarity also speeds up the most prominent stage of the cortical processing of ordinary speech by approximately 5 ms. Both effects disappear when listeners have no or little prior experience with the speech segment. The results are consistent with adaptive expectation mechanisms that consolidate detailed representations about speech sounds as identifiable factors assisting automatic restoration over ecologically relevant timescales.

Tracking Temporal Hazard in the Human Electroencephalogram Using a Forward Encoding Model

Human observers automatically extract temporal contingencies from the environment and predict the onset of future events. Temporal predictions are modeled by the hazard function, which describes the instantaneous probability for an event to occur given it has not occurred yet. Here, we tackle the question of whether and how the human brain tracks continuous temporal hazard on a moment-to-moment basis, and how flexibly it adjusts to strictly implicit variations in the hazard function. We applied an encoding-model approach to human electroencephalographic data recorded during a pitch-discrimination task, in which we implicitly manipulated temporal predictability of the target tones by varying the interval between cue and target tone (i.e. the foreperiod). Critically, temporal predictability either was driven solely by the passage of time (resulting in a monotonic hazard function) or was modulated to increase at intermediate foreperiods (resulting in a modulated hazard function with a peak at the intermediate foreperiod). Forward-encoding models trained to predict the recorded EEG signal from different temporal hazard functions were able to distinguish between experimental conditions, showing that implicit variations of temporal hazard bear tractable signatures in the human electroencephalogram. Notably, this tracking signal was reconstructed best from the supplementary motor area, underlining this area’s link to cognitive processing of time. Our results underline the relevance of temporal hazard to cognitive processing and show that the predictive accuracy of the encoding-model approach can be utilized to track abstract time-resolved stimuli.

Cardio-audio synchronization drives neural surprise response

Successful prediction of future events depends on the brain’s capacity to extract temporal regularities from sensory inputs. Neuroimaging studies mainly investigated regularity processing for exteroceptive sensory inputs (i.e. from outside the body). Here we investigated whether interoceptive signals (i.e. from inside the body) can mediate auditory regularity processing. Human participants passively listened to sound sequences presented in synchrony or asynchrony to their heartbeat while concomitant electroencephalography was recorded. We hypothesized that the cardio-audio synchronicity would induce a brain expectation of future sounds. Electrical neuroimaging analysis revealed a surprise response at 158–270 ms upon omission of the expected sounds in the synchronous condition only. Control analyses ruled out that this effect was trivially based on expectation from the auditory temporal structure or on differences in heartbeat physiological signals. Implicit neural monitoring of temporal regularities across interoceptive and exteroceptive signals drives prediction of future events in auditory sequences.

Integrating speech in time depends on temporal expectancies and attention

Sensory information that unfolds in time, such as in speech perception, relies on efficient chunking mechanisms in order to yield optimally-sized units for further processing. Whether or not two successive acoustic events receive a one-unit or a two-unit interpretation seems to depend on the fit between their temporal extent and a stipulated temporal window of integration. However, there is ongoing debate on how flexible this temporal window of integration should be, especially for the processing of speech sounds. Furthermore, there is no direct evidence of whether attention may modulate the temporal constraints on the integration window. For this reason, we here examine how different word durations, which lead to different temporal separations of sound onsets, interact with attention. In an Electroencephalography (EEG) study, participants actively and passively listened to words where word-final consonants were occasionally omitted. Words had either a natural duration or were artificially prolonged in order to increase the separation of speech sound onsets. Omission responses to incomplete speech input, originating in left temporal cortex, decreased when the critical speech sound was separated from previous sounds by more than 250 msec, i.e., when the separation was larger than the stipulated temporal window of integration (125-150 msec). Attention, on the other hand, only increased omission responses for stimuli with natural durations. We complemented the event-related potential (ERP) analyses by a frequency-domain analysis on the stimulus presentation rate. Notably, the power of stimulation frequency showed the same duration and attention effects than the omission responses. We interpret these findings on the background of existing research on temporal integration windows and further suggest that our findings may be accounted for within the framework of predictive coding.

Laryngeal Features Are Phonetically Abstract: Mismatch Negativity Evidence from Arabic, English, and Russian

Many theories of phonology assume that the sound structure of language is made up of distinctive features, but there is considerable debate about how much articulatory detail distinctive features encode in long-term memory. Laryngeal features such as voicing provide a unique window into this question: while many languages have two-way contrasts that can be given a simple binary feature account [±VOICE], the precise articulatory details underlying these contrasts can vary significantly across languages. Here, we investigate a series of two-way voicing contrasts in English, Arabic, and Russian, three languages that implement their voicing contrasts very differently at the articulatory-phonetic level. In three event-related potential experiments contrasting English, Arabic, and Russian fricatives along with Russian stops, we observe a consistent pattern of asymmetric mismatch negativity (MMN) effects that is compatible with an articulatorily abstract and cross-linguistically uniform way of marking two-way voicing contrasts, as opposed to an articulatorily precise and cross-linguistically diverse way of encoding them. Regardless of whether a language is theorized to encode [VOICE] over [SPREAD GLOTTIS], the data is consistent with a universal marking of the [SPREAD GLOTTIS] feature.

Predictive coding of phonological rules in auditory cortex: A mismatch negativity study

The brain is constantly generating predictions of future sensory input to enable efficient adaptation. In the auditory domain, this applies also to the processing of speech. Here we aimed to determine whether the brain predicts the following segments of speech input on the basis of language-specific phonological rules that concern non-adjacent phonemes. Auditory event-related potentials (ERP) were recorded in a mismatch negativity (MMN) paradigm, where the Finnish vowel harmony, determined by the first syllables of pseudowords, either constrained or did not constrain the phonological composition of pseudoword endings. The phonological rule of vowel harmony was expected to create predictions about phonologically legal pseudoword endings. Results showed that MMN responses were larger for phonologically illegal than legal pseudowords, and P3a was elicited only for illegal pseudowords. This supports the hypothesis that speech input is evaluated against context-dependent phonological predictions that facilitate speech processing.

Dialect Variation Influences the Phonological and Lexical-Semantic Word Processing in Sentences. Electrophysiological Evidence from a Cross-Dialectal Comprehension Study

This event-related potential (ERP) study examines the influence of dialectal competence differences (merged vs. unmerged dialect group) on cross-dialectal comprehension between Southern German dialects. It focuses on the question as to whether certain dialect phonemes (/oa⌢/, /oƱ⌢/), which are attributed to different lexemes in two dialect areas (Central Bavarian, Bavarian-Alemannic transition zone) evoke increased neural costs during sentence processing. In this context, the phonological and semantic processing of lexemes is compared in three types of potentially problematic communication settings (misunderstanding, incomprehension, allophonic variation = potential comprehension). For this purpose, an oddball design including whole sentences was combined with a semantic rating task. Listeners from the unmerged Central Bavarian dialect area heard sentences including either native or non-native lexemes from the merged neighboring dialect. These had to be evaluated with regard to their context acceptability. The main difference between the lexemes can be attributed to the fact that they have different meanings in the respective dialect areas or are non-existent in the linguistic competence of the Central Bavarians. The results provide evidence for the fact that non-native lexemes containing the /oa⌢/-diphthong lead to enhanced neural costs during sentence processing. The ERP results show a biphasic pattern (N2b/N400, LPC) for non-existent lexemes (incomprehension) as well as for semantically incongruous lexemes (misunderstanding), reflecting an early error detection mechanism and enhanced costs for semantic integration and evaluation. In contrast, allophonic /oƱ⌢/ deviations show reduced negativities and no LPC, indexing an unproblematic categorization and evaluation process. In the light of these results, an observed change of /oa⌢/ to /oƱ⌢/ in the Bavarian-Alemannic transition zone can be interpreted as a facilitation strategy of cross-dialectal comprehension to reduce both misunderstandings as well as neural costs in processing, which might be interpreted as the initial trigger for this particular phoneme change.

Predictions interact with missing sensory evidence in semantic processing areas

Human brain function draws on predictive mechanisms that exploit higher-level context during lower-level perception. These mechanisms are particularly relevant for situations in which sensory information is compromised or incomplete, as for example in natural speech where speech segments may be omitted due to sluggish articulation. Here, we investigate which brain areas support the processing of incomplete words that were predictable from semantic context, compared with incomplete words that were unpredictable. During functional magnetic resonance imaging (fMRI), participants heard sentences that orthogonally varied in predictability (semantically predictable vs. unpredictable) and completeness (complete vs. incomplete, i.e. missing their final consonant cluster). The effects of predictability and completeness interacted in heteromodal semantic processing areas, including left angular gyrus and left precuneus, where activity did not differ between complete and incomplete words when they were predictable. The same regions showed stronger activity for incomplete than for complete words when they were unpredictable. The interaction pattern suggests that for highly predictable words, the speech signal does not need to be complete for neural processing in semantic processing areas. Hum Brain Mapp 37:704-716, 2016. © 2015 Wiley Periodicals, Inc.

Auditory perceptual objects as generative models: Setting the stage for communication by sound

Communication by sounds requires that the communication channels (i.e. speech/speakers and other sound sources) had been established. This allows to separate concurrently active sound sources, to track their identity, to assess the type of message arriving from them, and to decide whether and when to react (e.g., reply to the message). We propose that these functions rely on a common generative model of the auditory environment. This model predicts upcoming sounds on the basis of representations describing temporal/sequential regularities. Predictions help to identify the continuation of the previously discovered sound sources to detect the emergence of new sources as well as changes in the behavior of the known ones. It produces auditory event representations which provide a full sensory description of the sounds, including their relation to the auditory context and the current goals of the organism. Event representations can be consciously perceived and serve as objects in various cognitive operations.

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

The McGurk effect is a textbook illustration of the automaticity with which the human brain integrates audio-visual speech. It shows that even incongruent audiovisual (AV) speech stimuli can be combined into percepts that correspond neither to the auditory nor to the visual input, but to a mix of both. Typically, when presented with, e.g., visual /aga/ and acoustic /aba/ we perceive an illusory /ada/. In the inverse situation, however, when acoustic /aga/ is paired with visual /aba/, we perceive a combination of both stimuli, i.e., /abga/ or /agba/. Here we assessed the role of dynamic cross-modal predictions in the outcome of AV speech integration using a computational model that processes continuous audiovisual speech sensory inputs in a predictive coding framework. The model involves three processing levels: sensory units, units that encode the dynamics of stimuli, and multimodal recognition/identity units. The model exhibits a dynamic prediction behavior because evidence about speech tokens can be asynchronous across sensory modality, allowing for updating the activity of the recognition units from one modality while sending top-down predictions to the other modality. We explored the model's response to congruent and incongruent AV stimuli and found that, in the two-dimensional feature space spanned by the speech second formant and lip aperture, fusion stimuli are located in the neighborhood of congruent /ada/, which therefore provides a valid match. Conversely, stimuli that lead to combination percepts do not have a unique valid neighbor. In that case, acoustic and visual cues are both highly salient and generate conflicting predictions in the other modality that cannot be fused, forcing the elaboration of a combinatorial solution. We propose that dynamic predictive mechanisms play a decisive role in the dichotomous perception of incongruent audiovisual inputs.

Attention and prediction in human audition: a lesson from cognitive psychophysiology

Attention is a hypothetical mechanism in the service of perception that facilitates the processing of relevant information and inhibits the processing of irrelevant information. Prediction is a hypothetical mechanism in the service of perception that considers prior information when interpreting the sensorial input. Although both (attention and prediction) aid perception, they are rarely considered together. Auditory attention typically yields enhanced brain activity, whereas auditory prediction often results in attenuated brain responses. However, when strongly predicted sounds are omitted, brain responses to silence resemble those elicited by sounds. Studies jointly investigating attention and prediction revealed that these different mechanisms may interact, e.g. attention may magnify the processing differences between predicted and unpredicted sounds. Following the predictive coding theory, we suggest that prediction relates to predictions sent down from predictive models housed in higher levels of the processing hierarchy to lower levels and attention refers to gain modulation of the prediction error signal sent up to the higher level. As predictions encode contents and confidence in the sensory data, and as gain can be modulated by the intention of the listener and by the predictability of the input, various possibilities for interactions between attention and prediction can be unfolded. From this perspective, the traditional distinction between bottom-up/exogenous and top-down/endogenous driven attention can be revisited and the classic concepts of attentional gain and attentional trace can be integrated.

The role of temporal structure in the investigation of sensory memory, auditory scene analysis, and speech perception: a healthy-aging perspective

Listening situations with multiple talkers or background noise are common in everyday communication and are particularly demanding for older adults. Here we review current research on auditory perception in aging individuals in order to gain insights into the challenges of listening under noisy conditions. Informationally rich temporal structure in auditory signals--over a range of time scales from milliseconds to seconds--renders temporal processing central to perception in the auditory domain. We discuss the role of temporal structure in auditory processing, in particular from a perspective relevant for hearing in background noise, and focusing on sensory memory, auditory scene analysis, and speech perception. Interestingly, these auditory processes, usually studied in an independent manner, show considerable overlap of processing time scales, even though each has its own 'privileged' temporal regimes. By integrating perspectives on temporal structure processing in these three areas of investigation, we aim to highlight similarities typically not recognized.

ERP Indices of Stimulus Prediction in Letter Sequences

Given the current focus on anticipation in perception, action and cognition, including language processing, there is a need for a method to tap into predictive processing in situations in which cue and feedback stimuli are not explicitly marked as such. To this aim, event related potentials (ERPs) were obtained while participants viewed alphabetic letter sequences (“A”, “B”, “C”, “D”, “E”, …), in which the letters were highly predictable, and random sequences (“S”, “B”, “A”, “I”, “F”, “M”, …), without feedback. Occasionally, the presentation of a letter in a sequence was delayed by 300 ms. During this delay period, an increased negativity was observed for predictive versus random sequences. In addition, the early positivity following the delay was larger for predictive compared with random sequences. These results suggest that expectation-sensitive ERP modulations can be elicited in anticipation of stimuli that are not explicit targets, rewards, feedback or instructions, and that a delay can strengthen the prediction for a particular stimulus. Applications to language processing will be discussed.

Mismatch negativity (MMN) as an index of cognitive dysfunction

Cognition is often affected in a variety of neuropsychiatric, neurological, and neurodevelopmental disorders. The neural discriminative response, reflected in mismatch negativity (MMN) and its magnetoencephalographic equivalent (MMNm), has been used as a tool to study a variety of disorders involving auditory cognition. MMN/MMNm is an involuntary brain response to auditory change or, more generally, to pattern regularity violation. For a number of disorders, MMN/MMNm amplitude to sound deviance has been shown to be attenuated or the peak-latency of the component prolonged compared to controls. This general finding suggests that while not serving as a specific marker to any particular disorder, MMN may be useful for understanding factors of cognition in various disorders, and has potential to serve as an indicator of risk. This review presents a brief history of the MMN, followed by a description of how MMN has been used to index auditory processing capability in a range of neuropsychiatric, neurological, and neurodevelopmental disorders. Finally, we suggest future directions for research to further enhance our understanding of the neural substrate of deviance detection that could lead to improvements in the use of MMN as a clinical tool.