Early electrophysiological indicators for predictive processing in audition: A review

A Duet for one

This paper considers communication in terms of inference about the behaviour of others (and our own behaviour). It is based on the premise that our sensations are largely generated by other agents like ourselves. This means, we are trying to infer how our sensations are caused by others, while they are trying to infer our behaviour: for example, in the dialogue between two speakers. We suggest that the infinite regress induced by modelling another agent - who is modelling you - can be finessed if you both possess the same model. In other words, the sensations caused by others and oneself are generated by the same process. This leads to a view of communication based upon a narrative that is shared by agents who are exchanging sensory signals. Crucially, this narrative transcends agency - and simply involves intermittently attending to and attenuating sensory input. Attending to sensations enables the shared narrative to predict the sensations generated by another (i.e. to listen), while attenuating sensory input enables one to articulate the narrative (i.e. to speak). This produces a reciprocal exchange of sensory signals that, formally, induces a generalised synchrony between internal (neuronal) brain states generating predictions in both agents. We develop the arguments behind this perspective, using an active (Bayesian) inference framework and offer some simulations (of birdsong) as proof of principle.

Complex linguistic rules modulate early auditory brain responses

During speech perception, listeners compensate for phonological rules of their language. For instance, English place assimilation causes green boat to be typically pronounced as greem boat; English listeners, however, perceptually compensate for this rule and retrieve the intended sound (n). Previous research using EEG has focused on rules with clear phonetic underpinnings, showing that perceptual compensation occurs at an early stage of speech perception. We tested whether this early mechanism also accounts for the compensation for more complex rules. We examined compensation for French voicing assimilation, a rule with abstract phonological restrictions on the contexts in which it applies. Our results reveal that perceptual compensation for this rule by French listeners modulates an early ERP component. This is evidence that early stages of speech sound categorization are sensitive to complex phonological rules of the native language.

Bridging prediction and attention in current research on perception and action

Prediction and attention are fundamental brain functions in the service of perception and action. Theories on prediction relate to neural (mental) models inferring about (present or future) sensory or action-related information, whereas theories of attention are about the control of information flow underlying perception and action. Both concepts are related and not always clearly distinguishable. The special issue includes current research on prediction and attention in various subfields of perception and action. It especially considers interactions between predictive and attentive processes, which constitute a newly emerging and highly interesting field of research. As outlined in this editorial, the contributions in this special issue allow specifying as well as bridging concepts on prediction and attention. The joint consideration of prediction and attention also reveals common functional principles of perception and action.

Auditory perceptual objects as generative models: Setting the stage for communication by sound

Communication by sounds requires that the communication channels (i.e. speech/speakers and other sound sources) had been established. This allows to separate concurrently active sound sources, to track their identity, to assess the type of message arriving from them, and to decide whether and when to react (e.g., reply to the message). We propose that these functions rely on a common generative model of the auditory environment. This model predicts upcoming sounds on the basis of representations describing temporal/sequential regularities. Predictions help to identify the continuation of the previously discovered sound sources to detect the emergence of new sources as well as changes in the behavior of the known ones. It produces auditory event representations which provide a full sensory description of the sounds, including their relation to the auditory context and the current goals of the organism. Event representations can be consciously perceived and serve as objects in various cognitive operations.

Happiness increases distraction by auditory deviant stimuli

Rare and unexpected changes (deviants) in an otherwise repeated stream of task-irrelevant auditory distractors (standards) capture attention and impair behavioural performance in an ongoing visual task. Recent evidence indicates that this effect is increased by sadness in a task involving neutral stimuli. We tested the hypothesis that such effect may not be limited to negative emotions but reflect a general depletion of attentional resources by examining whether a positive emotion (happiness) would increase deviance distraction too. Prior to performing an auditory-visual oddball task, happiness or a neutral mood was induced in participants by means of the exposure to music and the recollection of an autobiographical event. Results from the oddball task showed significantly larger deviance distraction following the induction of happiness. Interestingly, the small amount of distraction typically observed on the standard trial following a deviant trial (post-deviance distraction) was not increased by happiness. We speculate that happiness might interfere with the disengagement of attention from the deviant sound back towards the target stimulus (through the depletion of cognitive resources and/or mind wandering) but help subsequent cognitive control to recover from distraction.

Noise occlusion in discrete tone sequences as a tool towards auditory predictive processing?

The notion of predictive coding is a common feature of many theories of auditory information processing. Experimental demonstrations of predictive auditory processing often rest on omitting predictable input in order to uncover the prediction made by the brain. Findings show that auditory cortical activity elicited by the omission of a predictable tone resembles the activity elicited by the actual tone. Here we attempted to extend this approach towards using noises instead of omissions in order to capture a more prevalent case of degraded sensory input. By applying a subtraction approach to remove ERP effects of the noise itself, auditory cortical activity elicited "behind" the noise was uncovered. We hypothesized that ERPs elicited behind noise stimuli covering predictable tones should be more similar to ERPs elicited by the actual tones than when the same comparison is made for unpredictable tones. ERP results during passive listening partly confirm this hypothesis, but also point towards some methodological caveats in this particular approach towards studying neural correlates of predictive auditory processing due to contributions from predictability-unrelated factors. A follow-up active listening condition indicated that participants were not more likely to perceive the tone sequence as continuous when a predictable tone was covered with noise than when this pertained to an unpredictable tone. Overall, the noise-based paradigm in its present form was not shown to be successful in revealing predictive processing in perceptual judgments or early neural correlates of sound processing. We discuss these findings in the contexts of predictive processing and illusory auditory continuity. This article is part of a Special Issue entitled SI: Prediction and Attention.

Active inference, communication and hermeneutics

Hermeneutics refers to interpretation and translation of text (typically ancient scriptures) but also applies to verbal and non-verbal communication. In a psychological setting it nicely frames the problem of inferring the intended content of a communication. In this paper, we offer a solution to the problem of neural hermeneutics based upon active inference. In active inference, action fulfils predictions about how we will behave (e.g., predicting we will speak). Crucially, these predictions can be used to predict both self and others--during speaking and listening respectively. Active inference mandates the suppression of prediction errors by updating an internal model that generates predictions--both at fast timescales (through perceptual inference) and slower timescales (through perceptual learning). If two agents adopt the same model, then--in principle--they can predict each other and minimise their mutual prediction errors. Heuristically, this ensures they are singing from the same hymn sheet. This paper builds upon recent work on active inference and communication to illustrate perceptual learning using simulated birdsongs. Our focus here is the neural hermeneutics implicit in learning, where communication facilitates long-term changes in generative models that are trying to predict each other. In other words, communication induces perceptual learning and enables others to (literally) change our minds and vice versa.

Expectation Suppression in Early Visual Cortex Depends on Task Set

Stimulus expectation can modulate neural responses in early sensory cortical regions, with expected stimuli often leading to a reduced neural response. However, it is unclear whether this expectation suppression is an automatic phenomenon or is instead dependent on the type of task a subject is engaged in. To investigate this, human subjects were presented with visual grating stimuli in the periphery that were either predictable or non-predictable while they performed three tasks that differently engaged cognitive resources. In two of the tasks, the predictable stimulus was task-irrelevant and spatial attention was engaged at fixation, with a high load on either perceptual or working memory resources. In the third task, the predictable stimulus was task-relevant, and therefore spatially attended. We observed that expectation suppression is dependent on the cognitive resources engaged by a subjects’ current task. When the grating was task-irrelevant, expectation suppression for predictable items was visible in retinotopically specific areas of early visual cortex (V1-V3) during the perceptual task, but it was abolished when working memory was loaded. When the grating was task-relevant and spatially attended, there was no significant effect of expectation in early visual cortex. These results suggest that expectation suppression is not an automatic phenomenon, but dependent on attentional state and type of available cognitive resources.

Experience-based Auditory Predictions Modulate Brain Activity to Silence as do Real Sounds

Interactions between stimuli's acoustic features and experience-based internal models of the environment enable listeners to compensate for the disruptions in auditory streams that are regularly encountered in noisy environments. However, whether auditory gaps are filled in predictively or restored a posteriori remains unclear. The current lack of positive statistical evidence that internal models can actually shape brain activity as would real sounds precludes accepting predictive accounts of filling-in phenomenon. We investigated the neurophysiological effects of internal models by testing whether single-trial electrophysiological responses to omitted sounds in a rule-based sequence of tones with varying pitch could be decoded from the responses to real sounds and by analyzing the ERPs to the omissions with data-driven electrical neuroimaging methods. The decoding of the brain responses to different expected, but omitted, tones in both passive and active listening conditions was above chance based on the responses to the real sound in active listening conditions. Topographic ERP analyses and electrical source estimations revealed that, in the absence of any stimulation, experience-based internal models elicit an electrophysiological activity different from noise and that the temporal dynamics of this activity depend on attention. We further found that the expected change in pitch direction of omitted tones modulated the activity of left posterior temporal areas 140-200 msec after the onset of omissions. Collectively, our results indicate that, even in the absence of any stimulation, internal models modulate brain activity as do real sounds, indicating that auditory filling in can be accounted for by predictive activity.

Music-evoked emotions: principles, brain correlates, and implications for therapy

This paper describes principles underlying the evocation of emotion with music: evaluation, resonance, memory, expectancy/tension, imagination, understanding, and social functions. Each of these principles includes several subprinciples, and the framework on music-evoked emotions emerging from these principles and subprinciples is supposed to provide a starting point for a systematic, coherent, and comprehensive theory on music-evoked emotions that considers both reception and production of music, as well as the relevance of emotion-evoking principles for music therapy.

Differences in the strength of cortical and brainstem inputs to SSA and non-SSA neurons in the inferior colliculus

In an ever changing auditory scene, change detection is an ongoing task performed by the auditory brain. Neurons in the midbrain and auditory cortex that exhibit stimulus-specific adaptation (SSA) may contribute to this process. Those neurons adapt to frequent sounds while retaining their excitability to rare sounds. Here, we test whether neurons exhibiting SSA and those without are part of the same networks in the inferior colliculus (IC). We recorded the responses to frequent and rare sounds and then marked the sites of these neurons with a retrograde tracer to correlate the source of projections with the physiological response. SSA neurons were confined to the non-lemniscal subdivisions and exhibited broad receptive fields, while the non-SSA were confined to the central nucleus and displayed narrow receptive fields. SSA neurons receive strong inputs from auditory cortical areas and very poor or even absent projections from the brainstem nuclei. On the contrary, the major sources of inputs to the neurons that lacked SSA were from the brainstem nuclei. These findings demonstrate that auditory cortical inputs are biased in favor of IC synaptic domains that are populated by SSA neurons enabling them to compare top-down signals with incoming sensory information from lower areas.

Sensitivity to the temporal structure of rapid sound sequences - An MEG study

To probe sensitivity to the time structure of ongoing sound sequences, we measured MEG responses, in human listeners, to the offset of long tone-pip sequences containing various forms of temporal regularity. If listeners learn sequence temporal properties and form expectancies about the arrival time of an upcoming tone, sequence offset should be detectable as soon as an expected tone fails to arrive. Therefore, latencies of offset responses are indicative of the extent to which the temporal pattern has been acquired. In Exp1, sequences were isochronous with tone inter-onset-interval (IOI) set to 75, 125 or 225ms. Exp2 comprised of non-isochronous, temporally regular sequences, comprised of the IOIs above. Exp3 used the same sequences as Exp2 but listeners were required to monitor them for occasional frequency deviants. Analysis of the latency of offset responses revealed that the temporal structure of (even rather simple) regular sequences is not learnt precisely when the sequences are ignored. Pattern coding, supported by a network of temporal, parietal and frontal sources, improved considerably when the signals were made behaviourally pertinent. Thus, contrary to what might be expected in the context of an 'early warning system' framework, learning of temporal structure is not automatic, but affected by the signal's behavioural relevance.

Action-related auditory ERP attenuation: Paradigms and hypotheses

A number studies have shown that the auditory N1 event-related potential (ERP) is attenuated when elicited by self-induced or self-generated sounds. Because N1 is a correlate of auditory feature- and event-detection, it was generally assumed that N1-attenuation reflected the cancellation of auditory re-afference, enabled by the internal forward modeling of the predictable sensory consequences of the given action. Focusing on paradigms utilizing non-speech actions, the present review summarizes recent progress on action-related auditory attenuation. Following a critical analysis of the most widely used, contingent paradigm, two further hypotheses on the possible causes of action-related auditory ERP attenuation are presented. The attention hypotheses suggest that auditory ERP attenuation is brought about by a temporary division of attention between the action and the auditory stimulation. The pre-activation hypothesis suggests that the attenuation is caused by the activation of a sensory template during the initiation of the action, which interferes with the incoming stimulation. Although each hypothesis can account for a number of findings, none of them can accommodate the whole spectrum of results. It is suggested that a better understanding of auditory ERP attenuation phenomena could be achieved by systematic investigations of the types of actions, the degree of action-effect contingency, and the temporal characteristics of action-effect contingency representation-buildup and -deactivation. This article is part of a Special Issue entitled SI: Prediction and Attention.

The Timing of Regular Sequences: Production, Perception, and Covariation

The temporal structure of behavior provides information that allows the tracking of temporal regularity in the sensory and sensorimotor domains. In turn, temporal regularity allows the generation of predictions about upcoming events and to adjust behavior accordingly. These mechanisms are essential to ensure behavior beyond the level of mere reaction. However, efficient temporal processing is required to establish adequate internal representations of temporal structure. The current study used two simple paradigms, namely, finger-tapping at a regular self-chosen rate (spontaneous motor tempo) and ERPs of the EEG (EEG/ERP) recorded during attentive listening to temporally regular and irregular "oddball" sequences to explore the capacity to encode and use temporal regularity in production and perception. The results show that specific aspects of the ability to time a regular sequence of events in production covary with the ability to time a regular sequence in perception, probably pointing toward the engagement of domain-general mechanisms.

Temporal expectation and attention jointly modulate auditory oscillatory activity in the beta band

The neural response to a stimulus is influenced by endogenous factors such as expectation and attention. Current research suggests that expectation and attention exert their effects in opposite directions, where expectation decreases neural activity in sensory areas, while attention increases it. However, expectation and attention are usually studied either in isolation or confounded with each other. A recent study suggests that expectation and attention may act jointly on sensory processing, by increasing the neural response to expected events when they are attended, but decreasing it when they are unattended. Here we test this hypothesis in an auditory temporal cueing paradigm using magnetoencephalography in humans. In our study participants attended to, or away from, tones that could arrive at expected or unexpected moments. We found a decrease in auditory beta band synchrony to expected (versus unexpected) tones if they were unattended, but no difference if they were attended. Modulations in beta power were already evident prior to the expected onset times of the tones. These findings suggest that expectation and attention jointly modulate sensory processing.

Predictive processing of pitch trends in newborn infants

The notion of predictive sound processing suggests that the auditory system prepares for upcoming sounds once it has detected regular features within a sequence. Here we investigated whether predictive processes are operating at birth in the human auditory system. Event-related potentials (ERP) were recorded from healthy newborns to occasional ascending pitch steps occurring in the 2nd or the 5th position within trains of tones with otherwise monotonously descending pitch. If the trains were processed in a predictive manner only deviant pitch steps occurring in the later train position would elicit the discriminative mismatch response (MMR). Deviants delivered in the 5th but not in the 2nd position of the tone trains elicited a significant MMR response. These results suggest that newborns represent pitch trends within sound sequences and they process them in a predictive manner. This article is part of a Special Issue entitled SI: Prediction and Attention.

Attention and prediction in human audition: a lesson from cognitive psychophysiology

Attention is a hypothetical mechanism in the service of perception that facilitates the processing of relevant information and inhibits the processing of irrelevant information. Prediction is a hypothetical mechanism in the service of perception that considers prior information when interpreting the sensorial input. Although both (attention and prediction) aid perception, they are rarely considered together. Auditory attention typically yields enhanced brain activity, whereas auditory prediction often results in attenuated brain responses. However, when strongly predicted sounds are omitted, brain responses to silence resemble those elicited by sounds. Studies jointly investigating attention and prediction revealed that these different mechanisms may interact, e.g. attention may magnify the processing differences between predicted and unpredicted sounds. Following the predictive coding theory, we suggest that prediction relates to predictions sent down from predictive models housed in higher levels of the processing hierarchy to lower levels and attention refers to gain modulation of the prediction error signal sent up to the higher level. As predictions encode contents and confidence in the sensory data, and as gain can be modulated by the intention of the listener and by the predictability of the input, various possibilities for interactions between attention and prediction can be unfolded. From this perspective, the traditional distinction between bottom-up/exogenous and top-down/endogenous driven attention can be revisited and the classic concepts of attentional gain and attentional trace can be integrated.

The semantic aftermath of distraction by deviant sounds: Crosstalk interference is mediated by the predictability of semantic congruency

Rare changes in a stream of otherwise repeated task-irrelevant sounds break through selective attention and disrupt performance in an unrelated visual task. This deviance distraction effect emerges because deviant sounds violate the cognitive system's predictions. In this study we sought to examine whether predictability also mediate the so-called semantic effect whereby behavioral performance suffers from the clash between the involuntary semantic evaluation of irrelevant sounds and the voluntary processing of visual targets (e.g., when participants must categorize a right visual arrow following the presentation of the deviant sound "left"). By manipulating the conditional probabilities of the congruent and incongruent deviant sounds in a left/right arrow categorization task, we elicited implicit predictions about the upcoming target and related response. We observed a linear increase of the semantic effect with the proportion of congruent deviant trials (i.e., as deviant sounds increasingly predicted congruent targets). We conclude that deviant sounds affect response times based on a combination of crosstalk interference and two types of prediction violations: stimulus violations (violations of predictions regarding the identity of upcoming irrelevant sounds) and semantic violations (violations of predictions regarding the target afforded by deviant sounds). We report a three-parameter model that captures all key features of the observed RTs. Overall, our results fit with the view that the brain builds forward models of the environment in order to optimize cognitive processing and that control of one's attention and actions is called upon when predictions are violated. This article is part of a Special Issue entitled SI: Prediction and Attention.

Automatic prediction regarding the next state of a visual object: Electrophysiological indicators of prediction match and mismatch

Behavioral phenomena such as representational momentum suggest that the brain can automatically predict the next state of a visual object, based on sequential rules embedded in its preceding spatiotemporal context. To identify electrophysiological indicators of automatic visual prediction in terms of prediction match and mismatch, we recorded event-related brain potentials (ERPs) while participants passively viewed three types of task-irrelevant sequences of a bar stimulus: (1) an oddball sequence, which contained a sequential rule defined by stimulus repetition, providing repetition-rule-conforming (standard) and -violating (deviant) stimuli; (2) a rotating-oddball sequence, which contained a sequential rule defined by stimulus change (i.e., rotation), providing change-rule-conforming (regular) and -violating (irregular) stimuli; and (3) a random sequence, which did not contain a sequential rule, providing a neutral (control) stimulus. This protocol allowed us to expect that (1) an ERP effect that reflects a prediction-mismatch process should be exclusively observed in both the deviant-minus-control and irregular-minus-control comparisons and (2) an ERP effect that reflects a prediction-match process should be exclusively observed in both the standard-minus-control and regular-minus-control comparisons. The results showed that the ERP effect that met the criterion for prediction mismatch was an occipito-temporal negative deflection at around 170-300ms (visual mismatch negativity), while the ERP effect that met the criterion for prediction match was a frontal/central negative deflection at around 150-270ms (probably, the reduction of P2). These two contrasting ERP effects support a hypothetical view that automatic visual prediction would involve both an increase in the neural response to prediction-incongruent (i.e., novel) events and a decrease in the neural response to prediction-congruent (i.e., redundant) events. This article is part of a Special Issue entitled SI: Prediction and Attention.

The effects of prediction on the perception for own-race and other-race faces

Human beings do not passively perceive important social features about others such as race and age in social interactions. Instead, it is proposed that humans might continuously generate predictions about these social features based on prior similar experiences. Pre-awareness of racial information conveyed by others' faces enables individuals to act in “culturally appropriate” ways, which is useful for interpersonal relations in different ethnicity groups. However, little is known about the effects of prediction on the perception for own-race and other-race faces. Here, we addressed this issue using high temporal resolution event-related potential techniques. In total, data from 24 participants (13 women and 11 men) were analyzed. It was found that the N170 amplitudes elicited by other-race faces, but not own-race faces, were significantly smaller in the predictable condition compared to the unpredictable condition, reflecting a switch to holistic processing of other-race faces when those faces were predictable. In this respect, top-down prediction about face race might contribute to the elimination of the other-race effect (one face recognition impairment). Furthermore, smaller P300 amplitudes were observed for the predictable than for unpredictable conditions, which suggested that the prediction of race reduced the neural responses of human brains.

Separate and concurrent symbolic predictions of sound features are processed differently

The studies investigated the impact of predictive visual information about the pitch and location of a forthcoming sound on the sound processing. In Symbol-to-Sound matching paradigms, symbols induced predictions of particular sounds. The brain's error signals (IR and N2b components of the event-related potential) were measured in response to occasional violations of the prediction, i.e., when a sound was incongruent to the corresponding symbol. IR and N2b index the detection of prediction violations at different levels, IR at a sensory and N2b at a cognitive level. Participants evaluated the congruency between prediction and actual sound by button press. When the prediction referred to only the pitch or only the location feature (Experiment 1), the violation of each feature elicited IR and N2b. The IRs to pitch and location violations revealed differences in the in time course and topography, suggesting that they were generated in feature-specific sensory areas. When the prediction referred to both features concurrently (Experiment 2), that is, the symbol predicted the sound's pitch and location, either one or both predictions were violated. Unexpectedly, no significant effects in the IR range were obtained. However, N2b was elicited in response to all violations. N2b in response to concurrent violations of pitch and location had a shorter latency. We conclude that associative predictions can be established by arbitrary rule-based symbols and for different sound features, and that concurrent violations are processed in parallel. In complex situations as in Experiment 2, capacity limitations appear to affect processing in a hierarchical manner. While predictions were presumably not reliably established at sensory levels (absence of IR), they were established at more cognitive levels, where sounds are represented categorially (presence of N2b).

What we expect is not always what we get: evidence for both the direction-of-change and the specific-stimulus hypotheses of auditory attentional capture

Participants were requested to respond to a sequence of visual targets while listening to a well-known lullaby. One of the notes in the lullaby was occasionally exchanged with a pattern deviant. Experiment 1 found that deviants capture attention as a function of the pitch difference between the deviant and the replaced/expected tone. However, when the pitch difference between the expected tone and the deviant tone is held constant, a violation to the direction-of-pitch change across tones can also capture attention (Experiment 2). Moreover, in more complex auditory environments, wherein it is difficult to build a coherent neural model of the sound environment from which expectations are formed, deviations can capture attention but it appears to matter less whether this is a violation from a specific stimulus or a violation of the current direction-of-change (Experiment 3). The results support the expectation violation account of auditory distraction and suggest that there are at least two different expectations that can be violated: One appears to be bound to a specific stimulus and the other would seem to be bound to a more global cross-stimulus rule such as the direction-of-change based on a sequence of preceding sound events. Factors like base-rate probability of tones within the sound environment might become the driving mechanism of attentional capture—rather than violated expectations—in complex sound environments.

Neural correlates of perceptual grouping effects in the processing of sound omission by musicians and nonmusicians

Perceptual grouping is the process of organizing sounds into perceptually meaningful elements. Psychological studies have found that tones presented as a regular frequency or temporal pattern are grouped according to gestalt principles, such as similarity, proximity, and good continuity. Predictive coding theory suggests that this process helps create an internal model for the prediction of sounds in a tone sequence and that an omission-related brain response reflects the violation of this prediction. However, it remains unclear which brain areas are related to this process, especially in paying attention to the stimuli. To clarify this uncertainty, the present study investigated the neural correlates of perceptual grouping effects. Using magnetoencephalography (MEG), we recorded the evoked response fields (ERFs) of amateur musicians and nonmusicians to sound omissions in tone sequences with a regular or random pattern of three different frequencies during an omission detection task. Omissions in the regular sequences were detected faster and evoked greater activity in the left Heschl's gyrus (HG), right postcentral gyrus, and bilateral superior temporal gyrus (STG) than did omissions in the irregular sequences. Additionally, an interaction between musical experience and regularity was found in the left HG/STG. Tone-evoked responses did not show this difference, indicating that the expertise effect did not reflect the superior tone processing acquired by amateur musicians due to musical training. These results suggest that perceptual grouping based on repetition of a pattern of frequencies affects the processing of omissions in tone sequences and induces more activation of the bilateral auditory cortex by violating internal models. The interaction in the left HG/STG may suggest different styles of processing for musicians and nonmusicians, although this difference was not reflected at the behavioral level.

Preferential encoding of behaviorally relevant predictions revealed by EEG

Statistical regularities in the environment guide perceptual processing; however, some predictions are bound to be more important than others. In this electroencephalogram (EEG) study, we test how task relevance influences the way predictions are learned from the statistics of visual input, and exploited for behavior. We developed a novel task in which participants are simply instructed to respond to a designated target stimulus embedded in a serial stream of non-target stimuli. Presentation probabilities were manipulated such that a designated target cue stimulus predicted the target onset with 70% validity. We also included a corresponding control contingency: a pre-designated control cue predicted a specific non-target stimulus with 70% validity. Participants were not informed about these contingencies. This design allowed us to examine the neural response to task-relevant predictive (cue) and predicted stimuli (target), relative to task-irrelevant predictive (control cue) and predicted stimuli (control non-target). The behavioral results confirmed that participants learned and exploited task-relevant predictions even when not explicitly defined. The EEG results further showed that target-relevant predictions are coded more strongly than statistically equivalent regularities between non-target stimuli. There was a robust modulation of the response for predicted targets associated with learning, enhancing the response to cued stimuli just after 200 ms post-stimulus in central and posterior electrodes, but no corresponding effects for predicted non-target stimuli. These effects of target prediction were preceded by a sustained frontal negativity following presentation of the predictive cue stimulus. These results show that task relevance critically influences how the brain extracts predictive structure from the environment, and exploits these regularities for optimized behavior.

Encoding of nested levels of acoustic regularity in hierarchically organized areas of the human auditory cortex

Our auditory system is able to encode acoustic regularity of growing levels of complexity to model and predict incoming events. Recent evidence suggests that early indices of deviance detection in the time range of the middle-latency responses (MLR) precede the mismatch negativity (MMN), a well-established error response associated with deviance detection. While studies suggest that only the MMN, but not early deviance-related MLR, underlie complex regularity levels, it is not clear whether these two mechanisms interplay during scene analysis by encoding nested levels of acoustic regularity, and whether neuronal sources underlying local and global deviations are hierarchically organized. We registered magnetoencephalographic evoked fields to rapidly presented four-tone local sequences containing a frequency change. Temporally integrated local events, in turn, defined global regularities, which were infrequently violated by a tone repetition. A global magnetic mismatch negativity (MMNm) was obtained at 140-220 ms when breaking the global regularity, but no deviance-related effects were shown in early latencies. Conversely, Nbm (45-55 ms) and Pbm (60-75 ms) deflections of the MLR, and an earlier MMNm response at 120-160 ms, responded to local violations. Distinct neuronal generators in the auditory cortex underlay the processing of local and global regularity violations, suggesting that nested levels of complexity of auditory object representations are represented in separated cortical areas. Our results suggest that the different processing stages and anatomical areas involved in the encoding of auditory representations, and the subsequent detection of its violations, are hierarchically organized in the human auditory cortex.

Investigating bottom-up auditory attention

Bottom-up attention is a sensory-driven selection mechanism that directs perception toward a subset of the stimulus that is considered salient, or attention-grabbing. Most studies of bottom-up auditory attention have adapted frameworks similar to visual attention models whereby local or global “contrast” is a central concept in defining salient elements in a scene. In the current study, we take a more fundamental approach to modeling auditory attention; providing the first examination of the space of auditory saliency spanning pitch, intensity and timbre; and shedding light on complex interactions among these features. Informed by psychoacoustic results, we develop a computational model of auditory saliency implementing a novel attentional framework, guided by processes hypothesized to take place in the auditory pathway. In particular, the model tests the hypothesis that perception tracks the evolution of sound events in a multidimensional feature space, and flags any deviation from background statistics as salient. Predictions from the model corroborate the relationship between bottom-up auditory attention and statistical inference, and argues for a potential role of predictive coding as mechanism for saliency detection in acoustic scenes.

Brain correlates of music-evoked emotions

Music is a universal feature of human societies, partly owing to its power to evoke strong emotions and influence moods. During the past decade, the investigation of the neural correlates of music-evoked emotions has been invaluable for the understanding of human emotion. Functional neuroimaging studies on music and emotion show that music can modulate activity in brain structures that are known to be crucially involved in emotion, such as the amygdala, nucleus accumbens, hypothalamus, hippocampus, insula, cingulate cortex and orbitofrontal cortex. The potential of music to modulate activity in these structures has important implications for the use of music in the treatment of psychiatric and neurological disorders.

A hierarchy of cortical responses to sequence violations in three-month-old infants

The adult human brain quickly adapts to regular temporal sequences, and emits a sequence of novelty responses when these regularities are violated. These novelty responses have been interpreted as error signals that reflect the difference between the incoming signal and predictions generated at multiple cortical levels. Do infants already possess such a hierarchy of violation-detection mechanisms? Using high-density recordings of event-related potentials during an auditory local-global violation paradigm, we show that three-month-old infants process novelty in temporal sequences at two distinct levels. Violations of local expectancies, such as perceiving a deviant vowel "a" after repeated presentation of another vowel i-i-i, elicited an early auditory mismatch response. Conversely, violations of global expectancies, such as hearing the rare sequence a-a-a-a instead of the frequent sequence a-a-a-i, modulated this early mismatch response and led to a late frontal negative slow wave, whose cortical sources included the left inferior frontal region. These results suggest that the infant brain already possesses two dissociable systems for temporal sequence learning.

Mismatch negativity latency and cognitive function in schizophrenia

Background

The Mismatch Negativity (MMN) is an event-related potential (ERP) sensitive to early auditory deviance detection and has been shown to be reduced in schizophrenia patients. Moreover, MMN amplitude reduction to duration deviant tones was found to be related to functional outcomes particularly, to neuropsychological (working memory and verbal domains) and psychosocial measures. While MMN amplitude is thought to be correlated with deficits of early sensory processing, the functional significance of MMN latency remains unclear so far. The present study focused on the investigation of MMN in relation to neuropsychological function in schizophrenia.

Method

Forty schizophrenia patients and 16 healthy controls underwent a passive oddball paradigm (2400 binaural tones; 88% standards [1 kHz, 80 db, 80 ms], 11% frequency deviants [1.2 kHz], 11% duration deviants [40 ms]) and a neuropsychological test-battery. Patients were assessed with regard to clinical symptoms.

Results

Compared to healthy controls schizophrenia patients showed diminished MMN amplitude and shorter MMN latency to both deviants as well as an impaired neuropsychological test performance. Severity of positive symptoms was related to decreased MMN amplitude to duration deviants. Furthermore, enhanced verbal memory performance was associated with prolonged MMN latency to frequency deviants in patients.

Conclusion

The present study corroborates previous results of a diminished MMN amplitude and its association with positive symptoms in schizophrenia patients. Both, the findings of a shorter latency to duration and frequency deviants and the relationship of the latter with verbal memory in patients, emphasize the relevance of the temporal aspect of early auditory discrimination processing in schizophrenia.

Predictability effects in auditory scene analysis: a review

Many sound sources emit signals in a predictable manner. The idea that predictability can be exploited to support the segregation of one source's signal emissions from the overlapping signals of other sources has been expressed for a long time. Yet experimental evidence for a strong role of predictability within auditory scene analysis (ASA) has been scarce. Recently, there has been an upsurge in experimental and theoretical work on this topic resulting from fundamental changes in our perspective on how the brain extracts predictability from series of sensory events. Based on effortless predictive processing in the auditory system, it becomes more plausible that predictability would be available as a cue for sound source decomposition. In the present contribution, empirical evidence for such a role of predictability in ASA will be reviewed. It will be shown that predictability affects ASA both when it is present in the sound source of interest (perceptual foreground) and when it is present in other sound sources that the listener wishes to ignore (perceptual background). First evidence pointing toward age-related impairments in the latter capacity will be addressed. Moreover, it will be illustrated how effects of predictability can be shown by means of objective listening tests as well as by subjective report procedures, with the latter approach typically exploiting the multi-stable nature of auditory perception. Critical aspects of study design will be delineated to ensure that predictability effects can be unambiguously interpreted. Possible mechanisms for a functional role of predictability within ASA will be discussed, and an analogy with the old-plus-new heuristic for grouping simultaneous acoustic signals will be suggested.

Suppression of the auditory N1-component for heartbeat-related sounds reflects interoceptive predictive coding

Although many studies have elucidated the neurocognitive mechanisms supporting the processing of externally generated sensory signals, less is known about the processing of interoceptive signals related to the viscera. Drawing a parallel with research on agency and the perception of self-generated action effects, in the present EEG study we report a reduced auditory N1 component when participants listened to heartbeat-related sounds compared to externally generated sounds. The auditory suppression for heartbeat sounds was robust and persisted after controlling for ECG-related artifacts, the number of trials involved and the phase of the cardiac cycle. In addition, the auditory N1 suppression for heartbeat-related sounds had a comparable scalp distribution as the N1 suppression observed for actively generated sounds. This finding indicates that the brain automatically differentiates between heartbeat-related and externally generated sounds through a process of sensory suppression, suggesting that a comparable predictive mechanism may underlie the processing of heartbeat and action-related information. Extending recent behavioral data about cardio-visual integration, the present cardio-auditory EEG data reveal that the processing of sounds in auditory cortex is systematically modulated by an interoceptive cardiac signal. The findings are discussed with respect to theories of interoceptive awareness, emotion, predictive coding, and their relevance to bodily self-consciousness.

A hierarchy of responses to auditory regularities in the macaque brain

Can monkeys learn simple auditory sequences and detect when a new sequence deviates from the stored pattern? Here we tested the predictive-coding hypothesis, which postulates that cortical areas encode internal models of sensory sequences at multiple hierarchical levels, and use these predictive models to detect deviant stimuli. In humans, hierarchical predictive coding has been supported by studies of auditory sequence processing, but it is unclear whether internal hierarchical models of auditory sequences are also available to nonhuman animals. Using fMRI, we evaluated the encoding of auditory regularities in awake monkeys listening to first- and second-order sequence violations. We observed distinct fMRI responses to first-order violations in auditory cortex and to second-order violations in a frontoparietal network, a distinction only demonstrated in conscious humans so far. The results indicate that the capacity to represent and predict the structure of auditory sequences is shared by humans and nonhuman primates.

Prediction in the service of comprehension: modulated early brain responses to omitted speech segments

Speech signals are often compromised by disruptions originating from external (e.g., masking noise) or internal (e.g., inaccurate articulation) sources. Speech comprehension thus entails detecting and replacing missing information based on predictive and restorative neural mechanisms. The present study targets predictive mechanisms by investigating the influence of a speech segment's predictability on early, modality-specific electrophysiological responses to this segment's omission. Predictability was manipulated in simple physical terms in a single-word framework (Experiment 1) or in more complex semantic terms in a sentence framework (Experiment 2). In both experiments, final consonants of the German words Lachs ([laks], salmon) or Latz ([lats], bib) were occasionally omitted, resulting in the syllable La ([la], no semantic meaning), while brain responses were measured with multi-channel electroencephalography (EEG). In both experiments, the occasional presentation of the fragment La elicited a larger omission response when the final speech segment had been predictable. The omission response occurred ∼125-165 msec after the expected onset of the final segment and showed characteristics of the omission mismatch negativity (MMN), with generators in auditory cortical areas. Suggestive of a general auditory predictive mechanism at work, this main observation was robust against varying source of predictive information or attentional allocation, differing between the two experiments. Source localization further suggested the omission response enhancement by predictability to emerge from left superior temporal gyrus and left angular gyrus in both experiments, with additional experiment-specific contributions. These results are consistent with the existence of predictive coding mechanisms in the central auditory system, and suggestive of the general predictive properties of the auditory system to support spoken word recognition.

What determines auditory distraction? On the roles of local auditory changes and expectation violations

Both the acoustic variability of a distractor sequence and the degree to which it violates expectations are important determinants of auditory distraction. In four experiments we examined the relative contribution of local auditory changes on the one hand and expectation violations on the other hand in the disruption of serial recall by irrelevant sound. We present evidence for a greater disruption by auditory sequences ending in unexpected steady state distractor repetitions compared to auditory sequences with expected changing state endings even though the former contained fewer local changes. This effect was demonstrated with piano melodies (Experiment 1) and speech distractors (Experiment 2). Furthermore, it was replicated when the expectation violation occurred after the encoding of the target items (Experiment 3), indicating that the items' maintenance in short-term memory was disrupted by attentional capture and not their encoding. This seems to be primarily due to the violation of a model of the specific auditory distractor sequences because the effect vanishes and even reverses when the experiment provides no opportunity to build up a specific neural model about the distractor sequence (Experiment 4). Nevertheless, the violation of abstract long-term knowledge about auditory regularities seems to cause a small and transient capture effect: Disruption decreased markedly over the course of the experiments indicating that participants habituated to the unexpected distractor repetitions across trials. The overall pattern of results adds to the growing literature that the degree to which auditory distractors violate situation-specific expectations is a more important determinant of auditory distraction than the degree to which a distractor sequence contains local auditory changes.

Perceiving temporal regularity in music: the role of auditory event-related potentials (ERPs) in probing beat perception

The aim of this chapter is to give an overview of how the perception of a regular beat in music can be studied in humans adults, human newborns, and nonhuman primates using event-related brain potentials (ERPs). Next to a review of the recent literature on the perception of temporal regularity in music, we will discuss in how far ERPs, and especially the component called mismatch negativity (MMN), can be instrumental in probing beat perception. We conclude with a discussion on the pitfalls and prospects of using ERPs to probe the perception of a regular beat, in which we present possible constraints on stimulus design and discuss future perspectives.

Left-hemisphere activation is associated with enhanced vocal pitch error detection in musicians with absolute pitch

The ability to process auditory feedback for vocal pitch control is crucial during speaking and singing. Previous studies have suggested that musicians with absolute pitch (AP) develop specialized left-hemisphere mechanisms for pitch processing. The present study adopted an auditory feedback pitch perturbation paradigm combined with ERP recordings to test the hypothesis whether the neural mechanisms of the left-hemisphere enhance vocal pitch error detection and control in AP musicians compared with relative pitch (RP) musicians and non-musicians (NM). Results showed a stronger N1 response to pitch-shifted voice feedback in the right-hemisphere for both AP and RP musicians compared with the NM group. However, the left-hemisphere P2 component activation was greater in AP and RP musicians compared with NMs and also for the AP compared with RP musicians. The NM group was slower in generating compensatory vocal reactions to feedback pitch perturbation compared with musicians, and they failed to re-adjust their vocal pitch after the feedback perturbation was removed. These findings suggest that in the earlier stages of cortical neural processing, the right hemisphere is more active in musicians for detecting pitch changes in voice feedback. In the later stages, the left-hemisphere is more active during the processing of auditory feedback for vocal motor control and seems to involve specialized mechanisms that facilitate pitch processing in the AP compared with RP musicians. These findings indicate that the left hemisphere mechanisms of AP ability are associated with improved auditory feedback pitch processing during vocal pitch control in tasks such as speaking or singing.