Response from Schröger

Automatic prediction regarding the next state of a visual object: Electrophysiological indicators of prediction match and mismatch

Behavioral phenomena such as representational momentum suggest that the brain can automatically predict the next state of a visual object, based on sequential rules embedded in its preceding spatiotemporal context. To identify electrophysiological indicators of automatic visual prediction in terms of prediction match and mismatch, we recorded event-related brain potentials (ERPs) while participants passively viewed three types of task-irrelevant sequences of a bar stimulus: (1) an oddball sequence, which contained a sequential rule defined by stimulus repetition, providing repetition-rule-conforming (standard) and -violating (deviant) stimuli; (2) a rotating-oddball sequence, which contained a sequential rule defined by stimulus change (i.e., rotation), providing change-rule-conforming (regular) and -violating (irregular) stimuli; and (3) a random sequence, which did not contain a sequential rule, providing a neutral (control) stimulus. This protocol allowed us to expect that (1) an ERP effect that reflects a prediction-mismatch process should be exclusively observed in both the deviant-minus-control and irregular-minus-control comparisons and (2) an ERP effect that reflects a prediction-match process should be exclusively observed in both the standard-minus-control and regular-minus-control comparisons. The results showed that the ERP effect that met the criterion for prediction mismatch was an occipito-temporal negative deflection at around 170-300ms (visual mismatch negativity), while the ERP effect that met the criterion for prediction match was a frontal/central negative deflection at around 150-270ms (probably, the reduction of P2). These two contrasting ERP effects support a hypothetical view that automatic visual prediction would involve both an increase in the neural response to prediction-incongruent (i.e., novel) events and a decrease in the neural response to prediction-congruent (i.e., redundant) events. This article is part of a Special Issue entitled SI: Prediction and Attention.

Visual mismatch negativity: a predictive coding view

An increasing number of studies investigate the visual mismatch negativity (vMMN) or use the vMMN as a tool to probe various aspects of human cognition. This paper reviews the theoretical underpinnings of vMMN in the light of methodological considerations and provides recommendations for measuring and interpreting the vMMN. The following key issues are discussed from the experimentalist's point of view in a predictive coding framework: (1) experimental protocols and procedures to control "refractoriness" effects; (2) methods to control attention; (3) vMMN and veridical perception.

Prediction in the service of comprehension: modulated early brain responses to omitted speech segments

Speech signals are often compromised by disruptions originating from external (e.g., masking noise) or internal (e.g., inaccurate articulation) sources. Speech comprehension thus entails detecting and replacing missing information based on predictive and restorative neural mechanisms. The present study targets predictive mechanisms by investigating the influence of a speech segment's predictability on early, modality-specific electrophysiological responses to this segment's omission. Predictability was manipulated in simple physical terms in a single-word framework (Experiment 1) or in more complex semantic terms in a sentence framework (Experiment 2). In both experiments, final consonants of the German words Lachs ([laks], salmon) or Latz ([lats], bib) were occasionally omitted, resulting in the syllable La ([la], no semantic meaning), while brain responses were measured with multi-channel electroencephalography (EEG). In both experiments, the occasional presentation of the fragment La elicited a larger omission response when the final speech segment had been predictable. The omission response occurred ∼125-165 msec after the expected onset of the final segment and showed characteristics of the omission mismatch negativity (MMN), with generators in auditory cortical areas. Suggestive of a general auditory predictive mechanism at work, this main observation was robust against varying source of predictive information or attentional allocation, differing between the two experiments. Source localization further suggested the omission response enhancement by predictability to emerge from left superior temporal gyrus and left angular gyrus in both experiments, with additional experiment-specific contributions. These results are consistent with the existence of predictive coding mechanisms in the central auditory system, and suggestive of the general predictive properties of the auditory system to support spoken word recognition.

Task difficulty affects the predictive process indexed by visual mismatch negativity

Visual mismatch negativity (MMN) is an event-related brain potential (ERP) component that is elicited by prediction-incongruent events in successive visual stimulation. Previous oddball studies have shown that visual MMN in response to task-irrelevant deviant stimuli is insensitive to the manipulation of task difficulty, which supports the notion that visual MMN reflects attention-independent predictive processes. In these studies, however, visual MMN was evaluated in deviant-minus-standard difference waves, which may lead to an underestimation of the effects of task difficulty due to the possible superposition of N1-difference reflecting refractory effects. In the present study, we investigated the effects of task difficulty on visual MMN, less contaminated by N1-difference. While the participant performed a size-change detection task regarding a continuously-presented central fixation circle, we presented oddball sequences consisting of deviant and standard bar stimuli with different orientations (9.1 and 90.9%) and equiprobable sequences consisting of 11 types of control bar stimuli with different orientations (9.1% each) at the surrounding visual fields. Task difficulty was manipulated by varying the magnitude of the size-change. We found that the peak latencies of visual MMN evaluated in the deviant-minus-control difference waves were delayed as a function of task difficulty. Therefore, in contrast to the previous understanding, the present findings support the notion that visual MMN is associated with attention-demanding predictive processes.

Non-linear laws of echoic memory and auditory change detection in humans

BACKGROUND

The detection of any abrupt change in the environment is important to survival. Since memory of preceding sensory conditions is necessary for detecting changes, such a change-detection system relates closely to the memory system. Here we used an auditory change-related N1 subcomponent (change-N1) of event-related brain potentials to investigate cortical mechanisms underlying change detection and echoic memory.

RESULTS

Change-N1 was elicited by a simple paradigm with two tones, a standard followed by a deviant, while subjects watched a silent movie. The amplitude of change-N1 elicited by a fixed sound pressure deviance (70 dB vs. 75 dB) was negatively correlated with the logarithm of the interval between the standard sound and deviant sound (1, 10, 100, or 1000 ms), while positively correlated with the logarithm of the duration of the standard sound (25, 100, 500, or 1000 ms). The amplitude of change-N1 elicited by a deviance in sound pressure, sound frequency, and sound location was correlated with the logarithm of the magnitude of physical differences between the standard and deviant sounds.

CONCLUSIONS

The present findings suggest that temporal representation of echoic memory is non-linear and Weber-Fechner law holds for the automatic cortical response to sound changes within a suprathreshold range. Since the present results show that the behavior of echoic memory can be understood through change-N1, change-N1 would be a useful tool to investigate memory systems.

Human Visual System Automatically Encodes Sequential Regularities of Discrete Events

For our adaptive behavior in a dynamically changing environment, an essential task of the brain is to automatically encode sequential regularities inherent in the environment into a memory representation. Recent studies in neuroscience have suggested that sequential regularities embedded in discrete sensory events are automatically encoded into a memory representation at the level of the sensory system. This notion is largely supported by evidence from investigations using auditory mismatch negativity (auditory MMN), an event-related brain potential (ERP) correlate of an automatic memory-mismatch process in the auditory sensory system. However, it is still largely unclear whether or not this notion can be generalized to other sensory modalities. The purpose of the present study was to investigate the contribution of the visual sensory system to the automatic encoding of sequential regularities using visual mismatch negativity (visual MMN), an ERP correlate of an automatic memory-mismatch process in the visual sensory system. To this end, we conducted a sequential analysis of visual MMN in an oddball sequence consisting of infrequent deviant and frequent standard stimuli, and tested whether the underlying memory representation of visual MMN generation contains only a sensory memory trace of standard stimuli (trace-mismatch hypothesis) or whether it also contains sequential regularities extracted from the repetitive standard sequence (regularity-violation hypothesis). The results showed that visual MMN was elicited by first deviant (deviant stimuli following at least one standard stimulus), second deviant (deviant stimuli immediately following first deviant), and first standard (standard stimuli immediately following first deviant), but not by second standard (standard stimuli immediately following first standard). These results are consistent with the regularity-violation hypothesis, suggesting that the visual sensory system automatically encodes sequential regularities. In combination with a wide range of auditory MMN studies, the present study highlights the critical role of sensory systems in automatically encoding sequential regularities when modeling the world.

Comparator and non-comparator mechanisms of change detection in the context of speech--an ERP study

Automatic change detection reflects a cognitive memory-based comparison mechanism as well as a sensorial non-comparator mechanism based on differential states of refractoriness. The purpose of this study was to examine whether the comparator mechanism of the mismatch negativity component (MMN) is differentially affected by the lexical status of the deviant. Event-related potential (ERP) data was collected during an "oddball" paradigm designed to elicit the MMN from 15 healthy subjects that were involved in a counting task. Topography pattern analysis and source estimation were utilized to examine the deviance (deviants vs. standards), cognitive (deviants vs. control counterparts) and refractoriness (standards vs. control counterparts) effects elicited by standard-deviant pairs ("deh-day"; "day-deh"; "teh-tay") embedded within "oddball" blocks. Our results showed that when the change was salient regardless of lexical status (i.e., the /e:/ to /eI/ transition) the response tapped the comparator based-mechanism of the MMN which was located in the cuneus/posterior cingulate, reflected sensitivity to the novelty of the auditory object, appeared in the P2 latency range and mainly involved topography modulations. In contrast, when the novelty was low (i.e., the /eI/ to /e:/ transition) an acoustic change complex was elicited which involved strength modulations over the P1/N1 range and implicated the middle temporal gyrus. This result pattern also resembled the one displayed by the non-comparator mechanism. These findings suggest spatially and temporally distinct brain activities of comparator mechanisms of change detection in the context of speech.

MMN or no MMN: no magnitude of deviance effect on the MMN amplitude

Based on results showing that the "deviant-minus-standard" estimate of the mismatch negativity (MMN) amplitude increases with increasing amounts of deviance, it has been suggested that the MMN amplitude reflects the amount of difference between the neural representations of the standard and the deviant sound. However, the deviant-minus-standard waveform also includes an N1 difference. We tested the effects of the magnitude of deviance on MMN while minimizing this N1 confound. We found no significant magnitude of deviance effect on the genuine MMN amplitude. Thus we suggest that the average MMN amplitude does not reflect the difference between neural stimulus representations; rather it may index the percentage of detected deviants, each of which elicits an MMN response of uniform amplitude. These results are compatible with an explanation suggesting that MMN is involved in maintaining a neural representation of the auditory environment.

Localizing pre-attentive auditory memory-based comparison: Magnetic mismatch negativity to pitch change

Changes in the pitch of repetitive sounds elicit the mismatch negativity (MMN) of the event-related brain potential (ERP). There exist two alternative accounts for this index of automatic change detection: (1) A sensorial, non-comparator account according to which ERPs in oddball sequences are affected by differential refractory states of frequency-specific afferent cortical neurons. (2) A cognitive, comparator account stating that MMN reflects the outcome of a memory comparison between a neuronal model of the frequently presented standard sound with the sensory memory representation of the changed sound. Using a condition controlling for refractoriness effects, the two contributions to MMN can be disentangled. The present study used whole-head MEG to further elucidate the sensorial and cognitive contributions to frequency MMN. Results replicated ERP findings that MMN to pitch change is a compound of the activity of a sensorial, non-comparator mechanism and a cognitive, comparator mechanism which could be separated in time. The sensorial part of frequency MMN consisting of spatially dipolar patterns was maximal in the late N1 range (105-125 ms), while the cognitive part peaked in the late MMN-range (170-200 ms). Spatial principal component analyses revealed that the early part of the traditionally measured MMN (deviant minus standard) is mainly due to the sensorial mechanism while the later mainly due to the cognitive mechanism. Inverse modeling revealed sources for both MMN contributions in the gyrus temporales transversus, bilaterally. These MEG results suggest temporally distinct but spatially overlapping activities of non-comparator-based and comparator-based mechanisms of automatic frequency change detection in auditory cortex.

Mismatch Negativity

Since its discovery by Näätänen and colleagues in 1978, the mismatch negativity (MMN) has been used as an index of auditory sensory memory. The present paper explicates various possibilities of how MMN can assess memory functions, it reveals possible traps when interpreting MMN as an index of auditory memory, and it reviews recent developments of paradigms showing that memory on a short time-scale, consolidation of memory traces, and even implicit memory can be probed with MMN.

Early phase of spatial mismatch negativity is localized to a posterior “where” auditory pathway

The auditory mismatch negativity (MMN) is an event-related potential that reflects early processing of changes in acoustic stimulus features. Although the MMN has been well characterized by previous work, the number, roles, and anatomical locations of its cortical generators remain unresolved. Here, we report that the MMN elicited by occasional deviations in sound location is comprised of two temporally and anatomically distinct phases: an early phase with a generator posterior to auditory cortex and contralateral to the deviant stimulus, and a later phase with generators that are more frontal and bilaterally symmetric. The posterior location of the early-phase generator suggests the engagement of neurons within a putative "where" pathway for processing spatial auditory information.

Memory-based or afferent processes in mismatch negativity (MMN): a review of the evidence

The mismatch negativity (MMN) is an electromagnetic response to any discriminable change in regular auditory input. This response is usually interpreted as being generated by an automatic cortical change-detection process in which a difference is found between the current input and the representation of the regular aspects of the preceding auditory input. Recently, this interpretation was questioned by Jääskeläinen et al. (2004) who proposed that the MMN is a product of an N1 (N1a) difference wave emerging in the subtraction procedure used to visualize and quantify the MMN. We now evaluate this "adaptation hypothesis" of the MMN in the light of the available data. It is shown that the MMN cannot be accounted for by differential activation of the afferent N1 transient detectors by repetitive ("standard") stimuli and deviant ("novel") stimuli and that the presence of a memory representation of the standard is required for the elicitation of MMN.

Input to verbal working memory: Preattentive construction of the central speech representation

Working memory uses central sound representations as an informational basis. The central sound representation is the temporally and feature-integrated mental representation that corresponds to phenomenal perception. It is used in (higher-order) mental operations and stored in long-term memory. In the bottom-up processing path, the central sound representation can be probed at the level of auditory sensory memory with the mismatch negativity (MMN) of the event-related potential. The present paper reviews a newly developed MMN paradigm to tap into the processing of speech sound representations. Preattentive vowel categorization based on F1-F2 formant information occurs in speech sounds and complex tones even under conditions of high variability of the auditory input. However, an additional experiment demonstrated the limits of the preattentive categorization of language-relevant information. It tested whether the system categorizes complex tones containing the F1 and F2 formant components of the vowel /a/ differently than six sounds with nonlanguage-like F1-F2 combinations. From the absence of an MMN in this experiment, it is concluded that no adequate vowel representation was constructed. This shows limitations of the capability of preattentive vowel categorization.

Roughness perception in sounds: behavioral and ERP evidence

The mismatch negativity (MMN) correlates of the perception of roughness, the unpleasant character of sounds caused by the perception of amplitude fluctuation in the range of 20-200 Hz, were studied on the basis of a variation in the degree of modulation (=modulation index m), which is a main parameter influencing roughness. A psychophysical study showed that perceived roughness of tones increased with modulation index for m-values from 0 up to 1.2. For larger values of m, roughness perception remained stable. In a subsequent ERP-study, infrequent amplitude modulated (AM) tones with varying modulation index were presented in the context of a series of pure tones in an ignore condition. The amplitude of the mismatch negativity correlated highly with the roughness ratings (r = -0.93) and did not increase monotonously with increasing modulation index. We conclude that perceived roughness rather than its physical correlate in sounds is reflected by the MMN and that roughness is thus preattentively encoded.

Preattentive memory-based comparison of sound intensity

Changes in the intensity of repeated, ignored sounds elicit the mismatch negativity (MMN) brain response which reflects preattentive detection of the change. It is generally assumed that the MMN in response to intensity changes reflects a memory-based comparison mechanism rather than being due to differential states of refractoriness of intensity-specific cortical neurons. In the present study, an experimental protocol consisting of 4 oddball blocks and 1 control block was used in order to separate memory-comparison-related effects from refractoriness-related ones. This design allowed an assessment of intensity MMN using physically identical stimuli with equal probability of occurrence in separate blocks, while avoiding contamination by refractoriness. Results were consistent with an MMN in response to intensity change that reflects genuine memory-based comparison.

Measuring duration mismatch negativity

OBJECTIVE

Automatic comparisons of sound duration in auditory sensory memory are typically investigated by comparing event-related potentials (ERPs) to standard and deviant stimuli presented in oddball blocks. Deviants elicit mismatch negativity (MMN). This procedure might overestimate an MMN contribution reflecting automatic sensory memory processes because of differential states of refractoriness of respectively recruited neural populations [Neuroreport 1996;7:3005; Psychophysiology 2001;38:723]. Here, memory-comparison-based Duration MMN contributions were investigated using various experimental protocols.

METHODS

Memory-comparison-based first-order Duration MMN was investigated using 4 blocked conditions: (a) descending Deviant (100 ms, P=0.14), 150 ms Standard; (b) reverse ascending Deviant (150 ms), 100 ms Standard; (c) Control comprised of 7 equiprobable durations between 25 and 175 ms; and additionally (d) equiprobable tones between 100 and 400 ms. Using the former 3 conditions, Deviants, Standards and Controls were physically identical.

RESULTS

Comparing Deviants and Controls excluded potential refractoriness effects, and a decomposition of memory-comparison-based MMN and residual MMN was demonstrated. Genuine Duration MMN was also obtained in the deviant-standard-reverse comparison.

CONCLUSIONS

Using a blocked control condition yielded equivalent results to reversing the role of deviant and standard in two separate oddball blocks. Using the reverse ascending deviant condition is thus sufficient as a control.

Behavioral and electrophysiological effects of task-irrelevant sound change: a new distraction paradigm

A distraction paradigm was utilized that is suited to yield reliable auditory distraction on an individual level even with rather small frequency deviances (7%). Distraction to these tiny deviants was achieved by embedding task-relevant aspects and task-irrelevant, distracting aspects of stimulation into the same perceptual object. Event-related potential (ERP) and behavioral effects of this newly developed paradigm were determined. Subjects received tones that could be of short or long duration equiprobably. They were instructed to press a response button to long-duration tones (targets). In oddball blocks, tones could be of standard frequency or of low-probability (p=0.1), deviant frequency. The task-irrelevant frequency deviants elicited MMN, N2b, and P3a components, and caused impoverished behavioral performance to targets. The usage of tiny distractors permits an interpretation of auditory distraction in terms of attention switching due to a particular memory-related change-detection process. On the basis of the results from an additional condition in which tones were of 10 different frequencies (involving those frequencies which served as standard and deviant in oddball blocks), it is argued that one important prerequisite for linking the neural mechanisms reflected in change-related brain waves to behavioral distraction effects may be regarded as fulfilled. The robustness of the distraction effects to tiny deviations was confirmed in two control experiments.

Pre-attentive processing of spectrally complex sounds with asynchronous onsets: an event-related potential study with human subjects

Neuronal mechanisms involved in the processing of complex sounds with asynchronous onsets were studied in reading subjects. The sound onset asynchrony (SOA) between the leading partial and the remaining complex tone was varied between 0 and 360 ms. Infrequently occurring deviant sounds (in which one out of 10 harmonics was different in pitch relative to the frequently occurring standard sound) elicited the mismatch negativity (MMN), a change-specific cortical event-related potential (ERP) component. This indicates that the pitch of standard stimuli had been pre-attentively coded by sensory-memory traces. Moreover, when the complex-tone onset fell within temporal integration window initiated by the leading-partial onset, the deviants elicited the N2b component. This indexes that involuntary attention switch towards the sound change occurred. In summary, the present results support the existence of pre-perceptual integration mechanism of 100-200 ms duration and emphasize its importance in switching attention towards the stimulus change.