Stimulus Processing Constraints in Audition

Spatial and frequency specificity of the ventriloquism aftereffect revisited

Exposure to audiovisual stimuli with a consistent spatial misalignment seems to result in a recalibration of unisensory auditory spatial representations. The previous studies have suggested that this so-called ventriloquism aftereffect is confined to the trained region of space, but yielded inconsistent results as to whether or not recalibration generalizes to untrained sound frequencies. Here, we reassessed the spatial and frequency specificity of the ventriloquism aftereffect by testing whether auditory spatial perception can be independently recalibrated for two different sound frequencies and/or at two different spatial locations. Recalibration was confined to locations within the trained hemifield, suggesting that spatial representations were independently adjusted for the two hemifields. The frequency specificity of the ventriloquism aftereffect depended on the presence or the absence of conflicting audiovisual adaptation stimuli within the same hemifield. Moreover, adaptation of two different sound frequencies in opposite directions (leftward vs. rightward) resulted in a selective suppression of leftward recalibration, even when the adapting stimuli were presented in different hemifields. Thus, instead of representing a fixed stimulus-driven process, cross-modal recalibration seems to critically depend on the sensory context and takes into account inconsistencies in the cross-modal input.

Attending to auditory memory

Attention to memory describes the process of attending to memory traces when the object is no longer present. It has been studied primarily for representations of visual stimuli with only few studies examining attention to sound object representations in short-term memory. Here, we review the interplay of attention and auditory memory with an emphasis on 1) attending to auditory memory in the absence of related external stimuli (i.e., reflective attention) and 2) effects of existing memory on guiding attention. Attention to auditory memory is discussed in the context of change deafness, and we argue that failures to detect changes in our auditory environments are most likely the result of a faulty comparison system of incoming and stored information. Also, objects are the primary building blocks of auditory attention, but attention can also be directed to individual features (e.g., pitch). We review short-term and long-term memory guided modulation of attention based on characteristic features, location, and/or semantic properties of auditory objects, and propose that auditory attention to memory pathways emerge after sensory memory. A neural model for auditory attention to memory is developed, which comprises two separate pathways in the parietal cortex, one involved in attention to higher-order features and the other involved in attention to sensory information. This article is part of a Special Issue entitled SI: Auditory working memory.

The time course of activation of object shape and shape+colour representations during memory retrieval

Little is known about the timing of activating memory for objects and their associated perceptual properties, such as colour, and yet this is important for theories of human cognition. We investigated the time course associated with early cognitive processes related to the activation of object shape and object shape+colour representations respectively, during memory retrieval as assessed by repetition priming in an event-related potential (ERP) study. The main findings were as follows: (1) we identified a unique early modulation of mean ERP amplitude during the N1 that was associated with the activation of object shape independently of colour; (2) we also found a subsequent early P2 modulation of mean amplitude over the same electrode clusters associated with the activation of object shape+colour representations; (3) these findings were apparent across both familiar (i.e., correctly coloured – yellow banana) and novel (i.e., incorrectly coloured - blue strawberry) objects; and (4) neither of the modulations of mean ERP amplitude were evident during the P3. Together the findings delineate the timing of object shape and colour memory systems and support the notion that perceptual representations of object shape mediate the retrieval of temporary shape+colour representations for familiar and novel objects.

Decomposing the Garner interference paradigm: evidence for dissociations between macrolevel and microlevel performance

Three Garner interference experiments are described in which baseline, filtering, and correlated performance were assessed at both a macrolevel (condition average) and microlevel (intertrial contingency), using the pair-wise combinations of auditory pitch, loudness, and location. Discrepancies between pairs of dimensions were revealed between macro- and microlevel estimates of performance and, also, between filtering costs and correlated benefits, relative to baseline. The examination of the intertrial effects associated with filtering costs suggested that effects of increased stimulus uncertainty were mandatory, whereas effects of irrelevant variation were not. The examination of the intertrial effects associated with correlated benefits suggested that the detection of stimulus repetition took precedence over that of stimulus change. Violations of standard horse race accounts of processing did not appear to stem from differences in the absolute or relative speeds of processing between dimensions but, rather, from the special role that certain dimensions (e.g., pitch) may play in certain modalities (e.g., audition). The utility of examining repetition effects is demonstrated by revealing a level of understanding regarding stimulus processing typically hidden by aggregated measures of performance.

Trial after trial: General processing consequences as a function of repetition and change in multidimensional sound

While there are pointers relating to the consequences of repetition, a general framework regarding the cognitive implications of processing multidimensional stimuli as a function of previous stimulus history is currently lacking. Three experiments using sounds varying in location and pitch were carried out, in which the immediate consequences of repeating or changing task-relevant and task-irrelevant attributes were orthogonally examined. A consistent pattern of data was shown, in that the magnitude of selective attention failure was larger when the task-relevant value repeated across trials, while differences between dimensions were larger when the task-relevant value changed across trials. These effects of irrelevance and dimension as a function of intertrial contingency are summarized in a model depicting the dynamic allocation of processing resource.

To switch or not to switch: brain potential indices of attentional control after task-relevant and task-irrelevant changes of stimulus features

Attention is controlled by the interplay of sensory input and top-down processes. We compared attentional control processes during task switching and reorientation after distraction. The primary task was to discriminate laterally and centrally presented tones; these stimuli were composed of a frequent standard or an infrequent deviant pitch. In the distraction condition, pitch was irrelevant and could be ignored. In the switch condition, pitch changes were relevant: whenever a deviant tone was presented, participants had to discriminate its pitch and not its direction. The task in standard trials remained unchanged. In both conditions, deviants elicited mismatch negativity (MMN), P3a, P3b, and reorienting negativity (RON). We, therefore, suggest that distraction and switching are triggered by the same system of attentional control. In addition, remarkable differences were observable between the two conditions: In the switch condition the MMN was followed by a more pronounced N2b and P3a. The differences between these components support the idea that in the distraction condition, a switch of attention is only initiated but not completely performed.

It all sounds the same to me: sequential ERP and behavioral effects during pitch and harmonicity judgments

The representation of complex sounds was examined by comparing both behavioral and event-related brain potentials (ERPs) to the change or repetition of fundamental frequency (f0) and harmonicity. In the pitch task, participants were asked to categorize the incoming stimulus as either low or high, regardless of harmonicity, and in the harmonicity task, participants indicated whether the stimulus was tuned or mistuned, regardless of pitch. Over three experiments, participants were faster in responding to pitch than to harmonicity. As a result of this asymmetry, behavioral and ERP data showed that irrelevant changes in harmonicity had little impact on performance during the pitch task, whereas harmonicity judgments were impeded by irrelevant changes in f0. These data are consistent with both general horse-race accounts of processing and specific accounts of mistuning detection that posit prior f0 registration. In addition, ERP components N2 and P3 were modulated by both intertrial contingency and task instructions, revealing the further influence of top-down mechanisms on concurrent sound segregation.

Feature- and object-based attentional modulation in the human auditory "where" pathway

Attending to a visual stimulus feature, such as color or motion, enhances the processing of that feature in the visual cortex. Moreover, the processing of the attended object's other, unattended, features is also enhanced. Here, we used functional magnetic resonance imaging to show that attentional modulation in the auditory system may also exhibit such feature- and object-specific effects. Specifically, we found that attending to auditory motion increases activity in nonprimary motion-sensitive areas of the auditory cortical "where" pathway. Moreover, activity in these motion-sensitive areas was also increased when attention was directed to a moving rather than a stationary sound object, even when motion was not the attended feature. An analysis of effective connectivity revealed that the motion-specific attentional modulation was brought about by an increase in connectivity between the primary auditory cortex and nonprimary motion-sensitive areas, which, in turn, may have been mediated by the paracingulate cortex in the frontal lobe. The current results indicate that auditory attention can select both objects and features. The finding of feature-based attentional modulation implies that attending to one feature of a sound object does not necessarily entail an exhaustive processing of the object's unattended features.

Two separate mechanisms underlie auditory change detection and involuntary control of attention

We used behavioral and event-related potential (ERP) measures to study the neural mechanisms of involuntary attention switching to changes in unattended sounds. Our subjects discriminated two equiprobable sounds differing in frequency (fundamental frequency 186 or 196 Hz) while task-irrelevant intensity decrements or increments (-3, -6, -9, +3, +6, or +9 dB, standard intensity 60 dB HL) infrequently occurred in the same sounds. In line with the results of previous studies, discrimination performance deteriorated with increasing magnitude of the task-irrelevant intensity change. However, these distraction effects were dissimilar for intensity increments and decrements: while there were no differences in reaction time (RT) between intensity decrements and increments, hit rates (HR) were lower for large intensity increments than for large decrements. ERPs to task-irrelevant intensity increments and decrements were also distinctly different: the response to intensity increments consisted of an N1 enhancement, mismatch negativity (MMN), and P3a, while the response to intensity decrements consisted only of MMN. These results are consistent with the assumption that two separate mechanisms (indexed by N1 and MMN) underlie auditory change detection. However, the finding that distinct distraction effects were obtained for both intensity decrements and increments but that the P3a is elicited only by the intensity increments seems to suggest that P3a may not be regarded as a general index of attentional shift but rather it is only generated in conditions in which an enhanced N1 is elicited, too.

Effects of visual attentional load on low-level auditory scene analysis

The sharing of processing resources between the senses was investigated by examining the effects of visual task load on auditory event-related brain potentials (ERPs). In Experiment 1, participants completed both a zero-back and a one-back visual task while a tone pattern or a harmonic series was presented. N1 and P2 waves were modulated by visual task difficulty, but neither mismatch negativity (MMN) elicited by deviant stimuli from the tone pattern nor object-related negativity (ORN) elicited by mistuning from the harmonic series was affected. In Experiment 2, participants responded to identity (what) or location (where) in vision, while ignoring sounds alternating in either pitch (what) or location (where). Auditory ERP modulations were consistent with task difficulty, rather than with task specificity. In Experiment 3, we investigated auditory ERP generation under conditions of no visual task. The results are discussed with respect to a distinction between process-general (N1 and P2) and process-specific (MMN and ORN) auditory ERPs.