Effects of visual attentional load on low-level auditory scene analysis

Auditory change detection and visual selective attention: association between MMN and N2pc

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

Neural correlates of concurrent sound perception: A review and guidelines for future research

The perception of concurrent sound sources depends on processes (i.e., auditory scene analysis) that fuse and segregate acoustic features according to harmonic relations, temporal coherence, and binaural cues (encompass dichotic pitch, location difference, simulated echo). The object-related negativity (ORN) and P400 are electrophysiological indices of concurrent sound perception. Here, we review the different paradigms used to study concurrent sound perception and the brain responses obtained from these paradigms. Recommendations regarding the design and recording parameters of the ORN and P400 are made, and their clinical applications in assessing central auditory processing ability in different populations are discussed.

Spatial release from informational masking enhances the early cortical representation of speech sounds

INTRODUCTION

Spatial separation between competing speech streams reduces their confusion (informational masking), improving speech processing under challenging listening conditions. The precise stages of auditory processing involved in this benefit are not fully understood. This study used event-related potentials to examine the processing of target speech under conditions of informational masking and its spatial release.

METHODS

Participants detected noise-vocoded target speech presented with two-talker noise-vocoded masking speech. In separate conditions, the same set of targets were spatially co-located with maskers to produce informational masking and spatially separated from maskers using a perceptual manipulation to release the informational masking.

RESULTS

An increase in N1 and P2 amplitude, consistent with cortical auditory evoked potentials, and a later sustained positivity (P300) were observed in response to target onsets only under conditions supporting release from informational masking. At target intensities above masking threshold in both spatial conditions, N1 and P2 latencies were shorter when targets and maskers were perceptually separated.

DISCUSSION

These results indicate that spatial release from informational masking benefits speech representation beginning in the early stages of auditory perception. Additionally, these results suggest that the auditory evoked potential itself may be heavily dependent upon how information is perceptually organized rather than physically organized.

Visual Perceptual Load Does Not Affect the Frequency Mismatch Negativity

The mismatch negativity (MMN) has been of particular interest in auditory perception because of its sensitivity to auditory change. It is typically measured in an oddball task and is computed as the difference of deviant minus standard tones. Previous studies suggest that the oddball MMN can be reduced by crossmodal attention to a concurrent, difficult visual task. However, more recent studies did not replicate this effect. Because previous findings seem to be biased, we preregistered the present study and used Bayesian hypothesis testing to measure the strength of evidence for or against an effect of visual task difficulty. We manipulated visual perceptual load (high and low load). In the task, the visual stimuli were identical for both loads to avoid confounding effects from physical differences of the visual stimuli. We also measured the corrected MMN because the oddball MMN may be confounded by physical differences between deviant and standard tones. The corrected MMN is obtained with a separate control condition in which the same tone as the deviant (critical tone) is equiprobable with other tones. The corrected MMN is computed as deviant minus critical tones. Furthermore, we assessed working memory capacity to examine its moderating role. In our large sample (N = 49), the evidential strength in support of no effect of visual load was moderate for the oddball MMN (9.09 > BF01 > 3.57) and anecdotal to moderate for the corrected MMN (4.55 > BF01 > 2.17). Also, working memory capacity did not correlate with the visual load effect on the oddball MMN and the corrected MMN. The present findings support the robustness of the auditory frequency MMN to manipulations of crossmodal, visual attention and suggest that this relationship is not moderated by working memory capacity.

Auditory Figure-Ground Segregation Is Impaired by High Visual Load

Figure-ground segregation is fundamental to listening in complex acoustic environments. An ongoing debate pertains to whether segregation requires attention or is "automatic" and preattentive. In this magnetoencephalography study, we tested a prediction derived from load theory of attention (e.g., Lavie, 1995) that segregation requires attention but can benefit from the automatic allocation of any "leftover" capacity under low load. Complex auditory scenes were modeled with stochastic figure-ground stimuli (Teki et al., 2013), which occasionally contained repeated frequency component "figures." Naive human participants (both sexes) passively listened to these signals while performing a visual attention task of either low or high load. While clear figure-related neural responses were observed under conditions of low load, high visual load substantially reduced the neural response to the figure in auditory cortex (planum temporale, Heschl's gyrus). We conclude that fundamental figure-ground segregation in hearing is not automatic but draws on resources that are shared across vision and audition.SIGNIFICANCE STATEMENT This work resolves a long-standing question of whether figure-ground segregation, a fundamental process of auditory scene analysis, requires attention or is underpinned by automatic, encapsulated computations. Task-irrelevant sounds were presented during performance of a visual search task. We revealed a clear magnetoencephalography neural signature of figure-ground segregation in conditions of low visual load, which was substantially reduced in conditions of high visual load. This demonstrates that, although attention does not need to be actively allocated to sound for auditory segregation to occur, segregation depends on shared computational resources across vision and hearing. The findings further highlight that visual load can impair the computational capacity of the auditory system, even when it does not simply dampen auditory responses as a whole.

The effect of distraction on change detection in crowded acoustic scenes

In this series of behavioural experiments we investigated the effect of distraction on the maintenance of acoustic scene information in short-term memory. Stimuli are artificial acoustic ‘scenes’ composed of several (up to twelve) concurrent tone-pip streams (‘sources’). A gap (1000 ms) is inserted partway through the ‘scene’; Changes in the form of an appearance of a new source or disappearance of an existing source, occur after the gap in 50% of the trials. Listeners were instructed to monitor the unfolding ‘soundscapes’ for these events. Distraction was measured by presenting distractor stimuli during the gap. Experiments 1 and 2 used a dual task design where listeners were required to perform a task with varying attentional demands (‘High Demand’ vs. ‘Low Demand’) on brief auditory (Experiment 1a) or visual (Experiment 1b) signals presented during the gap. Experiments 2 and 3 required participants to ignore distractor sounds and focus on the change detection task. Our results demonstrate that the maintenance of scene information in short-term memory is influenced by the availability of attentional and/or processing resources during the gap, and that this dependence appears to be modality specific. We also show that these processes are susceptible to bottom up driven distraction even in situations when the distractors are not novel, but occur on each trial. Change detection performance is systematically linked with the, independently determined, perceptual salience of the distractor sound. The findings also demonstrate that the present task may be a useful objective means for determining relative perceptual salience.

•

Distraction is measured by presenting distractor stimuli during a scene gap.

•

Scene maintenance in memory depends on availability of resources during the gap.

•

This dependence appears to be modality specific.

•

Scene maintenance also prone to bottom up distraction even when distractors not novel.

•

Performance depends on the perceptual salience of the distractor sound.

Inattentional Deafness: Visual Load Leads to Time-Specific Suppression of Auditory Evoked Responses

Due to capacity limits on perception, conditions of high perceptual load lead to reduced processing of unattended stimuli (Lavie et al., 2014). Accumulating work demonstrates the effects of visual perceptual load on visual cortex responses, but the effects on auditory processing remain poorly understood. Here we establish the neural mechanisms underlying "inattentional deafness"--the failure to perceive auditory stimuli under high visual perceptual load. Participants performed a visual search task of low (target dissimilar to nontarget items) or high (target similar to nontarget items) load. On a random subset (50%) of trials, irrelevant tones were presented concurrently with the visual stimuli. Brain activity was recorded with magnetoencephalography, and time-locked responses to the visual search array and to the incidental presence of unattended tones were assessed. High, compared to low, perceptual load led to increased early visual evoked responses (within 100 ms from onset). This was accompanied by reduced early (∼ 100 ms from tone onset) auditory evoked activity in superior temporal sulcus and posterior middle temporal gyrus. A later suppression of the P3 "awareness" response to the tones was also observed under high load. A behavioral experiment revealed reduced tone detection sensitivity under high visual load, indicating that the reduction in neural responses was indeed associated with reduced awareness of the sounds. These findings support a neural account of shared audiovisual resources, which, when depleted under load, leads to failures of sensory perception and awareness.

SIGNIFICANCE STATEMENT

The present work clarifies the neural underpinning of inattentional deafness under high visual load. The findings of near-simultaneous load effects on both visual and auditory evoked responses suggest shared audiovisual processing capacity. Temporary depletion of shared capacity in perceptually demanding visual tasks leads to a momentary reduction in sensory processing of auditory stimuli, resulting in inattentional deafness. The dynamic "push-pull" pattern of load effects on visual and auditory processing furthers our understanding of both the neural mechanisms of attention and of cross-modal effects across visual and auditory processing. These results also offer an explanation for many previous failures to find cross-modal effects in experiments where the visual load effects may not have coincided directly with auditory sensory processing.

Visual Task Demands and the Auditory Mismatch Negativity: An Empirical Study and a Meta-Analysis

Because the auditory system is particularly useful in monitoring the environment, previous research has examined whether task-irrelevant, auditory distracters are processed even if subjects focus their attention on visual stimuli. This research suggests that attentionally demanding visual tasks decrease the auditory mismatch negativity (MMN) to simultaneously presented auditory distractors. Because a recent behavioral study found that high visual perceptual load decreased detection sensitivity of simultaneous tones, we used a similar task (n = 28) to determine if high visual perceptual load would reduce the auditory MMN. Results suggested that perceptual load did not decrease the MMN. At face value, these nonsignificant findings may suggest that effects of perceptual load on the MMN are smaller than those of other demanding visual tasks. If so, effect sizes should differ systematically between the present and previous studies. We conducted a selective meta-analysis of published studies in which the MMN was derived from the EEG, the visual task demands were continuous and varied between high and low within the same task, and the task-irrelevant tones were presented in a typical oddball paradigm simultaneously with the visual stimuli. Because the meta-analysis suggested that the present (null) findings did not differ systematically from previous findings, the available evidence was combined. Results of this meta-analysis confirmed that demanding visual tasks reduce the MMN to auditory distracters. However, because the meta-analysis was based on small studies and because of the risk for publication biases, future studies should be preregistered with large samples (n > 150) to provide confirmatory evidence for the results of the present meta-analysis. These future studies should also use control conditions that reduce confounding effects of neural adaptation, and use load manipulations that are defined independently from their effects on the MMN.

Sound segregation via embedded repetition is robust to inattention

The segregation of sound sources from the mixture of sounds that enters the ear is a core capacity of human hearing, but the extent to which this process is dependent on attention remains unclear. This study investigated the effect of attention on the ability to segregate sounds via repetition. We utilized a dual task design in which stimuli to be segregated were presented along with stimuli for a "decoy" task that required continuous monitoring. The task to assess segregation presented a target sound 10 times in a row, each time concurrent with a different distractor sound. McDermott, Wrobleski, and Oxenham (2011) demonstrated that repetition causes the target sound to be segregated from the distractors. Segregation was queried by asking listeners whether a subsequent probe sound was identical to the target. A control task presented similar stimuli but probed discrimination without engaging segregation processes. We present results from 3 different decoy tasks: a visual multiple object tracking task, a rapid serial visual presentation (RSVP) digit encoding task, and a demanding auditory monitoring task. Load was manipulated by using high- and low-demand versions of each decoy task. The data provide converging evidence of a small effect of attention that is nonspecific, in that it affected the segregation and control tasks to a similar extent. In all cases, segregation performance remained high despite the presence of a concurrent, objectively demanding decoy task. The results suggest that repetition-based segregation is robust to inattention.

Auditory perceptual objects as generative models: Setting the stage for communication by sound

Communication by sounds requires that the communication channels (i.e. speech/speakers and other sound sources) had been established. This allows to separate concurrently active sound sources, to track their identity, to assess the type of message arriving from them, and to decide whether and when to react (e.g., reply to the message). We propose that these functions rely on a common generative model of the auditory environment. This model predicts upcoming sounds on the basis of representations describing temporal/sequential regularities. Predictions help to identify the continuation of the previously discovered sound sources to detect the emergence of new sources as well as changes in the behavior of the known ones. It produces auditory event representations which provide a full sensory description of the sounds, including their relation to the auditory context and the current goals of the organism. Event representations can be consciously perceived and serve as objects in various cognitive operations.

Attentional Capacity Limits Gap Detection during Concurrent Sound Segregation

Detecting a brief silent interval (i.e., a gap) is more difficult when listeners perceive two concurrent sounds rather than one in a sound containing a mistuned harmonic in otherwise in-tune harmonics. This impairment in gap detection may reflect the interaction of low-level encoding or the division of attention between two sound objects, both of which could interfere with signal detection. To distinguish between these two alternatives, we compared ERPs during active and passive listening with complex harmonic tones that could include a gap, a mistuned harmonic, both features, or neither. During active listening, participants indicated whether they heard a gap irrespective of mistuning. During passive listening, participants watched a subtitled muted movie of their choice while the same sounds were presented. Gap detection was impaired when the complex sounds included a mistuned harmonic that popped out as a separate object. The ERP analysis revealed an early gap-related activity that was little affected by mistuning during the active or passive listening condition. However, during active listening, there was a marked decrease in the late positive wave that was thought to index attention and response-related processes. These results suggest that the limitation in detecting the gap is related to attentional processing, possibly divided attention induced by the concurrent sound objects, rather than deficits in preattentional sensory encoding.

Paying attention to MMN in schizophrenia

The aim of this review is to explore the phenomenon of reduced mismatch negativity (MMN) in persons with schizophrenia and the possible relationship it has with attention impairments. In doing so we discuss (i) the prediction error account of MMN, (ii) reduced MMN as a faulty predictive processing system in persons with schizophrenia, (iii) the role of these systems in relevance filtering and attentional resource protection, (iv) attentional impairments in persons with schizophrenia, and (v) research that has explored MMN and attention in schizophrenia groups. Our review of the literature suggests that no study has appropriately examined the functional impact of smaller MMN in schizophrenia on the performance of a concurrent attention task. We conclude that future research should explore this notion further in the hope that it might embed MMN findings within outcomes of functional significance to individuals with the illness and those providing treatment. This article is part of a Special Issue entitled SI: Prediction and Attention.

Attention is critical for spatial auditory object formation

The precedence effect provides a novel way to examine the role of attention in auditory object formation. When presented with two identical sounds from different locations separated by a short stimulus onset asynchrony (SOA), listeners report a single auditory object at the location of the lead sound. When the SOA is above the echo threshold, listeners report hearing two auditory objects with different locations. Event-related potential (ERP) studies have shown that the number of perceived auditory objects is reflected in an object-related negativity (ORN) 100-250 ms after onset and in a posterior late positivity (LP) 300-500 ms after onset. In the present study, we tested whether these ERP effects are modulated by attention by presenting lead/lag click pairs at and around listeners' echo thresholds, while in separate blocks the listeners (1) attended to the sounds and reported whether the lag sound was a separate source, and (2) performed a two-back visual task. When attention was directed away from the sounds, neither the ORN nor the LP observed in the attend condition was evident. Instead, unattended click pairs above the echo threshold elicited an anterior positivity 250-450 ms after onset. However, an effect resembling an ORN was found in comparing the ERPs elicited by unattended click pairs with SOAs below the attended echo threshold, indicating that the echo threshold may have been lowered when attention was directed away from the sounds. These results suggest that attention modulates early perceptual processes that are critical for auditory object formation.

Mismatch negativity (MMN) as an index of cognitive dysfunction

Cognition is often affected in a variety of neuropsychiatric, neurological, and neurodevelopmental disorders. The neural discriminative response, reflected in mismatch negativity (MMN) and its magnetoencephalographic equivalent (MMNm), has been used as a tool to study a variety of disorders involving auditory cognition. MMN/MMNm is an involuntary brain response to auditory change or, more generally, to pattern regularity violation. For a number of disorders, MMN/MMNm amplitude to sound deviance has been shown to be attenuated or the peak-latency of the component prolonged compared to controls. This general finding suggests that while not serving as a specific marker to any particular disorder, MMN may be useful for understanding factors of cognition in various disorders, and has potential to serve as an indicator of risk. This review presents a brief history of the MMN, followed by a description of how MMN has been used to index auditory processing capability in a range of neuropsychiatric, neurological, and neurodevelopmental disorders. Finally, we suggest future directions for research to further enhance our understanding of the neural substrate of deviance detection that could lead to improvements in the use of MMN as a clinical tool.

Attention to memory: orienting attention to sound object representations

Despite a growing acceptance that attention and memory interact, and that attention can be focused on an active internal mental representation (i.e., reflective attention), there has been a paucity of work focusing on reflective attention to 'sound objects' (i.e., mental representations of actual sound sources in the environment). Further research on the dynamic interactions between auditory attention and memory, as well as its degree of neuroplasticity, is important for understanding how sound objects are represented, maintained, and accessed in the brain. This knowledge can then guide the development of training programs to help individuals with attention and memory problems. This review article focuses on attention to memory with an emphasis on behavioral and neuroimaging studies that have begun to explore the mechanisms that mediate reflective attentional orienting in vision and more recently, in audition. Reflective attention refers to situations in which attention is oriented toward internal representations rather than focused on external stimuli. We propose four general principles underlying attention to short-term memory. Furthermore, we suggest that mechanisms involved in orienting attention to visual object representations may also apply for orienting attention to sound object representations.

The five myths of MMN: redefining how to use MMN in basic and clinical research

The goal of this review article is to redefine what the mismatch negativity (MMN) component of event-related potentials reflects in auditory scene analysis, and to provide an overview of how the MMN serves as a valuable tool in Cognitive Neuroscience research. In doing so, some of the old beliefs (five common 'myths') about MMN will be dispelled, such as the notion that MMN is a simple feature discriminator and that attention itself modulates MMN elicitation. A revised description of what MMN truly reflects will be provided, which includes a principal focus onto the highly context-dependent nature of MMN elicitation and new terminology to discuss MMN and attention. This revised framework will help clarify what has been a long line of seemingly contradictory results from studies in which behavioral ability to hear differences between sounds and passive elicitation of MMN have been inconsistent. Understanding what MMN is will also benefit clinical research efforts by providing a new picture of how to design appropriate paradigms suited to various clinical populations.

Turning down the noise: the benefit of musical training on the aging auditory brain

Age-related decline in hearing abilities is a ubiquitous part of aging, and commonly impacts speech understanding, especially when there are competing sound sources. While such age effects are partially due to changes within the cochlea, difficulties typically exist beyond measurable hearing loss, suggesting that central brain processes, as opposed to simple peripheral mechanisms (e.g., hearing sensitivity), play a critical role in governing hearing abilities late into life. Current training regimens aimed to improve central auditory processing abilities have experienced limited success in promoting listening benefits. Interestingly, recent studies suggest that in young adults, musical training positively modifies neural mechanisms, providing robust, long-lasting improvements to hearing abilities as well as to non-auditory tasks that engage cognitive control. These results offer the encouraging possibility that musical training might be used to counteract age-related changes in auditory cognition commonly observed in older adults. Here, we reviewed studies that have examined the effects of age and musical experience on auditory cognition with an emphasis on auditory scene analysis. We infer that musical training may offer potential benefits to complex listening and might be utilized as a means to delay or even attenuate declines in auditory perception and cognition that often emerge later in life.

Neural time course of visually enhanced echo suppression

Auditory spatial perception plays a critical role in day-to-day communication. For instance, listeners utilize acoustic spatial information to segregate individual talkers into distinct auditory "streams" to improve speech intelligibility. However, spatial localization is an exceedingly difficult task in everyday listening environments with numerous distracting echoes from nearby surfaces, such as walls. Listeners' brains overcome this unique challenge by relying on acoustic timing and, quite surprisingly, visual spatial information to suppress short-latency (1-10 ms) echoes through a process known as "the precedence effect" or "echo suppression." In the present study, we employed electroencephalography (EEG) to investigate the neural time course of echo suppression both with and without the aid of coincident visual stimulation in human listeners. We find that echo suppression is a multistage process initialized during the auditory N1 (70-100 ms) and followed by space-specific suppression mechanisms from 150 to 250 ms. Additionally, we find a robust correlate of listeners' spatial perception (i.e., suppressing or not suppressing the echo) over central electrode sites from 300 to 500 ms. Contrary to our hypothesis, vision's powerful contribution to echo suppression occurs late in processing (250-400 ms), suggesting that vision contributes primarily during late sensory or decision making processes. Together, our findings support growing evidence that echo suppression is a slow, progressive mechanism modifiable by visual influences during late sensory and decision making stages. Furthermore, our findings suggest that audiovisual interactions are not limited to early, sensory-level modulations but extend well into late stages of cortical processing.

Working memory capacity and visual-verbal cognitive load modulate auditory-sensory gating in the brainstem: toward a unified view of attention

Two fundamental research questions have driven attention research in the past: One concerns whether selection of relevant information among competing, irrelevant, information takes place at an early or at a late processing stage; the other concerns whether the capacity of attention is limited by a central, domain-general pool of resources or by independent, modality-specific pools. In this article, we contribute to these debates by showing that the auditory-evoked brainstem response (an early stage of auditory processing) to task-irrelevant sound decreases as a function of central working memory load (manipulated with a visual-verbal version of the n-back task). Furthermore, individual differences in central/domain-general working memory capacity modulated the magnitude of the auditory-evoked brainstem response, but only in the high working memory load condition. The results support a unified view of attention whereby the capacity of a late/central mechanism (working memory) modulates early precortical sensory processing.

Attention, awareness, and the perception of auditory scenes

Auditory perception and cognition entails both low-level and high-level processes, which are likely to interact with each other to create our rich conscious experience of soundscapes. Recent research that we review has revealed numerous influences of high-level factors, such as attention, intention, and prior experience, on conscious auditory perception. And recently, studies have shown that auditory scene analysis tasks can exhibit multistability in a manner very similar to ambiguous visual stimuli, presenting a unique opportunity to study neural correlates of auditory awareness and the extent to which mechanisms of perception are shared across sensory modalities. Research has also led to a growing number of techniques through which auditory perception can be manipulated and even completely suppressed. Such findings have important consequences for our understanding of the mechanisms of perception and also should allow scientists to precisely distinguish the influences of different higher-level influences.

Cortical responses to changes in acoustic regularity are differentially modulated by attentional load

This study investigates how acoustic change-events are represented in a listener's brain when attention is strongly focused elsewhere. Using magneto-encephalography (MEG) we examine whether cortical responses to different kinds of changes in stimulus statistics are similarly influenced by attentional load, and whether the processing of such acoustic changes in auditory cortex depends on modality-specific or general processing resources. We investigated these issues by examining cortical responses to two basic forms of acoustic transitions: (1) Violations of a simple acoustic pattern and (2) the emergence of a regular pattern from a random one. To simulate a complex sensory environment, these patterns were presented concurrently with streams of auditory and visual decoys. Listeners were required to perform tasks of high- and low-attentional-load in these domains. Results demonstrate that while auditory attentional-load does not influence the cortical representation of simple violations of regularity, it significantly reduces the magnitude of responses to the emergence of a regular acoustic pattern, suggesting a fundamentally skewed representation of the unattended auditory scene. In contrast, visual attentional-load had no effect on either transition response, consistent with the hypothesis that processing resources necessary for change detection are modality-specific.

The advantage of ambiguity? Enhanced neural responses to multi-stable percepts correlate with the degree of perceived instability

Artwork can often pique the interest of the viewer or listener as a result of the ambiguity or instability contained within it. Our engagement with uncertain sensory experiences might have its origins in early cortical responses, in that perceptually unstable stimuli might preclude neural habituation and maintain activity in early sensory areas. To assess this idea, participants engaged with an ambiguous visual stimulus wherein two squares alternated with one another, in terms of simultaneously opposing vertical and horizontal locations relative to fixation (i.e., stroboscopic alternating motion; von Schiller, 1933). At each trial, participants were invited to interpret the movement of the squares in one of five ways: traditional vertical or horizontal motion, novel clockwise or counter-clockwise motion, and, a free-view condition in which participants were encouraged to switch the direction of motion as often as possible. Behavioral reports of perceptual stability showed clockwise and counter-clockwise motion to possess an intermediate level of stability compared to relatively stable vertical and horizontal motion, and, relatively unstable motion perceived during free-view conditions. Early visual evoked components recorded at parietal–occipital sites such as C1, P1, and N1 modulated as a function of visual intention. Both at a group and individual level, increased perceptual instability was related to increased negativity in all three of these early visual neural responses. Engagement with increasingly ambiguous input may partly result from the underlying exaggerated neural response to it. The study underscores the utility of combining neuroelectric recording with the presentation of perceptually multi-stable yet physically identical stimuli, in revealing brain activity associated with the purely internal process of interpreting and appreciating the sensory world that surrounds us.

Neural generators underlying concurrent sound segregation

Although an object-based account of auditory attention has become an increasingly popular model for understanding how temporally overlapping sounds are segregated, relatively little is known about the cortical circuit that supports such ability. In the present study, we applied a beamformer spatial filter to magnetoencephalography (MEG) data recorded during an auditory paradigm that used inharmonicity to promote the formation of multiple auditory objects. Using this unconstrained, data-driven approach, the evoked field component linked with the perception of multiple auditory objects (i.e., the object-related negativity; ORNm), was found to be associated with bilateral auditory cortex sources that were distinct from those coinciding with the P1m, N1m, and P2m responses elicited by sound onset. The right hemispheric ORNm source in particular was consistently positioned anterior to the other sources across two experiments. These findings are consistent with earlier proposals of multiple auditory object detection being associated with generators in the auditory cortex and further suggest that these neural populations are distinct from the long latency evoked responses reflecting the detection of sound onset.

N2ac: an ERP component associated with the focusing of attention within an auditory scene

Humans must often focus attention onto relevant sensory signals in the presence of simultaneous irrelevant signals. This type of attention has been explored in vision with the N2pc component, and the present study sought to find an analogous auditory effect. In Experiment 1, two 750-ms sounds were presented simultaneously, one from each of two lateral speakers. On each trial, participants indicated whether one of the two sounds was a pre-defined target. We found that targets elicited an N2ac component: a negativity in the N2 latency range at anterior contralateral electrodes. We also observed a later and more posterior contralateral positivity. Experiment 2 replicated these effects and demonstrated that they arose from competition between attended and unattended tones rather than reflecting lateralized effects of attention for individual tones. The N2ac component may provide a useful tool for studying selective attention within auditory scenes.

Manipulations of listeners' echo perception are reflected in event-related potentials

To gain information from complex auditory scenes, it is necessary to determine which of the many loudness, pitch, and timbre changes originate from a single source. Grouping sound into sources based on spatial information is complicated by reverberant energy bouncing off multiple surfaces and reaching the ears from directions other than the source's location. The ability to localize sounds despite these echoes has been explored with the precedence effect: Identical sounds presented from two locations with a short stimulus onset asynchrony (e.g., 1-5 ms) are perceived as a single source with a location dominated by the lead sound. Importantly, echo thresholds, the shortest onset asynchrony at which a listener reports hearing the lag sound as a separate source about half of the time, can be manipulated by presenting sound pairs in contexts. Event-related brain potentials elicited by physically identical sounds in contexts that resulted in listeners reporting either one or two sources were compared. Sound pairs perceived as two sources elicited a larger anterior negativity 100-250 ms after onset, previously termed the object-related negativity, and a larger posterior positivity 250-500 ms. These results indicate that the models of room acoustics listeners form based on recent experience with the spatiotemporal properties of sound modulate perceptual as well as later higher-level processing.

Activations of human auditory cortex during visual and auditory selective attention tasks with varying difficulty

The present study was designed to directly test the hypothesis that suppression of activations to task-irrelevant sounds contributes to the attention-related modulations of auditory cortex (AC) activations observed in previous fMRI studies. Subjects selectively attended to auditory (broadband noise bursts with pitch) or visual (Gabor gratings) asynchronous fast-rate stimulus streams concurrently presented to left-ear, right-ear, above-fixation, or below-fixation. Auditory and visual task difficulty was parametrically manipulated in three levels. Behavioral data obtained during fMRI indicated that subjects achieved acceptable performance levels in all tasks and that the task-difficulty manipulation was effective. Consistent with previous studies, AC activations strongly depended on the direction of attention. AC activations to sounds were higher during auditory than during visual tasks and AC activations were higher in the hemisphere contralateral to the attended ear. However, the effects of task difficulty on AC activations were weak or non-existent. In particular, increasing task difficulty was not associated with a systematic decrease of AC activations in areas that were modulated by attention. These results suggest that suppression of AC activations to task-irrelevant sounds is likely to be small or negligible as compared with the strong activation enhancements observed in fMRI during active auditory tasks.

Momentary Fluctuations in Allocation of Attention: Cross-modal Effects of Visual Task Load on Auditory Discrimination

Even when our attention is dedicated to an important task, background processes monitor the environment for significant events. The mismatch negativity (MMN) event-related potential is thought to reflect such a monitoring process. Nevertheless, there is continuing debate concerning the susceptibility of the MMN to attentional manipulation. We investigated the trial-by-trial relationship between brain activity related to change detection, reflected in the MMN, and visual psychophysical performance—while varying task difficulty. We find that auditory change detection is indeed “automatic” in that MMN remains robust despite increasing (visual) task load. However, the MMN amplitude and latency are susceptible to both visual load and to momentary attentional fluctuations as reflected in success or failure to identify a following visual target. We conclude that background central auditory processing is sensitive to the demands of a visual task, and fluctuates based on moment-to-moment allocation of attentional resources to the visual task.

One sound or two? Object-related negativity indexes echo perception

The ability to isolate a single sound source among concurrent sources and reverberant energy is necessary for understanding the auditory world. The precedence effect describes a related experimental finding, that when presented with identical sounds from two locations with a short onset asynchrony (on the order of milliseconds), listeners report a single source with a location dominated by the lead sound. Single-cell recordings in multiple animal models have indicated that there are low-level mechanisms that may contribute to the precedence effect, yet psychophysical studies in humans have provided evidence that top-down cognitive processes have a great deal of influence on the perception of simulated echoes. In the present study, event-related potentials evoked by click pairs at and around listeners' echo thresholds indicate that perception of the lead and lag sound as individual sources elicits a negativity between 100 and 250 msec, previously termed the object-related negativity (ORN). Even for physically identical stimuli, the ORN is evident when listeners report hearing, as compared with not hearing, a second sound source. These results define a neural mechanism related to the conscious perception of multiple auditory objects.

It all sounds the same to me: sequential ERP and behavioral effects during pitch and harmonicity judgments

The representation of complex sounds was examined by comparing both behavioral and event-related brain potentials (ERPs) to the change or repetition of fundamental frequency (f0) and harmonicity. In the pitch task, participants were asked to categorize the incoming stimulus as either low or high, regardless of harmonicity, and in the harmonicity task, participants indicated whether the stimulus was tuned or mistuned, regardless of pitch. Over three experiments, participants were faster in responding to pitch than to harmonicity. As a result of this asymmetry, behavioral and ERP data showed that irrelevant changes in harmonicity had little impact on performance during the pitch task, whereas harmonicity judgments were impeded by irrelevant changes in f0. These data are consistent with both general horse-race accounts of processing and specific accounts of mistuning detection that posit prior f0 registration. In addition, ERP components N2 and P3 were modulated by both intertrial contingency and task instructions, revealing the further influence of top-down mechanisms on concurrent sound segregation.

The influence of strongly focused visual attention on the detection of change in an auditory pattern

The mismatch negativity, an ERP that reflects the detection of change in the auditory environment, is considered to be a relatively automatic process. Its automaticity has by in large been studied using the oddball paradigm, in which a physical feature of a frequently presented standard stimulus is changed. In the present study, the automaticity of the MMN is tested using a MMN elicited by a violation of a more abstract auditory pattern. Fourteen subjects were presented with an alternating pattern of two tones (ABABAB) that was occasionally broken by deviant repetitions (e.g., ABABABBBAB). The alternating tones were separated by 1 or 6 semitones in different conditions. The subjects were engaged in a continuous multiple object tracking (MOT) task and thus ignored the auditory stimuli. Difficulty of the MOT task was manipulated by increasing the number of objects to be tracked. Subjects were also asked to read a text and ignore the auditory stimuli in another condition. A much larger MMN was elicited by pattern violations in the 6 than in the 1 semitone condition. The difficult visual task should have presumably required greater attentional focus than the easy task, and performance did deteriorate during the difficult MOT. The MMN, however, was not affected by the demands of the MOT task. This finding suggests that the MMN elicited by the violation of a pattern is not affected by the presumed attentional demands of a difficult continuous task such as multiple object tracking.

Workload assessment of computer gaming using a single-stimulus event-related potential paradigm

Behavioral and event-related potential (ERP) measures were used to assess cognitive workload from expert computer gamers playing a "first-person shooter" video game. Game difficulty level was manipulated in separate conditions by adjusting the number of enemies (view, easy, medium, and hard). Infrequently presented single-stimulus tones were either ignored or counted across difficulty conditions. Game performance and tone-counting accuracy declined as game difficulty increased. ERP component amplitudes diminished for both the tone ignore and counting conditions as game difficulty increased. The findings suggest that cognitive workload induced by video gaming can be reliably assessed through behavioral and neuroelectric means, and that the single-stimulus paradigm can be a useful tool for evaluating workload in an immersive stimulus environment with less distraction than conventional tools.

The mismatch negativity (MMN) in basic research of central auditory processing: a review

In the present article, the basic research using the mismatch negativity (MMN) and analogous results obtained by using the magnetoencephalography (MEG) and other brain-imaging technologies is reviewed. This response is elicited by any discriminable change in auditory stimulation but recent studies extended the notion of the MMN even to higher-order cognitive processes such as those involving grammar and semantic meaning. Moreover, MMN data also show the presence of automatic intelligent processes such as stimulus anticipation at the level of auditory cortex. In addition, the MMN enables one to establish the brain processes underlying the initiation of attention switch to, conscious perception of, sound change in an unattended stimulus stream.

Perceptual load affects exogenous spatial orienting while working memory load does not

We examined whether or not increasing visual perceptual load or visual working memory (WM) load would affect the exogenous orienting of visuo-spatial attention, in order to assess whether or not exogenous orienting is genuinely automatic. In Experiment 1, we manipulated visual perceptual load by means of a central morphing shape that in some trials morphed into a particular target shape (a rectangle) that participants had to detect. In Experiment 2, the possibility that the presentation of any changing stimulus at fixation would eliminate exogenous orienting was ruled out, by presenting two alternating letters at fixation. In Experiment 3, we manipulated visual WM load by means of arrays consisting of three (low-load) or five (high-load) randomly located coloured squares. The participants had to remember these items in order to judge whether a cued square had been presented in the same or different colour at the end of each trial. In all the experiments, exogenous visuo-spatial attentional orienting was measured by means of an orthogonal spatial cuing task, in which the participants had to discriminate the elevation (up vs. down) of a visual target previously cued by a spatially nonpredictive visual cue. The results showed that increasing the perceptual load of the task eliminated the exogenous orienting of visuo-spatial attention. By contrast, increasing the WM load had no effect on spatial orienting. These results are discussed in terms of the light that they shed on claims regarding the automaticity of visuo-spatial exogenous orienting.

Neurophysiological indices of attention to speech in children with specific language impairment

OBJECTIVE

The aim was to determine whether children with specific language impairment (SLI) differed from children with typical language development (TLD) in their allocation of attention to speech sounds.

METHODS

Event-related potentials were recorded to non-target speech sounds in two tasks (passive-watch a video and attend to target tones among speech sounds) in two experiments, one using 50-ms duration vowels and the second using 250-ms vowels. The difference in ERPs across tasks was examined in the latency range of the early negative difference wave (Nd) found in adults. Analyses of the data using selected superior and inferior sites were compared to those using electrical field power (i.e., global field power or GFP). The topography of the ERP at the maximum GFP was also examined.

RESULTS

A negative difference, comparable to the adult Nd, was observed in the attend compared to the passive task for both types of analysis, suggesting allocation of attentional resources to processing the speech stimuli in the attend task. Children with TLD also showed greater negativity than those with SLI in the passive task for the long vowels, suggesting that they allocated more attentional resources to processing the speech in this task than the SLI group. This effect was only significant using the GFP analysis and was seen as smaller GFP for the TLD than SLI group. The SLI group also showed significantly later latency than the TLD group in reaching the maximum GFP. In addition, a significantly greater proportion of children with SLI compared to those with typical language showed left-greater-than-right frontocentral amplitude at the latency determined from each child's maximum GFP peak.

CONCLUSIONS

Children generally showed greater attention to speech sounds when attention is directed to the auditory modality compared to the visual modality. However, children with TLD, unlike SLI, also appear to devote some attentional resources to speech even in a task in which they are instructed to attend to visual information and ignore the speech.

SIGNIFICANCE

These findings suggest that children with SLI have limited attentional resources, that they are poorer at dividing attention, or that they are less automatic in allocating resources to speech compared to children with typically developing language skills.

Age-related differences in neuromagnetic brain activity underlying concurrent sound perception

Deficits in parsing concurrent auditory events are believed to contribute to older adults' difficulties in understanding speech in adverse listening conditions (e.g., cocktail party). To explore the level at which aging impairs sound segregation, we measured auditory evoked fields (AEFs) using magnetoencephalography while young, middle-aged, and older adults were presented with complex sounds that either had all of their harmonics in tune or had the third harmonic mistuned by 4 or 16% of its original value. During the recording, participants were asked to ignore the stimuli and watch a muted subtitled movie of their choice. For each participant, the AEFs were modeled with a pair of dipoles in the superior temporal plane, and the effects of age and mistuning were examined on the amplitude and latency of the resulting source waveforms. Mistuned stimuli generated an early positivity (60-100 ms), an object-related negativity (ORN) (140-180 ms) that overlapped the N1 and P2 waves, and a positive displacement that peaked at approximately 230 ms (P230) after sound onset. The early mistuning-related enhancement was similar in all three age groups, whereas the subsequent modulations (ORN and P230) were reduced in older adults. These age differences in auditory cortical activity were associated with a reduced likelihood of hearing two sounds as a function of mistuning. The results reveal that inharmonicity is rapidly and automatically registered in all three age groups but that the perception of concurrent sounds declines with age.

Breaking the wave: effects of attention and learning on concurrent sound perception

The auditory surrounding is often complex with many sound sources active simultaneously. Yet listeners are proficient in breaking apart the composite acoustic wave reaching the ears. This achievement is thought to be the result of bottom-up as well as top-down processes that reflect listeners' experience and knowledge of the auditory environment. Here, specific findings concerning the role of bottom-up and top-down (schema-driven) processes on concurrent sound perception are reviewed, with particular emphasis on studies that have used scalp recording of event-related brain potentials. Findings from several studies indicate that frequency periodicity, upon which concurrent sound perception partly depends, is quickly and automatically registered in primary auditory cortex. Moreover, success in identifying concurrent vowels is accompanied by enhanced neural activity, as revealed by functional magnetic resonance imaging, in thalamus, primary auditory cortex and planum temporale. Lastly, listeners' ability to segregate concurrent vowels improves with training and these neuroplastic changes occur rapidly, demonstrating the flexibility of human speech segregation mechanisms. Together, these studies suggest that the primary auditory cortex and the planum temporale play an important role in concurrent sound perception, and reveal a link between thalamo-cortical activation and the successful separation and identification of speech sounds presented simultaneously.