Manipulations of listeners' echo perception are reflected in event-related potentials

Original speech and its echo are segregated and separately processed in the human brain

Speech recognition crucially relies on slow temporal modulations (<16 Hz) in speech. Recent studies, however, have demonstrated that the long-delay echoes, which are common during online conferencing, can eliminate crucial temporal modulations in speech but do not affect speech intelligibility. Here, we investigated the underlying neural mechanisms. MEG experiments demonstrated that cortical activity can effectively track the temporal modulations eliminated by an echo, which cannot be fully explained by basic neural adaptation mechanisms. Furthermore, cortical responses to echoic speech can be better explained by a model that segregates speech from its echo than by a model that encodes echoic speech as a whole. The speech segregation effect was observed even when attention was diverted but would disappear when segregation cues, i.e., speech fine structure, were removed. These results strongly suggested that, through mechanisms such as stream segregation, the auditory system can build an echo-insensitive representation of speech envelope, which can support reliable speech recognition.

Temporal characteristics of contextual effects in sound localization

Two experiments examined plasticity induced by context in a simple target localization task. The context was represented by interleaved localization trials with the target preceded by a distractor. In a previous study, the context induced large response shifts when the target and distractor stimuli were identical 2-ms-noise clicks [Kopčo, Best, and Shinn-Cunningham (2007). J. Acoust. Soc. Am. 121, 420-432]. Here, the temporal characteristics of the contextual effect were examined for the same stimuli. Experiment 1 manipulated the context presentation rate and the distractor-target inter-stimulus interval (ISI). Experiment 2 manipulated the temporal structure of the context stimulus, replacing the one-click distractor either by a distractor consisting of eight sequentially presented clicks or by a noise burst with total energy and duration identical to the eight-click distractor. In experiment 1, the contextual shift size increased with increasing context rate while being largely independent of ISI. In experiment 2, the eight-click-distractor induced a stronger shift than the one-click-distractor context, while the noise-distractor context induced a very small shift. These results suggest that contextual plasticity is an adaptation driven both by low-level factors like spatiotemporal context distribution and higher-level factors like perceptual similarity between the stimuli, possibly related to precedence buildup.

Promoting the perception of two and three concurrent sound objects: An event-related potential study

The auditory environment typically comprises several simultaneously active sound sources. In contrast to the perceptual segregation of two concurrent sounds, the perception of three simultaneous sound objects has not yet been studied systematically. We conducted two experiments in which participants were presented with complex sounds containing sound segregation cues (mistuning, onset asynchrony, differences in frequency or amplitude modulation or in sound location), which were set up to promote the perceptual organization of the tonal elements into one, two, or three concurrent sounds. In Experiment 1, listeners indicated whether they heard one, two, or three concurrent sounds. In Experiment 2, participants watched a silent subtitled movie while EEG was recorded to extract the object-related negativity (ORN) component of the event-related potential. Listeners predominantly reported hearing two sounds when the segregation promoting manipulations were applied to the same tonal element. When two different tonal elements received manipulations promoting them to be heard as separate auditory objects, participants reported hearing two and three concurrent sounds objects with equal probability. The ORN was elicited in most conditions; sounds that included the amplitude- or the frequency-modulation cue generated the smallest ORN amplitudes. Manipulating two different tonal elements yielded numerically and often significantly smaller ORNs than the sum of the ORNs elicited when the same cues were applied on a single tonal element. These results suggest that ORN reflects the presence of multiple concurrent sounds, but not their number. The ORN results are compatible with the horse-race principle of combining different cues of concurrent sound segregation.

Neural realignment of spatially separated sound components

Natural auditory scenes often consist of several sound sources overlapping in time, but separated in space. Yet, location is not fully exploited in auditory grouping: spatially separated sounds can get perceptually fused into a single auditory object and this leads to difficulties in the identification and localization of concurrent sounds. Here, the brain mechanisms responsible for grouping across spatial locations were explored in magnetoencephalography (MEG) recordings. The results show that the cortical representation of a vowel spatially separated into two locations reflects the perceived location of the speech sound rather than the physical locations of the individual components. In other words, the auditory scene is neurally rearranged to bring components into spatial alignment when they were deemed to belong to the same object. This renders the original spatial information unavailable at the level of the auditory cortex and may contribute to difficulties in concurrent sound segregation.

Attention is critical for spatial auditory object formation

The precedence effect provides a novel way to examine the role of attention in auditory object formation. When presented with two identical sounds from different locations separated by a short stimulus onset asynchrony (SOA), listeners report a single auditory object at the location of the lead sound. When the SOA is above the echo threshold, listeners report hearing two auditory objects with different locations. Event-related potential (ERP) studies have shown that the number of perceived auditory objects is reflected in an object-related negativity (ORN) 100-250 ms after onset and in a posterior late positivity (LP) 300-500 ms after onset. In the present study, we tested whether these ERP effects are modulated by attention by presenting lead/lag click pairs at and around listeners' echo thresholds, while in separate blocks the listeners (1) attended to the sounds and reported whether the lag sound was a separate source, and (2) performed a two-back visual task. When attention was directed away from the sounds, neither the ORN nor the LP observed in the attend condition was evident. Instead, unattended click pairs above the echo threshold elicited an anterior positivity 250-450 ms after onset. However, an effect resembling an ORN was found in comparing the ERPs elicited by unattended click pairs with SOAs below the attended echo threshold, indicating that the echo threshold may have been lowered when attention was directed away from the sounds. These results suggest that attention modulates early perceptual processes that are critical for auditory object formation.

Newborn Infants Detect Cues of Concurrent Sound Segregation

Separating concurrent sounds is fundamental for a veridical perception of one's auditory surroundings. Sound components that are harmonically related and start at the same time are usually grouped into a common perceptual object, whereas components that are not in harmonic relation or have different onset times are more likely to be perceived in terms of separate objects. Here we tested whether neonates are able to pick up the cues supporting this sound organization principle. We presented newborn infants with a series of complex tones with their harmonics in tune (creating the percept of a unitary sound object) and with manipulated variants, which gave the impression of two concurrently active sound sources. The manipulated variant had either one mistuned partial (single-cue condition) or the onset of this mistuned partial was also delayed (double-cue condition). Tuned and manipulated sounds were presented in random order with equal probabilities. Recording the neonates' electroencephalographic responses allowed us to evaluate their processing of the sounds. Results show that, in both conditions, mistuned sounds elicited a negative displacement of the event-related potential (ERP) relative to tuned sounds from 360 to 400 ms after sound onset. The mistuning-related ERP component resembles the object-related negativity (ORN) component in adults, which is associated with concurrent sound segregation. Delayed onset additionally led to a negative displacement from 160 to 200 ms, which was probably more related to the physical parameters of the sounds than to their perceptual segregation. The elicitation of an ORN-like response in newborn infants suggests that neonates possess the basic capabilities of segregating concurrent sounds by detecting inharmonic relations between the co-occurring sounds.

The precedence effect in sound localization

In ordinary listening environments, acoustic signals reaching the ears directly from real sound sources are followed after a few milliseconds by early reflections arriving from nearby surfaces. Early reflections are spectrotemporally similar to their source signals but commonly carry spatial acoustic cues unrelated to the source location. Humans and many other animals, including nonmammalian and even invertebrate animals, are nonetheless able to effectively localize sound sources in such environments, even in the absence of disambiguating visual cues. Robust source localization despite concurrent or nearly concurrent spurious spatial acoustic information is commonly attributed to an assortment of perceptual phenomena collectively termed "the precedence effect," characterizing the perceptual dominance of spatial information carried by the first-arriving signal. Here, we highlight recent progress and changes in the understanding of the precedence effect and related phenomena.

The effects of preceding lead-alone and lag-alone click trains on the buildup of echo suppression

Spatial perception in echoic environments is influenced by recent acoustic history. For instance, echo suppression becomes more effective or "builds up" with repeated exposure to echoes having a consistent acoustic relationship to a temporally leading sound. Four experiments were conducted to investigate how buildup is affected by prior exposure to unpaired lead-alone or lag-alone click trains. Unpaired trains preceded lead-lag click trains designed to evoke and assay buildup. Listeners reported how many sounds they heard from the echo hemifield during the lead-lag trains. Stimuli were presented in free field (experiments 1 and 4) or dichotically through earphones (experiments 2 and 3). In experiment 1, listeners reported more echoes following a lead-alone train compared to a period of silence. In contrast, listeners reported fewer echoes following a lag-alone train; similar results were observed with earphones. Interestingly, the effects of lag-alone click trains on buildup were qualitatively different when compared to a no-conditioner trial type in experiment 4. Finally, experiment 3 demonstrated that the effects of preceding click trains on buildup cannot be explained by a change in counting strategy or perceived click salience. Together, these findings demonstrate that echo suppression is affected by prior exposure to unpaired stimuli.

Effects of multiple congruent cues on concurrent sound segregation during passive and active listening: an event-related potential (ERP) study

In two experiments, we assessed the effects of combining different cues of concurrent sound segregation on the object-related negativity (ORN) and the P400 event-related potential components. Participants were presented with sequences of complex tones, half of which contained some manipulation: one or two harmonic partials were mistuned, delayed, or presented from a different location than the rest. In separate conditions, one, two, or three of these manipulations were combined. Participants watched a silent movie (passive listening) or reported after each tone whether they perceived one or two concurrent sounds (active listening). ORN was found in almost all conditions except for location difference alone during passive listening. Combining several cues or manipulating more than one partial consistently led to sub-additive effects on the ORN amplitude. These results support the view that ORN reflects a combined, feature-unspecific assessment of the auditory system regarding the contribution of two sources to the incoming sound.

The precedence effect and its buildup and breakdown in ferrets and humans

Although many studies have examined the precedence effect (PE), few have tested whether it shows a buildup and breakdown in nonhuman animals comparable to that seen in humans. These processes are thought to reflect the ability of the auditory system to adjust to a listener's acoustic environment, and their mechanisms are still poorly understood. In this study, ferrets were trained on a two-alternative forced-choice task to discriminate the azimuthal direction of brief sounds. In one experiment, pairs of noise bursts were presented from two loudspeakers at different interstimulus delays (ISDs). Results showed that localization performance changed as a function of ISD in a manner consistent with the PE being operative. A second experiment investigated buildup and breakdown of the PE by measuring the ability of ferrets to discriminate the direction of a click pair following presentation of a conditioning train. Human listeners were also tested using this paradigm. In both species, performance was better when the test clicks and conditioning train had the same ISD but deteriorated following a switch in the direction of the leading and lagging sounds between the conditioning train and test clicks. These results suggest that ferrets, like humans, experience a buildup and breakdown of the PE.

Local inhibition of GABA affects precedence effect in the inferior colliculus

The precedence effect is a prerequisite for faithful sound localization in a complex auditory environment, and is a physiological phenomenon in which the auditory system selectively suppresses the directional information from echoes. Here we investigated how neurons in the inferior colliculus respond to the paired sounds that produce precedence-effect illusions, and whether their firing behavior can be modulated through inhibition with gamma-aminobutyric acid (GABA). We recorded extracellularly from 36 neurons in rat inferior colliculus under three conditions: no injection, injection with saline, and injection with gamma-aminobutyric acid. The paired sounds that produced precedence effects were two identical 4-ms noise bursts, which were delivered contralaterally or ipsilaterally to the recording site. The normalized neural responses were measured as a function of different inter-stimulus delays and half-maximal interstimulus delays were acquired. Neuronal responses to the lagging sounds were weak when the inter-stimulus delay was short, but increased gradually as the delay was lengthened. Saline injection produced no changes in neural responses, but after local gamma-aminobutyric acid application, responses to the lagging stimulus were suppressed. Application of gamma-aminobutyric acid affected the normalized response to lagging sounds, independently of whether they or the paired sounds were contralateral or ipsilateral to the recording site. These observations suggest that local inhibition by gamma-aminobutyric acid in the rat inferior colliculus shapes the neural responses to lagging sounds, and modulates the precedence effect.

The precedence effect: fusion and lateralization measures for headphone stimuli lateralized by interaural time and level differences

The present investigation assessed fusion and localization dominance aspects of the precedence effect under headphones across a variety of stimulus conditions in 10 normal-hearing listeners. Listeners were presented with "lead-lag" pairs of brief (123 μs) impulses or trains of such pairs lateralized by interaural time or level differences (ITD or ILD). Listeners used a touch-sensitive display to indicate for the final lead-lag pair presented on each trial (1) whether one or two locations were perceived and (2) the location perceived. In the event two locations were perceived, subjects were further instructed to indicate the left-most location perceived. Results demonstrated that lead-lag fusion was more robust for stimuli lateralized by ITD than ILD, particularly when cues of the test stimulus differed from cues of the preceding "buildup" stimulus, consistent with Krumbholz and Nobbe [(2002). J. Acoust. Soc. Am. 112, 654-663]. Unexpectedly, results also demonstrated reduced localization dominance with increasing lead-lag delay, suggesting that the fusion aspect of the precedence effect may be dissociated from the localization dominance aspect under buildup. It is thus argued that buildup of fusion might be understood more generally as an example of auditory object formation rather than a special facility for enhanced sound localization.

Neural time course of visually enhanced echo suppression

Auditory spatial perception plays a critical role in day-to-day communication. For instance, listeners utilize acoustic spatial information to segregate individual talkers into distinct auditory "streams" to improve speech intelligibility. However, spatial localization is an exceedingly difficult task in everyday listening environments with numerous distracting echoes from nearby surfaces, such as walls. Listeners' brains overcome this unique challenge by relying on acoustic timing and, quite surprisingly, visual spatial information to suppress short-latency (1-10 ms) echoes through a process known as "the precedence effect" or "echo suppression." In the present study, we employed electroencephalography (EEG) to investigate the neural time course of echo suppression both with and without the aid of coincident visual stimulation in human listeners. We find that echo suppression is a multistage process initialized during the auditory N1 (70-100 ms) and followed by space-specific suppression mechanisms from 150 to 250 ms. Additionally, we find a robust correlate of listeners' spatial perception (i.e., suppressing or not suppressing the echo) over central electrode sites from 300 to 500 ms. Contrary to our hypothesis, vision's powerful contribution to echo suppression occurs late in processing (250-400 ms), suggesting that vision contributes primarily during late sensory or decision making processes. Together, our findings support growing evidence that echo suppression is a slow, progressive mechanism modifiable by visual influences during late sensory and decision making stages. Furthermore, our findings suggest that audiovisual interactions are not limited to early, sensory-level modulations but extend well into late stages of cortical processing.

Spatial attention modulates the precedence effect

Communication and navigation in real environments rely heavily on the ability to distinguish objects in acoustic space. However, auditory spatial information is often corrupted by conflicting cues and noise such as acoustic reflections. Fortunately the brain can apply mechanisms at multiple levels to emphasize target information and mitigate such interference. In a rapid phenomenon known as the precedence effect, reflections are perceptually fused with the veridical primary sound. The brain can also use spatial attention to highlight a target sound at the expense of distracters. Although attention has been shown to modulate many auditory perceptual phenomena, rarely does it alter how acoustic energy is first parsed into objects, as with the precedence effect. This brief report suggests that both endogenous (voluntary) and exogenous (stimulus-driven) spatial attention have a profound influence on the precedence effect depending on where they are oriented. Moreover, we observed that both types of attention could enhance perceptual fusion while only exogenous attention could hinder it. These results demonstrate that attention, by altering how auditory objects are formed, guides the basic perceptual organization of our acoustic environment.