Hearing Two Things at Once: Neurophysiological Indices of Speech Segregation and Identification

The Effect of Combined Auditory Training on Concurrent Sound Segregation in Young old: A Single-Blinded Randomized Clinical Trial

This study aimed to investigate the behavioral results of perceptual learning in young old using double-vowel discrimination tasks in combined auditory training programs. In a single-blind randomized clinical trial, 35 participants were randomly divided into three groups and received different auditory training programs for six sessions using the software. To compare the double-vowel discrimination score, CV in noise test, and reaction time to the first and second vowels pre- and post-intervention, an analysis of variance was conducted. The discrimination score in the double vowel task and CV in noise test improved after training with no significant difference between the groups. After auditory training, the lowest RT1 was observed in the first intervention group, whereas RT2 decreased only in the second intervention. The present study showed that combined auditory training programs are as effective as conventional auditory training programs in improving speech perception in the elderly. Modifications in the sensory cortex could be investigated using electrophysiological recordings, but this was not conducted because of the pandemic.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12070-023-03923-x.

Neural correlates of concurrent sound perception: A review and guidelines for future research

The perception of concurrent sound sources depends on processes (i.e., auditory scene analysis) that fuse and segregate acoustic features according to harmonic relations, temporal coherence, and binaural cues (encompass dichotic pitch, location difference, simulated echo). The object-related negativity (ORN) and P400 are electrophysiological indices of concurrent sound perception. Here, we review the different paradigms used to study concurrent sound perception and the brain responses obtained from these paradigms. Recommendations regarding the design and recording parameters of the ORN and P400 are made, and their clinical applications in assessing central auditory processing ability in different populations are discussed.

Children with developmental language disorder: a frequency following response in the noise study

Introduction

Children with developmental language disorder have been reported to have poor temporal auditory processing. This study aimed to examine the frequency following response.

Objective

This work aimed to investigate speech processing in quiet and in noise.

Methods

Two groups of children were included in this work: the control group (15 children with normal language development) and the study group (25 children diagnosed with developmental language disorder). All children were submitted to intelligence scale, language assessment, full audiological evaluation, and frequency following response in quiet and noise (+5QNR and +10QNR).

Results

Results showed no statically significant difference between both groups as regards IQ or PTA. In the study group, the advanced analysis of frequency following response showed reduced F0 and F2 amplitudes. Results also showed that noise has an impact on both the transient and sustained components of the frequency following response in the same group.

Conclusion

Children with developmental language disorder have difficulty in speech processing especially in the presence of background noise. Frequency following response is an efficient procedure that can be used to address speech processing problems in children with developmental language disorder.

Simultaneous EEG and MEG recordings reveal vocal pitch elicited cortical gamma oscillations in young and older adults

The frequency-following response with origin in the auditory brainstem represents the pitch contour of voice and can be recorded with electrodes from the scalp. MEG studies also revealed a cortical contribution to the high gamma oscillations at the fundamental frequency (f₀) of a vowel stimulus. Therefore, studying the cortical component of the frequency-following response could provide insights into how pitch information is encoded at the cortical level. Comparing how aging affects the different responses may help to uncover the neural mechanisms underlying speech understanding deficits in older age. We simultaneously recorded EEG and MEG responses to the syllable /ba/. MEG beamformer analysis localized sources in bilateral auditory cortices and the midbrain. Time-frequency analysis showed a faithful representation of the pitch contour between 106 Hz and 138 Hz in the cortical activity. A cross-correlation revealed a latency of 20 ms. Furthermore, stimulus onsets elicited cortical 40-Hz responses. Both the 40-Hz and the f₀ response amplitudes increased in older age and were larger in the right hemisphere. The effects of aging and laterality of the f₀ response were evident in the MEG only, suggesting that both effects were characteristics of the cortical response. After comparing f₀ and N1 responses in EEG and MEG, we estimated that approximately one-third of the scalp-recorded f₀ response could be cortical in origin. We attributed the significance of the cortical f₀ response to the precise timing of cortical neurons that serve as a time-sensitive code for pitch.

Brainstem correlates of concurrent speech identification in adverse listening conditions

When two voices compete, listeners can segregate and identify concurrent speech sounds using pitch (fundamental frequency, F0) and timbre (harmonic) cues. Speech perception is also hindered by the signal-to-noise ratio (SNR). How clear and degraded concurrent speech sounds are represented at early, pre-attentive stages of the auditory system is not well understood. To this end, we measured scalp-recorded frequency-following responses (FFR) from the EEG while human listeners heard two concurrently presented, steady-state (time-invariant) vowels whose F0 differed by zero or four semitones (ST) presented diotically in either clean (no noise) or noise-degraded (+5dB SNR) conditions. Listeners also performed a speeded double vowel identification task in which they were required to identify both vowels correctly. Behavioral results showed that speech identification accuracy increased with F0 differences between vowels, and this perceptual F0 benefit was larger for clean compared to noise degraded (+5dB SNR) stimuli. Neurophysiological data demonstrated more robust FFR F0 amplitudes for single compared to double vowels and considerably weaker responses in noise. F0 amplitudes showed speech-on-speech masking effects, along with a non-linear constructive interference at 0ST, and suppression effects at 4ST. Correlations showed that FFR F0 amplitudes failed to predict listeners' identification accuracy. In contrast, FFR F1 amplitudes were associated with faster reaction times, although this correlation was limited to noise conditions. The limited number of brain-behavior associations suggests subcortical activity mainly reflects exogenous processing rather than perceptual correlates of concurrent speech perception. Collectively, our results demonstrate that FFRs reflect pre-attentive coding of concurrent auditory stimuli that only weakly predict the success of identifying concurrent speech.

Age Effects on Concurrent Speech Segregation by Onset Asynchrony

Purpose For elderly listeners, it is more challenging to listen to 1 voice surrounded by other voices than for young listeners. This could be caused by a reduced ability to use acoustic cues-such as slight differences in onset time-for the segregation of concurrent speech signals. Here, we study whether the ability to benefit from onset asynchrony differs between young (18-33 years) and elderly (55-74 years) listeners. Method We investigated young (normal hearing, N = 20) and elderly (mildly hearing impaired, N = 26) listeners' ability to segregate 2 vowels with onset asynchronies ranging from 20 to 100 ms. Behavioral measures were complemented by a specific event-related brain potential component, the object-related negativity, indicating the perception of 2 distinct auditory objects. Results Elderly listeners' behavioral performance (identification accuracy of the 2 vowels) was considerably poorer than young listeners'. However, both age groups showed the same amount of improvement with increasing onset asynchrony. Object-related negativity amplitude also increased similarly in both age groups. Conclusion Both age groups benefit to a similar extent from onset asynchrony as a cue for concurrent speech segregation during active (behavioral measurement) and during passive (electroencephalographic measurement) listening.

Auditory spatial attention modulates the unmasking effect of perceptual separation in a "cocktail party" environment

The perceptual separation between a signal speech and a competing speech (masker), induced by the precedence effect, plays an important role in releasing the signal speech from the masker, especially in a reverberant environment. The perceptual-separation-induced unmasking effect has been suggested to involve multiple cognitive processes, such as selective attention. However, whether listeners' spatial attention modulate the perceptual-separation-induced unmasking effect is not clear. The present study investigated how perceptual separation and auditory spatial attention interact with each other to facilitate speech perception under a simulated noisy and reverberant environment by analyzing the cortical auditory evoked potentials to the signal speech. The results showed that the N1 wave was significantly enhanced by perceptual separation between the signal and masker regardless of whether the participants' spatial attention was directed to the signal or not. However, the P2 wave was significantly enhanced by perceptual separation only when the participants attended to the signal speech. The results indicate that the perceptual-separation-induced facilitation of P2 needs more attentional resource than that of N1. The results also showed that the signal speech caused an enhanced N1 in the contralateral hemisphere regardless of whether participants' attention was directed to the signal or not. In contrast, the signal speech caused an enhanced P2 in the contralateral hemisphere only when the participant attended to the signal. The results indicate that the hemispheric distribution of N1 is mainly affected by the perceptual features of the acoustic stimuli, while that of P2 is affected by the listeners' attentional status.

Saliency of Vowel Features in Neural Responses of Cochlear Implant Users

Cochlear implants restore hearing in deaf individuals, but speech perception remains challenging. Poor discrimination of spectral components is thought to account for limitations of speech recognition in cochlear implant users. We investigated how combined variations of spectral components along two orthogonal dimensions can maximize neural discrimination between two vowels, as measured by mismatch negativity. Adult cochlear implant users and matched normal-hearing listeners underwent electroencephalographic event-related potentials recordings in an optimum-1 oddball paradigm. A standard /a/ vowel was delivered in an acoustic free field along with stimuli having a deviant fundamental frequency (+3 and +6 semitones), a deviant first formant making it a /i/ vowel or combined deviant fundamental frequency and first formant (+3 and +6 semitones /i/ vowels). Speech recognition was assessed with a word repetition task. An analysis of variance between both amplitude and latency of mismatch negativity elicited by each deviant vowel was performed. The strength of correlations between these parameters of mismatch negativity and speech recognition as well as participants' age was assessed. Amplitude of mismatch negativity was weaker in cochlear implant users but was maximized by variations of vowels' first formant. Latency of mismatch negativity was later in cochlear implant users and was particularly extended by variations of the fundamental frequency. Speech recognition correlated with parameters of mismatch negativity elicited by the specific variation of the first formant. This nonlinear effect of acoustic parameters on neural discrimination of vowels has implications for implant processor programming and aural rehabilitation.

Electrophysiological Correlates of Speaker Segregation and Foreground-Background Selection in Ambiguous Listening Situations

In everyday listening environments, a main task for our auditory system is to follow one out of multiple speakers talking simultaneously. The present study was designed to find electrophysiological indicators of two central processes involved - segregating the speech mixture into distinct speech sequences corresponding to the two speakers, and then attending to one of the speech sequences. We generated multistable speech stimuli that were set up to create ambiguity as to whether only one or two speakers are talking. Thereby we were able to investigate three perceptual alternatives (no segregation, segregated - speaker A in the foreground, segregated - speaker B in the foreground) without any confounding stimulus changes. Participants listened to a continuously repeating sequence of syllables, which were uttered alternately by two human speakers, and indicated whether they perceived the sequence as an inseparable mixture or as originating from two separate speakers. In the latter case, they distinguished which speaker was in their attentional foreground. Our data show a long-lasting event-related potential (ERP) modulation starting at 130ms after stimulus onset, which can be explained by the perceptual organization of the two speech sequences into attended foreground and ignored background streams. Our paradigm extends previous work with pure-tone sequences toward speech stimuli and adds the possibility to obtain neural correlates of the difficulty to segregate a speech mixture into distinct streams.

Listening under difficult conditions: An activation likelihood estimation meta-analysis

The brain networks supporting speech identification and comprehension under difficult listening conditions are not well specified. The networks hypothesized to underlie effortful listening include regions responsible for executive control. We conducted meta-analyses of auditory neuroimaging studies to determine whether a common activation pattern of the frontal lobe supports effortful listening under different speech manipulations. Fifty-three functional neuroimaging studies investigating speech perception were divided into three independent Activation Likelihood Estimate analyses based on the type of speech manipulation paradigm used: Speech-in-noise (SIN, 16 studies, involving 224 participants); spectrally degraded speech using filtering techniques (15 studies involving 270 participants); and linguistic complexity (i.e., levels of syntactic, lexical and semantic intricacy/density, 22 studies, involving 348 participants). Meta-analysis of the SIN studies revealed higher effort was associated with activation in left inferior frontal gyrus (IFG), left inferior parietal lobule, and right insula. Studies using spectrally degraded speech demonstrated increased activation of the insula bilaterally and the left superior temporal gyrus (STG). Studies manipulating linguistic complexity showed activation in the left IFG, right middle frontal gyrus, left middle temporal gyrus and bilateral STG. Planned contrasts revealed left IFG activation in linguistic complexity studies, which differed from activation patterns observed in SIN or spectral degradation studies. Although there were no significant overlap in prefrontal activation across these three speech manipulation paradigms, SIN and spectral degradation showed overlapping regions in left and right insula. These findings provide evidence that there is regional specialization within the left IFG and differential executive networks underlie effortful listening.

Low- and high-frequency cortical brain oscillations reflect dissociable mechanisms of concurrent speech segregation in noise

Parsing simultaneous speech requires listeners use pitch-guided segregation which can be affected by the signal-to-noise ratio (SNR) in the auditory scene. The interaction of these two cues may occur at multiple levels within the cortex. The aims of the current study were to assess the correspondence between oscillatory brain rhythms and determine how listeners exploit pitch and SNR cues to successfully segregate concurrent speech. We recorded electrical brain activity while participants heard double-vowel stimuli whose fundamental frequencies (F0s) differed by zero or four semitones (STs) presented in either clean or noise-degraded (+5 dB SNR) conditions. We found that behavioral identification was more accurate for vowel mixtures with larger pitch separations but F0 benefit interacted with noise. Time-frequency analysis decomposed the EEG into different spectrotemporal frequency bands. Low-frequency (θ, β) responses were elevated when speech did not contain pitch cues (0ST > 4ST) or was noisy, suggesting a correlate of increased listening effort and/or memory demands. Contrastively, γ power increments were observed for changes in both pitch (0ST > 4ST) and SNR (clean > noise), suggesting high-frequency bands carry information related to acoustic features and the quality of speech representations. Brain-behavior associations corroborated these effects; modulations in low-frequency rhythms predicted the speed of listeners' perceptual decisions with higher bands predicting identification accuracy. Results are consistent with the notion that neural oscillations reflect both automatic (pre-perceptual) and controlled (post-perceptual) mechanisms of speech processing that are largely divisible into high- and low-frequency bands of human brain rhythms.

Noise and pitch interact during the cortical segregation of concurrent speech

Behavioral studies reveal listeners exploit intrinsic differences in voice fundamental frequency (F0) to segregate concurrent speech sounds-the so-called "F0-benefit." More favorable signal-to-noise ratio (SNR) in the environment, an extrinsic acoustic factor, similarly benefits the parsing of simultaneous speech. Here, we examined the neurobiological substrates of these two cues in the perceptual segregation of concurrent speech mixtures. We recorded event-related brain potentials (ERPs) while listeners performed a speeded double-vowel identification task. Listeners heard two concurrent vowels whose F0 differed by zero or four semitones presented in either clean (no noise) or noise-degraded (+5 dB SNR) conditions. Behaviorally, listeners were more accurate in correctly identifying both vowels for larger F0 separations but F0-benefit was more pronounced at more favorable SNRs (i.e., pitch × SNR interaction). Analysis of the ERPs revealed that only the P2 wave (∼200 ms) showed a similar F0 x SNR interaction as behavior and was correlated with listeners' perceptual F0-benefit. Neural classifiers applied to the ERPs further suggested that speech sounds are segregated neurally within 200 ms based on SNR whereas segregation based on pitch occurs later in time (400-700 ms). The earlier timing of extrinsic SNR compared to intrinsic F0-based segregation implies that the cortical extraction of speech from noise is more efficient than differentiating speech based on pitch cues alone, which may recruit additional cortical processes. Findings indicate that noise and pitch differences interact relatively early in cerebral cortex and that the brain arrives at the identities of concurrent speech mixtures as early as ∼200 ms.

Promoting the perception of two and three concurrent sound objects: An event-related potential study

The auditory environment typically comprises several simultaneously active sound sources. In contrast to the perceptual segregation of two concurrent sounds, the perception of three simultaneous sound objects has not yet been studied systematically. We conducted two experiments in which participants were presented with complex sounds containing sound segregation cues (mistuning, onset asynchrony, differences in frequency or amplitude modulation or in sound location), which were set up to promote the perceptual organization of the tonal elements into one, two, or three concurrent sounds. In Experiment 1, listeners indicated whether they heard one, two, or three concurrent sounds. In Experiment 2, participants watched a silent subtitled movie while EEG was recorded to extract the object-related negativity (ORN) component of the event-related potential. Listeners predominantly reported hearing two sounds when the segregation promoting manipulations were applied to the same tonal element. When two different tonal elements received manipulations promoting them to be heard as separate auditory objects, participants reported hearing two and three concurrent sounds objects with equal probability. The ORN was elicited in most conditions; sounds that included the amplitude- or the frequency-modulation cue generated the smallest ORN amplitudes. Manipulating two different tonal elements yielded numerically and often significantly smaller ORNs than the sum of the ORNs elicited when the same cues were applied on a single tonal element. These results suggest that ORN reflects the presence of multiple concurrent sounds, but not their number. The ORN results are compatible with the horse-race principle of combining different cues of concurrent sound segregation.

Modeling of speech localization in a multi-talker mixture using periodicity and energy-based auditory features

A recent study showed that human listeners are able to localize a short speech target simultaneously masked by four speech tokens in reverberation [Kopčo, Best, and Carlile (2010). J. Acoust. Soc. Am. 127, 1450-1457]. Here, an auditory model for solving this task is introduced. The model has three processing stages: (1) extraction of the instantaneous interaural time difference (ITD) information, (2) selection of target-related ITD information ("glimpses") using a template-matching procedure based on periodicity, spectral energy, or both, and (3) target location estimation. The model performance was compared to the human data, and to the performance of a modified model using an ideal binary mask (IBM) at stage (2). The IBM-based model performed similarly to the subjects, indicating that the binaural model is able to accurately estimate source locations. Template matching using spectral energy and using a combination of spectral energy and periodicity achieved good results, while using periodicity alone led to poor results. Particularly, the glimpses extracted from the initial portion of the signal were critical for good performance. Simulation data show that the auditory features investigated here are sufficient to explain human performance in this challenging listening condition and thus may be used in models of auditory scene analysis.

Effects of Physiological Internal Noise on Model Predictions of Concurrent Vowel Identification for Normal-Hearing Listeners

Previous studies have shown that concurrent vowel identification improves with increasing temporal onset asynchrony of the vowels, even if the vowels have the same fundamental frequency. The current study investigated the possible underlying neural processing involved in concurrent vowel perception. The individual vowel stimuli from a previously published study were used as inputs for a phenomenological auditory-nerve (AN) model. Spectrotemporal representations of simulated neural excitation patterns were constructed (i.e., neurograms) and then matched quantitatively with the neurograms of the single vowels using the Neurogram Similarity Index Measure (NSIM). A novel computational decision model was used to predict concurrent vowel identification. To facilitate optimum matches between the model predictions and the behavioral human data, internal noise was added at either neurogram generation or neurogram matching using the NSIM procedure. The best fit to the behavioral data was achieved with a signal-to-noise ratio (SNR) of 8 dB for internal noise added at the neurogram but with a much smaller amount of internal noise (SNR of 60 dB) for internal noise added at the level of the NSIM computations. The results suggest that accurate modeling of concurrent vowel data from listeners with normal hearing may partly depend on internal noise and where internal noise is hypothesized to occur during the concurrent vowel identification process.

Vowel perception in listeners with normal hearing and in listeners with hearing loss: a preliminary study

OBJECTIVES

To determine the influence of hearing loss on perception of vowel slices.

METHODS

Fourteen listeners aged 20-27 participated; ten (6 males) had hearing within normal limits and four (3 males) had moderate-severe sensorineural hearing loss (SNHL). Stimuli were six naturally-produced words consisting of the vowels /i a u æ ɛ ʌ/ in a /b V b/ context. Each word was presented as a whole and in eight slices: the initial transition, one half and one fourth of initial transition, full central vowel, one-half central vowel, ending transition, one half and one fourth of ending transition. Each of the 54 stimuli was presented 10 times at 70 dB SPL (sound press level); listeners were asked to identify the word. Stimuli were shaped using signal processing software for the listeners with SNHL to mimic gain provided by an appropriately-fitting hearing aid.

RESULTS

Listeners with SNHL had a steeper rate of decreasing vowel identification with decreasing slice duration as compared to listeners with normal hearing, and the listeners with SNHL showed different patterns of vowel identification across vowels when compared to listeners with normal hearing.

CONCLUSION

Abnormal temporal integration is likely affecting vowel identification for listeners with SNHL, which in turn affects vowel internal representation at different levels of the auditory system.

Sleep-dependent neuroplastic changes during auditory perceptual learning

Auditory perceptual learning is accompanied by a significant increase in the amplitude of sensory evoked responses on the second day of training. This is thought to reflect memory consolidation after the first practice session. However, it is unclear whether the changes in sensory evoked responses depend on sleep per se or whether a break between training sessions would sufficiently yield similar changes. To assess the relative contributions of sleep and passage of time (wakefulness) on the sensory evoked responses, we recorded auditory evoked fields using magnetoencephalography while participants performed a vowel segregation task in three different sessions separated by 12h over two consecutive days. The first two practice sessions were scheduled in the morning and evening of the same day for one group and the evening and morning of subsequent days for the other group. For each participant, we modeled the auditory evoked magnetic field with single dipoles in bilateral superior temporal planes. We then examined the amplitudes and latencies of the resulting source waveforms as a function of sleep and passage of time. In both groups, performance gradually improved with repeated testing. Auditory learning was paralleled by increased sustained field between 250 and 350ms after sound onset as well as sensory evoked fields around 200ms after sound onset (i.e., P2m amplitude) for sessions taking place on the same and different days, respectively. These neuromagnetic changes suggest that auditory learning involves a consolidation phase that occurs during the wake state, which is followed by a sleep-dependent consolidation stage indexed by the P2m amplitude.

The impact of musicianship on the cortical mechanisms related to separating speech from background noise

Musicians have enhanced auditory processing abilities. In some studies, these abilities are paralleled by an improved understanding of speech in noisy environments, partially due to more robust encoding of speech signals in noise at the level of the brainstem. Little is known about the impact of musicianship on attention-dependent cortical activity related to lexical access during a speech-in-noise task. To address this issue, we presented musicians and nonmusicians with single words mixed with three levels of background noise, across two conditions, while monitoring electrical brain activity. In the active condition, listeners repeated the words aloud, and in the passive condition, they ignored the words and watched a silent film. When background noise was most intense, musicians repeated more words correctly compared with nonmusicians. Auditory evoked responses were attenuated and delayed with the addition of background noise. In musicians, P1 amplitude was marginally enhanced during active listening and was related to task performance in the most difficult listening condition. By comparing ERPs from the active and passive conditions, we isolated an N400 related to lexical access. The amplitude of the N400 was not influenced by the level of background noise in musicians, whereas N400 amplitude increased with the level of background noise in nonmusicians. In nonmusicians, the increase in N400 amplitude was related to a reduction in task performance. In musicians only, there was a rightward shift of the sources contributing to the N400 as the level of background noise increased. This pattern of results supports the hypothesis that encoding of speech in noise is more robust in musicians and suggests that this facilitates lexical access. Moreover, the shift in sources suggests that musicians, to a greater extent than nonmusicians, may increasingly rely on acoustic cues to understand speech in noise.

Effects of multiple congruent cues on concurrent sound segregation during passive and active listening: an event-related potential (ERP) study

In two experiments, we assessed the effects of combining different cues of concurrent sound segregation on the object-related negativity (ORN) and the P400 event-related potential components. Participants were presented with sequences of complex tones, half of which contained some manipulation: one or two harmonic partials were mistuned, delayed, or presented from a different location than the rest. In separate conditions, one, two, or three of these manipulations were combined. Participants watched a silent movie (passive listening) or reported after each tone whether they perceived one or two concurrent sounds (active listening). ORN was found in almost all conditions except for location difference alone during passive listening. Combining several cues or manipulating more than one partial consistently led to sub-additive effects on the ORN amplitude. These results support the view that ORN reflects a combined, feature-unspecific assessment of the auditory system regarding the contribution of two sources to the incoming sound.

Neural correlates of sound localization in complex acoustic environments

Listening to and understanding people in a “cocktail-party situation” is a remarkable feature of the human auditory system. Here we investigated the neural correlates of the ability to localize a particular sound among others in an acoustically cluttered environment with healthy subjects. In a sound localization task, five different natural sounds were presented from five virtual spatial locations during functional magnetic resonance imaging (fMRI). Activity related to auditory stream segregation was revealed in posterior superior temporal gyrus bilaterally, anterior insula, supplementary motor area, and frontoparietal network. Moreover, the results indicated critical roles of left planum temporale in extracting the sound of interest among acoustical distracters and the precuneus in orienting spatial attention to the target sound. We hypothesized that the left-sided lateralization of the planum temporale activation is related to the higher specialization of the left hemisphere for analysis of spectrotemporal sound features. Furthermore, the precuneus − a brain area known to be involved in the computation of spatial coordinates across diverse frames of reference for reaching to objects − seems to be also a crucial area for accurately determining locations of auditory targets in an acoustically complex scene of multiple sound sources. The precuneus thus may not only be involved in visuo-motor processes, but may also subserve related functions in the auditory modality.

Age-related deficits in the processing of fundamental frequency differences for the intelligibility of competing voices

A common complaint of older listeners is that they can hear speech, yet cannot understand it, especially when listening to speech in a background noise. When target and competing speech signals are concurrently presented, a difference in the fundamental frequency (ΔF0) between competing speech signals, which determines the pitch of voice, can be an important and commonly occurring cue to facilitate the separation of the target message from the interfering message, consequently improving intelligibility of the target message. To address the question of whether the older listeners have reduced ability to use ΔF0 and how the age-related deficits in the processing of ΔF0 are theoretically explained, this paper is divided into three parts. The first part of this article summarizes how the speech-communication difficulties that older listeners have are theoretically explained. In the second part, literatures on the perceptual benefits from ΔF0 and the age-related deficits on the use of ΔF0 are reviewed. As a final part, three theoretical models explaining the general processing of ΔF0 are compared to discuss which better explains the age-related deficits in the processing of ΔF0.

The influence of lifelong musicianship on neurophysiological measures of concurrent sound segregation

The ability to separate concurrent sounds based on periodicity cues is critical for parsing complex auditory scenes. This ability is enhanced in young adult musicians and reduced in older adults. Here, we investigated the impact of lifelong musicianship on concurrent sound segregation and perception using scalp-recorded ERPs. Older and younger musicians and nonmusicians were presented with periodic harmonic complexes where the second harmonic could be tuned or mistuned by 1-16% of its original value. The likelihood of perceiving two simultaneous sounds increased with mistuning, and musicians, both older and younger, were more likely to detect and report hearing two sounds when the second harmonic was mistuned at or above 2%. The perception of a mistuned harmonic as a separate sound was paralleled by an object-related negativity that was larger and earlier in younger musicians compared with the other three groups. When listeners made a judgment about the harmonic stimuli, the perception of the mistuned harmonic as a separate sound was paralleled by a positive wave at about 400 msec poststimulus (P400), which was enhanced in both older and younger musicians. These findings suggest attention-dependent processing of a mistuned harmonic is enhanced in older musicians and provides further evidence that age-related decline in hearing abilities are mitigated by musical training.

Effect of fundamental-frequency and sentence-onset differences on speech-identification performance of young and older adults in a competing-talker background

This study investigated the benefits of differences between sentences in fundamental frequency (F0) and temporal onset for sentence pairs among listener groups differing in age and hearing sensitivity. Two experiments were completed with the primary difference between experiments being the way in which the stimuli were presented. Experiment 1 used blocked stimulus presentation, which ultimately provided redundant acoustic cues to mark the target sentence in each pair, whereas Experiment 2 sampled a slightly more restricted stimulus space, but in a completely randomized presentation order. For both experiments, listeners were required to detect a cue word ("Baron") for the target sentence in each pair and to then identify the target words (color, number) that appeared later in the target sentence. Results of Experiment 1 showed that F0 or onset separation cues were beneficial to both cue-word detection and color-number identification performance. There were no significant differences across groups in the ability to detect the cue word, but groups differed in their ability to identify the correct color-number words. Elderly adults with impaired hearing had the greatest difficulty with the identification task despite the application of spectral shaping to restore the audibility of the speech stimuli. For the most part, the primary results of Experiment 1 were replicated in Experiment 2, although, in the latter experiment, all older adults, whether they had normal or impaired hearing, performed worse than young adults with normal hearing. From Experiment 2, the benefits received for a difference in F0 between talkers of 6 semitones were equivalent to those received for an onset asynchrony of 300 ms between sentences and, for such conditions, the combination of both sound-segregation cues resulted in an additive benefit.

Functional brain networks underlying perceptual switching: auditory streaming and verbal transformations

Recent studies have shown that auditory scene analysis involves distributed neural sites below, in, and beyond the auditory cortex (AC). However, it remains unclear what role each site plays and how they interact in the formation and selection of auditory percepts. We addressed this issue through perceptual multistability phenomena, namely, spontaneous perceptual switching in auditory streaming (AS) for a sequence of repeated triplet tones, and perceptual changes for a repeated word, known as verbal transformations (VTs). An event-related fMRI analysis revealed brain activity timelocked to perceptual switching in the cerebellum for AS, in frontal areas for VT, and the AC and thalamus for both. The results suggest that motor-based prediction, produced by neural networks outside the auditory system, plays essential roles in the segmentation of acoustic sequences both in AS and VT. The frequency of perceptual switching was determined by a balance between the activation of two sites, which are proposed to be involved in exploring novel perceptual organization and stabilizing current perceptual organization. The effect of the gene polymorphism of catechol-O-methyltransferase (COMT) on individual variations in switching frequency suggests that the balance of exploration and stabilization is modulated by catecholamines such as dopamine and noradrenalin. These mechanisms would support the noteworthy flexibility of auditory scene analysis.

Auditory grouping mechanisms reflect a sound's relative position in a sequence

The human brain uses acoustic cues to decompose complex auditory scenes into its components. For instance to improve communication, a listener can select an individual “stream,” such as a talker in a crowded room, based on cues such as pitch or location. Despite numerous investigations into auditory streaming, few have demonstrated clear correlates of perception; instead, in many studies perception covaries with changes in physical stimulus properties (e.g., frequency separation). In the current report, we employ a classic ABA streaming paradigm and human electroencephalography (EEG) to disentangle the individual contributions of stimulus properties from changes in auditory perception. We find that changes in perceptual state—that is the perception of one versus two auditory streams with physically identical stimuli—and changes in physical stimulus properties are reflected independently in the event-related potential (ERP) during overlapping time windows. These findings emphasize the necessity of controlling for stimulus properties when studying perceptual effects of streaming. Furthermore, the independence of the perceptual effect from stimulus properties suggests the neural correlates of streaming reflect a tone's relative position within a larger sequence (1st, 2nd, 3rd) rather than its acoustics. By clarifying the role of stimulus attributes along with perceptual changes, this study helps explain precisely how the brain is able to distinguish a sound source of interest in an auditory scene.

Attention, awareness, and the perception of auditory scenes

Auditory perception and cognition entails both low-level and high-level processes, which are likely to interact with each other to create our rich conscious experience of soundscapes. Recent research that we review has revealed numerous influences of high-level factors, such as attention, intention, and prior experience, on conscious auditory perception. And recently, studies have shown that auditory scene analysis tasks can exhibit multistability in a manner very similar to ambiguous visual stimuli, presenting a unique opportunity to study neural correlates of auditory awareness and the extent to which mechanisms of perception are shared across sensory modalities. Research has also led to a growing number of techniques through which auditory perception can be manipulated and even completely suppressed. Such findings have important consequences for our understanding of the mechanisms of perception and also should allow scientists to precisely distinguish the influences of different higher-level influences.

Neural Encoding of Speech and Music: Implications for Hearing Speech in Noise

Understanding speech in a background of competing noise is challenging, especially for individuals with hearing loss or deficits in auditory processing ability. The ability to hear in background noise cannot be predicted from the audiogram, an assessment of peripheral hearing ability; therefore, it is important to consider the impact of central and cognitive factors on speech-in-noise perception. Auditory processing in complex environments is reflected in neural encoding of pitch, timing, and timbre, the crucial elements of speech and music. Musical expertise in processing pitch, timing, and timbre may transfer to enhancements in speech-in-noise perception due to shared neural pathways for speech and music. Through cognitive-sensory interactions, musicians develop skills enabling them to selectively listen to relevant signals embedded in a network of melodies and harmonies, and this experience leads in turn to enhanced ability to focus on one voice in a background of other voices. Here we review recent work examining the biological mechanisms of speech and music perception and the potential for musical experience to ameliorate speech-in-noise listening difficulties.

Effects of age on concurrent vowel perception in acoustic and simulated electroacoustic hearing

PURPOSE

In this study, the authors investigated the effects of age on the use of fundamental frequency differences (ΔF(0)) in the perception of competing synthesized vowels in simulations of electroacoustic and cochlear-implant hearing.

METHOD

Twelve younger listeners with normal hearing and 13 older listeners with (near) normal hearing were evaluated in their use of ΔF(0) in the perception of competing synthesized vowels for 3 conditions: unprocessed synthesized vowels (UNP), envelope-vocoded synthesized vowels that simulated a cochlear implant (VOC), and synthesized vowels processed to simulate electroacoustic stimulation (EAS) hearing. Tasks included (a) multiplicity, which required listeners to identify whether a stimulus contained 1 or 2 sounds and (b) double-vowel identification, which required listeners to attach phonemic labels to the competing synthesized vowels.

RESULTS

Multiplicity perception was facilitated by ΔF(0) in UNP and EAS but not in VOC, with no age-related deficits evident. Double-vowel identification was facilitated by ΔF(0), with ΔF(0) benefit largest in UNP, reduced in EAS, and absent in VOC. Age adversely affected overall identification and ΔF(0) benefit on the double-vowel task.

CONCLUSIONS

Some but not all older listeners derived ΔF(0) benefit in EAS hearing. This variability may partly be due to how listeners are able to draw on higher-level processing resources in extracting and integrating cues in EAS hearing.

Brainstem correlates of speech-in-noise perception in children

Children often have difficulty understanding speech in challenging listening environments. In the absence of peripheral hearing loss, these speech perception difficulties may arise from dysfunction at more central levels in the auditory system, including subcortical structures. We examined brainstem encoding of pitch in a speech syllable in 38 school-age children. In children with poor speech-in-noise perception, we find impaired encoding of the fundamental frequency and the second harmonic, two important cues for pitch perception. Pitch, an essential factor in speaker identification, aids the listener in tracking a specific voice from a background of voices. These results suggest that the robustness of subcortical neural encoding of pitch features in time-varying signals is a key factor in determining success with perceiving speech in noise.

Behind the scenes of auditory perception

'Auditory scenes' often contain contributions from multiple acoustic sources. These are usually heard as separate auditory 'streams', which can be selectively followed over time. How and where these auditory streams are formed in the auditory system is one of the most fascinating questions facing auditory scientists today. Findings published within the past two years indicate that both cortical and subcortical processes contribute to the formation of auditory streams, and they raise important questions concerning the roles of primary and secondary areas of auditory cortex in this phenomenon. In addition, these findings underline the importance of taking into account the relative timing of neural responses, and the influence of selective attention, in the search for neural correlates of the perception of auditory streams.

Concurrent sound segregation based on inharmonicity and onset asynchrony

To explore the neural processes underlying concurrent sound segregation, auditory evoked fields (AEFs) were measured using magnetoencephalography (MEG). To induce the segregation of two auditory objects we manipulated harmonicity and onset synchrony. Participants were presented with complex sounds with (i) all harmonics in-tune (ii) the third harmonic mistuned by 8% of its original value (iii) the onset of the third harmonic delayed by 160 ms compared to the other harmonics. During recording, participants listened to the sounds and performed an auditory localisation task whereas in another session they ignored the sounds and performed a visual localisation task. Active and passive listening was chosen to evaluate the contribution of attention on sound segregation. Both cues - inharmonicity and onset asynchrony - elicited sound segregation, as participants were more likely to report correctly on which side they heard the third harmonic when it was mistuned or delayed compared to being in-tune with all other harmonics. AEF activity associated with concurrent sound segregation was identified over both temporal lobes. We found an early deflection at approximately 75 ms (P75m) after sound onset, probably reflecting an automatic registration of the mistuned harmonic. Subsequent deflections, the object-related negativity (ORNm) and a later displacement (P230m) seem to be more general markers of concurrent sound segregation, as they were elicited by both mistuning and delaying the third harmonic. Results indicate that the ORNm reflects relatively automatic, bottom-up sound segregation processes, whereas the P230m is more sensitive to attention, especially with inharmonicity as the cue for concurrent sound segregation.

Pitch, harmonicity and concurrent sound segregation: psychoacoustical and neurophysiological findings

Harmonic complex tones are a particularly important class of sounds found in both speech and music. Although these sounds contain multiple frequency components, they are usually perceived as a coherent whole, with a pitch corresponding to the fundamental frequency (F0). However, when two or more harmonic sounds occur concurrently, e.g., at a cocktail party or in a symphony, the auditory system must separate harmonics and assign them to their respective F0s so that a coherent and veridical representation of the different sounds sources is formed. Here we review both psychophysical and neurophysiological (single-unit and evoked-potential) findings, which provide some insight into how, and how well, the auditory system accomplishes this task. A survey of computational models designed to estimate multiple F0s and segregate concurrent sources is followed by a review of the empirical literature on the perception and neural coding of concurrent harmonic sounds, including vowels, as well as findings obtained using single complex tones with mistuned harmonics.

Exploring transient transfer entropy based on a group-wise ICA decomposition of EEG data

This paper presents a data-driven pipeline for studying asymmetries in mutual interdependencies between distinct components of EEG signal. Due to volume conductance, estimating coherence between scalp electrodes may lead to spurious results. A group-based independent component analysis (ICA), which is conducted across all subjects and conditions simultaneously, is an alternative representation of the EEG measurements. Within this approach, the extracted components are independent in a global sense while short-lived or transient interdependencies may still be present between the components. In this paper, functional roles of the ICA components are specified through a partial least squares (PLS) analysis of task effects within the time course of the derived components. Functional integration is estimated within the information-theoretic approach using transfer entropy analysis based on asymmetries in mutual interdependencies of reconstructed phase dynamics. A secondary PLS analysis is performed to assess robust task-specific changes in transfer entropy estimates between functionally specific components.

Effect of vowel identity and onset asynchrony on concurrent vowel identification

PURPOSE

The purpose of the current study was to determine the effects of vowel identity and temporal onset asynchrony on identification of vowels overlapped in time.

METHOD

Fourteen listeners with normal hearing, with a mean age of 24 years, participated. The listeners were asked to identify both of a pair of 200-ms vowels (referred to as double vowels) presented either simultaneously or with a temporal asynchrony ranging from 25 ms to 150 ms in 25-ms steps. The stimuli were synthetic steady-state vowels /i ae u / arranged in seven combinations: /u i/, /ae /, / /, / ae/, /ae i/, / i/, and / u/.

RESULTS

Listeners' responses revealed that one vowel of a pair was identified correctly more often than the other vowel (known as vowel dominance). Vowel dominance effects were seen for 6 of the 7 vowel pairs, and there was improvement of vowel identification with increasing temporal separation between vowels for 5 of the 7 pairs. Vowel pairs with the vowel // consistently yielded improved identification with increases in temporal asynchrony.

DISCUSSION

Peripheral masking cannot explain the patterns of results of this study. A more parsimonious explanation may be perceptual anchoring.

Comparator and non-comparator mechanisms of change detection in the context of speech--an ERP study

Automatic change detection reflects a cognitive memory-based comparison mechanism as well as a sensorial non-comparator mechanism based on differential states of refractoriness. The purpose of this study was to examine whether the comparator mechanism of the mismatch negativity component (MMN) is differentially affected by the lexical status of the deviant. Event-related potential (ERP) data was collected during an "oddball" paradigm designed to elicit the MMN from 15 healthy subjects that were involved in a counting task. Topography pattern analysis and source estimation were utilized to examine the deviance (deviants vs. standards), cognitive (deviants vs. control counterparts) and refractoriness (standards vs. control counterparts) effects elicited by standard-deviant pairs ("deh-day"; "day-deh"; "teh-tay") embedded within "oddball" blocks. Our results showed that when the change was salient regardless of lexical status (i.e., the /e:/ to /eI/ transition) the response tapped the comparator based-mechanism of the MMN which was located in the cuneus/posterior cingulate, reflected sensitivity to the novelty of the auditory object, appeared in the P2 latency range and mainly involved topography modulations. In contrast, when the novelty was low (i.e., the /eI/ to /e:/ transition) an acoustic change complex was elicited which involved strength modulations over the P1/N1 range and implicated the middle temporal gyrus. This result pattern also resembled the one displayed by the non-comparator mechanism. These findings suggest spatially and temporally distinct brain activities of comparator mechanisms of change detection in the context of speech.

One sound or two? Object-related negativity indexes echo perception

The ability to isolate a single sound source among concurrent sources and reverberant energy is necessary for understanding the auditory world. The precedence effect describes a related experimental finding, that when presented with identical sounds from two locations with a short onset asynchrony (on the order of milliseconds), listeners report a single source with a location dominated by the lead sound. Single-cell recordings in multiple animal models have indicated that there are low-level mechanisms that may contribute to the precedence effect, yet psychophysical studies in humans have provided evidence that top-down cognitive processes have a great deal of influence on the perception of simulated echoes. In the present study, event-related potentials evoked by click pairs at and around listeners' echo thresholds indicate that perception of the lead and lag sound as individual sources elicits a negativity between 100 and 250 msec, previously termed the object-related negativity (ORN). Even for physically identical stimuli, the ORN is evident when listeners report hearing, as compared with not hearing, a second sound source. These results define a neural mechanism related to the conscious perception of multiple auditory objects.

Responses of inferior colliculus neurons to double harmonic tones

The auditory system can segregate sounds that overlap in time and frequency, if the sounds differ in acoustic properties such as fundamental frequency (f0). However, the neural mechanisms that underlie this ability are poorly understood. Responses of neurons in the inferior colliculus (IC) of the anesthetized chinchilla were measured. The stimuli were harmonic tones, presented alone (single harmonic tones) and in the presence of a second harmonic tone with a different f0 (double harmonic tones). Responses to single harmonic tones exhibited no stimulus-related temporal pattern, or in some cases, a simple envelope modulated at f0. Responses to double harmonic tones exhibited complex slowly modulated discharge patterns. The discharge pattern varied with the difference in f0 and with characteristic frequency. The discharge pattern also varied with the relative levels of the two tones; complex temporal patterns were observed when levels were equal, but as the level difference increased, the discharge pattern reverted to that associated with single harmonic tones. The results indicated that IC neurons convey information about simultaneous sounds in their temporal discharge patterns and that the patterns are produced by interactions between adjacent components in the spectrum. The representation is "low-resolution," in that it does not convey information about single resolved components from either individual sound.

Breaking the wave: effects of attention and learning on concurrent sound perception

The auditory surrounding is often complex with many sound sources active simultaneously. Yet listeners are proficient in breaking apart the composite acoustic wave reaching the ears. This achievement is thought to be the result of bottom-up as well as top-down processes that reflect listeners' experience and knowledge of the auditory environment. Here, specific findings concerning the role of bottom-up and top-down (schema-driven) processes on concurrent sound perception are reviewed, with particular emphasis on studies that have used scalp recording of event-related brain potentials. Findings from several studies indicate that frequency periodicity, upon which concurrent sound perception partly depends, is quickly and automatically registered in primary auditory cortex. Moreover, success in identifying concurrent vowels is accompanied by enhanced neural activity, as revealed by functional magnetic resonance imaging, in thalamus, primary auditory cortex and planum temporale. Lastly, listeners' ability to segregate concurrent vowels improves with training and these neuroplastic changes occur rapidly, demonstrating the flexibility of human speech segregation mechanisms. Together, these studies suggest that the primary auditory cortex and the planum temporale play an important role in concurrent sound perception, and reveal a link between thalamo-cortical activation and the successful separation and identification of speech sounds presented simultaneously.

Age-related changes in neural activity associated with concurrent vowel segregation

Older adults exhibit degraded speech comprehension in complex sound environments, which may be related to overall age-related declines in low-level sound segregation. This hypothesis was tested by measuring event-related potentials (ERPs) while listeners identified two different vowels presented simultaneously. Older adults were less accurate than young adults at identifying the two vowels, although both groups improved similarly with increasing fundamental frequency differences (Deltaf0) between vowels. Reaction time data showed that older adults took more time to process stimuli, especially those with smaller Deltaf0. A negative ERP wave indexing the automatic registration of Deltaf0 (the object-related negativity) was reduced in older adults. In contrast, young and older adults showed a similar pattern of neural activity indexing attentive processing of Deltaf0. The results suggest that aging affects the ability to automatically segregate speech sounds.