EEG-based assessment of temporal fine structure and envelope effect in mandarin syllable and tone perception

In recent years, speech perception research has benefited from low-frequency rhythm entrainment tracking of the speech envelope. However, speech perception is still controversial regarding the role of speech envelope and temporal fine structure, especially in Mandarin. This study aimed to discuss the dependence of Mandarin syllables and tones perception on the speech envelope and the temporal fine structure. We recorded the electroencephalogram (EEG) of the subjects under three acoustic conditions using the sound chimerism analysis, including (i) the original speech, (ii) the speech envelope and the sinusoidal modulation, and (iii) the fine structure of time and the modulation of the non-speech (white noise) sound envelope. We found that syllable perception mainly depended on the speech envelope, while tone perception depended on the temporal fine structure. The delta bands were prominent, and the parietal and prefrontal lobes were the main activated brain areas, regardless of whether syllable or tone perception was involved. Finally, we decoded the spatiotemporal features of Mandarin perception from the microstate sequence. The spatiotemporal feature sequence of the EEG caused by speech material was found to be specific, suggesting a new perspective for the subsequent auditory brain-computer interface. These results provided a new scheme for the coding strategy of new hearing aids for native Mandarin speakers.

HIGHLIGHTS

Cortical representation of speech temporal information through high gamma-band activity and its temporal modulation

Numerous studies have investigated low-frequency (theta-band) and high-frequency (gamma-band) neural activities that are phase-locked to temporal structures, including the temporal envelope and fine structure (TFS) of speech signals. Nonetheless, the neural mechanisms underlying the interaction between envelope and TFS processing remain elusive. Here we examined high gamma-band activities and their low-frequency amplitude modulations while listening to monotone speech (MS) with a fundamental frequency (F0) of 80 Hz and non-speech sounds with similar temporal characteristics to MS, namely an amplitude-modulated click train (AMC). Additionally, we utilized noise-vocoded speech (NVS) to evaluate the impact of eliminating the TFS from MS on the high gamma-band activity. We observed discernible high gamma-band activity at the same frequency as F0 of MS and the train frequency of AMC (80 Hz). Furthermore, source localization analysis revealed that the high gamma-band activities exhibited left hemisphere dominance in both MS and AMC conditions. Finally, high gamma-band activities exhibited amplitude-modulation at the same rate as the stimulus envelope of MS and AMC (5 Hz), though such modulation was not observed in NVS. Our findings indicate that the high gamma-band activity in the left hemisphere is pivotal in the interaction of envelope and TFS information processing, regardless of the nature of the stimulus being speech or non-speech.

A Linear Superposition Model of Envelope and Frequency Following Responses May Help Identify Generators Based on Latency

Envelope and frequency-following responses (FFR_ENV and FFR_TFS) are scalp-recorded electrophysiological potentials that closely follow the periodicity of complex sounds such as speech. These signals have been established as important biomarkers in speech and learning disorders. However, despite important advances, it has remained challenging to map altered FFR_ENV and FFR_TFS to altered processing in specific brain regions. Here we explore the utility of a deconvolution approach based on the assumption that FFR_ENV and FFR_TFS reflect the linear superposition of responses that are triggered by the glottal pulse in each cycle of the fundamental frequency (F0 responses). We tested the deconvolution method by applying it to FFR_ENV and FFR_TFS of rhesus monkeys to human speech and click trains with time-varying pitch patterns. Our analyses show that F0_ENV responses could be measured with high signal-to-noise ratio and featured several spectro-temporally and topographically distinct components that likely reflect the activation of brainstem (<5 ms; 200-1000 Hz), midbrain (5-15 ms; 100-250 Hz), and cortex (15-35 ms; ~90 Hz). In contrast, F0_TFS responses contained only one spectro-temporal component that likely reflected activity in the midbrain. In summary, our results support the notion that the latency of F0 components map meaningfully onto successive processing stages. This opens the possibility that pathologically altered FFR_ENV or FFR_TFS may be linked to altered F0_ENV or F0_TFS and from there to specific processing stages and ultimately spatially targeted interventions.

Neural processing and perception of Schroeder-phase harmonic tone complexes in the gerbil: Relating single-unit neurophysiology to behavior

Schroeder-phase harmonic tone complexes have been used in physiological and psychophysical studies in several species to gain insight into cochlear function. Each pitch period of the Schroeder stimulus contains a linear frequency sweep; the duty cycle, sweep velocity, and direction are controlled by parameters of the phase spectrum. Here, responses to a range of Schroeder-phase harmonic tone complexes were studied both behaviorally and in neural recordings from the auditory nerve and inferior colliculus of Mongolian gerbils. Gerbils were able to discriminate Schroeder-phase harmonic tone complexes based on sweep direction, duty cycle, and/or velocity for fundamental frequencies up to 200 Hz. Temporal representation in neural responses based on the van Rossum spike-distance metric, with time constants of either 1 ms or related to the stimulus' period, was compared to average discharge rates. Neural responses and behavioral performance were both expressed in terms of sensitivity, d', to allow direct comparisons. Our results suggest that in the auditory nerve, stimulus fine structure is represented by spike timing while envelope is represented by rate. In the inferior colliculus, both temporal fine structure and envelope appear to be represented best by rate. However, correlations between neural d' values and behavioral sensitivity for sweep direction were strongest for both temporal metrics, for both auditory nerve and inferior colliculus. Furthermore, the high sensitivity observed in the inferior colliculus neural rate-based discrimination suggests that these neurons integrate across multiple inputs arising from the auditory periphery.

Relationships Among Temporal Fine Structure Sensitivity, Transient Storage Capacity, and Ultra-High Frequency Hearing Thresholds in Tinnitus Patients and Normal Adults of Different Ages

Background

Elderlies and tinnitus patients often find it challenging to process acoustic signals in noisy environments. The sensitivity to temporal fine structure (TFS), the transient storage capacity for TFS, and the ultra-high frequency (UHF) thresholds are all associated with aging-related damage, evidenced by speech-in-noise perception deficits. In the present study, we aimed to investigate the relationships among TFS sensitivity, transient storage capacity, and UHF thresholds in tinnitus patients and normal adults of different ages.

Methods

In the present study, 38 tinnitus patients (age ranging from 21 to 65) and 23 non-tinnitus adults (age ranging from 22 to 56) were enrolled, and some of their auditory indicators were examined, including the TFS-adaptive frequency (TFS-AF), break in interaural correlation (BIAC) delay threshold, and UHF thresholds.

Results

We found no significant difference in TFS-AF thresholds and BIAC delay thresholds between the tinnitus group and normal group, while their relationships with age were more evident in the tinnitus group. Moreover, these two tests were only significantly correlated in the tinnitus group. UHF thresholds were significantly correlated with TFS-AF thresholds only in the tinnitus group, suggesting that the UHF hearing was positively associated with the TFS sensitivity.

Conclusion

These findings indicated that the influencing factors, such as tinnitus and UHF thresholds, should be fully considered when examining age-related hearing decline, because the combination of tinnitus and poor UHF hearing might play a role in affecting hearing ability, such as TFS sensitivity.

Individualized Assays of Temporal Coding in the Ascending Human Auditory System

Neural phase-locking to temporal fluctuations is a fundamental and unique mechanism by which acoustic information is encoded by the auditory system. The perceptual role of this metabolically expensive mechanism, the neural phase-locking to temporal fine structure (TFS) in particular, is debated. Although hypothesized, it is unclear whether auditory perceptual deficits in certain clinical populations are attributable to deficits in TFS coding. Efforts to uncover the role of TFS have been impeded by the fact that there are no established assays for quantifying the fidelity of TFS coding at the individual level. While many candidates have been proposed, for an assay to be useful, it should not only intrinsically depend on TFS coding, but should also have the property that individual differences in the assay reflect TFS coding per se over and beyond other sources of variance. Here, we evaluate a range of behavioral and electroencephalogram (EEG)-based measures as candidate individualized measures of TFS sensitivity. Our comparisons of behavioral and EEG-based metrics suggest that extraneous variables dominate both behavioral scores and EEG amplitude metrics, rendering them ineffective. After adjusting behavioral scores using lapse rates, and extracting latency or percent-growth metrics from EEG, interaural timing sensitivity measures exhibit robust behavior-EEG correlations. Together with the fact that unambiguous theoretical links can be made relating binaural measures and phase-locking to TFS, our results suggest that these "adjusted" binaural assays may be well suited for quantifying individual TFS processing.

Noninvasive Measures of Distorted Tonotopic Speech Coding Following Noise-Induced Hearing Loss

Animal models of noise-induced hearing loss (NIHL) show a dramatic mismatch between cochlear characteristic frequency (CF, based on place of innervation) and the dominant response frequency in single auditory-nerve-fiber responses to broadband sounds (i.e., distorted tonotopy, DT). This noise trauma effect is associated with decreased frequency-tuning-curve (FTC) tip-to-tail ratio, which results from decreased tip sensitivity and enhanced tail sensitivity. Notably, DT is more severe for noise trauma than for metabolic (e.g., age-related) losses of comparable degree, suggesting that individual differences in DT may contribute to speech intelligibility differences in patients with similar audiograms. Although DT has implications for many neural-coding theories for real-world sounds, it has primarily been explored in single-neuron studies that are not viable with humans. Thus, there are no noninvasive measures to detect DT. Here, frequency following responses (FFRs) to a conversational speech sentence were recorded in anesthetized male chinchillas with either normal hearing or NIHL. Tonotopic sources of FFR envelope and temporal fine structure (TFS) were evaluated in normal-hearing chinchillas. Results suggest that FFR envelope primarily reflects activity from high-frequency neurons, whereas FFR-TFS receives broad tonotopic contributions. Representation of low- and high-frequency speech power in FFRs was also assessed. FFRs in hearing-impaired animals were dominated by low-frequency stimulus power, consistent with oversensitivity of high-frequency neurons to low-frequency power. These results suggest that DT can be diagnosed noninvasively. A normalized DT metric computed from speech FFRs provides a potential diagnostic tool to test for DT in humans. A sensitive noninvasive DT metric could be used to evaluate perceptual consequences of DT and to optimize hearing-aid amplification strategies to improve tonotopic coding for hearing-impaired listeners.

Relationship between sensitivity to temporal fine structure and spoken language abilities in children with mild-to-moderate sensorineural hearing loss

Children with sensorineural hearing loss show considerable variability in spoken language outcomes. The present study tested whether specific deficits in supra-threshold auditory perception might contribute to this variability. In a previous study by Halliday, Rosen, Tuomainen, and Calcus [(2019). J. Acoust. Soc. Am. 146, 4299], children with mild-to-moderate sensorineural hearing loss (MMHL) were shown to perform more poorly than those with normal hearing (NH) on measures designed to assess sensitivity to the temporal fine structure (TFS; the rapid oscillations in the amplitude of narrowband signals over short time intervals). However, they performed within normal limits on measures assessing sensitivity to the envelope (E; the slow fluctuations in the overall amplitude). Here, individual differences in unaided sensitivity to the TFS accounted for significant variance in the spoken language abilities of children with MMHL after controlling for nonverbal intelligence quotient, family history of language difficulties, and hearing loss severity. Aided sensitivity to the TFS and E cues was equally important for children with MMHL, whereas for children with NH, E cues were more important. These findings suggest that deficits in TFS perception may contribute to the variability in spoken language outcomes in children with sensorineural hearing loss.

Temporal Coding of Single Auditory Nerve Fibers Is Not Degraded in Aging Gerbils

People suffering from age-related hearing loss typically present with deficits in temporal processing tasks. Temporal processing deficits have also been shown in single-unit studies at the level of the auditory brainstem, midbrain, and cortex of aged animals. In this study, we explored whether temporal coding is already affected at the level of the input to the central auditory system. Single-unit auditory nerve fiber recordings were obtained from 41 Mongolian gerbils of either sex, divided between young, middle-aged, and old gerbils. Temporal coding quality was evaluated as vector strength in response to tones at best frequency, and by constructing shuffled and cross-stimulus autocorrelograms, and reverse correlations, from responses to 1 s noise bursts at 10-30 dB sensation level (dB above threshold). At comparable sensation levels, all measures showed that temporal coding was not altered in auditory nerve fibers of aging gerbils. Furthermore, both temporal fine structure and envelope coding remained unaffected. However, spontaneous rates were decreased in aging gerbils. Importantly, despite elevated pure tone thresholds, the frequency tuning of auditory nerve fibers was not affected. These results suggest that age-related temporal coding deficits arise more centrally, possibly due to a loss of auditory nerve fibers (or their peripheral synapses) but not due to qualitative changes in the responses of remaining auditory nerve fibers. The reduced spontaneous rate and elevated thresholds, but normal frequency tuning, of aged auditory nerve fibers can be explained by the well known reduction of endocochlear potential due to strial dysfunction in aged gerbils.SIGNIFICANCE STATEMENT As our society ages, age-related hearing deficits become ever more prevalent. Apart from decreased hearing sensitivity, elderly people often suffer from a reduced ability to communicate in daily settings, which is thought to be caused by known age-related deficits in auditory temporal processing. The current study demonstrated, using several different stimuli and analysis techniques, that these putative temporal processing deficits are not apparent in responses of single-unit auditory nerve fibers of quiet-aged gerbils. This suggests that age-related temporal processing deficits may develop more central to the auditory nerve, possibly due to a reduced population of active auditory nerve fibers, which will be of importance for the development of treatments for age-related hearing disorders.

Temporal Coding of Single Auditory Nerve Fibers Is Not Degraded in Aging Gerbils

People suffering from age-related hearing loss typically present with deficits in temporal processing tasks. Temporal processing deficits have also been shown in single-unit studies at the level of the auditory brainstem, midbrain, and cortex of aged animals. In this study, we explored whether temporal coding is already affected at the level of the input to the central auditory system. Single-unit auditory nerve fiber recordings were obtained from 41 Mongolian gerbils of either sex, divided between young, middle-aged, and old gerbils. Temporal coding quality was evaluated as vector strength in response to tones at best frequency, and by constructing shuffled and cross-stimulus autocorrelograms, and reverse correlations, from responses to 1 s noise bursts at 10-30 dB sensation level (dB above threshold). At comparable sensation levels, all measures showed that temporal coding was not altered in auditory nerve fibers of aging gerbils. Furthermore, both temporal fine structure and envelope coding remained unaffected. However, spontaneous rates were decreased in aging gerbils. Importantly, despite elevated pure tone thresholds, the frequency tuning of auditory nerve fibers was not affected. These results suggest that age-related temporal coding deficits arise more centrally, possibly due to a loss of auditory nerve fibers (or their peripheral synapses) but not due to qualitative changes in the responses of remaining auditory nerve fibers. The reduced spontaneous rate and elevated thresholds, but normal frequency tuning, of aged auditory nerve fibers can be explained by the well known reduction of endocochlear potential due to strial dysfunction in aged gerbils.SIGNIFICANCE STATEMENT As our society ages, age-related hearing deficits become ever more prevalent. Apart from decreased hearing sensitivity, elderly people often suffer from a reduced ability to communicate in daily settings, which is thought to be caused by known age-related deficits in auditory temporal processing. The current study demonstrated, using several different stimuli and analysis techniques, that these putative temporal processing deficits are not apparent in responses of single-unit auditory nerve fibers of quiet-aged gerbils. This suggests that age-related temporal processing deficits may develop more central to the auditory nerve, possibly due to a reduced population of active auditory nerve fibers, which will be of importance for the development of treatments for age-related hearing disorders.

Divergent Auditory Nerve Encoding Deficits Between Two Common Etiologies of Sensorineural Hearing Loss

Speech intelligibility can vary dramatically between individuals with similar clinically defined severity of hearing loss based on the audiogram. These perceptual differences, despite equal audiometric-threshold elevation, are often assumed to reflect central-processing variations. Here, we compared peripheral-processing in auditory nerve (AN) fibers of male chinchillas between two prevalent hearing loss etiologies: metabolic hearing loss (MHL) and noise-induced hearing loss (NIHL). MHL results from age-related reduction of the endocochlear potential due to atrophy of the stria vascularis. MHL in the present study was induced using furosemide, which provides a validated model of age-related MHL in young animals by reversibly inhibiting the endocochlear potential. Effects of MHL on peripheral processing were assessed using Wiener-kernel (system identification) analyses of single AN fiber responses to broadband noise, for direct comparison to previously published AN responses from animals with NIHL. Wiener-kernel analyses show that even mild NIHL causes grossly abnormal coding of low-frequency stimulus components. In contrast, for MHL the same abnormal coding was only observed with moderate to severe loss. For equal sensitivity loss, coding impairment was substantially less severe with MHL than with NIHL, probably due to greater preservation of the tip-to-tail ratio of cochlear frequency tuning with MHL compared with NIHL rather than different intrinsic AN properties. Differences in peripheral neural coding between these two pathologies-the more severe of which, NIHL, is preventable-likely contribute to individual speech perception differences. Our results underscore the need to minimize noise overexposure and for strategies to personalize diagnosis and treatment for individuals with sensorineural hearing loss.SIGNIFICANCE STATEMENT Differences in speech perception ability between individuals with similar clinically defined severity of hearing loss are often assumed to reflect central neural-processing differences. Here, we demonstrate for the first time that peripheral neural processing of complex sounds differs dramatically between the two most common etiologies of hearing loss. Greater processing impairment with noise-induced compared with an age-related (metabolic) hearing loss etiology may explain heightened speech perception difficulties in people overexposed to loud environments. These results highlight the need for public policies to prevent noise-induced hearing loss, an entirely avoidable hearing loss etiology, and for personalized strategies to diagnose and treat sensorineural hearing loss.

Effects of Musical Training and Hearing Loss on Fundamental Frequency Discrimination and Temporal Fine Structure Processing: Psychophysics and Modeling

Several studies have shown that musical training leads to improved fundamental frequency (F0) discrimination for young listeners with normal hearing (NH). It is unclear whether a comparable effect of musical training occurs for listeners whose sensory encoding of F0 is degraded. To address this question, the effect of musical training was investigated for three groups of listeners (young NH, older NH, and older listeners with hearing impairment, HI). In a first experiment, F0 discrimination was investigated using complex tones that differed in harmonic content and phase configuration (sine, positive, or negative Schroeder). Musical training was associated with significantly better F0 discrimination of complex tones containing low-numbered harmonics for all groups of listeners. Part of this effect was caused by the fact that musicians were more robust than non-musicians to harmonic roving. Despite the benefit relative to their non-musicians counterparts, the older musicians, with or without HI, performed worse than the young musicians. In a second experiment, binaural sensitivity to temporal fine structure (TFS) cues was assessed for the same listeners by estimating the highest frequency at which an interaural phase difference was perceived. Performance was better for musicians for all groups of listeners and the use of TFS cues was degraded for the two older groups of listeners. These findings suggest that musical training is associated with an enhancement of both TFS cues encoding and F0 discrimination in young and older listeners with or without HI, although the musicians' benefit decreased with increasing hearing loss. Additionally, models of the auditory periphery and midbrain were used to examine the effect of HI on F0 encoding. The model predictions reflected the worsening in F0 discrimination with increasing HI and accounted for up to 80 % of the variance in the data.

Cantonese Tone Identification in Three Temporal Cues in Quiet, Speech-Shaped Noise and Two-Talker Babble

Purpose: Cochlear implant processors deliver mostly temporal envelope information and limited fundamental frequency (F0) information to the users, which make pitch and lexical tone perception challenging for cochlear implantees. Different factors have been found to affect Mandarin tone perception in temporal cues but the most effective temporal cues for lexical tone identification across different backgrounds remained unclear because no study has comprehensively examined the effects and interactions of these factors, particularly, in languages that use both pitch heights and pitch shapes to differentiate lexical meanings. The present study compared identification of Cantonese tones in naturally produced stimuli, and in three temporal cues, namely the amplitude contour cue (TE50), the periodicity cue (TE500), and the temporal fine structure cue (TFS), in three different numbers of frequency bands (B04, B08, B16) in quiet and two types of noise (two male talker-babble and speech-shaped noise). Method: Naturally produced Cantonese tones and synthetic tones that combined different acoustic cues and different number of frequency bands were presented to 18 young native Cantonese speakers for tone identification in quiet and noise. Results: Among the three temporal cues, TFS was the most effective for Cantonese tone identification in quiet and noise, except for T4 (LF) identification. Its effect was even stronger when the tones were presented in 4 or 8 bands rather than 16 bands. Neither TE500 nor TE50 was effective for Cantonese tone identification in quiet or noise. In noise, most tones in TE500 and TE50 were misheard as T4 (LF), demonstrating errors in both tone shapes and tone heights. Types of noise had limited effect on tone identification. Conclusions: Findings on Mandarin tone perception in temporal cues may not be applicable to other tone languages with more complex tonal systems. TFS presented in four bands was the most effective temporal cue for Cantonese tone identification in quiet and noise. Temporal envelope cues were not effective for tone, tone shape or tone height identification in Cantonese. These findings have implications for future design of cochlear implants for tone speakers who use pitch heights or a combination of pitch heights and pitch shapes to differentiate meanings.

Adaptation to Noise in Human Speech Recognition Unrelated to the Medial Olivocochlear Reflex

Sensory systems constantly adapt their responses to the current environment. In hearing, adaptation may facilitate communication in noisy settings, a benefit frequently (but controversially) attributed to the medial olivocochlear reflex (MOCR) enhancing the neural representation of speech. Here, we show that human listeners (N = 14; five male) recognize more words presented monaurally in ipsilateral, contralateral, and bilateral noise when they are given some time to adapt to the noise. This finding challenges models and theories that claim that speech intelligibility in noise is invariant over time. In addition, we show that this adaptation to the noise occurs also for words processed to maintain the slow-amplitude modulations in speech (the envelope) disregarding the faster fluctuations (the temporal fine structure). This demonstrates that noise adaptation reflects an enhancement of amplitude modulation speech cues and is unaffected by temporal fine structure cues. Last, we show that cochlear implant users (N = 7; four male) show normal monaural adaptation to ipsilateral noise. Because the electrical stimulation delivered by cochlear implants is independent from the MOCR, this demonstrates that noise adaptation does not require the MOCR. We argue that noise adaptation probably reflects adaptation of the dynamic range of auditory neurons to the noise level statistics.SIGNIFICANCE STATEMENT People find it easier to understand speech in noisy environments when they are given some time to adapt to the noise. This benefit is frequently but controversially attributed to the medial olivocochlear efferent reflex enhancing the representation of speech cues in the auditory nerve. Here, we show that the adaptation to noise reflects an enhancement of the slow fluctuations in amplitude over time that are present in speech. In addition, we show that adaptation to noise for cochlear implant users is not statistically different from that for listeners with normal hearing. Because the electrical stimulation delivered by cochlear implants is independent from the medial olivocochlear efferent reflex, this demonstrates that adaptation to noise does not require this reflex.

Dual Coding of Frequency Modulation in the Ventral Cochlear Nucleus

Frequency modulation (FM) is a common acoustic feature of natural sounds and is known to play a role in robust sound source recognition. Auditory neurons show precise stimulus-synchronized discharge patterns that may be used for the representation of low-rate FM. However, it remains unclear whether this representation is based on synchronization to slow temporal envelope (ENV) cues resulting from cochlear filtering or phase locking to faster temporal fine structure (TFS) cues. To investigate the plausibility of those encoding schemes, single units of the ventral cochlear nucleus of guinea pigs of either sex were recorded in response to sine FM tones centered at the unit's best frequency (BF). The results show that, in contrast to high-BF units, for modulation depths within the receptive field, low-BF units (<4 kHz) demonstrate good phase locking to TFS. For modulation depths extending beyond the receptive field, the discharge patterns follow the ENV and fluctuate at the modulation rate. The receptive field proved to be a good predictor of the ENV responses for most primary-like and chopper units. The current in vivo data also reveal a high level of diversity in responses across unit types. TFS cues are mainly conveyed by low-frequency and primary-like units and ENV cues by chopper and onset units. The diversity of responses exhibited by cochlear nucleus neurons provides a neural basis for a dual-coding scheme of FM in the brainstem based on both ENV and TFS cues.SIGNIFICANCE STATEMENT Natural sounds, including speech, convey informative temporal modulations in frequency. Understanding how the auditory system represents those frequency modulations (FM) has important implications as robust sound source recognition depends crucially on the reception of low-rate FM cues. Here, we recorded 115 single-unit responses from the ventral cochlear nucleus in response to FM and provide the first physiological evidence of a dual-coding mechanism of FM via synchronization to temporal envelope cues and phase locking to temporal fine structure cues. We also demonstrate a diversity of neural responses with different coding specializations. These results support the dual-coding scheme proposed by psychophysicists to account for FM sensitivity in humans and provide new insights on how this might be implemented in the early stages of the auditory pathway.

Predictions of Speech Chimaera Intelligibility Using Auditory Nerve Mean-Rate and Spike-Timing Neural Cues

Perceptual studies of speech intelligibility have shown that slow variations of acoustic envelope (ENV) in a small set of frequency bands provides adequate information for good perceptual performance in quiet, whereas acoustic temporal fine-structure (TFS) cues play a supporting role in background noise. However, the implications for neural coding are prone to misinterpretation because the mean-rate neural representation can contain recovered ENV cues from cochlear filtering of TFS. We investigated ENV recovery and spike-time TFS coding using objective measures of simulated mean-rate and spike-timing neural representations of chimaeric speech, in which either the ENV or the TFS is replaced by another signal. We (a) evaluated the levels of mean-rate and spike-timing neural information for two categories of chimaeric speech, one retaining ENV cues and the other TFS; (b) examined the level of recovered ENV from cochlear filtering of TFS speech; (c) examined and quantified the contribution to recovered ENV from spike-timing cues using a lateral inhibition network (LIN); and (d) constructed linear regression models with objective measures of mean-rate and spike-timing neural cues and subjective phoneme perception scores from normal-hearing listeners. The mean-rate neural cues from the original ENV and recovered ENV partially accounted for perceptual score variability, with additional variability explained by the recovered ENV from the LIN-processed TFS speech. The best model predictions of chimaeric speech intelligibility were found when both the mean-rate and spike-timing neural cues were included, providing further evidence that spike-time coding of TFS cues is important for intelligibility when the speech envelope is degraded.

Role of Binaural Temporal Fine Structure and Envelope Cues in Cocktail-Party Listening

While conversing in a crowded social setting, a listener is often required to follow a target speech signal amid multiple competing speech signals (the so-called "cocktail party" problem). In such situations, separation of the target speech signal in azimuth from the interfering masker signals can lead to an improvement in target intelligibility, an effect known as spatial release from masking (SRM). This study assessed the contributions of two stimulus properties that vary with separation of sound sources, binaural envelope (ENV) and temporal fine structure (TFS), to SRM in normal-hearing (NH) human listeners. Target speech was presented from the front and speech maskers were either colocated with or symmetrically separated from the target in azimuth. The target and maskers were presented either as natural speech or as "noise-vocoded" speech in which the intelligibility was conveyed only by the speech ENVs from several frequency bands; the speech TFS within each band was replaced with noise carriers. The experiments were designed to preserve the spatial cues in the speech ENVs while retaining/eliminating them from the TFS. This was achieved by using the same/different noise carriers in the two ears. A phenomenological auditory-nerve model was used to verify that the interaural correlations in TFS differed across conditions, whereas the ENVs retained a high degree of correlation, as intended. Overall, the results from this study revealed that binaural TFS cues, especially for frequency regions below 1500 Hz, are critical for achieving SRM in NH listeners. Potential implications for studying SRM in hearing-impaired listeners are discussed.

SIGNIFICANCE STATEMENT

Acoustic signals received by the auditory system pass first through an array of physiologically based band-pass filters. Conceptually, at the output of each filter, there are two principal forms of temporal information: slowly varying fluctuations in the envelope (ENV) and rapidly varying fluctuations in the temporal fine structure (TFS). The importance of these two types of information in everyday listening (e.g., conversing in a noisy social situation; the "cocktail-party" problem) has not been established. This study assessed the contributions of binaural ENV and TFS cues for understanding speech in multiple-talker situations. Results suggest that, whereas the ENV cues are important for speech intelligibility, binaural TFS cues are critical for perceptually segregating the different talkers and thus for solving the cocktail party problem.

Assessing the Role of Place and Timing Cues in Coding Frequency and Amplitude Modulation as a Function of Age

Natural sounds can be characterized by their fluctuations in amplitude and frequency. Ageing may affect sensitivity to some forms of fluctuations more than others. The present study used individual differences across a wide age range (20-79 years) to test the hypothesis that slow-rate, low-carrier frequency modulation (FM) is coded by phase-locked auditory-nerve responses to temporal fine structure (TFS), whereas fast-rate FM is coded via rate-place (tonotopic) cues, based on amplitude modulation (AM) of the temporal envelope after cochlear filtering. Using a low (500 Hz) carrier frequency, diotic FM and AM detection thresholds were measured at slow (1 Hz) and fast (20 Hz) rates in 85 listeners. Frequency selectivity and TFS coding were assessed using forward masking patterns and interaural phase disparity tasks (slow dichotic FM), respectively. Comparable interaural level disparity tasks (slow and fast dichotic AM and fast dichotic FM) were measured to control for effects of binaural processing not specifically related to TFS coding. Thresholds in FM and AM tasks were correlated, even across tasks thought to use separate peripheral codes. Age was correlated with slow and fast FM thresholds in both diotic and dichotic conditions. The relationship between age and AM thresholds was generally not significant. Once accounting for AM sensitivity, only diotic slow-rate FM thresholds remained significantly correlated with age. Overall, results indicate stronger effects of age on FM than AM. However, because of similar effects for both slow and fast FM when not accounting for AM sensitivity, the effects cannot be unambiguously ascribed to TFS coding.

Do we hear what birds hear in birdsong?

Peter Marler's fascination with richness of birdsong included the notion that birds attended to some acoustic features of birdsong, likely in the time domain, which were inaccessible to human listeners. While a considerable amount is known about hearing and vocal communication in birds, how exactly birds perceive their auditory world still remains somewhat of a mystery. For sure, field and laboratory studies suggest that birds hear the spectral, gross temporal features (i.e. envelope) and perhaps syntax of birdsong much like we do. However, there is also ample anecdotal evidence that birds are consistently more sensitive than humans to at least some aspects of their song. Here we review several psychophysical studies supporting Marler's intuitions that birds have both an exquisite sensitivity to temporal fine structure and may be able to focus their auditory attention on critical acoustic details of their vocalizations. Zebra finches, Taeniopygia guttata, particularly, seem to be extremely sensitive to temporal fine structure in both synthetic stimuli and natural vocalizations. This finding, together with recent research highlighting the complexity of zebra finch vocalizations across contexts, raises interesting questions about what information zebra finches may be communicating in temporal fine structure. Together these findings show there is an acoustic richness in bird vocalizations that is available to birds but likely out of reach for human listeners. Depending on the universality of these findings, it raises questions about how we approach the study of birdsong and whether potentially significant information is routinely being encoded in the temporal fine structure of avian vocal signals.

Relation Between Cochlear Mechanics and Performance of Temporal Fine Structure-Based Tasks

This study examined whether the mechanical characteristics of the cochlea could influence individual variation in the ability to use temporal fine structure (TFS) information. Cochlear mechanical functioning was evaluated by swept-tone evoked otoacoustic emissions (OAEs), which are thought to comprise linear reflection by micromechanical impedance perturbations, such as spatial variations in the number or geometry of outer hair cells, on the basilar membrane (BM). Low-rate (2 Hz) frequency modulation detection limens (FMDLs) were measured for carrier frequency of 1000 Hz and interaural phase difference (IPD) thresholds as indices of TFS sensitivity and high-rate (16 Hz) FMDLs and amplitude modulation detection limens (AMDLs) as indices of sensitivity to non-TFS cues. Significant correlations were found among low-rate FMDLs, low-rate AMDLs, and IPD thresholds (R = 0.47-0.59). A principal component analysis was used to show a common factor that could account for 81.1, 74.1, and 62.9 % of the variance in low-rate FMDLs, low-rate AMDLs, and IPD thresholds, respectively. An OAE feature, specifically a characteristic dip around 2-2.5 kHz in OAE spectra, showed a significant correlation with the common factor (R = 0.54). High-rate FMDLs and AMDLs were correlated with each other (R = 0.56) but not with the other measures. The results can be interpreted as indicating that (1) the low-rate AMDLs, as well as the IPD thresholds and low-rate FMDLs, depend on the use of TFS information coded in neural phase locking and (2) the use of TFS information is influenced by a particular aspect of cochlear mechanics, such as mechanical irregularity along the BM.

Spectrotemporal Modulation Sensitivity as a Predictor of Speech-Reception Performance in Noise With Hearing Aids

The audiogram predicts <30% of the variance in speech-reception thresholds (SRTs) for hearing-impaired (HI) listeners fitted with individualized frequency-dependent gain. The remaining variance could reflect suprathreshold distortion in the auditory pathways or nonauditory factors such as cognitive processing. The relationship between a measure of suprathreshold auditory function—spectrotemporal modulation (STM) sensitivity—and SRTs in noise was examined for 154 HI listeners fitted with individualized frequency-specific gain. SRTs were measured for 65-dB SPL sentences presented in speech-weighted noise or four-talker babble to an individually programmed master hearing aid, with the output of an ear-simulating coupler played through insert earphones. Modulation-depth detection thresholds were measured over headphones for STM (2cycles/octave density, 4-Hz rate) applied to an 85-dB SPL, 2-kHz lowpass-filtered pink-noise carrier. SRTs were correlated with both the high-frequency (2–6 kHz) pure-tone average (HFA; R²= .31) and STM sensitivity (R²= .28). Combined with the HFA, STM sensitivity significantly improved the SRT prediction (ΔR²= .13; total R²= .44). The remaining unaccounted variance might be attributable to variability in cognitive function and other dimensions of suprathreshold distortion. STM sensitivity was most critical in predicting SRTs for listeners < 65 years old or with HFA <53 dB HL. Results are discussed in the context of previous work suggesting that STM sensitivity for low rates and low-frequency carriers is impaired by a reduced ability to use temporal fine-structure information to detect dynamic spectra. STM detection is a fast test of suprathreshold auditory function for frequencies <2 kHz that complements the HFA to predict variability in hearing-aid outcomes for speech perception in noise.

Preferred Compression Speed for Speech and Music and Its Relationship to Sensitivity to Temporal Fine Structure

Multichannel amplitude compression is widely used in hearing aids. The preferred compression speed varies across individuals. Moore (2008) suggested that reduced sensitivity to temporal fine structure (TFS) may be associated with preference for slow compression. This idea was tested using a simulated hearing aid. It was also assessed whether preferences for compression speed depend on the type of stimulus: speech or music. Twenty-two hearing-impaired subjects were tested, and the stimulated hearing aid was fitted individually using the CAM2A method. On each trial, a given segment of speech or music was presented twice. One segment was processed with fast compression and the other with slow compression, and the order was balanced across trials. The subject indicated which segment was preferred and by how much. On average, slow compression was preferred over fast compression, more so for music, but there were distinct individual differences, which were highly correlated for speech and music. Sensitivity to TFS was assessed using the difference limen for frequency at 2000 Hz and by two measures of sensitivity to interaural phase at low frequencies. The results for the difference limens for frequency, but not the measures of sensitivity to interaural phase, supported the suggestion that preference for compression speed is affected by sensitivity to TFS.

The Influence of Cochlear Mechanical Dysfunction, Temporal Processing Deficits, and Age on the Intelligibility of Audible Speech in Noise for Hearing-Impaired Listeners

The aim of this study was to assess the relative importance of cochlear mechanical dysfunction, temporal processing deficits, and age on the ability of hearing-impaired listeners to understand speech in noisy backgrounds. Sixty-eight listeners took part in the study. They were provided with linear, frequency-specific amplification to compensate for their audiometric losses, and intelligibility was assessed for speech-shaped noise (SSN) and a time-reversed two-talker masker (R2TM). Behavioral estimates of cochlear gain loss and residual compression were available from a previous study and were used as indicators of cochlear mechanical dysfunction. Temporal processing abilities were assessed using frequency modulation detection thresholds. Age, audiometric thresholds, and the difference between audiometric threshold and cochlear gain loss were also included in the analyses. Stepwise multiple linear regression models were used to assess the relative importance of the various factors for intelligibility. Results showed that (a) cochlear gain loss was unrelated to intelligibility, (b) residual cochlear compression was related to intelligibility in SSN but not in a R2TM, (c) temporal processing was strongly related to intelligibility in a R2TM and much less so in SSN, and (d) age per se impaired intelligibility. In summary, all factors affected intelligibility, but their relative importance varied across maskers.

Temporal Fine-Structure Coding and Lateralized Speech Perception in Normal-Hearing and Hearing-Impaired Listeners

This study investigated the relationship between speech perception performance in spatially complex, lateralized listening scenarios and temporal fine-structure (TFS) coding at low frequencies. Young normal-hearing (NH) and two groups of elderly hearing-impaired (HI) listeners with mild or moderate hearing loss above 1.5 kHz participated in the study. Speech reception thresholds (SRTs) were estimated in the presence of either speech-shaped noise, two-, four-, or eight-talker babble played reversed, or a nonreversed two-talker masker. Target audibility was ensured by applying individualized linear gains to the stimuli, which were presented over headphones. The target and masker streams were lateralized to the same or to opposite sides of the head by introducing 0.7-ms interaural time differences between the ears. TFS coding was assessed by measuring frequency discrimination thresholds and interaural phase difference thresholds at 250 Hz. NH listeners had clearly better SRTs than the HI listeners. However, when maskers were spatially separated from the target, the amount of SRT benefit due to binaural unmasking differed only slightly between the groups. Neither the frequency discrimination threshold nor the interaural phase difference threshold tasks showed a correlation with the SRTs or with the amount of masking release due to binaural unmasking, respectively. The results suggest that, although HI listeners with normal hearing thresholds below 1.5 kHz experienced difficulties with speech understanding in spatially complex environments, these limitations were unrelated to TFS coding abilities and were only weakly associated with a reduction in binaural-unmasking benefit for spatially separated competing sources.

Hearing impairment, cognition and speech understanding: exploratory factor analyses of a comprehensive test battery for a group of hearing aid users, the n200 study

OBJECTIVE

The aims of the current n200 study were to assess the structural relations between three classes of test variables (i.e. HEARING, COGNITION and aided speech-in-noise OUTCOMES) and to describe the theoretical implications of these relations for the Ease of Language Understanding (ELU) model.

STUDY SAMPLE

Participants were 200 hard-of-hearing hearing-aid users, with a mean age of 60.8 years. Forty-three percent were females and the mean hearing threshold in the better ear was 37.4 dB HL.

DESIGN

LEVEL1 factor analyses extracted one factor per test and/or cognitive function based on a priori conceptualizations. The more abstract LEVEL 2 factor analyses were performed separately for the three classes of test variables.

RESULTS

The HEARING test variables resulted in two LEVEL 2 factors, which we labelled SENSITIVITY and TEMPORAL FINE STRUCTURE; the COGNITIVE variables in one COGNITION factor only, and OUTCOMES in two factors, NO CONTEXT and CONTEXT. COGNITION predicted the NO CONTEXT factor to a stronger extent than the CONTEXT outcome factor. TEMPORAL FINE STRUCTURE and SENSITIVITY were associated with COGNITION and all three contributed significantly and independently to especially the NO CONTEXT outcome scores (R(2) = 0.40).

CONCLUSIONS

All LEVEL 2 factors are important theoretically as well as for clinical assessment.

Differential Group Delay of the Frequency Following Response Measured Vertically and Horizontally

The frequency following response (FFR) arises from the sustained neural activity of a population of neurons that are phase locked to periodic acoustic stimuli. Determining the source of the FFR noninvasively may be useful for understanding the function of phase locking in the auditory pathway to the temporal envelope and fine structure of sounds. The current study compared the FFR recorded with a horizontally aligned (mastoid-to-mastoid) electrode montage and a vertically aligned (forehead-to-neck) electrode montage. Unlike previous studies, envelope and fine structure latencies were derived simultaneously from the same narrowband stimuli to minimize differences in cochlear delay. Stimuli were five amplitude-modulated tones centered at 576 Hz, each with a different modulation rate, resulting in different side-band frequencies across stimulus conditions. Changes in response phase across modulation frequency and side-band frequency (group delay) were used to determine the latency of the FFR reflecting phase locking to the envelope and temporal fine structure, respectively. For the FFR reflecting phase locking to the temporal fine structure, the horizontal montage had a shorter group delay than the vertical montage, suggesting an earlier generation source within the auditory pathway. For the FFR reflecting phase locking to the envelope, group delay was longer than that for the fine structure FFR, and no significant difference in group delay was found between montages. However, it is possible that multiple sources of FFR (including the cochlear microphonic) were recorded by each montage, complicating interpretations of the group delay.

The history and future of neural modeling for cochlear implants

This special issue of Network: Computation in Neural Systems on the topic of "Computational models of the electrically stimulated auditory system" incorporates review articles spanning a wide range of approaches to modeling cochlear implant stimulation of the auditory system. The purpose of this overview paper is to provide a historical context for the different modeling endeavors and to point toward how computational modeling could play a key role in the understanding, evaluation, and improvement of cochlear implants in the future.

Auditory midbrain representation of a break in interaural correlation

The auditory peripheral system filters broadband sounds into narrowband waves and decomposes narrowband waves into quickly varying temporal fine structures (TFSs) and slowly varying envelopes. When a noise is presented binaurally (with the interaural correlation being 1), human listeners can detect a transient break in interaural correlation (BIC), which does not alter monaural inputs substantially. The central correlates of BIC are unknown. This study examined whether phase locking-based frequency-following responses (FFRs) of neuron populations in the rat auditory midbrain [inferior colliculus (IC)] to interaurally correlated steady-state narrowband noises are modulated by introduction of a BIC. The results showed that the noise-induced FFR exhibited both a TFS component (FFRTFS) and an envelope component (FFREnv), signaling the center frequency and bandwidth, respectively. Introduction of either a BIC or an interaurally correlated amplitude gap (which had the summated amplitude matched to the BIC) significantly reduced both FFRTFS and FFREnv. However, the BIC-induced FFRTFS reduction and FFREnv reduction were not correlated with the amplitude gap-induced FFRTFS reduction and FFREnv reduction, respectively. Thus, although introduction of a BIC does not affect monaural inputs, it causes a temporary reduction in sustained responses of IC neuron populations to the noise. This BIC-induced FFR reduction is not based on a simple linear summation of noise signals.

Age-group differences in speech identification despite matched audiometrically normal hearing: contributions from auditory temporal processing and cognition

Hearing loss with increasing age adversely affects the ability to understand speech, an effect that results partly from reduced audibility. The aims of this study were to establish whether aging reduces speech intelligibility for listeners with normal audiograms, and, if so, to assess the relative contributions of auditory temporal and cognitive processing. Twenty-one older normal-hearing (ONH; 60-79 years) participants with bilateral audiometric thresholds ≤ 20 dB HL at 0.125-6 kHz were matched to nine young (YNH; 18-27 years) participants in terms of mean audiograms, years of education, and performance IQ. Measures included: (1) identification of consonants in quiet and in noise that was unmodulated or modulated at 5 or 80 Hz; (2) identification of sentences in quiet and in co-located or spatially separated two-talker babble; (3) detection of modulation of the temporal envelope (TE) at frequencies 5-180 Hz; (4) monaural and binaural sensitivity to temporal fine structure (TFS); (5) various cognitive tests. Speech identification was worse for ONH than YNH participants in all types of background. This deficit was not reflected in self-ratings of hearing ability. Modulation masking release (the improvement in speech identification obtained by amplitude modulating a noise background) and spatial masking release (the benefit obtained from spatially separating masker and target speech) were not affected by age. Sensitivity to TE and TFS was lower for ONH than YNH participants, and was correlated positively with speech-in-noise (SiN) identification. Many cognitive abilities were lower for ONH than YNH participants, and generally were correlated positively with SiN identification scores. The best predictors of the intelligibility of SiN were composite measures of cognition and TFS sensitivity. These results suggest that declines in speech perception in older persons are partly caused by cognitive and perceptual changes separate from age-related changes in audiometric sensitivity.

Factors that account for inter-individual variability of lateralization performance revealed by correlations of performance among multiple psychoacoustical tasks

This study explored the source of inter-listener variability in the performance of lateralization tasks based on interaural time or level differences (ITDs or ILDs) by examining correlation of performance between pairs of multiple psychoacoustical tasks. The ITD, ILD, Time, and Level tasks were intended to measure sensitivities to ITD; ILD; temporal fine structure or envelope of the stimulus encoded by the neural phase locking; and stimulus level, respectively. Stimuli in low- and high-frequency regions were tested. The low-frequency stimulus was a harmonic complex (F 0 = 100 Hz) that was spectrally shaped for the frequency region around the 11th harmonic. The high frequency stimulus was a "transposed stimulus," which was a 4-kHz tone amplitude-modulated with a half-wave rectified 125-Hz sinusoid. The task procedures were essentially the same between the low- and high-frequency stimuli. Generally, the thresholds for pairs of ITD and ILD tasks, across cues or frequencies, exhibited significant positive correlations, suggesting a common mechanism across cues and frequencies underlying the lateralization tasks. For the high frequency stimulus, there was a significant positive correlation of performance between the ITD and Time tasks. A significant positive correlation was found also in the pair of ILD and Level tasks for the low- frequency stimulus. These results indicate that the inter-listener variability of ITD and ILD sensitivities could be accounted for partially by the variability of monaural efficiency of neural phase locking and intensity coding, respectively, depending of frequency.

Training changes processing of speech cues in older adults with hearing loss

Aging results in a loss of sensory function, and the effects of hearing impairment can be especially devastating due to reduced communication ability. Older adults with hearing loss report that speech, especially in noisy backgrounds, is uncomfortably loud yet unclear. Hearing loss results in an unbalanced neural representation of speech: the slowly-varying envelope is enhanced, dominating representation in the auditory pathway and perceptual salience at the cost of the rapidly-varying fine structure. We hypothesized that older adults with hearing loss can be trained to compensate for these changes in central auditory processing through directed attention to behaviorally-relevant speech sounds. To that end, we evaluated the effects of auditory-cognitive training in older adults (ages 55-79) with normal hearing and hearing loss. After training, the auditory training group with hearing loss experienced a reduction in the neural representation of the speech envelope presented in noise, approaching levels observed in normal hearing older adults. No changes were noted in the control group. Importantly, changes in speech processing were accompanied by improvements in speech perception. Thus, central processing deficits associated with hearing loss may be partially remediated with training, resulting in real-life benefits for everyday communication.

Effects of sensorineural hearing loss on temporal coding of narrowband and broadband signals in the auditory periphery

People with sensorineural hearing loss have substantial difficulty understanding speech under degraded listening conditions. Behavioral studies suggest that this difficulty may be caused by changes in auditory processing of the rapidly-varying temporal fine structure (TFS) of acoustic signals. In this paper, we review the presently known effects of sensorineural hearing loss on processing of TFS and slower envelope modulations in the peripheral auditory system of mammals. Cochlear damage has relatively subtle effects on phase locking by auditory-nerve fibers to the temporal structure of narrowband signals under quiet conditions. In background noise, however, sensorineural loss does substantially reduce phase locking to the TFS of pure-tone stimuli. For auditory processing of broadband stimuli, sensorineural hearing loss has been shown to severely alter the neural representation of temporal information along the tonotopic axis of the cochlea. Notably, auditory-nerve fibers innervating the high-frequency part of the cochlea grow increasingly responsive to low-frequency TFS information and less responsive to temporal information near their characteristic frequency (CF). Cochlear damage also increases the correlation of the response to TFS across fibers of varying CF, decreases the traveling-wave delay between TFS responses of fibers with different CFs, and can increase the range of temporal modulation frequencies encoded in the periphery for broadband sounds. Weaker neural coding of temporal structure in background noise and degraded coding of broadband signals along the tonotopic axis of the cochlea are expected to contribute considerably to speech perception problems in people with sensorineural hearing loss. This article is part of a Special Issue entitled "Annual Reviews 2013".