Concurrent-vowel and tone recognitions in acoustic and simulated electric hearing

Frequency importance functions in simulated bimodal cochlear-implant users with spectral holes

Frequency importance functions (FIFs) for simulated bimodal hearing were derived using sentence perception scores measured in quiet and noise. Acoustic hearing was simulated using low-pass filtering. Electric hearing was simulated using a six-channel vocoder with three input frequency ranges, resulting in overlap, meet, and gap maps, relative to the acoustic cutoff frequency. Spectral holes present in the speech spectra were created within electric stimulation by setting amplitude(s) of channels to zero. FIFs were significantly different between frequency maps. In quiet, the three FIFs were similar with gradually increasing weights with channels 5 and 6 compared to the first three channels. However, the most and least weighted channels slightly varied depending on the maps. In noise, the patterns of the three FIFs were similar to those in quiet, with steeper increasing weights with channels 5 and 6 compared to the first four channels. Thus, channels 5 and 6 contributed to speech perception the most, while channels 1 and 2 contributed the least, regardless of frequency maps. Results suggest that the contribution of cochlear implant frequency bands for bimodal speech perception depends on the degree of frequency overlap between acoustic and electric stimulation and if noise is absent or present.

The P300 Auditory Event-Related Potential May Predict Segregation of Competing Speech by Bimodal Cochlear Implant Listeners

Compared to normal-hearing (NH) listeners, cochlear implant (CI) listeners have greater difficulty segregating competing speech. Neurophysiological studies have largely investigated the neural foundations for CI listeners' speech recognition in quiet, mainly using the P300 component of event-related potentials (ERPs). P300 is closely related to cognitive processes involving auditory discrimination, selective attention, and working memory. Different from speech perception in quiet, little is known about the neurophysiological foundations for segregation of competing speech by CI listeners. In this study, ERPs were measured for a 1 vs. 2 kHz contrast in 11 Mandarin-speaking bimodal CI listeners and 11 NH listeners. Speech reception thresholds (SRTs) for a male target talker were measured in steady noise or with a male or female masker. Results showed that P300 amplitudes were significantly larger and latencies were significantly shorter for the NH than for the CI group. Similarly, SRTs were significantly better for the NH than for the CI group. Across all participants, P300 amplitude was significantly correlated with SRTs in steady noise (r = -0.65, p = 0.001) and with the competing male (r = -0.62, p = 0.002) and female maskers (r = -0.60, p = 0.003). Within the CI group, there was a significant correlation between P300 amplitude and SRTs with the male masker (r = -0.78, p = 0.005), which produced the most informational masking. The results suggest that P300 amplitude may be a clinically useful neural correlate of central auditory processing capabilities (e.g., susceptibility to informational masking) in bimodal CI patients.

Reverberation Degrades Pitch Perception but Not Mandarin Tone and Vowel Recognition of Cochlear Implant Users

OBJECTIVES

The primary goal of this study was to investigate the effects of reverberation on Mandarin tone and vowel recognition of cochlear implant (CI) users and normal-hearing (NH) listeners. To understand the performance of Mandarin tone recognition, this study also measured participants' pitch perception and the availability of temporal envelope cues in reverberation.

DESIGN

Fifteen CI users and nine NH listeners, all Mandarin speakers, were asked to recognize Mandarin single-vowels produced in four lexical tones and rank harmonic complex tones in pitch with different reverberation times (RTs) from 0 to 1 second. Virtual acoustic techniques were used to simulate rooms with different degrees of reverberation. Vowel duration and correlation between amplitude envelope and fundamental frequency (F0) contour were analyzed for different tones as a function of the RT.

RESULTS

Vowel durations of different tones significantly increased with longer RTs. Amplitude-F0 correlation remained similar for the falling Tone 4 but greatly decreased for the other tones in reverberation. NH listeners had robust pitch-ranking, tone recognition, and vowel recognition performance as the RT increased. Reverberation significantly degraded CI users' pitch-ranking thresholds but did not significantly affect the overall scores of tone and vowel recognition with CIs. Detailed analyses of tone confusion matrices showed that CI users reduced the flat Tone-1 responses but increased the falling Tone-4 responses in reverberation, possibly due to the falling amplitude envelope of late reflections after the original vowel segment. CI users' tone recognition scores were not correlated with their pitch-ranking thresholds.

CONCLUSIONS

NH listeners can reliably recognize Mandarin tones in reverberation using salient pitch cues from spectral and temporal fine structures. However, CI users have poorer pitch perception using F0-related amplitude modulations that are reduced in reverberation. Reverberation distorts speech amplitude envelopes, which affect the distribution of tone responses but not the accuracy of tone recognition with CIs. Recognition of vowels with stationary formant trajectories is not affected by reverberation for both NH listeners and CI users, regardless of the available spectral resolution. Future studies should test how the relatively stable vowel and tone recognition may contribute to sentence recognition in reverberation of Mandarin-speaking CI users.

Meta-Analysis on the Identification of Linguistic and Emotional Prosody in Cochlear Implant Users and Vocoder Simulations

OBJECTIVES

This study quantitatively assesses how cochlear implants (CIs) and vocoder simulations of CIs influence the identification of linguistic and emotional prosody in nontonal languages. By means of meta-analysis, it was explored how accurately CI users and normal-hearing (NH) listeners of vocoder simulations (henceforth: simulation listeners) identify prosody compared with NH listeners of unprocessed speech (henceforth: NH listeners), whether this effect of electric hearing differs between CI users and simulation listeners, and whether the effect of electric hearing is influenced by the type of prosody that listeners identify or by the availability of specific cues in the speech signal.

DESIGN

Records were found by searching the PubMed Central, Web of Science, Scopus, Science Direct, and PsycINFO databases (January 2018) using the search terms "cochlear implant prosody" and "vocoder prosody." Records (published in English) were included that reported results of experimental studies comparing CI users' and/or simulation listeners' identification of linguistic and/or emotional prosody in nontonal languages to that of NH listeners (all ages included). Studies that met the inclusion criteria were subjected to a multilevel random-effects meta-analysis.

RESULTS

Sixty-four studies reported in 28 records were included in the meta-analysis. The analysis indicated that CI users and simulation listeners were less accurate in correctly identifying linguistic and emotional prosody compared with NH listeners, that the identification of emotional prosody was more strongly compromised by the electric hearing speech signal than linguistic prosody was, and that the low quality of transmission of fundamental frequency (f0) through the electric hearing speech signal was the main cause of compromised prosody identification in CI users and simulation listeners. Moreover, results indicated that the accuracy with which CI users and simulation listeners identified linguistic and emotional prosody was comparable, suggesting that vocoder simulations with carefully selected parameters can provide a good estimate of how prosody may be identified by CI users.

CONCLUSIONS

The meta-analysis revealed a robust negative effect of electric hearing, where CIs and vocoder simulations had a similar negative influence on the identification of linguistic and emotional prosody, which seemed mainly due to inadequate transmission of f0 cues through the degraded electric hearing speech signal of CIs and vocoder simulations.

Segregation of competing speech in adults and children with normal hearing and in children with cochlear implants

Children with normal hearing (CNH) have greater difficulty segregating competing speech than do adults with normal hearing (ANH). Children with cochlear implants (CCI) have greater difficulty segregating competing speech than do CNH. In the present study, speech reception thresholds (SRTs) in competing speech were measured in Chinese Mandarin-speaking ANH, CNH, and CCIs. Target sentences were produced by a male Mandarin-speaking talker. Maskers were time-forward or -reversed sentences produced by a native Mandarin-speaking male (different from the target) or female or a non-native English-speaking male. The SRTs were lowest (best) for the ANH group, followed by the CNH and CCI groups. The masking release (MR) was comparable between the ANH and CNH group, but much poorer in the CCI group. The temporal properties differed between the native and non-native maskers and between forward and reversed speech. The temporal properties of the maskers were significantly associated with the SRTs for the CCI and CNH groups but not for the ANH group. Whereas the temporal properties of the maskers were significantly associated with the MR for all three groups, the association was stronger for the CCI and CNH groups than for the ANH group.

Masking Effects in the Perception of Multiple Simultaneous Talkers in Normal-Hearing and Cochlear Implant Listeners

For normal-hearing (NH) listeners, monaural factors, such as voice pitch cues, may play an important role in the segregation of speech signals in multitalker environments. However, cochlear implant (CI) users experience difficulties in segregating speech signals in multitalker environments in part due to the coarse spectral resolution. The present study examined how the vocal characteristics of the target and masking talkers influence listeners’ ability to extract information from a target phrase in a multitalker environment. Speech recognition thresholds (SRTs) were measured with one, two, or four masker talkers for different combinations of target-masker vocal characteristics in 10 adult Mandarin-speaking NH listeners and 12 adult Mandarin-speaking CI users. The results showed that CI users performed significantly poorer than NH listeners in the presence of competing talkers. As the number of masker talkers increased, the mean SRTs significantly worsened from –22.0 dB to –5.2 dB for NH listeners but significantly improved from 5.9 dB to 2.8 dB for CI users. The results suggest that the flattened peaks and valleys with increased numbers of competing talkers may reduce NH listeners’ ability to use dips in the spectral and temporal envelopes that allow for “glimpses” of the target speech. However, the flattened temporal envelope of the resultant masker signals may be less disruptive to the amplitude contour of the target speech, which is important for Mandarin-speaking CI users’ lexical tone recognition. The amount of masking release was further estimated by comparing SRTs between the same-sex maskers and the different-sex maskers. There was a large amount of masking release in NH adults (12 dB) and a small but significant amount of masking release in CI adults (2 dB). These results suggest that adult CI users may significantly benefit from voice pitch differences between target and masker speech.

Mandarin Tone and Vowel Recognition in Cochlear Implant Users: Effects of Talker Variability and Bimodal Hearing

OBJECTIVES

For cochlear implant (CI) users with residual low-frequency acoustic hearing in the nonimplanted ear, bimodal hearing combining the use of a CI and a contralateral hearing aid (HA) may provide more salient talker voice cues than CI alone to handle the variability of talker identity across trials. This study tested the effects of talker variability, bimodal hearing, and their interaction on response accuracy and time of CI users' Mandarin tone, vowel, and syllable recognition (i.e., combined Mandarin tone and vowel recognition in this study).

DESIGN

Fifteen prelingually deafened native Mandarin-speaking CI users (at age 20 or lower) participated in this study. Four talkers each produced six Mandarin single-vowel syllables in four lexical tones. The stimuli were presented in quiet via a single loudspeaker. To study the effects of talker variability, Mandarin tone, vowel, and syllable recognition was tested in two presentation conditions: with stimuli blocked according to talker (blocked-talker condition) or mixed across talkers from trial to trial (mixed-talker condition). To explore the effects of bimodal hearing, two processor conditions were tested: CI alone or CI + HA. The cumulative response time was recorded as an indirect indicator of the cognitive load or listening effort in each condition. The correlations were computed between demographic/hearing factors (e.g., hearing thresholds in the nonimplanted ear) and bimodal performance/benefits (where bimodal benefits refer to the performance differences between CI alone and CI + HA).

RESULTS

Mandarin tone recognition with both CI alone and CI + HA was significantly poorer in the mixed-talker condition than in the blocked-talker condition, while vowel recognition was comparable in the two presentation conditions. Bimodal hearing significantly improved Mandarin tone recognition but not vowel recognition. Mandarin syllable recognition was significantly affected by both talker variability and bimodal hearing. The cumulative response time significantly reduced with CI + HA compared with CI alone, but remained invariant with respect to talker variability. There was no interaction between talker variability and bimodal hearing for any performance measure adopted in this study. Correlation analyses revealed that the bimodal performance and benefits in Mandarin tone, vowel, and syllable recognition could not be predicted by the hearing thresholds in the nonimplanted ear or by the demographic factors of the participants.

CONCLUSIONS

Talker variability from trial to trial significantly degraded Mandarin tone and syllable recognition performance in both the CI alone and CI + HA conditions. While bimodal hearing did not reduce the talker variability effects on Mandarin tone and syllable recognition, generally better Mandarin tone and syllable recognition performance with shorter response time (an indicator of less listening effort) was observed when a contralateral HA was used in conjunction with the CI. On the other hand, vowel recognition was not significantly affected by either talker variability or bimodal hearing, because ceiling effects could not be counted out of the vowel recognition results.

Auditory implant research at the House Ear Institute 1989-2013

The House Ear Institute (HEI) had a long and distinguished history of auditory implant innovation and development. Early clinical innovations include being one of the first cochlear implant (CI) centers, being the first center to implant a child with a cochlear implant in the US, developing the auditory brainstem implant, and developing multiple surgical approaches and tools for Otology. This paper reviews the second stage of auditory implant research at House - in-depth basic research on perceptual capabilities and signal processing for both cochlear implants and auditory brainstem implants. Psychophysical studies characterized the loudness and temporal perceptual properties of electrical stimulation as a function of electrical parameters. Speech studies with the noise-band vocoder showed that only four bands of tonotopically arrayed information were sufficient for speech recognition, and that most implant users were receiving the equivalent of 8-10 bands of information. The noise-band vocoder allowed us to evaluate the effects of the manipulation of the number of bands, the alignment of the bands with the original tonotopic map, and distortions in the tonotopic mapping, including holes in the neural representation. Stimulation pulse rate was shown to have only a small effect on speech recognition. Electric fields were manipulated in position and sharpness, showing the potential benefit of improved tonotopic selectivity. Auditory training shows great promise for improving speech recognition for all patients. And the Auditory Brainstem Implant was developed and improved and its application expanded to new populations. Overall, the last 25 years of research at HEI helped increase the basic scientific understanding of electrical stimulation of hearing and contributed to the improved outcomes for patients with the CI and ABI devices. This article is part of a Special Issue entitled .

Identification and multiplicity of double vowels in cochlear implant users

PURPOSE

The present study examined cochlear implant (CI) users' perception of vowels presented concurrently (i.e., double vowels) to further our understanding of auditory grouping in electric hearing.

METHOD

Identification of double vowels and single vowels was measured with 10 CI subjects. Fundamental frequencies (F0s) of vowels were either 100 + 100 Hz or 100 + 300 Hz. Vowels were presented either synchronously or with a time delay. In "Double" sessions, subjects were given only double vowels. In "Double + Single" sessions, while double and single vowels were presented, subjects reported the number and identity of the vowel(s). In addition to clinical settings, stimuli were delivered via an experimental method that interleaved pulse streams of two vowels.

RESULTS

Although the time delay between vowels had a large effect on identification, the effect of change in fundamental frequency (ΔF0) was modest. Enumeration was poor in general, and identification of synchronous vowels was above chance in only the Double sessions with a priori knowledge about presentation. Interleaved presentation of vowel streams provided no benefit for identification and a marginal benefit for enumeration.

CONCLUSIONS

The results demonstrate the importance of episodic context for CI users. Unreliable perception of multiplicity observed in the present results suggests that auditory grouping in CIs may be driven by a schema-based process.

Rate and onset cues can improve cochlear implant synthetic vowel recognition in noise

Understanding speech-in-noise is difficult for most cochlear implant (CI) users. Speech-in-noise segregation cues are well understood for acoustic hearing but not for electric hearing. This study investigated the effects of stimulation rate and onset delay on synthetic vowel-in-noise recognition in CI subjects. In experiment I, synthetic vowels were presented at 50, 145, or 795 pulse/s and noise at the same three rates, yielding nine combinations. Recognition improved significantly if the noise had a lower rate than the vowel, suggesting that listeners can use temporal gaps in the noise to detect a synthetic vowel. This hypothesis is supported by accurate prediction of synthetic vowel recognition using a temporal integration window model. Using lower rates a similar trend was observed in normal hearing subjects. Experiment II found that for CI subjects, a vowel onset delay improved performance if the noise had a lower or higher rate than the synthetic vowel. These results show that differing rates or onset times can improve synthetic vowel-in-noise recognition, indicating a need to develop speech processing strategies that encode or emphasize these cues.

Using Zebra-speech to study sequential and simultaneous speech segregation in a cochlear-implant simulation

Previous studies have suggested that cochlear implant users may have particular difficulties exploiting opportunities to glimpse clear segments of a target speech signal in the presence of a fluctuating masker. Although it has been proposed that this difficulty is associated with a deficit in linking the glimpsed segments across time, the details of this mechanism are yet to be explained. The present study introduces a method called Zebra-speech developed to investigate the relative contribution of simultaneous and sequential segregation mechanisms in concurrent speech perception, using a noise-band vocoder to simulate cochlear implants. One experiment showed that the saliency of the difference between the target and the masker is a key factor for Zebra-speech perception, as it is for sequential segregation. Furthermore, forward masking played little or no role, confirming that intelligibility was not limited by energetic masking but by across-time linkage abilities. In another experiment, a binaural cue was used to distinguish the target and the masker. It showed that the relative contribution of simultaneous and sequential segregation depended on the spectral resolution, with listeners relying more on sequential segregation when the spectral resolution was reduced. The potential of Zebra-speech as a segregation enhancement strategy for cochlear implants is discussed.

Perception of Mandarin Chinese with cochlear implants using enhanced temporal pitch cues

A cochlear implant (CI) signal processing strategy named F0 modulation (F0mod) was compared with the advanced combination encoder (ACE) strategy in a group of four post-lingually deafened Mandarin Chinese speaking CI listeners. F0 provides an enhanced temporal pitch cue by amplitude modulating the multichannel electrical stimulation pattern at the fundamental frequency (F0) of the incoming speech signal. Word and sentence recognition tests were carried out in quiet and in noise. The responses for the word-recognition test were further segmented into phoneme and tone scores. Off-line implementations of ACE and F0mod were used, and electrical stimulation patterns were directly streamed to the CI subject's implant. To focus on the feasibility of enhanced temporal cues for tonal language perception, idealized F0 information that was extracted from speech tokens in quiet was used in the F0mod processing of speech-in-noise mixtures. The results indicated significantly better lexical tone perception with the F0mod strategy than with ACE for the male voice (p<0.05). No significant differences in sentence recognition were found between F0mod and ACE.

Effects of spectral shifting on speech perception in noise

The present study used eight normal-hearing (NH) subjects, listening to acoustic cochlear implant (CI) simulations, to examine the effects of spectral shifting on speech recognition in noise. Speech recognition was measured using spectrally matched and shifted speech (vowels, consonants, and IEEE sentences), generated by 8-channel, sine-wave vocoder. Measurements were made in quiet and in noise (speech-shaped static noise and speech-babble at 5 dB signal-to-noise ratio). One spectral match condition and four spectral shift conditions were investigated: 2 mm, 3 mm, and 4 mm linear shift, and 3 mm shift with compression, in terms of cochlear distance. Results showed that speech recognition scores dropped because of noise and spectral shifting, and that the interactive effects of spectral shifting and background conditions depended on the degree/type of spectral shift, background conditions, and the speech test materials. There was no significant interaction between spectral shifting and two noise conditions for all speech test materials. However, significant interactions between linear spectral shifts and all background conditions were found in sentence recognition; significant interactions between spectral shift types and all background conditions were found in vowel recognition. Overall, the results suggest that tonotopic mismatch may affect performance of CI users in complex listening environments.

Concurrent-vowel and tone recognition by Mandarin-speaking cochlear implant users

In Mandarin Chinese, tonal patterns are lexically meaningful. In a multi-talker environment, competing tones may create interference in addition to competing vowels and consonants. The present study measured Mandarin-speaking cochlear implant (CI) users' ability to recognize concurrent vowels, tones, and syllables in a concurrent-syllable recognition test. Concurrent syllables were constructed by summing either one Chinese syllable each from one male and one female talker or two syllables from the same male talker. Each talker produced 16 different syllables (4 vowels combined with 4 tones); all syllables were normalized to have the same overall duration and amplitude. Both single- and concurrent-syllable recognition were measured in 4 adolescent and 4 adult CI subjects, using their clinically assigned speech processors. The results showed no significant difference in performance between the adolescent and adult CI subjects. With single syllables, mean vowel recognition was 90% correct, while tone and syllable recognition were only 63% and 57% correct, respectively. With concurrent syllables, vowel, tone, and syllable recognition scores dropped by 40-60 percentage points. Concurrent-syllable performance was significantly correlated with single-syllable performance. Concurrent-vowel and syllable recognition were not significantly different between the same- and different-talker conditions, while concurrent-tone recognition was significantly better with the same-talker condition. Vowel and tone recognition were better when concurrent syllables contained the same vowels or tones, respectively. Across the different vowel pairs, tone recognition was less variable than vowel recognition; across the different tone pairs, vowel recognition was less variable than tone recognition. The present results suggest that interference between concurrent tones may contribute to Mandarin-speaking CI users' susceptibility to competing-talker backgrounds.