Median-plane sound localization as a function of the number of spectral channels using a channel vocoder

How to vocode: Using channel vocoders for cochlear-implant research

The channel vocoder has become a useful tool to understand the impact of specific forms of auditory degradation-particularly the spectral and temporal degradation that reflect cochlear-implant processing. Vocoders have many parameters that allow researchers to answer questions about cochlear-implant processing in ways that overcome some logistical complications of controlling for factors in individual cochlear implant users. However, there is such a large variety in the implementation of vocoders that the term "vocoder" is not specific enough to describe the signal processing used in these experiments. Misunderstanding vocoder parameters can result in experimental confounds or unexpected stimulus distortions. This paper highlights the signal processing parameters that should be specified when describing vocoder construction. The paper also provides guidance on how to determine vocoder parameters within perception experiments, given the experimenter's goals and research questions, to avoid common signal processing mistakes. Throughout, we will assume that experimenters are interested in vocoders with the specific goal of better understanding cochlear implants.

Interaural Place-of-Stimulation Mismatch Estimates Using CT Scans and Binaural Perception, But Not Pitch, Are Consistent in Cochlear-Implant Users

Bilateral cochlear implants (BI-CIs) or a CI for single-sided deafness (SSD-CI; one normally functioning acoustic ear) can partially restore spatial-hearing abilities, including sound localization and speech understanding in noise. For these populations, however, interaural place-of-stimulation mismatch can occur and thus diminish binaural sensitivity that relies on interaurally frequency-matched neurons. This study examined whether plasticity-reorganization of central neural pathways over time-can compensate for peripheral interaural place mismatch. We hypothesized differential plasticity across two systems: none for binaural processing but adaptation for pitch perception toward frequencies delivered by the specific electrodes. Interaural place mismatch was evaluated in 19 BI-CI and 23 SSD-CI human subjects (both sexes) using binaural processing (interaural-time-difference discrimination with simultaneous bilateral stimulation), pitch perception (pitch ranking for single electrodes or acoustic tones with sequential bilateral stimulation), and physical electrode-location estimates from computed-tomography (CT) scans. On average, CT scans revealed relatively little BI-CI interaural place mismatch (26° insertion-angle mismatch) but a relatively large SSD-CI mismatch, particularly at low frequencies (166° for an electrode tuned to 300 Hz, decreasing to 14° at 7000 Hz). For BI-CI subjects, the three metrics were in agreement because there was little mismatch. For SSD-CI subjects, binaural and CT measurements were in agreement, suggesting little binaural-system plasticity induced by mismatch. The pitch measurements disagreed with binaural and CT measurements, suggesting place-pitch plasticity or a procedural bias. These results suggest that reducing interaural place mismatch and potentially improving binaural processing by reprogramming the CI frequency allocation would be better done using CT-scan than pitch information.SIGNIFICANCE STATEMENT Electrode-array placement for cochlear implants (bionic prostheses that partially restore hearing) does not explicitly align neural representations of frequency information. The resulting interaural place-of-stimulation mismatch can diminish spatial-hearing abilities. In this study, adults with two cochlear implants showed reasonable interaural alignment, whereas those with one cochlear implant but normal hearing in the other ear often showed mismatch. In cases of mismatch, binaural sensitivity was best when the same cochlear locations were stimulated in both ears, suggesting that binaural brainstem pathways do not experience plasticity to compensate for mismatch. In contrast, interaurally pitch-matched electrodes deviated from cochlear-location estimates and did not optimize binaural sensitivity. Clinical correction of interaural place mismatch using binaural or computed-tomography (but not pitch) information may improve spatial-hearing benefits.

Temporal quantization deteriorates the discrimination of interaural time differences

Cochlear implants (CIs) often replace acoustic temporal fine structure by a fixed-rate pulse train. If the pulse timing is arbitrary (that is, not based on the phase information of the acoustic signal), temporal information is quantized by the pulse period. This temporal quantization is probably imperceptible with current clinical devices. However, it could result in large temporal jitter for strategies that aim to improve bilateral and bimodal CI users' perception of interaural time differences (ITDs), such as envelope enhancement. In an experiment with 16 normal-hearing listeners, it is shown that such jitter could deteriorate ITD perception for temporal quantization that corresponds to the often-used stimulation rate of 900 pulses per second (pps): the just-noticeable difference in ITD with quantization was 177 μs as compared to 129 μs without quantization. For smaller quantization step sizes, no significant deterioration of ITD perception was found. In conclusion, the binaural system can only average out the effect of temporal quantization to some extent, such that pulse timing should be well-considered. As this psychophysical procedure was somewhat unconventional, different procedural parameters were compared by simulating a number of commonly used two-down one-up adaptive procedures in Appendix B.

Sound Localization in Real-Time Vocoded Cochlear-Implant Simulations With Normal-Hearing Listeners

Bilateral cochlear-implant (CI) users and single-sided deaf listeners with a CI are less effective at localizing sounds than normal-hearing (NH) listeners. This performance gap is due to the degradation of binaural and monaural sound localization cues, caused by a combination of device-related and patient-related issues. In this study, we targeted the device-related issues by measuring sound localization performance of 11 NH listeners, listening to free-field stimuli processed by a real-time CI vocoder. The use of a real-time vocoder is a new approach, which enables testing in a free-field environment. For the NH listening condition, all listeners accurately and precisely localized sounds according to a linear stimulus–response relationship with an optimal gain and a minimal bias both in the azimuth and in the elevation directions. In contrast, when listening with bilateral real-time vocoders, listeners tended to orient either to the left or to the right in azimuth and were unable to determine sound source elevation. When listening with an NH ear and a unilateral vocoder, localization was impoverished on the vocoder side but improved toward the NH side. Localization performance was also reflected by systematic variations in reaction times across listening conditions. We conclude that perturbation of interaural temporal cues, reduction of interaural level cues, and removal of spectral pinna cues by the vocoder impairs sound localization. Listeners seem to ignore cues that were made unreliable by the vocoder, leading to acute reweighting of available localization cues. We discuss how current CI processors prevent CI users from localizing sounds in everyday environments.

Effect of channel separation and interaural mismatch on fusion and lateralization in normal-hearing and cochlear-implant listeners

Bilateral cochlear implantation has provided access to some of the benefits of binaural hearing enjoyed by normal-hearing (NH) listeners. However, a gap in performance still exists between the two populations. Single-channel stimulation studies have shown that interaural place-of-stimulation mismatch (IPM) due to differences in implantation depth leads to decreased binaural fusion and lateralization of interaural time and level differences (ITDs and ILDs, respectively). While single-channel studies are informative, multi-channel stimulation is needed for good speech understanding with cochlear implants (CIs). Some multi-channel studies have shown that channel interaction due to current spread can affect ITD sensitivity. In this work, we studied the effect of IPM and channel spacing, along with their potential interaction, on binaural fusion and ITD/ILD lateralization. Experiments were conducted in adult NH listeners and CI listeners with a history of acoustic hearing. Results showed that IPM reduced the range of lateralization for ITDs but not ILDs. CI listeners were more likely to report a fused percept in the presence of IPM with multi-channel stimulation than NH listeners. However, no effect of channel spacing was found. These results suggest that IPM should be accounted for in clinical mapping practices in order to maximize bilateral CI benefits.

Binaural unmasking with temporal envelope and fine structure in listeners with cochlear implants

For normal-hearing (NH) listeners, interaural information in both temporal envelope and temporal fine structure contribute to binaural unmasking of target signals in background noise; however, in many conditions low-frequency interaural information in temporal fine structure produces greater binaural unmasking. For bilateral cochlear-implant (CI) listeners, interaural information in temporal envelope contributes to binaural unmasking; however, the effect of encoding temporal fine structure information in electrical pulse timing (PT) is not fully understood. In this study, diotic and dichotic signal detection thresholds were measured in CI listeners using bilaterally synchronized single-electrode stimulation for conditions in which the temporal envelope was presented without temporal fine structure encoded (constant-rate pulses) or with temporal fine structure encoded (pulses timed to peaks of the temporal fine structure). CI listeners showed greater binaural unmasking at 125 pps with temporal fine structure encoded than without. There was no significant effect of encoding temporal fine structure at 250 pps. A similar pattern of performance was shown by NH listeners presented with acoustic pulse trains designed to simulate CI stimulation. The results suggest a trade-off across low rates between interaural information obtained from temporal envelope and that obtained from temporal fine structure encoded in PT.

Interaural Time-Difference Discrimination as a Measure of Place of Stimulation for Cochlear-Implant Users With Single-Sided Deafness

Current clinical practice in programming a cochlear implant (CI) for individuals with single-sided deafness (SSD) is to maximize the transmission of speech information via the implant, with the implicit assumption that this will also result in improved spatial-hearing abilities. However, binaural sensitivity is reduced by interaural place-of-stimulation mismatch, a likely occurrence with a standard CI frequency-to-electrode allocation table (FAT). As a step toward reducing interaural mismatch, this study investigated whether a test of interaural-time-difference (ITD) discrimination could be used to estimate the acoustic frequency yielding the best place match for a given CI electrode. ITD-discrimination performance was measured by presenting 300-ms bursts of 100-pulses-per-second electrical pulse trains to a single CI electrode and band-limited pulse trains with variable carrier frequencies to the acoustic ear. Listeners discriminated between two reference intervals (four bursts each with constant ITD) and a moving target interval (four bursts with variable ITD). For 17 out of the 26 electrodes tested across eight listeners, the function describing the relationship between ITD-discrimination performance and carrier frequency had a discernable peak where listeners achieved 70% to 100% performance. On average, this peak occurred 1.15 octaves above the CI manufacturer’s default FAT. ITD discrimination shows promise as a method of estimating the cochlear place of stimulation for a given electrode, thereby providing information to optimize the FAT for SSD-CI listeners.

Encoding a Melody Using Only Temporal Information for Cochlear-Implant and Normal-Hearing Listeners

One way to provide pitch information to cochlear implant users is through amplitude-modulation rate. It is currently unknown whether amplitude-modulation rate can provide cochlear implant users with pitch information adequate for perceiving melodic information. In the present study, the notes of a song were encoded via amplitude-modulation rate of pulse trains on single electrodes at the apex or middle of long electrode arrays. The melody of the song was either physically correct or modified by compression or expansion. Nine cochlear implant users rated the extent to which the song was out of tune in the different conditions. Cochlear implant users on average did not show sensitivity to melody compression or expansion regardless of place of stimulation. These results were found despite the fact that three of the cochlear implant users showed the expected sensitivity to melody compression and expansion with the same task using acoustic pure tones in a contralateral acoustic ear. Normal-hearing listeners showed an inconsistent and weak effect of melody compression and expansion when the notes of the song were encoded with acoustic pulse rate. The results suggest that amplitude-modulation rate provides insufficient access to melodic information for cochlear-implant and normal-hearing listeners.

Modeling the Effects of Sensorineural Hearing Loss on Sound Localization in the Median Plane

Listeners use monaural spectral cues to localize sound sources in sagittal planes (along the up-down and front-back directions). How sensorineural hearing loss affects the salience of monaural spectral cues is unclear. To simulate the effects of outer-hair-cell (OHC) dysfunction and the contribution of different auditory-nerve fiber types on localization performance, we incorporated a nonlinear model of the auditory periphery into a model of sagittal-plane sound localization for normal-hearing listeners. The localization model was first evaluated in its ability to predict the effects of spectral cue modifications for normal-hearing listeners. Then, we used it to simulate various degrees of OHC dysfunction applied to different types of auditory-nerve fibers. Predicted localization performance was hardly affected by mild OHC dysfunction but was strongly degraded in conditions involving severe and complete OHC dysfunction. These predictions resemble the usually observed degradation in localization performance induced by sensorineural hearing loss. Predicted localization performance was best when preserving fibers with medium spontaneous rates, which is particularly important in view of noise-induced hearing loss associated with degeneration of this fiber type. On average across listeners, predicted localization performance was strongly related to level discrimination sensitivity of auditory-nerve fibers, indicating an essential role of this coding property for localization accuracy in sagittal planes.

Binaural release from masking with single- and multi-electrode stimulation in children with cochlear implants

Cochlear implants (CIs) provide children with access to speech information from a young age. Despite bilateral cochlear implantation becoming common, use of spatial cues in free field is smaller than in normal-hearing children. Clinically fit CIs are not synchronized across the ears; thus binaural experiments must utilize research processors that can control binaural cues with precision. Research to date has used single pairs of electrodes, which is insufficient for representing speech. Little is known about how children with bilateral CIs process binaural information with multi-electrode stimulation. Toward the goal of improving binaural unmasking of speech, this study evaluated binaural unmasking with multi- and single-electrode stimulation. Results showed that performance with multi-electrode stimulation was similar to the best performance with single-electrode stimulation. This was similar to the pattern of performance shown by normal-hearing adults when presented an acoustic CI simulation. Diotic and dichotic signal detection thresholds of the children with CIs were similar to those of normal-hearing children listening to a CI simulation. The magnitude of binaural unmasking was not related to whether the children with CIs had good interaural time difference sensitivity. Results support the potential for benefits from binaural hearing and speech unmasking in children with bilateral CIs.

Effects of Adaptation Rate and Noise Suppression on the Intelligibility of Compressed-Envelope Based Speech

Temporal envelope is the primary acoustic cue used in most cochlear implant (CI) speech processors to elicit speech perception for patients fitted with CI devices. Envelope compression narrows down envelope dynamic range and accordingly degrades speech understanding abilities of CI users, especially under challenging listening conditions (e.g., in noise). A new adaptive envelope compression (AEC) strategy was proposed recently, which in contrast to the traditional static envelope compression, is effective at enhancing the modulation depth of envelope waveform by making best use of its dynamic range and thus improving the intelligibility of envelope-based speech. The present study further explored the effect of adaptation rate in envelope compression on the intelligibility of compressed-envelope based speech. Moreover, since noise reduction is another essential unit in modern CI systems, the compatibility of AEC and noise reduction was also investigated. In this study, listening experiments were carried out by presenting vocoded sentences to normal hearing listeners for recognition. Experimental results demonstrated that the adaptation rate in envelope compression had a notable effect on the speech intelligibility performance of the AEC strategy. By specifying a suitable adaptation rate, speech intelligibility could be enhanced significantly in noise compared to when using static envelope compression. Moreover, results confirmed that the AEC strategy was suitable for combining with noise reduction to improve the intelligibility of envelope-based speech in noise.

Comparing sound localization deficits in bilateral cochlear-implant users and vocoder simulations with normal-hearing listeners

Bilateral cochlear-implant (BiCI) users are less accurate at localizing free-field (FF) sound sources than normal-hearing (NH) listeners. This performance gap is not well understood but is likely due to a combination of compromises in acoustic signal representation by the two independent speech processors and neural degradation of auditory pathways associated with a patient's hearing loss. To exclusively investigate the effect of CI speech encoding on horizontal-plane sound localization, the present study measured sound localization performance in NH subjects listening to vocoder processed and nonvocoded virtual acoustic space (VAS) stimuli. Various aspects of BiCI stimulation such as independently functioning devices, variable across-ear channel selection, and pulsatile stimulation were simulated using uncorrelated noise (Nu), correlated noise (N0), or Gaussian-enveloped tone (GET) carriers during vocoder processing. Additionally, FF sound localization in BiCI users was measured in the same testing environment for comparison. Distinct response patterns across azimuthal locations were evident for both listener groups and were analyzed using a multilevel regression analysis. Simulated implant speech encoding, regardless of carrier, was detrimental to NH localization and the GET vocoder best simulated BiCI FF performance in NH listeners. Overall, the detrimental effect of vocoder processing on NH performance suggests that sound localization deficits may persist even for BiCI patients who have minimal neural degradation associated with their hearing loss and indicates that CI speech encoding plays a significant role in the sound localization deficits experienced by BiCI users.

Modeling sound-source localization in sagittal planes for human listeners

Monaural spectral features are important for human sound-source localization in sagittal planes, including front-back discrimination and elevation perception. These directional features result from the acoustic filtering of incoming sounds by the listener's morphology and are described by listener-specific head-related transfer functions (HRTFs). This article proposes a probabilistic, functional model of sagittal-plane localization that is based on human listeners' HRTFs. The model approximates spectral auditory processing, accounts for acoustic and non-acoustic listener specificity, allows for predictions beyond the median plane, and directly predicts psychoacoustic measures of localization performance. The predictive power of the listener-specific modeling approach was verified under various experimental conditions: The model predicted effects on localization performance of band limitation, spectral warping, non-individualized HRTFs, spectral resolution, spectral ripples, and high-frequency attenuation in speech. The functionalities of vital model components were evaluated and discussed in detail. Positive spectral gradient extraction, sensorimotor mapping, and binaural weighting of monaural spatial information were addressed in particular. Potential applications of the model include predictions of psychophysical effects, for instance, in the context of virtual acoustics or hearing assistive devices.

Internalized elevation perception of simple stimuli in cochlear-implant and normal-hearing listeners

In normal-hearing (NH) listeners, elevation perception is produced by the spectral cues imposed by the pinna, head, and torso. Elevation perception in cochlear-implant (CI) listeners appears to be non-existent; this may be a result of poorly encoded spectral cues. In this study, an analog of elevation perception was investigated by having 15 CI and 8 NH listeners report the intracranial location of spectrally simple signals (single-electrode or bandlimited acoustic stimuli, respectively) in both horizontal and vertical dimensions. Thirteen CI listeners and all of the NH listeners showed an association between place of stimulation (i.e., stimulus frequency) and perceived elevation, generally responding with higher elevations for more basal stimulation. This association persisted in the presence of a randomized temporal pitch, suggesting that listeners were not associating pitch with elevation. These data provide evidence that CI listeners might perceive changes in elevation if they were presented stimuli with sufficiently salient elevation cues.

Acoustic and non-acoustic factors in modeling listener-specific performance of sagittal-plane sound localization

The ability of sound-source localization in sagittal planes (along the top-down and front-back dimension) varies considerably across listeners. The directional acoustic spectral features, described by head-related transfer functions (HRTFs), also vary considerably across listeners, a consequence of the listener-specific shape of the ears. It is not clear whether the differences in localization ability result from differences in the encoding of directional information provided by the HRTFs, i.e., an acoustic factor, or from differences in auditory processing of those cues (e.g., spectral-shape sensitivity), i.e., non-acoustic factors. We addressed this issue by analyzing the listener-specific localization ability in terms of localization performance. Directional responses to spatially distributed broadband stimuli from 18 listeners were used. A model of sagittal-plane localization was fit individually for each listener by considering the actual localization performance, the listener-specific HRTFs representing the acoustic factor, and an uncertainty parameter representing the non-acoustic factors. The model was configured to simulate the condition of complete calibration of the listener to the tested HRTFs. Listener-specifically calibrated model predictions yielded correlations of, on average, 0.93 with the actual localization performance. Then, the model parameters representing the acoustic and non-acoustic factors were systematically permuted across the listener group. While the permutation of HRTFs affected the localization performance, the permutation of listener-specific uncertainty had a substantially larger impact. Our findings suggest that across-listener variability in sagittal-plane localization ability is only marginally determined by the acoustic factor, i.e., the quality of directional cues found in typical human HRTFs. Rather, the non-acoustic factors, supposed to represent the listeners' efficiency in processing directional cues, appear to be important.

Effect of mismatched place-of-stimulation on the salience of binaural cues in conditions that simulate bilateral cochlear-implant listening

Although bilateral cochlear implantation has the potential to improve sound localization and speech understanding in noise, obstacles exist in presenting maximally useful binaural information to bilateral cochlear-implant (CI) users. One obstacle is that electrode arrays may differ in cochlear position by several millimeters, thereby stimulating different neural populations. Effects of interaural frequency mismatch on binaural processing were studied in normal-hearing (NH) listeners using band-limited pulse trains, thereby avoiding confounding factors that may occur in CI users. In experiment 1, binaural image fusion was measured to capture perceptual number, location, and compactness. Subjects heard a single, compact image on 73% of the trials. In experiment 2, intracranial image location was measured for different interaural time differences (ITDs) and interaural level differences (ILDs). For larger mismatch, locations perceptually shifted towards the ear with the higher carrier frequency. In experiment 3, ITD and ILD just-noticeable differences (JNDs) were measured. JNDs increased with decreasing bandwidth and increasing mismatch, but were always measurable up to 3 mm of mismatch. If binaural-hearing mechanisms are similar between NH and CI subjects, these results may explain reduced sensitivity of ITDs and ILDs in CI users. Large mismatches may lead to distorted spatial maps and reduced binaural image fusion.

Sound localization in individualized and non-individualized crosstalk cancellation systems

The sound-source localization provided by a crosstalk cancellation (CTC) system depends on the head-related transfer functions (HRTFs) used for the CTC filter calculation. In this study, the horizontal- and sagittal-plane localization performance was investigated in humans listening to individualized matched, individualized but mismatched, and non-individualized CTC systems. The systems were simulated via headphones in a binaural virtual environment with two virtual loudspeakers spatialized in front of the listener. The individualized mismatched system was based on two different sets of listener-individual HRTFs. Both sets provided similar binaural localization performance in terms of quadrant, polar, and lateral errors. The individualized matched systems provided performance similar to that from the binaural listening. For the individualized mismatched systems, the performance deteriorated, and for the non-individualized mismatched systems (based on HRTFs from other listeners), the performance deteriorated even more. The direction-dependent analysis showed that mismatch and lack of individualization yielded a substantially degraded performance for targets placed outside of the loudspeaker span and behind the listeners, showing relevance of individualized CTC systems for those targets. Further, channel separation was calculated for different frequency ranges and is discussed in the light of its use as a predictor for the localization performance provided by a CTC system.

The role of envelope statistics in detecting changes in interaural correlation

The role of envelope statistics in binaural masking-level differences (BMLDs) and correlation change detection was investigated in normal-hearing listeners. Thresholds and just-noticeable differences (JNDs) were measured for different bandwidths and center frequencies (500, 2000, 4000, and 8000 Hz) using Gaussian noises (GNs) and low-fluctuation noises (LFNs). At a 500-Hz center frequency, GN NoSo thresholds were higher than, NoSπ thresholds were lower than, and correlation change detection JNDs were the same as LFN thresholds and JNDs. At higher center frequencies, GN NoSπ thresholds were the same or higher than LFN thresholds and GN correlation change detection JNDs were much smaller than LFN JNDs. Using a pulsed sine vocoder, a second experiment was performed to investigate if binaural adaptation might contribute to the difference in GN and LFN detection. There was no effect of pulse rate, thus providing no clear evidence that binaural adaptation plays a role in these tasks. Both a cross-correlation model and a model that utilized the fluctuations in the interaural differences could explain a majority of the variance in the LFN correlation change JNDs.

Two-dimensional localization of virtual sound sources in cochlear-implant listeners

OBJECTIVE

To test localization of sound sources in horizontal and vertical dimensions in cochlear-implant (CI) listeners using clinical bilateral CI systems.

DESIGN

Five bilateral CI subjects listened via their clinical speech processors to noises filtered with subject-specific, behind-the-ear microphones and head-related transfer functions. Subjects were immersed in a visual virtual environment presented via a head-mounted display. Subjects used a manual pointer to respond to the perceived sound location and received visual response feedback via the head-mounted display during the tests. The target positions were randomly distributed in two-dimensional space over an azimuth range of 0° to 360° and over an elevation range of -30° to +80°. In experiment 1, the signal level was roved in the range of ±2.5 dB from trial to trial. In experiment 2, the signal level was roved in the range of ±5 dB.

RESULTS

CI subjects were generally worse at sound localization than normal-hearing listeners tested in a previous study, in both the horizontal and vertical dimensions. In the horizontal plane, subjects could determine the correct side and locate the target within the side at better than chance performance. In the vertical plane, with a smaller level-roving range, subjects could determine the correct hemifield at better than chance performance but could not locate the target within the correct hemifield. The target angle and response angle were correlated as expected. The response angle and signal level range were also correlated, raising concerns that subjects were using only level cues for the task. With a larger level-roving range, the number of front-back confusions increased. The correlation between the target and response angles decreased, whereas the correlation between the level and response angle did not change, which is an indication that the subjects were relying heavily on level cues.

CONCLUSIONS

For the horizontal plane, the results are in agreement with previous CI studies performed in the horizontal plane with a comparable range of targets. For the vertical plane, CI listeners could discriminate front from back at better than chance performance; however, there are strong indications that the broadband level, not the spectral profile, was used as the primary localization cue. This study indicates the necessity of new CI processing strategies that encode spectral localization cues.