A mathematical model of vowel identification by users of cochlear implants

Electro-Haptic Stimulation: A New Approach for Improving Cochlear-Implant Listening

Cochlear implants (CIs) have been remarkably successful at restoring speech perception for severely to profoundly deaf individuals. Despite their success, several limitations remain, particularly in CI users' ability to understand speech in noisy environments, locate sound sources, and enjoy music. A new multimodal approach has been proposed that uses haptic stimulation to provide sound information that is poorly transmitted by the implant. This augmenting of the electrical CI signal with haptic stimulation (electro-haptic stimulation; EHS) has been shown to improve speech-in-noise performance and sound localization in CI users. There is also evidence that it could enhance music perception. We review the evidence of EHS enhancement of CI listening and discuss key areas where further research is required. These include understanding the neural basis of EHS enhancement, understanding the effectiveness of EHS across different clinical populations, and the optimization of signal-processing strategies. We also discuss the significant potential for a new generation of haptic neuroprosthetic devices to aid those who cannot access hearing-assistive technology, either because of biomedical or healthcare-access issues. While significant further research and development is required, we conclude that EHS represents a promising new approach that could, in the near future, offer a non-invasive, inexpensive means of substantially improving clinical outcomes for hearing-impaired individuals.

Discrimination of Voice Pitch and Vocal-Tract Length in Cochlear Implant Users

OBJECTIVES

When listening to two competing speakers, normal-hearing (NH) listeners can take advantage of voice differences between the speakers. Users of cochlear implants (CIs) have difficulty in perceiving speech on speech. Previous literature has indicated sensitivity to voice pitch (related to fundamental frequency, F0) to be poor among implant users, while sensitivity to vocal-tract length (VTL; related to the height of the speaker and formant frequencies), the other principal voice characteristic, has not been directly investigated in CIs. A few recent studies evaluated F0 and VTL perception indirectly, through voice gender categorization, which relies on perception of both voice cues. These studies revealed that, contrary to prior literature, CI users seem to rely exclusively on F0 while not utilizing VTL to perform this task. The objective of the present study was to directly and systematically assess raw sensitivity to F0 and VTL differences in CI users to define the extent of the deficit in voice perception.

DESIGN

The just-noticeable differences (JNDs) for F0 and VTL were measured in 11 CI listeners using triplets of consonant-vowel syllables in an adaptive three-alternative forced choice method.

RESULTS

The results showed that while NH listeners had average JNDs of 1.95 and 1.73 semitones (st) for F0 and VTL, respectively, CI listeners showed JNDs of 9.19 and 7.19 st. These JNDs correspond to differences of 70% in F0 and 52% in VTL. For comparison to the natural range of voices in the population, the F0 JND in CIs remains smaller than the typical male-female F0 difference. However, the average VTL JND in CIs is about twice as large as the typical male-female VTL difference.

CONCLUSIONS

These findings, thus, directly confirm that CI listeners do not seem to have sufficient access to VTL cues, likely as a result of limited spectral resolution, and, hence, that CI listeners' voice perception deficit goes beyond poor perception of F0. These results provide a potential common explanation not only for a number of deficits observed in CI listeners, such as voice identification and gender categorization, but also for competing speech perception.

Saliency of Vowel Features in Neural Responses of Cochlear Implant Users

Cochlear implants restore hearing in deaf individuals, but speech perception remains challenging. Poor discrimination of spectral components is thought to account for limitations of speech recognition in cochlear implant users. We investigated how combined variations of spectral components along two orthogonal dimensions can maximize neural discrimination between two vowels, as measured by mismatch negativity. Adult cochlear implant users and matched normal-hearing listeners underwent electroencephalographic event-related potentials recordings in an optimum-1 oddball paradigm. A standard /a/ vowel was delivered in an acoustic free field along with stimuli having a deviant fundamental frequency (+3 and +6 semitones), a deviant first formant making it a /i/ vowel or combined deviant fundamental frequency and first formant (+3 and +6 semitones /i/ vowels). Speech recognition was assessed with a word repetition task. An analysis of variance between both amplitude and latency of mismatch negativity elicited by each deviant vowel was performed. The strength of correlations between these parameters of mismatch negativity and speech recognition as well as participants' age was assessed. Amplitude of mismatch negativity was weaker in cochlear implant users but was maximized by variations of vowels' first formant. Latency of mismatch negativity was later in cochlear implant users and was particularly extended by variations of the fundamental frequency. Speech recognition correlated with parameters of mismatch negativity elicited by the specific variation of the first formant. This nonlinear effect of acoustic parameters on neural discrimination of vowels has implications for implant processor programming and aural rehabilitation.

Deactivating cochlear implant electrodes to improve speech perception: A computational approach

A potential bottleneck to improving speech perception performance in cochlear implant (CI) users is that some of their electrodes may poorly encode speech information. Several studies have examined the effect of deactivating poorly encoding electrodes on speech perception with mixed results. Many of these studies focused on identifying poorly encoding electrodes by some measure (e.g. electrode discrimination, pitch ordering, threshold, CT-guided, masked modulation detection), but provide inconsistent criteria about which electrodes, and how many, should be deactivated, and without considering how speech information becomes distributed across the electrode array. The present simulation study addresses this issue using computational approaches. Previously validated models were used to generate predictions of speech scores as a function of all possible combinations of active electrodes in a 22-electrode array in three groups of hypothetical subjects representative of relatively better, moderate, and poorer performing CI users. Using high-performance computing, over 500 million predictions were generated. Although deactivation of the poorest encoding electrodes sometimes resulted in predicted benefit, this benefit was significantly less relative to predictions resulting from model-optimized deactivations. This trend persisted when using novel stimuli (i.e. other than those used for optimization) and when using different processing strategies. Optimum electrode deactivation patterns produced an average predicted increase in word scores of 10% with some scores increasing by more than 20%. Optimum electrode deactivation patterns typically included 11 to 19 (out of 22) active electrodes, depending on the performance group. Optimal active electrode combinations were those that maximized discrimination of speech cues, maintaining 80%-100% of the physical span of the array. The present study demonstrates the potential for further improving CI users' speech scores with appropriate selection of active electrodes.

Contribution of formant frequency information to vowel perception in steady-state noise by cochlear implant users

Cochlear implant (CI) recipients have difficulty understanding speech in noise even at moderate signal-to-noise ratios. Knowing the mechanisms they use to understand speech in noise may facilitate the search for better speech processing algorithms. In the present study, a computational model is used to assess whether CI users' vowel identification in noise can be explained by formant frequency cues (F1 and F2). Vowel identification was tested with 12 unilateral CI users in quiet and in noise. Formant cues were measured from vowels in each condition, specific to each subject's speech processor. Noise distorted the location of vowels in the F2 vs F1 plane in comparison to quiet. The best fit model to subjects' data in quiet produced model predictions in noise that were within 8% of actual scores on average. Predictions in noise were much better when assuming that subjects used a priori knowledge regarding how formant information is degraded in noise (experiment 1). However, the model's best fit to subjects' confusion matrices in noise was worse than in quiet, suggesting that CI users utilize formant cues to identify vowels in noise, but to a different extent than how they identify vowels in quiet (experiment 2).

Vowel and consonant confusions from spectrally manipulated stimuli designed to simulate poor cochlear implant electrode-neuron interfaces

Suboptimal interfaces between cochlear implant (CI) electrodes and auditory neurons result in a loss or distortion of spectral information in specific frequency regions, which likely decreases CI users' speech identification performance. This study exploited speech acoustics to model regions of distorted CI frequency transmission to determine the perceptual consequences of suboptimal electrode-neuron interfaces. Normal hearing adults identified naturally spoken vowels and consonants after spectral information was manipulated through a noiseband vocoder: either (1) low-, middle-, or high-frequency regions of information were removed by zeroing the corresponding channel outputs, or (2) the same regions were distorted by splitting filter outputs to neighboring filters. These conditions simulated the detrimental effects of suboptimal CI electrode-neuron interfaces on spectral transmission. Vowel and consonant confusion patterns were analyzed with sequential information transmission, perceptual distance, and perceptual vowel space analyses. Results indicated that both types of spectral manipulation were equally destructive. Loss or distortion of frequency information produced similar effects on phoneme identification performance and confusion patterns. Consonant error patterns were consistently based on place of articulation. Vowel confusions showed that perceptions gravitated away from the degraded frequency region in a predictable manner, indicating that vowels can probe frequency-specific regions of spectral degradations.

A physiological and behavioral system for hearing restoration with cochlear implants

Cochlear implants are neuroprosthetic devices that provide hearing to deaf patients, although outcomes are highly variable even with prolonged training and use. The central auditory system must process cochlear implant signals, but it is unclear how neural circuits adapt-or fail to adapt-to such inputs. The knowledge of these mechanisms is required for development of next-generation neuroprosthetics that interface with existing neural circuits and enable synaptic plasticity to improve perceptual outcomes. Here, we describe a new system for cochlear implant insertion, stimulation, and behavioral training in rats. Animals were first ensured to have significant hearing loss via physiological and behavioral criteria. We developed a surgical approach for multichannel (2- or 8-channel) array insertion, comparable with implantation procedures and depth in humans. Peripheral and cortical responses to stimulation were used to program the implant objectively. Animals fitted with implants learned to use them for an auditory-dependent task that assesses frequency detection and recognition in a background of environmentally and self-generated noise and ceased responding appropriately to sounds when the implant was temporarily inactivated. This physiologically calibrated and behaviorally validated system provides a powerful opportunity to study the neural basis of neuroprosthetic device use and plasticity.

Optimization of a Spectral Contrast Enhancement Algorithm for Cochlear Implants Based on a Vowel Identification Model

Speech intelligibility achieved with cochlear implants (CIs) shows large variability across different users. One reason that can explain this variability is the CI user's individual electrode nerve interface which can impact the spectral resolution they can achieve. Spectral resolution has been reported to be related to vowel and consonant recognition in CI listeners. One measure of spectral resolution is the spectral modulation threshold (SMT), which is defined as the smallest detectable spectral contrast in a stimulus. In this study we hypothesize that an algorithm that improves SMT may improve vowel identification, and consequently produce an improvement in speech understanding for CIs. With this purpose we implemented an algorithm, termed spectral contrast enhancement (SCE) that emphasizes peaks with respect to valleys in the audio spectrum. This algorithm can be configured with a single parameter: the amount of spectral contrast enhancement entitled "SCE factor". We would like to investigate whether the "SCE factor" can be individualized to each CI user. With this purpose we used a vowel identification model to predict the performance produced by the SCE algorithm with different "SCE factors" in a vowel identification task.In five CI users the new algorithm has been evaluated using a SMT task and a vowel identification task. The tasks were performed for SCE factors of 0 (no enhancement), 2 and 4. In general it seems that increasing the SCE factor produces a decrease in performance in both the SMT threshold and vowel identification.

Spectral contrast enhancement improves speech intelligibility in noise for cochlear implants

Spectral smearing causes, at least partially, that cochlear implant (CI) users require a higher signal-to-noise ratio to obtain the same speech intelligibility as normal hearing listeners. A spectral contrast enhancement (SCE) algorithm has been designed and evaluated as an additional feature for a standard CI strategy. The algorithm keeps the most prominent peaks within a speech signal constant while attenuating valleys in the spectrum. The goal is to partly compensate for the spectral smearing produced by the limited number of stimulation electrodes and the overlap of electrical fields produced in CIs. Twelve CI users were tested for their speech reception threshold (SRT) using the standard CI coding strategy with and without SCE. No significant differences in SRT were observed between conditions. However, an analysis of the electrical stimulation patterns shows a reduction in stimulation current when using SCE. In a second evaluation, 12 CI users were tested in a similar configuration of the SCE strategy with the stimulation being balanced between the SCE and the non-SCE variants such that the loudness perception delivered by the strategies was the same. Results show a significant improvement in SRT of 0.57 dB (p < 0.0005) for the SCE algorithm.

The neural encoding of formant frequencies contributing to vowel identification in normal-hearing listeners

Even though speech signals trigger coding in the cochlea to convey speech information to the central auditory structures, little is known about the neural mechanisms involved in such processes. The purpose of this study was to understand the encoding of formant cues and how it relates to vowel recognition in listeners. Neural representations of formants may differ across listeners; however, it was hypothesized that neural patterns could still predict vowel recognition. To test the hypothesis, the frequency-following response (FFR) and vowel recognition were obtained from 38 normal-hearing listeners using four different vowels, allowing direct comparisons between behavioral and neural data in the same individuals. FFR was employed because it provides an objective and physiological measure of neural activity that can reflect formant encoding. A mathematical model was used to describe vowel confusion patterns based on the neural responses to vowel formant cues. The major findings were (1) there were large variations in the accuracy of vowel formant encoding across listeners as indexed by the FFR, (2) these variations were systematically related to vowel recognition performance, and (3) the mathematical model of vowel identification was successful in predicting good vs poor vowel identification performers based exclusively on physiological data.

Bilateral cochlear implants with large asymmetries in electrode insertion depth: implications for the study of auditory plasticity

CONCLUSION

The human frequency-to-place map may be modified by experience, even in adult listeners. However, such plasticity has limitations. Knowledge of the extent and the limitations of human auditory plasticity can help optimize parameter settings in users of auditory prostheses.

OBJECTIVES

To what extent can adults adapt to sharply different frequency-to-place maps across ears? This question was investigated in two bilateral cochlear implant users who had a full electrode insertion in one ear, a much shallower insertion in the other ear, and standard frequency-to-electrode maps in both ears.

METHODS

Three methods were used to assess adaptation to the frequency-to-electrode maps in each ear: (1) pitch matching of electrodes in opposite ears, (2) listener-driven selection of the most intelligible frequency-to-electrode map, and (3) speech perception tests. Based on these measurements, one subject was fitted with an alternative frequency-to-electrode map, which sought to compensate for her incomplete adaptation to the standard frequency-to-electrode map.

RESULTS

Both listeners showed remarkable ability to adapt, but such adaptation remained incomplete for the ear with the shallower electrode insertion, even after extended experience. The alternative frequency-to-electrode map that was tested resulted in substantial increases in speech perception for one subject in the short insertion ear.

Current and planned cochlear implant research at New York University Laboratory for Translational Auditory Research

The Laboratory of Translational Auditory Research (LTAR/NYUSM) is part of the Department of Otolaryngology at the New York University School of Medicine and has close ties to the New York University Cochlear Implant Center. LTAR investigators have expertise in multiple related disciplines including speech and hearing science, audiology, engineering, and physiology. The lines of research in the laboratory deal mostly with speech perception by hearing impaired listeners, and particularly those who use cochlear implants (CIs) or hearing aids (HAs). Although the laboratory's research interests are diverse, there are common threads that permeate and tie all of its work. In particular, a strong interest in translational research underlies even the most basic studies carried out in the laboratory. Another important element is the development of engineering and computational tools, which range from mathematical models of speech perception to software and hardware that bypass clinical speech processors and stimulate cochlear implants directly, to novel ways of analyzing clinical outcomes data. If the appropriate tool to conduct an important experiment does not exist, we may work to develop it, either in house or in collaboration with academic or industrial partners. Another notable characteristic of the laboratory is its interdisciplinary nature where, for example, an audiologist and an engineer might work closely to develop an approach that would not have been feasible if each had worked singly on the project. Similarly, investigators with expertise in hearing aids and cochlear implants might join forces to study how human listeners integrate information provided by a CI and a HA. The following pages provide a flavor of the diversity and the commonalities of our research interests.

RFcap: a software analysis tool for multichannel cochlear implant signals

Being able to display and analyse the output of a speech processor that encodes the parameters of complex stimuli to be presented by a cochlear implant (CI) is useful for software and hardware development as well as for diagnostic purposes. This firstly requires appropriate hardware that is able to receive and decode the radio frequency (RF)-coded signals, and then processing the decoded data using suitable software. The PCI-IF6 clinical hardware for the Nucleus CI system, together with the Nucleus Implant Communicator and Nucleus Matlab Toolbox research software libraries, provide the necessary functionality. RFcap is a standalone Matlab application that encapsulates the relevant functions to capture, display, and analyse the RF-coded signals intended for the Nucleus CI24M/R, CI24RE, and CI500 multichannel CIs.

Cross-frequency integration for consonant and vowel identification in bimodal hearing

PURPOSE

Improved speech recognition in binaurally combined acoustic-electric stimulation (otherwise known as bimodal hearing) could arise when listeners integrate speech cues from the acoustic and electric hearing. The aims of this study were (a) to identify speech cues extracted in electric hearing and residual acoustic hearing in the low-frequency region and (b) to investigate cochlear implant (CI) users' ability to integrate speech cues across frequencies.

METHOD

Normal-hearing (NH) and CI subjects participated in consonant and vowel identification tasks. Each subject was tested in 3 listening conditions: CI alone (vocoder speech for NH), hearing aid (HA) alone (low-pass filtered speech for NH), and both. Integration ability for each subject was evaluated using a model of optimal integration--the PreLabeling integration model (Braida, 1991).

RESULTS

Only a few CI listeners demonstrated bimodal benefit for phoneme identification in quiet. Speech cues extracted from the CI and the HA were highly redundant for consonants but were complementary for vowels. CI listeners also exhibited reduced integration ability for both consonant and vowel identification compared with their NH counterparts.

CONCLUSION

These findings suggest that reduced bimodal benefits in CI listeners are due to insufficient complementary speech cues across ears, a decrease in integration ability, or both.

A mathematical model of medial consonant identification by cochlear implant users

The multidimensional phoneme identification model is applied to consonant confusion matrices obtained from 28 postlingually deafened cochlear implant users. This model predicts consonant matrices based on these subjects' ability to discriminate a set of postulated spectral, temporal, and amplitude speech cues as presented to them by their device. The model produced confusion matrices that matched many aspects of individual subjects' consonant matrices, including information transfer for the voicing, manner, and place features, despite individual differences in age at implantation, implant experience, device and stimulation strategy used, as well as overall consonant identification level. The model was able to match the general pattern of errors between consonants, but not the full complexity of all consonant errors made by each individual. The present study represents an important first step in developing a model that can be used to test specific hypotheses about the mechanisms cochlear implant users employ to understand speech.