Behavioral and physiological correlates of temporal pitch perception in electric and acoustic hearing

Optimization of Sound Coding Strategies to Make Singing Music More Accessible for Cochlear Implant Users

Cochlear implants (CIs) are implantable medical devices that can partially restore hearing to people suffering from profound sensorineural hearing loss. While these devices provide good speech understanding in quiet, many CI users face difficulties when listening to music. Reasons include poor spatial specificity of electric stimulation, limited transmission of spectral and temporal fine structure of acoustic signals, and restrictions in the dynamic range that can be conveyed via electric stimulation of the auditory nerve. The coding strategies currently used in CIs are typically designed for speech rather than music. This work investigates the optimization of CI coding strategies to make singing music more accessible to CI users. The aim is to reduce the spectral complexity of music by selecting fewer bands for stimulation, attenuating the background instruments by strengthening a noise reduction algorithm, and optimizing the electric dynamic range through a back-end compressor. The optimizations were evaluated through both objective and perceptual measures of speech understanding and melody identification of singing voice with and without background instruments, as well as music appreciation questionnaires. Consistent with the objective measures, results gathered from the perceptual evaluations indicated that reducing the number of selected bands and optimizing the electric dynamic range significantly improved speech understanding in music. Moreover, results obtained from questionnaires show that the new music back-end compressor significantly improved music enjoyment. These results have potential as a new CI program for improved singing music perception.

The burst gap is a peripheral temporal code for pitch perception that is shared across audition and touch

When tactile afferents were manipulated to fire in periodic bursts of spikes, we discovered that the perceived pitch corresponded to the inter-burst interval (burst gap) in a spike train, rather than the spike rate or burst periodicity as previously thought. Given that tactile frequency mechanisms have many analogies to audition, and indications that temporal frequency channels are linked across the two modalities, we investigated whether there is burst gap temporal encoding in the auditory system. To link this putative neural code to perception, human subjects (n = 13, 6 females) assessed pitch elicited by trains of temporally-structured acoustic pulses in psychophysical experiments. Each pulse was designed to excite a fixed population of cochlear neurons, precluding place of excitation cues, and to elicit desired temporal spike trains in activated afferents. We tested periodicities up to 150 Hz using a variety of burst patterns and found striking deviations from periodicity-predicted pitch. Like the tactile system, the duration of the silent gap between successive bursts of neural activity best predicted perceived pitch, emphasising the role of peripheral temporal coding in shaping pitch. This suggests that temporal patterning of stimulus pulses in cochlear implant users might improve pitch perception.

Cochlear Implant Research and Development in the Twenty-first Century: A Critical Update

Cochlear implants (CIs) are the world's most successful sensory prosthesis and have been the subject of intense research and development in recent decades. We critically review the progress in CI research, and its success in improving patient outcomes, from the turn of the century to the present day. The review focuses on the processing, stimulation, and audiological methods that have been used to try to improve speech perception by human CI listeners, and on fundamental new insights in the response of the auditory system to electrical stimulation. The introduction of directional microphones and of new noise reduction and pre-processing algorithms has produced robust and sometimes substantial improvements. Novel speech-processing algorithms, the use of current-focusing methods, and individualised (patient-by-patient) deactivation of subsets of electrodes have produced more modest improvements. We argue that incremental advances have and will continue to be made, that collectively these may substantially improve patient outcomes, but that the modest size of each individual advance will require greater attention to experimental design and power. We also briefly discuss the potential and limitations of promising technologies that are currently being developed in animal models, and suggest strategies for researchers to collectively maximise the potential of CIs to improve hearing in a wide range of listening situations.

Burst gap code predictions for tactile frequency are valid across the range of perceived frequencies attributed to two distinct tactile channels

Perceived frequency of vibrotactile stimuli can be divided into two distinctive cutaneous sensations-flutter (<60 Hz) and vibratory hum (>60 Hz), mediated by two different tactile afferent types [fast adapting type I (FA1) and fast adapting type II (FA2), respectively]. We recently demonstrated a novel form of neural coding in the human tactile system, where frequency perception of stimulus pulses grouped into periodic bursts in the flutter range depended on the duration of the silent gap between bursts, rather than the periodicity or mean impulse rate. Here, we investigated whether this interburst interval could also explain the perceived frequency of electrocutaneous pulse patterns delivered at frequencies above the flutter range. At stimulus rates of 50 to 190 pulses/s, the burst gap model correctly predicted the perceived frequency. This shows that the burst gap code represents a general coding strategy that spans the range of frequencies traditionally attributed to two different tactile channels.NEW & NOTEWORTHY We present evidence for a generalized frequency processing strategy on tactile afferent inputs that is shared across a broad range of frequencies extending beyond the flutter range, supporting the notion that spike timing has an important role in shaping tactile perception.

Tapping Into the Language of Touch: Using Non-invasive Stimulation to Specify Tactile Afferent Firing Patterns

The temporal pattern of action potentials can convey rich information in a variety of sensory systems. We describe a new non-invasive technique that enables precise, reliable generation of action potential patterns in tactile peripheral afferent neurons by brief taps on the skin. Using this technique, we demonstrate sophisticated coding of temporal information in the somatosensory system, that shows that perceived vibration frequency is not encoded in peripheral afferents as was expected by either their firing rate or the underlying periodicity of the stimulus. Instead, a burst gap or silent gap between trains of action potentials conveys frequency information. This opens the possibility of new encoding strategies that could be deployed to convey sensory information using mechanical or electrical stimulation in neural prostheses and brain-machine interfaces, and may extend to senses beyond artificial encoding of aspects of touch. We argue that a focus on appropriate use of effective temporal coding offers more prospects for rapid improvement in the function of these interfaces than attempts to scale-up existing devices.

Combined neural and behavioural measures of temporal pitch perception in cochlear implant users

Four experiments measured the perceptual and neural correlates of the temporal pattern of electrical stimulation applied to one cochlear-implant (CI) electrode, for several subjects. Neural effects were estimated from the electrically evoked compound action potential (ECAP) to each pulse. Experiment 1 attenuated every second pulse of a 200-pps pulse train. Increasing attenuation caused pitch to drop and the ECAP to become amplitude modulated, thereby providing an estimate of the relationship between neural modulation and pitch. Experiment 2 showed that the pitch of a 200-pps pulse train can be reduced by delaying every second pulse, so that the inter-pulse-intervals alternate between longer and shorter intervals. This caused the ECAP to become amplitude modulated, but not by enough to account for the change in pitch. Experiment 3 replicated the finding that rate discrimination deteriorates with increases in baseline rate. This was accompanied by an increase in ECAP modulation, but by an amount that produced only a small effect on pitch in experiment 1. Experiment 4 showed that preceding a pulse train with a carefully selected "pre-pulse" could reduce ECAP modulation, but did not improve rate discrimination. Implications for theories of pitch and for limitations of pitch perception in CI users are discussed.

Considering optogenetic stimulation for cochlear implants

Electrical cochlear implants are by far the most successful neuroprostheses and have been implanted in over 300,000 people worldwide. Cochlear implants enable open speech comprehension in most patients but are limited in providing music appreciation and speech understanding in noisy environments. This is generally considered to be due to low frequency resolution as a consequence of wide current spread from stimulation contacts. Accordingly, the number of independently usable stimulation channels is limited to less than a dozen. As light can be conveniently focused, optical stimulation might provide an alternative approach to cochlear implants with increased number of independent stimulation channels. Here, we focus on summarizing recent work on optogenetic stimulation as one way to develop optical cochlear implants. We conclude that proof of principle has been presented for optogenetic stimulation of the cochlea and central auditory neurons in rodents as well as for the technical realization of flexible μLED-based multichannel cochlear implants. Still, much remains to be done in order to advance the technique for auditory research and even more for eventual clinical translation. This article is part of a Special Issue entitled <Lasker Award>.

Re-examining the upper limit of temporal pitch

Five normally hearing listeners pitch-ranked harmonic complexes of different fundamental frequencies (F0s) filtered in three different frequency regions. Harmonics were summed either in sine, alternating sine-cosine (ALT), or pulse-spreading (PSHC) phase. The envelopes of ALT and PSHC complexes repeated at rates of 2F0 and 4F0. Pitch corresponded to those rates at low F0s, but, as F0 increased, there was a range of F0s over which pitch remained constant or dropped. Gammatone-filterbank simulations showed that, as F0 increased and the number of harmonics interacting in a filter dropped, the output of that filter switched from repeating at 2F0 or 4F0 to repeating at F0. A model incorporating this phenomenon accounted well for the data, except for complexes filtered into the highest frequency region (7800-10 800 Hz). To account for the data in that region it was necessary to assume either that auditory filters at very high frequencies are sharper than traditionally believed, and/or that the auditory system applies smaller weights to filters whose outputs repeat at high rates. The results also provide evidence on the highest pitch that can be derived from purely temporal cues, and corroborate recent reports that a complex pitch can be derived from very-high-frequency resolved harmonics.

Modulation frequency discrimination with modulated and unmodulated interference in normal hearing and in cochlear-implant users

Differences in fundamental frequency (F0) provide an important cue for segregating simultaneous sounds. Cochlear implants (CIs) transmit F0 information primarily through the periodicity of the temporal envelope of the electrical pulse trains. Successful segregation of sounds with different F0s requires the ability to process multiple F0s simultaneously, but it is unknown whether CI users have this ability. This study measured modulation frequency discrimination thresholds for half-wave rectified sinusoidal envelopes modulated at 115 Hz in CI users and normal-hearing (NH) listeners. The target modulation was presented in isolation or in the presence of an interferer. Discrimination thresholds were strongly affected by the presence of an interferer, even when it was unmodulated and spectrally remote. Interferer modulation increased interference and often led to very high discrimination thresholds, especially when the interfering modulation frequency was lower than that of the target. Introducing a temporal offset between the interferer and the target led to at best modest improvements in performance in CI users and NH listeners. The results suggest no fundamental difference between acoustic and electric hearing in processing single or multiple envelope-based F0s, but confirm that differences in F0 are unlikely to provide a robust cue for perceptual segregation in CI users.

Relationships between auditory nerve activity and temporal pitch perception in cochlear implant users

Cochlear implant (CI) users can derive a musical pitch from the temporal pattern of pulses delivered to one electrode. However, pitch perception deteriorates with increasing pulse rate, and most listeners cannot detect increases in pulse rate beyond about 300 pps. In addition, previous studies using irregular pulse trains suggest that pitch can be substantially influenced by neural refractory effects. We presented electric pulse trains to one CI electrode and measured rate discrimination, pitch perception, and auditory nerve (AN) activity in the same subjects and with the same stimuli. The measures of AN activity, obtained using the electrically evoked compound action potential (ECAP), replicated the well-known finding that the neural response to isochronous pulse trains at rates above about 200-300 pps is modulated, with the ECAP being larger to odd-numbered than to even-numbered pulses. This finding has been attributed to refractoriness. Behavioural results replicated the deterioration in rate discrimination at rates above 200-300 pps and the finding that pulse trains whose inter-pulse intervals (IPIs) alternate between a shorter and a longer value (e.g. 4 and 6 ms) have a pitch lower than that corresponding to the mean IPI. To link ECAP modulation to pitch, we physically modulated a 200-pps pulse train by attenuating every other pulse and measured both ECAPs and pitch as a function of modulation depth. Our results show that important aspects of temporal pitch perception cannot be explained in terms of the AN response, at least as measured by ECAPs, and suggest that pitch is influenced by refractory effects occurring central to the AN.

Perceived continuity and pitch shifts for complex tones with unresolved harmonics

Brief complex tone bursts with fundamental frequencies (F0s) of 100, 125, 166.7, and 250 Hz were bandpass filtered between the 22nd and 30th harmonics, to produce waveforms with five regularly occurring envelope peaks ("pitch pulses") that evoked pitches associated with their repetition period. Two such tone bursts were presented sequentially and separated by a silent interval of two periods (2/F0). When the relative phases of the two bursts were varied, such that the interpulse interval (IPI) between the last pulse of the first burst and the first pulse of the second burst was varied, the pitch of the whole sequence was little affected. This is consistent with previous results suggesting that the pitch integration window may be "reset" by a discontinuity. However, when the interval between the two bursts was filled with a noise with the same spectral envelope as the complex, variations in IPI had substantial effects on the pitch of the sequence. It is suggested that the presence of the noise causes the two tones bursts to appear continuous, hence, resetting does not occur, and the pitch mechanism is sensitive to the phase discontinuity across the silent interval.

Effect of stimulus level and place of stimulation on temporal pitch perception by cochlear implant users

Three experiments studied the effect of pulse rate on temporal pitch perception by cochlear implant users. Experiment 1 measured rate discrimination for pulse trains presented in bipolar mode to either an apical, middle, or basal electrode and for standard rates of 100 and 200 pps. In each block of trials the signals could have a level of -0.35, 0, or +0.35 dB re the standard, and performance for each signal level was recorded separately. Signal level affected performance for just over half of the combinations of subject, electrode, and standard rate studied. Performance was usually, but not always, better at the higher signal level. Experiment 2 showed that, for a given subject and condition, the direction of the effect was similar in monopolar and bipolar mode. Experiment 3 employed a pitch comparison procedure without feedback, and showed that the signal levels in experiment 1 that produced the best performance for a given subject and condition also led to the signal having a higher pitch. It is concluded that small level differences can have a robust and substantial effect on pitch judgments and argue that these effects are not entirely due to response biases or to co-variation of place-of-excitation with level.

The upper limit of temporal pitch for cochlear-implant listeners: stimulus duration, conditioner pulses, and the number of electrodes stimulated

Three experiments studied discrimination of changes in the rate of electrical pulse trains by cochlear-implant (CI) users and investigated the effect of manipulations that would be expected to substantially affect the pattern of auditory nerve (AN) activity. Experiment 1 used single-electrode stimulation and tested discrimination at baseline rates between 100 and 500 pps. Performance was generally similar for stimulus durations of 200 and 800 ms, and, for the longer duration, for stimuli that were gated on abruptly or with 300-ms ramps. Experiment 2 used a similar procedure and found that no substantial benefit was obtained by the addition of background 5000-pps "conditioning" pulses. Experiment 3 used a pitch-ranking procedure and found that the range of rates over which pitch increased with increasing rate was not greater for multiple-electrode than for single-electrode stimulation. The results indicate that the limitation on pulse-rate discrimination by CI users, at high baseline rates, is not specific to a particular temporal pattern of the AN response.

Understanding pitch perception as a hierarchical process with top-down modulation

Pitch is one of the most important features of natural sounds, underlying the perception of melody in music and prosody in speech. However, the temporal dynamics of pitch processing are still poorly understood. Previous studies suggest that the auditory system uses a wide range of time scales to integrate pitch-related information and that the effective integration time is both task- and stimulus-dependent. None of the existing models of pitch processing can account for such task- and stimulus-dependent variations in processing time scales. This study presents an idealized neurocomputational model, which provides a unified account of the multiple time scales observed in pitch perception. The model is evaluated using a range of perceptual studies, which have not previously been accounted for by a single model, and new results from a neurophysiological experiment. In contrast to other approaches, the current model contains a hierarchy of integration stages and uses feedback to adapt the effective time scales of processing at each stage in response to changes in the input stimulus. The model has features in common with a hierarchical generative process and suggests a key role for efferent connections from central to sub-cortical areas in controlling the temporal dynamics of pitch processing.

A cascade autocorrelation model of pitch perception

Autocorrelation algorithms, in combination with computational models of the auditory periphery, have been successfully used to predict the pitch of a wide range of complex stimuli. However, new stimuli are frequently offered as counterexamples to the viability of this approach. This study addresses the issue of whether in the light of these challenges the predictive power of autocorrelation can be preserved by changes to the peripheral model and the computational algorithm. An existing model is extended by the addition of a low-pass filter of the summary integration of the individual within-channel autocorrelations. Other recent developments are also incorporated, including nonlinear processing on the basilar membrane and the use of integration time constants that are proportional to the autocorrelation lags. The modified and extended model predicts with reasonable success the pitches of a range of stimuli that have proved problematic for earlier implementations of the autocorrelation principle. The evaluation stimuli include short tone sequences, click trains consisting of alternating interclick intervals, click trains consisting of mixtures of regular and irregular intervals, shuffled click trains, and transposed tones.