Encoding a Melody Using Only Temporal Information for Cochlear-Implant and Normal-Hearing Listeners

[Influence of overtones and undertones on melody recognition with a cochlear implant with SSD]

Many cochlear implant (CI) users have difficulties recognising pitches and melodies because pitch transmission is blurred and shifted. This study investigates whether postlingually deafened adult CI users recognize melodies better when overtones are removed or undertones are added.Fifteen unilaterally postlingually deafened CI users (single sided deafness = SSD) were included aged 22 to 73 years (MW 52, SD 11.6) with CI hearing experience between 3 and 75 months (MW 33, SD 21.0) with varying MED-EL devices. Three short piano melodies were presented to them firstly to the normal-hearing ear and then in modified overtone or undertone variants and the original variant to the CI ear. These variants should be identified as one of the three original melodies. In addition, musical experience and ability were assessed by the Munich Music Questionnaire and the MiniPROMS music tests.The CI users showed the best melody recognition in the fundamental frequency variant. The overtone variant with the third overtone was as good as the original variant with all overtones with regard to melody recognition (p=1). However, the undertone variant with the first undertone was recognised significantly worse than the fundamental version (p=0.032). Furthermore, there was no correlation between musical experience or musical ability and the number of melodies recognised (p>0.1).Since a reduction of overtones did not worsen the melody recognition, overtone reduction should be considered in future music processing programs for the CI. This could reduce the energy consumption of the CI.

A subjective evaluation of different music preprocessing approaches in cochlear implant listeners

Cochlear implants (CIs) can partially restore speech perception to relatively high levels in listeners with moderate to profound hearing loss. However, for most CI listeners, the perception and enjoyment of music remains notably poor. Since a number of technical and physiological restrictions of current implant designs cannot be easily overcome, a number of preprocessing methods for music signals have been proposed recently. They aim to emphasize the leading voice and rhythmic elements and to reduce their spectral complexity. In this study, CI listeners evaluated five remixing approaches in comparison to unprocessed signals. To identify potential explaining factors of CI preference ratings, different signal quality criteria of the processed signals were additionally assessed by normal-hearing listeners. Additional factors were investigated based on instrumental signal-level features. For three preprocessing methods, a significant improvement over the unprocessed reference was found. Especially, two deep neural network-based remix strategies proved to enhance music perception in CI listeners. These strategies provide remixes of the respective harmonic and percussive signal components of the four source stems "vocals," "bass," "drums," and "other accompaniment." Moreover, the results demonstrate that CI listeners prefer an attenuation of sustained components of drum source signals.

Musical Interval Perception With a Cochlear Implant Alone and With a Contralateral Normal Hearing Ear

Music through a cochlear implant (CI) is described as out-of-tune, suggesting that musical intervals are not accurately provided by a CI. One potential reason is that pitch may be insufficiently conveyed to provide reliable intervals. Another potential reason is that the size of intervals is distorted through a CI as they would be when produced by a mistuned piano. To measure intervals through a CI, listeners selected prerecorded vowels with different fundamental frequencies to represent each note in Happy Birthday. Each listener had contralateral normal hearing (NH); repeating the experiment with their NH ear allowed for a within-subject control. Additionally, the effect of listening simultaneously to both a CI and NH ear was measured. The resulting versions of Happy Birthday were analyzed in terms of their contours, interval sizes, magnitudes, consistency, and direction. Intervals with NH ears ranged from perfect to uncorrelated with target intervals. Chosen interval size with the CI was poorer than with the NH ear for all subjects. Across listeners, chosen intervals with the CI ranged from highly correlated to uncorrelated with target intervals. That CI intervals were highly correlated with target intervals for some listeners suggests that accurate intervals can be provided through a CI. For some listeners, chosen intervals were larger than target intervals, suggesting that intervals may be perceived as too small. Overall, intervals with the combination of the NH and CI ears were similar to those with the NH ear alone, suggesting that the addition of a CI has little-to-no effect on interval perception.

Computer-based musical interval training program for Cochlear implant users and listeners with no known hearing loss

A musical interval is the difference in pitch between two sounds. The way that musical intervals are used in melodies relative to the tonal center of a key can strongly affect the emotion conveyed by the melody. The present study examines musical interval identification in people with no known hearing loss and in cochlear implant users. Pitch resolution varies widely among cochlear implant users with average resolution an order of magnitude worse than in normal hearing. The present study considers the effect of training on musical interval identification and tests for correlations between low-level psychophysics and higher-level musical abilities. The overarching hypothesis is that cochlear implant users are limited in their ability to identify musical intervals both by low-level access to frequency cues for pitch as well as higher-level mapping of the novel encoding of pitch that implants provide. Participants completed a 2-week, online interval identification training. The benchmark tests considered before and after interval identification training were pure tone detection thresholds, pure tone frequency discrimination, fundamental frequency discrimination, tonal and rhythm comparisons, and interval identification. The results indicate strong correlations between measures of pitch resolution with interval identification; however, only a small effect of training on interval identification was observed for the cochlear implant users. Discussion focuses on improving access to pitch cues for cochlear implant users and on improving auditory training for musical intervals.

A versatile deep-neural-network-based music preprocessing and remixing scheme for cochlear implant listeners

While cochlear implants (CIs) have proven to restore speech perception to a remarkable extent, access to music remains difficult for most CI users. In this work, a methodology for the design of deep learning-based signal preprocessing strategies that simplify music signals and emphasize rhythmic information is proposed. It combines harmonic/percussive source separation and deep neural network (DNN) based source separation in a versatile source mixture model. Two different neural network architectures were assessed with regard to their applicability for this task. The method was evaluated with instrumental measures and in two listening experiments for both network architectures and six mixing presets. Normal-hearing subjects rated the signal quality of the processed signals compared to the original both with and without a vocoder which provides an approximation of the auditory perception in CI listeners. Four combinations of remix models and DNNs have been selected for an evaluation with vocoded signals and were all rated significantly better in comparison to the unprocessed signal. In particular, the two best-performing remix networks are promising candidates for further evaluation in CI listeners.

Periodicity Pitch Perception

This study presents a computational model to reproduce the biological dynamics of "listening to music." A biologically plausible model of periodicity pitch detection is proposed and simulated. Periodicity pitch is computed across a range of the auditory spectrum. Periodicity pitch is detected from subsets of activated auditory nerve fibers (ANFs). These activate connected model octopus cells, which trigger model neurons detecting onsets and offsets; thence model interval-tuned neurons are innervated at the right interval times; and finally, a set of common interval-detecting neurons indicate pitch. Octopus cells rhythmically spike with the pitch periodicity of the sound. Batteries of interval-tuned neurons stopwatch-like measure the inter-spike intervals of the octopus cells by coding interval durations as first spike latencies (FSLs). The FSL-triggered spikes synchronously coincide through a monolayer spiking neural network at the corresponding receiver pitch neurons.

Interactive Evaluation of a Music Preprocessing Scheme for Cochlear Implants Based on Spectral Complexity Reduction

Music is difficult to access for the majority of CI users as the reduced dynamic range and poor spectral resolution in cochlear implants (CI), amongst others constraints, severely impair their auditory perception. The reduction of spectral complexity is therefore a promising means to facilitate music enjoyment for CI listeners. We evaluate a spectral complexity reduction method for music signals based on principal component analysis that enforces spectral sparsity, emphasizes the melody contour and attenuates interfering accompanying voices. To cover a wide range of spectral complexity reduction levels a new experimental design for listening experiments was introduced. It allows CI users to select the preferred level of spectral complexity reduction interactively and in real-time. Ten adult CI recipients with post-lingual bilateral profound sensorineural hearing loss and CI experience of at least 6 months were enrolled in the study. In eight consecutive sessions over a period of 4 weeks they were asked to choose their preferred version out of 10 different complexity settings for a total number of 16 recordings of classical western chamber music. As the experiments were performed in consecutive sessions we also studied a potential long term effect. Therefore, we investigated the hypothesis that repeated engagement with music signals of reduced spectral complexity leads to a habituation effect which allows CI users to deal with music signals of increasing complexity. Questionnaires and tests about music listening habits and musical abilities complemented these experiments. The participants significantly preferred signals with high spectral complexity reduction levels over the unprocessed versions. While the results of earlier studies comprising only two preselected complexity levels were generally confirmed, this study revealed a tendency toward a selection of even higher spectral complexity reduction levels. Therefore, spectral complexity reduction for music signals is a useful strategy to enhance music enjoyment for CI users. Although there is evidence for a habituation effect in some subjects, such an effect has not been significant in general.

Pleasantness Ratings for Harmonic Intervals With Acoustic and Electric Hearing in Unilaterally Deaf Cochlear Implant Patients

Background

Harmony is an important part of tonal music that conveys context, form and emotion. Two notes sounded simultaneously form a harmonic interval. In normal-hearing (NH) listeners, some harmonic intervals (e.g., minor 2nd, tritone, major 7th) typically sound more dissonant than others (e.g., octave, major 3rd, 4th). Because of the limited spectro-temporal resolution afforded by cochlear implants (CIs), music perception is generally poor. However, CI users may still be sensitive to relative dissonance across intervals. In this study, dissonance ratings for harmonic intervals were measured in 11 unilaterally deaf CI patients, in whom ratings from the CI could be compared to those from the normal ear.

Methods

Stimuli consisted of pairs of equal amplitude MIDI piano tones. Intervals spanned a range of two octaves relative to two root notes (F3 or C4). Dissonance was assessed in terms of subjective pleasantness ratings for intervals presented to the NH ear alone, the CI ear alone, and both ears together (NH + CI). Ratings were collected for both root notes for within- and across-octave intervals (1–12 and 13–24 semitones). Participants rated the pleasantness of each interval by clicking on a line anchored with “least pleasant” and “most pleasant.” A follow-up experiment repeated the task with a smaller stimulus set.

Results

With NH-only listening, within-octave intervals minor 2nd, major 2nd, and major 7th were rated least pleasant; major 3rd, 5th, and octave were rated most pleasant. Across-octave counterparts were similarly rated. With CI-only listening, ratings were consistently lower and showed a reduced range. Mean ratings were highly correlated between NH-only and CI-only listening (r = 0.845, p < 0.001). Ratings were similar between NH-only and NH + CI listening, with no significant binaural enhancement/interference. The follow-up tests showed that ratings were reliable for the least and most pleasant intervals.

Discussion

Although pleasantness ratings were less differentiated for the CI ear than the NH ear, there were similarities between the two listening modes. Given the lack of spectro-temporal detail needed for harmonicity-based distinctions, temporal envelope interactions (within and across channels) associated with a perception of roughness may contribute to dissonance perception for harmonic intervals with CI-only listening.

Perceptual Differences Between Low-Frequency Analog and Pulsatile Stimulation as Shown by Single- and Multidimensional Scaling

Cochlear-implant users who have experienced both analog and pulsatile sound coding strategies often have strong preferences for the sound quality of one over the other. This suggests that analog and pulsatile stimulation may provide different information or sound quality to an implant listener. It has been well documented that many implant listeners both prefer and perform better with multichannel analog than multichannel pulsatile strategies, although the reasons for these differences remain unknown. Here, we examine the perceptual differences between analog and pulsatile stimulation on a single electrode. A multidimensional scaling task, analyzed across two dimensions, suggested that pulsatile stimulation was perceived to be considerably different from analog stimulation. Two associated tasks using single-dimensional scaling showed that analog stimulation was perceived to be less Clean on average than pulsatile stimulation and that the perceptual differences were not related to pitch. In a follow-up experiment, it was determined that the perceptual differences between analog and pulsatile stimulation were not dependent on the interpulse gap present in pulsatile stimulation. Although the results suggest that there is a large perceptual difference between analog and pulsatile stimulation, further work is needed to determine the nature of these differences.

Modeling Pitch Perception With an Active Auditory Model Extended by Octopus Cells

Pitch is an essential category for musical sensations. Models of pitch perception are vividly discussed up to date. Most of them rely on definitions of mathematical methods in the spectral or temporal domain. Our proposed pitch perception model is composed of an active auditory model extended by octopus cells. The active auditory model is the same as used in the Stimulation based on Auditory Modeling (SAM), a successful cochlear implant sound processing strategy extended here by modeling the functional behavior of the octopus cells in the ventral cochlear nucleus and by modeling their connections to the auditory nerve fibers (ANFs). The neurophysiological parameterization of the extended model is fully described in the time domain. The model is based on latency-phase en- and decoding as octopus cells are latency-phase rectifiers in their local receptive fields. Pitch is ubiquitously represented by cascaded firing sweeps of octopus cells. Based on the firing patterns of octopus cells, inter-spike interval histograms can be aggregated, in which the place of the global maximum is assumed to encode the pitch.