Melodic interval perception by normal-hearing listeners and cochlear implant users

The role of carrier spectral composition in the perception of musical pitch

Temporal envelope fluctuations of natural sounds convey critical information to speech and music processing. In particular, musical pitch perception is assumed to be primarily underlined by temporal envelope encoding. While increasing evidence demonstrates the importance of carrier fine structure to complex pitch perception, how carrier spectral information affects musical pitch perception is less clear. Here, transposed tones designed to convey identical envelope information across different carriers were used to assess the effects of carrier spectral composition to pitch discrimination and musical-interval and melody identifications. Results showed that pitch discrimination thresholds became lower (better) with increasing carrier frequencies from 1k to 10k Hz, with performance comparable to that of pure sinusoids. Musical interval and melody defined by the periodicity of sine- or harmonic complex envelopes across carriers were identified with greater than 85% accuracy even on a 10k-Hz carrier. Moreover, enhanced interval and melody identification performance was observed with increasing carrier frequency up to 6k Hz. Findings suggest a perceptual enhancement of temporal envelope information with increasing carrier spectral region in musical pitch processing, at least for frequencies up to 6k Hz. For carriers in the extended high-frequency region (8-20k Hz), the use of temporal envelope information to music pitch processing may vary depending on task requirement. Collectively, these results implicate the fidelity of temporal envelope information to musical pitch perception is more pronounced than previously considered, with ecological implications.

Why is the perceptual octave stretched? An account based on mismatched time constants within the auditory brainstem

This paper suggests an explanation for listeners' greater tolerance to positive than negative mistuning of the higher tone within an octave pair. It hypothesizes a neural circuit tuned to cancel the lower tone that also cancels the higher tone if that tone is in tune. Imperfect cancellation is the cue to mistuning of the octave. The circuit involves two neural pathways, one delayed with respect to the other, that feed a coincidence-sensitive neuron via excitatory and inhibitory synapses. A mismatch between the time constants of these two synapses results in an asymmetry in sensitivity to mismatch. Specifically, if the time constant of the delayed pathway is greater than that of the direct pathway, there is a greater tolerance to positive mistuning than to negative mistuning. The model is directly applicable to the harmonic octave (concurrent tones) but extending it to the melodic octave (successive tones) requires additional assumptions that are discussed. The paper reviews evidence from auditory psychophysics and physiology in favor-or against-this explanation.

Musical Interval Perception With a Cochlear Implant Alone and With a Contralateral Normal Hearing Ear

Music through a cochlear implant (CI) is described as out-of-tune, suggesting that musical intervals are not accurately provided by a CI. One potential reason is that pitch may be insufficiently conveyed to provide reliable intervals. Another potential reason is that the size of intervals is distorted through a CI as they would be when produced by a mistuned piano. To measure intervals through a CI, listeners selected prerecorded vowels with different fundamental frequencies to represent each note in Happy Birthday. Each listener had contralateral normal hearing (NH); repeating the experiment with their NH ear allowed for a within-subject control. Additionally, the effect of listening simultaneously to both a CI and NH ear was measured. The resulting versions of Happy Birthday were analyzed in terms of their contours, interval sizes, magnitudes, consistency, and direction. Intervals with NH ears ranged from perfect to uncorrelated with target intervals. Chosen interval size with the CI was poorer than with the NH ear for all subjects. Across listeners, chosen intervals with the CI ranged from highly correlated to uncorrelated with target intervals. That CI intervals were highly correlated with target intervals for some listeners suggests that accurate intervals can be provided through a CI. For some listeners, chosen intervals were larger than target intervals, suggesting that intervals may be perceived as too small. Overall, intervals with the combination of the NH and CI ears were similar to those with the NH ear alone, suggesting that the addition of a CI has little-to-no effect on interval perception.

Computer-based musical interval training program for Cochlear implant users and listeners with no known hearing loss

A musical interval is the difference in pitch between two sounds. The way that musical intervals are used in melodies relative to the tonal center of a key can strongly affect the emotion conveyed by the melody. The present study examines musical interval identification in people with no known hearing loss and in cochlear implant users. Pitch resolution varies widely among cochlear implant users with average resolution an order of magnitude worse than in normal hearing. The present study considers the effect of training on musical interval identification and tests for correlations between low-level psychophysics and higher-level musical abilities. The overarching hypothesis is that cochlear implant users are limited in their ability to identify musical intervals both by low-level access to frequency cues for pitch as well as higher-level mapping of the novel encoding of pitch that implants provide. Participants completed a 2-week, online interval identification training. The benchmark tests considered before and after interval identification training were pure tone detection thresholds, pure tone frequency discrimination, fundamental frequency discrimination, tonal and rhythm comparisons, and interval identification. The results indicate strong correlations between measures of pitch resolution with interval identification; however, only a small effect of training on interval identification was observed for the cochlear implant users. Discussion focuses on improving access to pitch cues for cochlear implant users and on improving auditory training for musical intervals.

Reverberation Degrades Pitch Perception but Not Mandarin Tone and Vowel Recognition of Cochlear Implant Users

OBJECTIVES

The primary goal of this study was to investigate the effects of reverberation on Mandarin tone and vowel recognition of cochlear implant (CI) users and normal-hearing (NH) listeners. To understand the performance of Mandarin tone recognition, this study also measured participants' pitch perception and the availability of temporal envelope cues in reverberation.

DESIGN

Fifteen CI users and nine NH listeners, all Mandarin speakers, were asked to recognize Mandarin single-vowels produced in four lexical tones and rank harmonic complex tones in pitch with different reverberation times (RTs) from 0 to 1 second. Virtual acoustic techniques were used to simulate rooms with different degrees of reverberation. Vowel duration and correlation between amplitude envelope and fundamental frequency (F0) contour were analyzed for different tones as a function of the RT.

RESULTS

Vowel durations of different tones significantly increased with longer RTs. Amplitude-F0 correlation remained similar for the falling Tone 4 but greatly decreased for the other tones in reverberation. NH listeners had robust pitch-ranking, tone recognition, and vowel recognition performance as the RT increased. Reverberation significantly degraded CI users' pitch-ranking thresholds but did not significantly affect the overall scores of tone and vowel recognition with CIs. Detailed analyses of tone confusion matrices showed that CI users reduced the flat Tone-1 responses but increased the falling Tone-4 responses in reverberation, possibly due to the falling amplitude envelope of late reflections after the original vowel segment. CI users' tone recognition scores were not correlated with their pitch-ranking thresholds.

CONCLUSIONS

NH listeners can reliably recognize Mandarin tones in reverberation using salient pitch cues from spectral and temporal fine structures. However, CI users have poorer pitch perception using F0-related amplitude modulations that are reduced in reverberation. Reverberation distorts speech amplitude envelopes, which affect the distribution of tone responses but not the accuracy of tone recognition with CIs. Recognition of vowels with stationary formant trajectories is not affected by reverberation for both NH listeners and CI users, regardless of the available spectral resolution. Future studies should test how the relatively stable vowel and tone recognition may contribute to sentence recognition in reverberation of Mandarin-speaking CI users.

Pure tone discrimination with cochlear implants and filter-band spread

For many cochlear implant (CI) users, frequency discrimination is still challenging. We studied the effect of frequency differences relative to the electrode frequency bands on pure tone discrimination. A single-center, prospective, controlled, psychoacoustic exploratory study was conducted in a tertiary university referral center. Thirty-four patients with Cochlear Ltd. and MED-EL CIs and 19 age-matched normal-hearing control subjects were included. Two sinusoidal tones were presented with varying frequency differences. The reference tone frequency was chosen according to the center frequency of basal or apical electrodes. Discrimination abilities were psychophysically measured in a three-interval, two-alternative, forced-choice procedure (3I-2AFC) for various CI electrodes. Hit rates were measured, particularly with respect to discrimination abilities at the corner frequency of the electrode frequency-bands. The mean rate of correct decision concerning pitch difference was about 60% for CI users and about 90% for the normal-hearing control group. In CI users, the difference limen was two semitones, while normal-hearing participants detected the difference of one semitone. No influence of the corner frequency of the CI electrodes was found. In CI users, pure tone discrimination seems to be independent of tone positions relative to the corner frequency of the electrode frequency-band. Differences of 2 semitones can be distinguished within one electrode.

Melodic interval perception with acoustic and electric hearing in bimodal and single-sided deaf cochlear implant listeners

Two notes sounded sequentially elicit melodic intervals and contours that form the basis of melody. Many previous studies have characterized pitch perception in cochlear implant (CI) users to be poor which may be due to the limited spectro-temporal resolution and/or spectral warping with electric hearing compared to acoustic hearing (AH). Poor pitch perception in CIs has been shown to distort melodic interval perception. To characterize this interval distortion, we recruited CI users with either normal (single sided deafness, SSD) or limited (bimodal) AH in the non-implanted ear. The contralateral AH allowed for a stable reference with which to compare melodic interval perception in the CI ear, within the same listener. Melodic interval perception was compared across acoustic and electric hearing in 9 CI listeners (4 bimodal and 5 SSD). Participants were asked to rank the size of a probe interval presented to the CI ear to a reference interval presented to the contralateral AH ear using a method of constant stimuli. Ipsilateral interval ranking was also measured within the AH ear to ensure that listeners understood the task and that interval ranking was stable and accurate within AH. Stimuli were delivered to the AH ear via headphones and to the CI ear via direct audio input (DAI) to participants' clinical processors. During testing, a reference and probe interval was presented and participants indicated which was larger. Ten comparisons for each reference-probe combination were presented. Psychometric functions were fit to the data to determine the probe interval size that matched the reference interval. Across all AH reference intervals, the mean matched CI interval was 1.74 times larger than the AH reference. However, there was great inter-subject variability. For some participants, CI interval distortion varied across different reference AH intervals; for others, CI interval distortion was constant. Within the AH ear, ipsilateral interval ranking was accurate, ensuring that participants understood the task. No significant differences in the patterns of results were observed between bimodal and SSD CI users. The present data show that much larger intervals were needed with the CI to match contralateral AH reference intervals. As such, input melodic patterns are likely to be perceived as frequency compressed and/or warped with electric hearing, with less variation among notes in the pattern. The high inter-subject variability in CI interval distortion suggests that CI signal processing should be optimized for individual CI users.

Place-Pitch Interval Perception With a Cochlear Implant

OBJECTIVES

Pitch is poorly perceived by cochlear implant (CI) users. However, as it is not well understood how pitch is encoded with electric stimulation, improving pitch representation with a CI is challenging. Changes in place of stimulation along the cochlea have been described as changes in pitch and can be accurately ranked by CI users. However, it remains unknown if place-pitch can be used to encode musical intervals, which are a necessary attribute of pitch. The objective of these experiments is to determine if place-pitch coding can be used to represent musical intervals with a CI.

DESIGN

In the first experiment, 10 CI users and 10 normal hearing (NH) controls were tested on their sensitivity to changes in the semitone spacing between each of the notes in the melody "Happy Birthday." The changes were implemented by uniformly expanding or compressing the frequency differences between each note in the melody. The participant's task was to scale how "out-of-tune" the melody was for various semitone spacing distortions. The notes were represented by pure-tones ≥440 Hz to minimize potential useful temporal information from the stimuli. A second experiment replicated the first experiment using single-sided deafened CI users allowing for a within-subject control. A third experiment verified that the CI users who participated in Experiment 1 were each able to determine pitch direction reliably.

RESULTS

Unlike NH listeners, CI listeners often ranked all distortions of interval spacing similarly in both the first and second experiment, and no effect of interval spacing was detected across CI users. Some participants found distorted interval spacings to be less out-of-tune than the nominally correct interval spacings. However, these patterns were inconsistent across listeners. Although performance was better for the NH listeners, the third experiment demonstrated that the CI listeners were able to reliably identify changes in pitch direction from place-pitch coding.

CONCLUSIONS

The data suggest that place-pitch intervals are not properly represented through a CI sound processor. Some limited support is found for place-pitch being useful for interval encoding as some participants demonstrated improved ratings for certain interval distortions. Presumably the interval representation for these participants could be improved by a change to the frequencies represented by each electrode. However, as these patterns vary across listeners, there is not a universal correction to frequency representation that will solve this issue. As results are similar for single-sided deafened CI users, the limitations in ratings are likely not limited by an eroded representation of the melody caused by an extended duration of deafness.

Effect of instrument timbre on musical emotion recognition in normal-hearing listeners and cochlear implant users

The effect of instrument timbre (e.g., piano, trumpet, organ, and violin) on musical emotion recognition was tested in normal-hearing (NH) listeners and cochlear implant (CI) users. NH performance was best with the piano, and did not change when melodies were normalized with a fixed tempo. CI performance was significantly better with the piano and trumpet than with the violin and organ when both tempo and mode cues were preserved, but not for the tempo-normalized melodies. The sharper temporal onsets of piano and trumpet may enhance CI users' perception of tempo cues important for musical emotion recognition.

Vibrotactile Stimulation Based on the Fundamental Frequency Can Improve Melodic Contour Identification of Normal-Hearing Listeners With a 4-Channel Cochlear Implant Simulation

Cochlear implant (CI) users' poor speech recognition in noise and music perception may be both due to their limited access to pitch cues such as the fundamental frequency (F0). Recent studies showed that similar to residual low-frequency acoustic hearing, vibrotactile presentation of the F0 significantly improved speech recognition in noise of CI users. The present study tested whether F0-based vibrotactile stimulation can improve melodic contour identification (MCI) of normal-hearing listeners with acoustically simulated CI processing. Each melodic contour consisted of five musical notes with one of nine contour patterns (rising, falling, or flat in each half of the contour). The F0 of the middle note was 220 or 880 Hz, and the frequency intervals between adjacent notes were 1, 3, or 5 semitones. The F0 of each note was extracted in real time and transposed to a vibration frequency centered around 110 Hz at the right forearm top. MCI was tested in five experimental conditions (with a 4- or 8-channel CI simulation alone, vibrotactile stimulation alone, and 4- or 8-channel CI simulation plus vibrotactile stimulation), each after the same amount of brief training was provided. Results showed that discrimination of vibrotactile stimuli significantly improved from chance to near perfect as the vibration frequency interval increased from 0.25 to 3 semitones. The MCI performance with vibrotactile stimulation alone was similar to that with the 4-channel CI simulation alone, but was significantly worse than that with the 8-channel CI simulation alone. Significant improvement in MCI performance with the addition of vibrotactile stimulation was only found with the 4-channel CI simulation when the middle F0 was 880 Hz and when the frequency intervals were 3 or 5 semitones. The improvement in MCI performance with than without vibrotactile stimulation was significantly correlated with the baseline MCI performance with 4-channel CI simulation alone or with the MCI performance difference between vibrotactile stimulation and 8-channel CI simulation. Therefore, when the simulated or real CI performance is relatively poor, vibrotactile stimulation based on the F0 may improve MCI with acoustic CI simulations and perhaps in real CI users as well.

High Oxygen Exchange to Music Indicates Auditory Distractibility in Acquired Brain Injury: An fNIRS Study with a Vector-Based Phase Analysis

Attention deficits due to auditory distractibility are pervasive among patients with acquired brain injury (ABI). It remains unclear, however, whether attention deficits following ABI specific to auditory modality are associated with altered haemodynamic responses. Here, we examined cerebral haemodynamic changes using functional near-infrared spectroscopy combined with a topological vector-based analysis method. A total of thirty-seven participants (22 healthy adults, 15 patients with ABI) performed a melodic contour identification task (CIT) that simulates auditory distractibility. Findings demonstrated that the melodic CIT was able to detect auditory distractibility in patients with ABI. The rate-corrected score showed that the ABI group performed significantly worse than the non-ABI group in both CIT1 (target contour identification against environmental sounds) and CIT2 (target contour identification against target-like distraction). Phase-associated response intensity during the CITs was greater in the ABI group than in the non-ABI group. Moreover, there existed a significant interaction effect in the left dorsolateral prefrontal cortex (DLPFC) during CIT1 and CIT2. These findings indicated that stronger hemodynamic responses involving oxygen exchange in the left DLPFC can serve as a biomarker for evaluating and monitoring auditory distractibility, which could potentially lead to the discovery of the underlying mechanism that causes auditory attention deficits in patients with ABI.

Interaction Between Pitch and Timbre Perception in Normal-Hearing Listeners and Cochlear Implant Users

Despite their mutually exclusive definitions, pitch and timbre perception interact with each other in normal-hearing (NH) listeners. Cochlear implant (CI) users have worse than normal pitch and timbre perception. However, the pitch-timbre interaction with CIs is not well understood. This study tested the interaction between pitch and sharpness (an aspect of timbre) perception related to the fundamental frequency (F0) and spectral slope of harmonic complex tones, respectively, in both NH listeners and CI users. In experiment 1, the F0 (and spectral slope) difference limens (DLs) were measured with a fixed spectral slope (and F0) and 20-dB amplitude roving. Then, the F0 and spectral slope were varied congruently or incongruently by the same multiple of individual DLs to assess the pitch and sharpness ranking sensitivity. Both NH and CI subjects had significantly higher pitch and sharpness ranking sensitivity with congruent than with incongruent F0 and spectral slope variations, and showed a similar symmetric interaction between pitch and timbre perception. In experiment 2, CI users' melodic contour identification (MCI) was tested in three spectral slope (no, congruent, and incongruent spectral slope variations by the same multiple of individual DLs as the F0 variations) and two amplitude conditions (0- and 20-dB amplitude roving). When there was no amplitude roving, the MCI scores were significantly higher with congruent than with no, and in turn than with incongruent spectral slope variations. The 20-dB amplitude roving significantly reduced the overall MCI scores and the effect of spectral slope variations. These results reflected a confusion between higher (or lower) pitch and sharper (or duller) timbre and offered important implications for understanding and enhancing pitch and timbre perception with CIs.

Mismatch negativity reflects asymmetric pre-attentive harmonic interval discrimination

Objective

Western music is based on intervals; thus, interval discrimination is important for distinguishing the character of melodies or tracking melodies in polyphonic music. In this study the encoding of intervals in simultaneously presented sound is studied.

Study design

In an electrophysiological experiment in 15 normal-hearing non-musicians, major thirds or fifths were presented in a controlled oddball paradigm. Harmonic intervals were created by simultaneously presented sinusoidals with randomized root frequency. Mismatch negativity (MMN) responses were measured with an EEG recording. The discrimination index was calculated in a psychoacoustic experiment.

Results

A clear MMN response was found for the major third but not for the fifth. The neural generators were located within the auditory cortices. Psychoacoustically, no evidence was found that the subjects were able to detect the deviants.

Conclusions

We conclude that pre-attentive discrimination of harmonic interval size is, in principle, possible in listeners without musical training although simultaneous presentation makes it harder to distinguish compared to non-overlapping intervals. Furthermore we see a difference in the response to infrequent dissonant stimuli in consonant standard stimuli compared to the opposite, rare consonant stimuli in dissonant standard stimuli.

Pitch matching in bimodal cochlear implant patients: Effects of frequency, spectral envelope, and level

This study systematically investigated the effects of frequency, level, and spectral envelope on pitch matching in twelve bimodal cochlear implant (CI) users. The participants were asked to vary the frequency and level of a pure or complex tone (adjustable sounds) presented in the non-implanted ear to match the pitch and loudness of different reference stimuli presented to the implanted ear. Three reference sounds were used: single electrode pulse trains, pure tones, and piano notes. The data showed a significant effect of the frequency and complexity of the reference sounds. No significant effect of the level of the reference sounds was found. The magnitude of effect of frequency was compressed in the implanted ear: on average a difference of seven semitones in the non-implanted ear induced the same pitch change as a difference of 19 to 24 semitones for a stimulus presented to the implanted ear. The spectral envelope of the adjustable sound presented to the non-implanted ear also had a significant effect. The matched frequencies were higher by an average of six semitones for the pure tone compared to a complex tone. Overall, the CI listeners might have matched the stimuli based on timbre characteristics such as brightness.

Standard-interval size affects interval-discrimination thresholds for pure-tone melodic pitch intervals

Our ability to discriminate between pitch intervals of different sizes is not only an important aspect of speech and music perception, but also a useful means of evaluating higher-level pitch perception. The current study examined how pitch-interval discrimination was affected by the size of the intervals being compared, and by musical training. Using an adaptive procedure, pitch-interval discrimination thresholds were measured for sequentially presented pure-tone intervals with standard intervals of 1 semitone (minor second), 6 semitones (the tri-tone), and 7 semitones (perfect fifth). Listeners were classified into three groups based on musical experience: non-musicians had less than 3 years of informal musical experience, amateur musicians had at least 10 years of experience but no formal music theory training, and expert musicians had at least 12 years of experience with 1 year of formal ear training, and were either currently pursuing or had earned a Bachelor's degree as either a music major or music minor. Consistent with previous studies, discrimination thresholds obtained from expert musicians were significantly lower than those from other listeners. Thresholds also significantly varied with the magnitude of the reference interval and were higher for conditions with a 6- or 7-semitone standard than a 1-semitone standard. These data show that interval-discrimination thresholds are strongly affected by the size of the standard interval.

Encoding a Melody Using Only Temporal Information for Cochlear-Implant and Normal-Hearing Listeners

One way to provide pitch information to cochlear implant users is through amplitude-modulation rate. It is currently unknown whether amplitude-modulation rate can provide cochlear implant users with pitch information adequate for perceiving melodic information. In the present study, the notes of a song were encoded via amplitude-modulation rate of pulse trains on single electrodes at the apex or middle of long electrode arrays. The melody of the song was either physically correct or modified by compression or expansion. Nine cochlear implant users rated the extent to which the song was out of tune in the different conditions. Cochlear implant users on average did not show sensitivity to melody compression or expansion regardless of place of stimulation. These results were found despite the fact that three of the cochlear implant users showed the expected sensitivity to melody compression and expansion with the same task using acoustic pure tones in a contralateral acoustic ear. Normal-hearing listeners showed an inconsistent and weak effect of melody compression and expansion when the notes of the song were encoded with acoustic pulse rate. The results suggest that amplitude-modulation rate provides insufficient access to melodic information for cochlear-implant and normal-hearing listeners.

Melodic Contour Identification Reflects the Cognitive Threshold of Aging

Cognitive decline is a natural phenomenon of aging. Although there exists a consensus that sensitivity to acoustic features of music is associated with such decline, no solid evidence has yet shown that structural elements and contexts of music explain this loss of cognitive performance. This study examined the extent and the type of cognitive decline that is related to the contour identification task (CIT) using tones with different pitches (i.e., melodic contours). Both younger and older adult groups participated in the CIT given in three listening conditions (i.e., focused, selective, and alternating). Behavioral data (accuracy and response times) and hemodynamic reactions were measured using functional near-infrared spectroscopy (fNIRS). Our findings showed cognitive declines in the older adult group but with a subtle difference from the younger adult group. The accuracy of the melodic CITs given in the target-like distraction task (CIT2) was significantly lower than that in the environmental noise (CIT1) condition in the older adult group, indicating that CIT2 may be a benchmark test for age-specific cognitive decline. The fNIRS findings also agreed with this interpretation, revealing significant increases in oxygenated hemoglobin (oxyHb) concentration in the younger (p < 0.05 for Δpre - on task; p < 0.01 for Δon – post task) rather than the older adult group (n.s for Δpre - on task; n.s for Δon – post task). We further concluded that the oxyHb difference was present in the brain regions near the right dorsolateral prefrontal cortex. Taken together, these findings suggest that CIT2 (i.e., the melodic contour task in the target-like distraction) is an optimized task that could indicate the degree and type of age-related cognitive decline.

Mandarin speech-in-noise and tone recognition using vocoder simulations of the temporal limits encoder for cochlear implants

Temporal envelope-based signal processing strategies are widely used in cochlear-implant (CI) systems. It is well recognized that the inability to convey temporal fine structure (TFS) in the stimuli limits CI users' performance, but it is still unclear how to effectively deliver the TFS. A strategy known as the temporal limits encoder (TLE), which employs an approach to derive the amplitude modulator to generate the stimuli coded in an interleaved-sampling strategy, has recently been proposed. The TLE modulator contains information related to the original temporal envelope and a slow-varying TFS from the band signal. In this paper, theoretical analyses are presented to demonstrate the superiority of TLE compared with two existing strategies, the clinically available continuous-interleaved-sampling (CIS) strategy and the experimental harmonic-single-sideband-encoder strategy. Perceptual experiments with vocoder simulations in normal-hearing listeners are conducted to compare the performance of TLE and CIS on two tasks (i.e., Mandarin speech reception in babble noise and tone recognition in quiet). The performance of the TLE modulator is mostly better than (for most tone-band vocoders) or comparable to (for noise-band vocoders) the CIS modulator on both tasks. This work implies that there is some potential for improving the representation of TFS with CIs by using a TLE strategy.