Second-order temporal modulation transfer functions

On mistuning detection and beat perception for harmonic complex tones at low and very high frequencies

This study assessed the detection of mistuning of a single harmonic in complex tones (CTs) containing either low-frequency harmonics or very high-frequency harmonics, for which phase locking to the temporal fine structure is weak or absent. CTs had F0s of either 280 or 1400 Hz and contained harmonics 6-10, the 8th of which could be mistuned. Harmonics were presented either diotically or dichotically (odd and even harmonics to different ears). In the diotic condition, mistuning-detection thresholds were very low for both F0s and consistent with detection of temporal interactions (beats) produced by peripheral interactions of components. In the dichotic condition, for which the components in each ear were more widely spaced and beats were not reported, the mistuned component was perceptually segregated from the complex for the low F0, but subjects reported no "popping out" for the high F0 and performance was close to chance. This is consistent with the idea that phase locking is required for perceptual segregation to occur. For diotic presentation, the perceived beat rate corresponded to the amount of mistuning (in Hz). It is argued that the beat percept cannot be explained solely by interactions between the mistuned component and its two closest harmonic neighbours.

Double-pass consistency for amplitude- and frequency-modulation detection in normal-hearing listeners

Amplitude modulation (AM) and frequency modulation (FM) provide crucial auditory information. If FM is encoded as AM, it should be possible to give a unified account of AM and FM perception both in terms of response consistency and performance. These two aspects of behavior were estimated for normal-hearing participants using a constant-stimuli, forced-choice detection task repeated twice with the same stimuli (double pass). Sinusoidal AM or FM with rates of 2 or 20 Hz were applied to a 500-Hz pure-tone carrier and presented at detection threshold. All stimuli were masked by a modulation noise. Percent agreement of responses across passes and percent-correct detection for the two passes were used to estimate consistency and performance, respectively. These data were simulated using a model implementing peripheral processes, a central modulation filterbank, an additive internal noise, and a template-matching device. Different levels of internal noise were required to reproduce AM and FM data, but a single level could account for the 2- and 20-Hz AM data. As for FM, two levels of internal noise were needed to account for detection at slow and fast rates. Finally, the level of internal noise yielding best predictions increased with the level of the modulation-noise masker. Overall, these results suggest that different sources of internal variability are involved for AM and FM detection at low audio frequencies.

Cascaded Tuning to Amplitude Modulation for Natural Sound Recognition

The auditory system converts the physical properties of a sound waveform to neural activities and processes them for recognition. During the process, the tuning to amplitude modulation (AM) is successively transformed by a cascade of brain regions. To test the functional significance of the AM tuning, we conducted single-unit recording in a deep neural network (DNN) trained for natural sound recognition. We calculated the AM representation in the DNN and quantitatively compared it with those reported in previous neurophysiological studies. We found that an auditory-system-like AM tuning emerges in the optimized DNN. Better-recognizing models showed greater similarity to the auditory system. We isolated the factors forming the AM representation in the different brain regions. Because the model was not designed to reproduce any anatomical or physiological properties of the auditory system other than the cascading architecture, the observed similarity suggests that the AM tuning in the auditory system might also be an emergent property for natural sound recognition during evolution and development.SIGNIFICANCE STATEMENT This study suggests that neural tuning to amplitude modulation may be a consequence of the auditory system evolving for natural sound recognition. We modeled the function of the entire auditory system; that is, recognizing sounds from raw waveforms with as few anatomical or physiological assumptions as possible. We analyzed the model using single-unit recording, which enabled a fair comparison with neurophysiological data with as few methodological biases as possible. Interestingly, our results imply that frequency decomposition in the inner ear might not be necessary for processing amplitude modulation. This implication could not have been obtained if we had used a model that assumes frequency decomposition.

Accounting for masking of frequency modulation by amplitude modulation with the modulation filter-bank concept

Frequency modulation (FM) is assumed to be detected through amplitude modulation (AM) created by cochlear filtering for modulation rates above 10 Hz and carrier frequencies (f_c) above 4 kHz. If this is the case, a model of modulation perception based on the concept of AM filters should predict masking effects between AM and FM. To test this, masking effects of sinusoidal AM on sinusoidal FM detection thresholds were assessed on normal-hearing listeners as a function of FM rate, f_c, duration, AM rate, AM depth, and phase difference between FM and AM. The data were compared to predictions of a computational model implementing an AM filter-bank. Consistent with model predictions, AM masked FM with some AM-masking-AM features (broad tuning and effect of AM-masker depth). Similar masking was predicted and observed at f_c = 0.5 and 5 kHz for a 2 Hz AM masker, inconsistent with the notion that additional (e.g., temporal fine-structure) cues drive slow-rate FM detection at low f_c. However, masking was lower than predicted and, unlike model predictions, did not show beating or phase effects. Broadly, the modulation filter-bank concept successfully explained some AM-masking-FM effects, but could not give a complete account of both AM and FM detection.

Sensorineural hearing loss impairs sensitivity but spares temporal integration for detection of frequency modulation

The effect of the number of modulation cycles (N) on frequency-modulation (FM) detection thresholds (FMDTs) was measured with and without interfering amplitude modulation (AM) for hearing-impaired (HI) listeners, using a 500-Hz sinusoidal carrier and FM rates of 2 and 20 Hz. The data were compared with FMDTs for normal-hearing (NH) listeners and AM detection thresholds (AMDTs) for NH and HI listeners [Wallaert, Moore, and Lorenzi (2016). J. Acoust. Soc. 139, 3088-3096; Wallaert, Moore, Ewert, and Lorenzi (2017). J. Acoust. Soc. 141, 971-980]. FMDTs were higher for HI than for NH listeners, but the effect of increasing N was similar across groups. In contrast, AMDTs were lower and the effect of increasing N was greater for HI listeners than for NH listeners. A model of temporal-envelope processing based on a modulation filter-bank and a template-matching decision strategy accounted better for the FMDTs at 20 Hz than at 2 Hz for young NH listeners and predicted greater temporal integration of FM than observed for all groups. These results suggest that different mechanisms underlie AM and FM detection at low rates and that hearing loss impairs FM-detection mechanisms, but preserves the memory and decision processes responsible for temporal integration of FM.

Cascaded Amplitude Modulations in Sound Texture Perception

Sound textures, such as crackling fire or chirping crickets, represent a broad class of sounds defined by their homogeneous temporal structure. It has been suggested that the perception of texture is mediated by time-averaged summary statistics measured from early auditory representations. In this study, we investigated the perception of sound textures that contain rhythmic structure, specifically second-order amplitude modulations that arise from the interaction of different modulation rates, previously described as “beating” in the envelope-frequency domain. We developed an auditory texture model that utilizes a cascade of modulation filterbanks that capture the structure of simple rhythmic patterns. The model was examined in a series of psychophysical listening experiments using synthetic sound textures—stimuli generated using time-averaged statistics measured from real-world textures. In a texture identification task, our results indicated that second-order amplitude modulation sensitivity enhanced recognition. Next, we examined the contribution of the second-order modulation analysis in a preference task, where the proposed auditory texture model was preferred over a range of model deviants that lacked second-order modulation rate sensitivity. Lastly, the discriminability of textures that included second-order amplitude modulations appeared to be perceived using a time-averaging process. Overall, our results demonstrate that the inclusion of second-order modulation analysis generates improvements in the perceived quality of synthetic textures compared to the first-order modulation analysis considered in previous approaches.

Sensorineural hearing loss enhances auditory sensitivity and temporal integration for amplitude modulation

Amplitude-modulation detection thresholds (AMDTs) were measured at 40 dB sensation level for listeners with mild-to-moderate sensorineural hearing loss (age: 50-64 yr) for a carrier frequency of 500 Hz and rates of 2 and 20 Hz. The number of modulation cycles, N, varied between two and nine. The data were compared with AMDTs measured for young and older normal-hearing listeners [Wallaert, Moore, and Lorenzi (2016). J. Acoust. Soc. Am. 139, 3088-3096]. As for normal-hearing listeners, AMDTs were lower for the 2-Hz than for the 20-Hz rate, and AMDTs decreased with increasing N. AMDTs were lower for hearing-impaired listeners than for normal-hearing listeners, and the effect of increasing N was greater for hearing-impaired listeners. A computational model based on the modulation-filterbank concept and a template-matching decision strategy was developed to account for the data. The psychophysical and simulation data suggest that the loss of amplitude compression in the impaired cochlea is mainly responsible for the enhanced sensitivity and temporal integration of temporal envelope cues found for hearing-impaired listeners. The data also suggest that, for AM detection, cochlear damage is associated with increased internal noise, but preserved short-term memory and decision mechanisms.

Amplitude modulation detection with concurrent frequency modulation

Human speech consists of concomitant temporal modulations in amplitude and frequency that are crucial for speech perception. In this study, amplitude modulation (AM) detection thresholds were measured for 550 and 5000 Hz carriers with and without concurrent frequency modulation (FM), at AM rates crucial for speech perception. Results indicate that adding 40 Hz FM interferes with AM detection, more so for 5000 Hz carrier and for frequency deviations exceeding the critical bandwidth of the carrier frequency. These findings suggest that future cochlear implant processors, encoding speech fine-structures may consider limiting the FM to narrow bandwidth and to low frequencies.

Dynamic Reweighting of Auditory Modulation Filters

Sound waveforms convey information largely via amplitude modulations (AM). A large body of experimental evidence has provided support for a modulation (bandpass) filterbank. Details of this model have varied over time partly reflecting different experimental conditions and diverse datasets from distinct task strategies, contributing uncertainty to the bandwidth measurements and leaving important issues unresolved. We adopt here a solely data-driven measurement approach in which we first demonstrate how different models can be subsumed within a common 'cascade' framework, and then proceed to characterize the cascade via system identification analysis using a single stimulus/task specification and hence stable task rules largely unconstrained by any model or parameters. Observers were required to detect a brief change in level superimposed onto random level changes that served as AM noise; the relationship between trial-by-trial noisy fluctuations and corresponding human responses enables targeted identification of distinct cascade elements. The resulting measurements exhibit a dynamic complex picture in which human perception of auditory modulations appears adaptive in nature, evolving from an initial lowpass to bandpass modes (with broad tuning, Q∼1) following repeated stimulus exposure.

The perception of speech modulation cues in lexical tones is guided by early language-specific experience

A number of studies showed that infants reorganize their perception of speech sounds according to their native language categories during their first year of life. Still, information is lacking about the contribution of basic auditory mechanisms to this process. This study aimed to evaluate when native language experience starts to noticeably affect the perceptual processing of basic acoustic cues [i.e., frequency-modulation (FM) and amplitude-modulation information] known to be crucial for speech perception in adults. The discrimination of a lexical-tone contrast (rising versus low) was assessed in 6- and 10-month-old infants learning either French or Mandarin using a visual habituation paradigm. The lexical tones were presented in two conditions designed to either keep intact or to severely degrade the FM and fine spectral cues needed to accurately perceive voice-pitch trajectory. A third condition was designed to assess the discrimination of the same voice-pitch trajectories using click trains containing only the FM cues related to the fundamental-frequency (F0) in French- and Mandarin-learning 10-month-old infants. Results showed that the younger infants of both language groups and the Mandarin-learning 10-month-olds discriminated the intact lexical-tone contrast while French-learning 10-month-olds failed. However, only the French 10-month-olds discriminated degraded lexical tones when FM, and thus voice-pitch cues were reduced. Moreover, Mandarin-learning 10-month-olds were found to discriminate the pitch trajectories as presented in click trains better than French infants. Altogether, these results reveal that the perceptual reorganization occurring during the first year of life for lexical tones is coupled with changes in the auditory ability to use speech modulation cues.

Lateralization based on interaural differences in the second-order amplitude modulator

Second-order amplitude modulation is a relatively slow variation of the modulation depth of a first-order amplitude modulation with higher frequency. In contrast to first-order modulation, which appears as a physical component in the stimulus spectrum after half-wave rectification, second-order modulation is not necessarily demodulated by the auditory periphery. For binaural processing of second-order amplitude modulated stimuli it is unknown whether interaural time differences (ITDs) in the second-order modulation result in a lateralized percept. Thus, second-order modulation can serve as a tool to investigate whether demodulation of interaurally delayed components is a prerequisite for lateralization. In most of the psychoacoustic experiments presented here, a 25 Hz sinusoidally amplitude-modulated (SAM) 160 Hz tone was either transposed to 4 kHz by half-wave rectifying this SAM waveform before multiplication with a 4 kHz tone (TSAM), or by adding an offset before multiplication (SAMAM). The experiments revealed an inability to lateralize the SAMAM based on ITDs in the 25 Hz component, whereas subjects could lateralize the TSAM. Given that only the TSAM results in a demodulated 25 Hz component after peripheral auditory processing, this result supports the hypothesis that demodulation is a prerequisite for lateralization, which has consequences for temporal modulation processing in models of binaural interaction.

Neural representations of complex temporal modulations in the human auditory cortex

Natural sounds such as speech contain multiple levels and multiple types of temporal modulations. Because of nonlinearities of the auditory system, however, the neural response to multiple, simultaneous temporal modulations cannot be predicted from the neural responses to single modulations. Here we show the cortical neural representation of an auditory stimulus simultaneously frequency modulated (FM) at a high rate, f(FM) approximately 40 Hz, and amplitude modulation (AM) at a slow rate, f(AM) <15 Hz. Magnetoencephalography recordings show fast FM and slow AM stimulus features evoke two separate but not independent auditory steady-state responses (aSSR) at f(FM) and f(AM), respectively. The power, rather than phase locking, of the aSSR of both decreases with increasing stimulus f(AM). The aSSR at f(FM) is itself simultaneously amplitude modulated and phase modulated with fundamental frequency f(AM), showing that the slow stimulus AM is not only encoded in the neural response at f(AM) but also encoded in the instantaneous amplitude and phase of the neural response at f(FM). Both the amplitude modulation and phase modulation of the aSSR at f(FM) are most salient for low stimulus f(AM) but remain observable at the highest tested f(AM) (13.8 Hz). The instantaneous amplitude of the aSSR at f(FM) is successfully predicted by a model containing temporal integration on two time scales, approximately 25 and approximately 200 ms, followed by a static compression nonlinearity.

Investigation of perceptual constancy in the temporal-envelope domain

The ability to discriminate complex temporal envelope patterns submitted to temporal compression or expansion was assessed in normal-hearing listeners. An XAB, matching-to-sample-procedure was used. X, the reference stimulus, is obtained by applying the sum of two, inharmonically related, sinusoids to a broadband noise carrier. A and B are obtained by multiplying the frequency of each modulation component of X by the same time expansion/compression factor, alpha (alphain[0.35-2.83]). For each trial, A or B is a time-reversed rendering of X, and the listeners' task is to choose which of the two is matched by X. Overall, the results indicate that discrimination performance degrades for increasing amounts of time expansion/compression (i.e., when alpha departs from 1), regardless of the frequency spacing of modulation components and the peak-to-trough ratio of the complex envelopes. An auditory model based on envelope extraction followed by a memory-limited, template-matching process accounted for results obtained without time scaling of stimuli, but generally underestimated discrimination ability with either time expansion or compression, especially with the longer stimulus durations. This result is consistent with partial or incomplete perceptual normalization of envelope patterns.

Cues for masked amplitude-modulation detection

The ability of psychoacoustic models to predict listeners' performance depends on two key stages: preprocessing and the generation of a decision variable. The goal of the current study was to determine the perceptually relevant decision variables in masked amplitude-modulation detection tasks in which the modulation depth of the masker was systematically varied. Potential cues were made unreliable by roving the overall modulation depth from trial to trial or were reduced in salience by equalizing the envelope energy of the standard and target after the signal was added. Listeners' performance was significantly degraded in both paradigms compared to the baseline (fixed-level modulation masker) condition, which was similar to those used in previous studies of masking in the envelope-frequency domain. Although this observation was broadly consistent with a simple long-term envelope power-spectrum model, there were several aspects of the data that were not. For example, the steep rate of change in threshold with masker depth and the fact that an optimal amount of envelope noise could enhance performance were not predicted by decision variables calculated directly from the stimulus envelope. A physiologically based processing model suggested a realistic nonlinear mechanism that could give rise to these second-order features of the data.

Effect of modulation maskers on the detection of second-order amplitude modulation with and without notched noise

The mechanisms underlying the detection of second-order amplitude modulation (AM) were explored. The detectability of second-order AM (fixed depth for each subject) was measured for first- and second-order modulation rates of 16 and 2 Hz, respectively (slow-rate pair), and 50 and 10 Hz, respectively (fast-rate pair), with no masker, a low-band modulation masker (centered at 2 or 10 Hz), and a high-band modulation masker (centered at 16 or 50 Hz). This was done in the absence and presence of an audio-frequency notched noise centered at the carrier frequency of 4000 Hz. Both modulation maskers were "low-noise" noises, to prevent overmodulation. In the absence of notched noise, both modulation maskers impaired performance for the slow-rate pair, but only the low-band masker impaired performance for the fast-rate pair. When notched noise was present, the low-band masker had no significant effect for either rate pair and the high-band masker had an effect only for the slow-rate pair. These results suggest that second-order AM detection is mediated both by an envelope distortion component at the second-order rate and by slow fluctuations in the output of a modulation filter tuned to the first-order rate. When notched noise is present, the distortion component plays little role.

Perception of the envelope-beat frequency of inharmonic complex temporal envelopes

Listeners can hear slow sinusoidal variations in the depth of sinusoidally amplitude-modulated (SAM) stimuli. Here, the SAM stimulus of frequency f(m) acts as the carrier, and the slow variation in depth of frequency f'm (referred to as "second-order" amplitude modulation) corresponds to a beat in the temporal envelope. Recent studies have suggested that second-order amplitude modulation perception is based on a modulation-distortion component or the "venelope" (the Hilbert envelope of the ac-coupled Hilbert envelope), both occurring at the envelope-beat frequency f'm. This was tested by transposing to the modulation domain the matching paradigm used by Schouten et al. [J. Acoust. Soc. Am. 34, 1418-1424 (1962)]. Listeners estimated the envelope-beat frequency evoked by a 5-Hz, second-order SAM white noise with f(m) either an integer multiple of f'm or shifted in frequency to make the complex envelope inharmonic. The results indicate that the perception of the envelope-beat frequency was affected by these shifts when f(m) < or = 20 Hz. This suggests that, at least at low modulation frequencies, the perceived envelope beat is not determined by a modulation-distortion or venelope component, but rather relies on the time intervals between the main peaks of the first-order envelope.

Masking release for consonant features in temporally fluctuating background noise

Consonant identification was measured for normal-hearing listeners using Vowel-Consonant-Vowel stimuli that were either unprocessed or spectrally degraded to force listeners to use temporal-envelope cues. Stimuli were embedded in a steady state or fluctuating noise masker and presented at a fixed signal-to-noise ratio. Fluctuations in the maskers were obtained by applying sinusoidal modulation to: (i) the amplitude of the noise (1st-order SAM masker) or (ii) the modulation depth of a 1st-order SAM noise (2nd-order SAM masker). The frequencies of the amplitude variation fm and the depth variation f'm were systematically varied. Consistent with previous studies, identification scores obtained with unprocessed speech were highest in an 8-Hz, 1st-order SAM masker. Reception of voicing and manner also peaked around fm=8 Hz, while the reception of place of articulation was maximal at a higher frequency (fm=32 Hz). When 2nd-order SAM maskers were used, identification scores and received information for each consonant feature were found to be independent of f'm. They decreased progressively with increasing carrier modulation frequency fm, and ranged between those obtained with the steady state and the 1st-order SAM maskers. Finally, the results obtained with spectrally degraded speech were similar across all types of maskers, although an 8% improvement in the reception of voicing was observed for modulated maskers with fm < 64 Hz compared to the steady-state masker. These data provide additional evidence that listeners take advantage of temporal minima in fluctuating background noises, and suggest that: (i) minima of different durations are required for an optimal reception of the three consonant features and (ii) complex (i.e., 2nd-order) envelope fluctuations in background noise do not degrade speech identification by interfering with speech-envelope processing.

Monaural and interaural temporal modulation transfer functions measured with 5-kHz carriers

Temporal modulation transfer functions (TMTFs) were measured for detection of monaural sinusoidal amplitude modulation and dynamically varying interaural level differences for a single set of listeners. For the interaural TMTFs, thresholds are the modulation depths at which listeners can just discriminate interaural envelope-phase differences of 0 and 180 degrees. A 5-kHz pure tone and narrowband noises, 30- and 300-Hz wide centered at 5 kHz, were used as carriers. In the interaural conditions, the noise carriers were either diotic or interaurally uncorrelated. The interaural TMTFs with tonal and diotic noise carriers exhibited a low-pass characteristic but the cutoff frequencies changed nonmonotonically with increasing bandwidth. The interaural TMTFs for the tonal carrier began rolling off approximately a half-octave lower than the tonal monaural TMTF (approximately 80 Hz vs approximately 120 Hz). Monaural TMTFs obtained with noise carriers showed effects attributable to masking of the signal modulation by intrinsic fluctuations of the carrier. In the interaural task with dichotic noise carriers, similar masking due to the interaural carrier fluctuations was observed. Although the mechanisms responsible for differences between the monaural and interaural TMTFs are unknown, the lower binaural TMTF cutoff frequency suggests that binaural processing exhibits greater temporal limitation than monaural processing.

Modulation masking produced by second-order modulators

Recent studies suggest that an auditory nonlinearity converts second-order sinusoidal amplitude modulation (SAM) (i.e., modulation of SAM depth) into a first-order SAM component, which contributes to the perception of second-order SAM. However, conversion may also occur in other ways such as cochlear filtering. The present experiments explored the source of the first-order SAM component by investigating the ability to detect a 5-Hz, first-order SAM probe in the presence of a second-order SAM masker beating at the probe frequency. Detection performance was measured as a function of masker-carrier modulation frequency, phase relationship between the probe and masker modulator, and probe modulation depth. In experiment 1, the carrier was a 5-kHz sinusoid presented either alone or within a notched-noise masker in order to restrict off-frequency listening. In experiment 2, the carrier was a white noise. The data obtained in both carrier conditions are consistent with the existence of a modulation distortion component. However, the phase yielding poorest detection performance varied across experimental conditions between 0 degrees and 180 degrees, confirming that, in addition to nonlinear mechanisms, cochlear filtering and off-frequency listening play a role in second-order SAM perception. The estimated magnitude of the modulation distortion component ranges from 5%-12%.

Estimation of the level and phase of the simple distortion tone the modulation domain

These experiments were designed to test the idea that nonlinearities in the auditory system can introduce a distortion component into the internal representation of the envelope of a sound, and to estimate the phase of the hypothetical distortion component. In experiment 1, a two-alternative forced-choice (2AFC) task with feedback was used to measure psychometric functions for detecting 5-Hz probe modulation of a 4-kHz sinusoidal carrier in the presence of a masker modulator with components at 50 and 55 Hz (m=0.3 for each component). Performance was measured as a function of the relative phase, delta[symbol see text], of the probe relative to the "venelope" (envelope of the envelope) of the masker. Performance was poorest for delta[symbol see text]= 135 degrees. In experiment 2, delta[symbol see text] was fixed at 135 degrees, m was set to 0.48 for each masker component, and psychometric functions for detecting probe modulation were measured using a 2AFC task without feedback. For small probe modulation depths (m approximately 0.03), the detectability index, d', was consistently negative, consistent with the existence of a weak distortion product which can "cancel" the probe modulation. The distortion component for the conditions of the experiment was estimated to have a phase of about -25 degrees relative to the venelope.

Effects of amplitude compression on first- and second-order modulation detection thresholds in cochlear implant listeners

The aim of this study was to examine the effects of instantaneous non-linear amplitude mapping on the detection of single-component and multicomponent temporal envelopes. To address this issue, first- and second-order amplitude modulation detection thresholds were measured in four cochlear implant users with the intervention of the compression device of the implant processor. The compression device is set to produce either a strongly or a weakly logarithmic mapping of stimulus amplitude to electrical amplitude. 'First-order' modulation detection thresholds indicate the ability of listeners to detect sinusoidal amplitude modulation (SAM) applied to a white noise carrier; they are measured as a function of the rate of that modulation, fm. 'Second-order' modulation detection thresholds indicate the ability to detect sinusoidal modulation applied to the depth of a sinusoidally amplitude-modulated signal (here, a 16-Hz sinusoidally amplitude-modulated white noise); they are measured as a function of the rate of the modulation applied to the modulation depth (referred to as fm'). In each task, stimuli are transformed by the implant processor and are presented through one electrode at approximately the same level. The results show that, in cochlear implant listeners, both first- and second-order modulation detection thresholds measured at the lower rates (< or =7 Hz) decrease slightly by about 3-6dB when the stronger compression is used. No effect of compression is observed at higher rates. These results suggest that instantaneous logarithmic amplitude mapping has beneficial- but limited-effects on the detection of single-component and multicomponent temporal envelopes. These results are discussed in light of current models of temporal envelope processing.

Modulation masking produced by complex tone modulators

Thresholds were measured for detecting sinusoidal amplitude modulation in the presence of a complex-tone masker modulation. Both modulations were applied to the same sinusoidal carrier. Two different masker modulations were used: (i) a pair of components beating at the difference frequency and (ii) a three-tone complex producing a sinusoidal amplitude modulation of the modulation depth at the difference frequency between adjacent components. Both maskers show a periodicity in the waveform that is not contained in the envelope spectrum itself but can be observed when the envelope of the envelope, referred to as the "venelope" [Ewert et al., J. Acoust. Soc. Am. 112, 2921-2931 (2002)], is calculated. For a signal frequency equal to the masker-venelope periodicity, modulation depth at threshold was measured as a function of the signal phase relative to the phase of the masker-venelope component. Signal frequencies of 5, 30, and 90 Hz were used. It was found that masking was phase dependent for all three signal frequencies. Thresholds were lower for the in-phase condition, where maxima in the signal waveform coincided with maxima in the masker-venelope waveform, than for the antiphase condition. The maximum threshold difference was 15 dB. The results are in contrast to recent data [Moore et al., J. Acoust. Soc. Am. 106, 908-918 (1999)], where lowest thresholds were reported for the antiphase condition in a similar experiment. The present data are in line with the idea that a nonlinearity prior to a modulation filterbank extracts the venelope of the masker modulator. However, a compressive nonlinearity such as that associated with the processing on the basilar membrane cannot account for the empirical findings, since it predicts the opposite phase effect.

Testing the concept of a modulation filter bank: the audibility of component modulation and detection of phase change in three-component modulators

Two experiments were performed to test the concept that the auditory system contains a "modulation filter bank" (MFB). Experiment 1 examined the ability to "hear out" the modulation frequency of the central component of a three-component modulator applied to a 4-kHz sinusoidal carrier. On each trial, three modulated stimuli were presented. The modulator of the first stimulus contained three components. Within a run the frequencies of the outer two components were fixed and the frequency of the central ("target") component was drawn randomly from one of five values. The modulators of second and third stimuli contained one component. One had a frequency equal to that of the target and the other had a frequency randomly selected from one of the other possible values. Subjects indicated whether the target corresponded to the second or third stimulus. Scores were around 80% correct when the components in the three-component modulator were widely spaced and when the frequencies of the target and comparison differed sufficiently. Experiment 2 examined the ability to hear a change in the relative phase of the components in a three-component modulator with harmonically spaced components, using a 31FC task. The frequency of the central component, f(c), was either 50 or 100 Hz. Scores were 80%-90% correct when the component spacing was < or = 0.5 f(c), but decreased markedly for greater spacings. Performance was only slightly impaired by randomizing the overall modulation depth from one stimulus to the next. The results of both experiments are broadly consistent with what would be expected from a MFB with a Q value of 1 or slightly less.

Effect of a noise modulation masker on the detection of second-order amplitude modulation

Amplitude modulation waveforms can contain complex patterns of modulation frequency and depth that are characteristic of many biologically relevant sounds. To investigate the mechanisms involved in the processing of such patterns, we measured detection thresholds for second-order amplitude modulation (AM), a sinusoidal AM in which AM depth varies with time at frequency f(m)'. Second-order AM generates sidebands in the modulation spectrum on either side of the frequency components introduced by the first-order AM. Previous masking studies suggested that a distortion product located at f(m)' contributes to the detection of second-order AM. This hypothesis was tested by masking the putative distortion product using a noise modulation masker centred on (1) the second-order modulation frequency (f(m)'=2 Hz) and (2) the first-order modulation frequency (f(m)=16 Hz). The second-order AM was applied to a 5-kHz pure-tone carrier. Increasing the depth of a 2-Hz-wide noise modulator masker centred on 2 Hz had little effect on detection thresholds for second-order AM, but increased detection thresholds for 2-Hz first-order AM six-fold. Increasing the depth of an 8-Hz-wide noise modulator masker centred on 16 Hz increased detection thresholds for both first- and second-order AM three-fold. These results show that the detection of the second-order AM, when f(m)' is 2 Hz, is not dependent on the detection of modulation at f(m)' but is dependent on the detection of modulation components centred on f(m).

Effect of cochlear damage on the detection of complex temporal envelopes

Recent studies have demonstrated that the detection of complex temporal envelopes relies - at least partially - on the perception of a distortion component generated by a peripheral (cochlear) and/or central (post-cochlear) non-linearity. In the present study, first- and second-order amplitude modulation (AM) detection thresholds were obtained in normally hearing (NH) and hearing-impaired (HI) listeners using a 2-kHz pure-tone carrier. In both groups of listeners, first-order AM detection thresholds were measured for AM rates fm ranging between 4 and 87 Hz, and second-order AM detection thresholds were measured for second-order AM rates fm' ranging between 4 and 23 Hz, using a fixed first-order 'carrier' AM rate fm of 64 Hz. When the sound pressure level was adjusted in order to yield equal detectability in both groups for the 64-Hz first-order carrier modulation, (i) first-order AM detection thresholds for the HI listeners were normal at fm=87 Hz, and better-than-normal at fm=4 and 16 Hz, and (ii) second-order AM detection thresholds were identical at all modulation rates in NH and HI listeners. Similar results were obtained when the audibility of the 2-kHz pure-tone carrier was equated for both groups, i.e. when listeners were tested at the same sensation level. These results demonstrate clearly that cochlear damage has no effect on the detection of complex temporal envelopes, and indicate that the distortion component must be generated by a more central non-linearity than cochlear compression, transduction, or short-term adaptation.

Spectro-temporal processing in the envelope-frequency domain

The frequency selectivity for amplitude modulation applied to tonal carriers and the role of beats between modulators in modulation masking were studied. Beats between the masker and signal modulation as well as intrinsic envelope fluctuations of narrow-band-noise modulators are characterized by fluctuations in the "second-order" envelope (referred to as the "venelope" in the following). In experiment 1, masked threshold patterns (MTPs), representing signal modulation threshold as a function of masker-modulation frequency, were obtained for signal-modulation frequencies of 4, 16, and 64 Hz in the presence of a narrow-band-noise masker modulation, both applied to the same sinusoidal carrier. Carrier frequencies of 1.4, 2.8, and 5.5 kHz were used. The shape and relative bandwidth of the MTPs were found to be independent of the signal-modulation frequency and the carrier frequency. Experiment 2 investigated the extent to which the detection of beats between signal and masker modulation is involved in tone-in-noise (TN), noise-in-tone (NT), and tone-in-tone (TT) modulation masking, whereby the TN condition was similar to the one used in the first experiment. A signal-modulation frequency of 64 Hz, applied to a 2.8-kHz carrier, was tested. Thresholds in the NT condition were always lower than in the TN condition, analogous to the masking effects known from corresponding experiments in the audio-frequency domain. TT masking conditions generally produced the lowest thresholds and were strongly influenced by the detection of beats between the signal and the masker modulation. In experiment 3, TT masked-threshold patterns were obtained in the presence of an additional sinusoidal masker at the beat frequency. Signal-modulation frequencies of 32, 64, and 128 Hz, applied to a 2.8-kHz carrier, were used. It was found that the presence of an additional modulation at the beat frequency hampered the subject's ability to detect the envelope beats and raised thresholds up to a level comparable to that found in the TN condition. The results of the current study suggest that (i) venelope fluctuations play a similar role in modulation masking as envelope fluctuations do in spectral masking, and (ii) envelope and venelope fluctuations are processed by a common mechanism. To interpret the empirical findings, a general model structure for the processing of envelope and venelope fluctuations is proposed.

Mechanisms of modulation gap detection

It has been postulated that the central auditory system contains an array of modulation filters, each responsive to a different range of modulation frequencies present at the outputs of the (peripheral) auditory filters. In the present experiments, we tested what we call the "dip hypothesis," that a gap in modulation is detected using the "dip" in the output of the modulation filter tuned to the modulator frequency. In experiment 1, the task was to detect a gap in the sinusoidal amplitude modulation imposed on a 4-kHz carrier. The modulator preceding the gap ended with a positive-going zero-crossing. There were three conditions, differing in the phase at which the modulator started at the end of the gap; zero-phase, at a positive-going zero-crossing; pi-phase, at a negative-going zero-crossing; and "preserved" phase, at the phase the modulator would have had if it had continued without interruption. Modulation frequencies were 5, 10, 20, and 40 Hz. Psychometric functions for detection of the gap were measured using a two-alternative forced-choice task. For the zero-phase and preserved-phase conditions, the detectability index, d', increased monotonically with increasing gap duration. For the pi-phase condition, performance was good (d' > 1) for small gap durations, and initially worsened with increasing gap duration, before improving again for longer gap durations. This is the pattern of results expected from the dip hypothesis, provided that the modulation filters have Q values of 2 or more. However, it is also possible that a rhythm cue was used to improve performance in the pi-phase condition for short gap durations; the introduction of the gap markedly disrupted the regular rhythm produced by the modulator peaks. In experiment 2, the rhythm cue was disrupted by varying the modulator period randomly around its nominal value, except for the modulator periods immediately before and after the gap. This markedly impaired performance, and resulted in psychometric functions that were very similar for the zero-phase and pi-phase conditions. This pattern of results is inconsistent with the dip hypothesis. For both experiments, modulation gap "thresholds" (d' approximately 1) were roughly constant when expressed as a proportion of the modulator period. Possible mechanisms of modulation gap detection are discussed and evaluated.

Second-order modulation detection thresholds for pure-tone and narrow-band noise carriers

Modulation perception has typically been characterized by measuring detection thresholds for sinusoidally amplitude-modulated (SAM) signals. This study uses multicomponent modulations. "Second-order" temporal modulation transfer functions (TMTFs) measure detection thresholds for a sinusoidal modulation of the modulation waveform of a SAM signal [Lorenzi et al., J. Acoust. Soc. Am. 110, 1030-2038 (2001)]. The SAM signal therefore acts as a "carrier" stimulus of frequency fm, and sinusoidal modulation of the SAM signal's modulation depth (at rate f'm) generates two additional components in the modulation spectrum at fm - f'm and fm + f'm. There is no spectral energy at the envelope beat frequency f'm in the modulation spectrum of the "physical" stimulus. In the present study, second-order TMTFs were measured for three listeners when fm was 16, 64, and 256 Hz. The carrier was either a 5-kHz pure tone or a narrow-band noise with center frequency and bandwidth of 5 kHz and 2 Hz, respectively. The narrow-band noise carrier was used to prevent listeners from detecting spectral energy at the beat frequency f'm in the "internal" stimuli's modulation spectrum. The results show that, for the 5-kHz pure-tone carrier, second-order TMTFs are nearly low pass in shape; the overall sensitivity and cutoff frequency measured on these second-order TMTFs increase when fm increases from 16 to 256 Hz. For the 2-Hz-wide narrow-band noise carrier, second-order TMTFs are nearly flat in shape for fm = 16 and 64 Hz, and they show a high-pass segment for fm = 256 Hz. These results suggest that detection of spectral energy at the envelope beat frequency contributes in part to the detection of second-order modulation. This is consistent with the idea that nonlinear mechanisms in the auditory pathway produce an audible distortion component at the envelope beat frequency in the internal modulation spectrum of the sounds.