Evolutionary conservation and neuronal mechanisms of auditory perceptual restoration

Restoration and Efficiency of the Neural Processing of Continuous Speech Are Promoted by Prior Knowledge

Sufficiently noisy listening conditions can completely mask the acoustic signal of significant parts of a sentence, and yet listeners may still report the perception of hearing the masked speech. This occurs even when the speech signal is removed entirely, if the gap is filled with stationary noise, a phenomenon known as perceptual restoration. At the neural level, however, it is unclear the extent to which the neural representation of missing extended speech sequences is similar to the dynamic neural representation of ordinary continuous speech. Using auditory magnetoencephalography (MEG), we show that stimulus reconstruction, a technique developed for use with neural representations of ordinary speech, works also for the missing speech segments replaced by noise, even when spanning several phonemes and words. The reconstruction fidelity of the missing speech, up to 25% of what would be attained if present, depends however on listeners' familiarity with the missing segment. This same familiarity also speeds up the most prominent stage of the cortical processing of ordinary speech by approximately 5 ms. Both effects disappear when listeners have no or little prior experience with the speech segment. The results are consistent with adaptive expectation mechanisms that consolidate detailed representations about speech sounds as identifiable factors assisting automatic restoration over ecologically relevant timescales.

Dynamic cortical representations of perceptual filling-in for missing acoustic rhythm

In the phenomenon of perceptual filling-in, missing sensory information can be reconstructed via interpolation or extrapolation from adjacent contextual cues by what is necessarily an endogenous, not yet well understood, neural process. In this investigation, sound stimuli were chosen to allow observation of fixed cortical oscillations driven by contextual (but missing) sensory input, thus entirely reflecting endogenous neural activity. The stimulus employed was a 5 Hz frequency-modulated tone, with brief masker probes (noise bursts) occasionally added. For half the probes, the rhythmic frequency modulation was moreover removed. Listeners reported whether the tone masked by each probe was perceived as being rhythmic or not. Time-frequency analysis of neural responses obtained by magnetoencephalography (MEG) shows that for maskers without the underlying acoustic rhythm, trials where rhythm was nonetheless perceived show higher evoked sustained rhythmic power than trials for which no rhythm was reported. The results support a model in which perceptual filling-in is aided by differential co-modulations of cortical activity at rates directly relevant to human speech communication. We propose that the presence of rhythmically-modulated neural dynamics predicts the subjective experience of a rhythmically modulated sound in real time, even when the perceptual experience is not supported by corresponding sensory data.

Female túngara frogs do not experience the continuity illusion

In humans and some nonhuman vertebrates, a sound containing brief silent gaps can be rendered perceptually continuous by inserting noise into the gaps. This so-called "continuity illusion" arises from a phenomenon known as "auditory induction" and results in the perception of complete auditory objects despite fragmentary or incomplete acoustic information. Previous studies of auditory induction in gray treefrogs (Hyla versicolor and H. chrysoscelis) have demonstrated an absence of this phenomenon. These treefrog species produce pulsatile (noncontinuous) vocalizations, whereas studies of auditory induction in other taxa, including humans, often present continuous sounds (e.g., frequency-modulated sweeps). This study investigated the continuity illusion in a frog (Physalaemus pustulosus) with an advertisement vocalization that is naturally continuous and thus similar to the tonal sweeps used in human psychophysical studies of auditory induction. In a series of playback experiments, female subjects were presented with sets of stimuli that included complete calls, calls with silent gaps, and calls with silent gaps filled with noise. The results failed to provide evidence of auditory induction. Current evidence, therefore, suggests that mammals and birds experience auditory induction, but frogs may not. This emerging pattern of taxonomic differences is considered in light of potential methodological, neurophysiological, and functional explanations.

Integration of Partial Information Within and Across Modalities: Contributions to Spoken and Written Sentence Recognition

PURPOSE

This study evaluated the extent to which partial spoken or written information facilitates sentence recognition under degraded unimodal and multimodal conditions.

METHOD

Twenty young adults with typical hearing completed sentence recognition tasks in unimodal and multimodal conditions across 3 proportions of preservation. In the unimodal condition, performance was examined when only interrupted text or interrupted speech stimuli were available. In the multimodal condition, performance was examined when both interrupted text and interrupted speech stimuli were concurrently presented. Sentence recognition scores were obtained from simultaneous and delayed response conditions.

RESULTS

Significantly better performance was obtained for unimodal speech-only compared with text-only conditions across all proportions preserved. The multimodal condition revealed better performance when responses were delayed. During simultaneous responses, participants received equal benefit from speech information when the text was moderately and significantly degraded. The benefit from text in degraded auditory environments occurred only when speech was highly degraded.

CONCLUSIONS

The speech signal, compared with text, is robust against degradation likely due to its continuous, versus discrete, features. Allowing time for offline linguistic processing is beneficial for the recognition of partial sensory information in unimodal and multimodal conditions. Despite the perceptual differences between the 2 modalities, the results highlight the utility of multimodal speech + text signals.

Perceptual asymmetry induced by the auditory continuity illusion

The challenges of daily communication require listeners to integrate both independent and complementary auditory information to form holistic auditory scenes. As part of this process listeners are thought to fill in missing information to create continuous perceptual streams, even when parts of messages are masked or obscured. One example of this filling-in process-the auditory continuity illusion-has been studied primarily using stimuli presented in isolation, leaving it unclear whether the illusion occurs in more complex situations with higher perceptual and attentional demands. In this study, young normal-hearing participants listened for long target tones, either real or illusory, in "clouds" of shorter masking tone and noise bursts with pseudorandom spectrotemporal locations. Patterns of detection suggest that illusory targets are salient within mixtures, although they do not produce the same level of performance as the real targets. The results suggest that the continuity illusion occurs in the presence of competing sounds and can be used to aid in the detection of partially obscured objects within complex auditory scenes.

Illusory auditory continuity despite neural evidence to the contrary

Many previous studies have shown that a tone that is momentarily -interrupted can be perceived as continuous if the interruption is completely masked by noise. It has been suggested this "continuity illusion" occurs only when peripheral neural responses contain no evidence that the signal was interrupted. In this study, we used a combination of psychophysical measures and computational simulations of peripheral auditory responses to examine whether the continuity illusion can be experienced under conditions where peripheral neural responses contain evidence that the signal did not continue through the masker. Our results provide an example of a salient continuity illusion despite evidence of an interruption in the peripheral representation, indicating that the illusion may depend more on global features of the interrupting sound, such as its long-term specific loudness, than on its fine-grained temporal structure.

Functional imaging of auditory scene analysis

Our auditory system is constantly faced with the task of decomposing the complex mixture of sound arriving at the ears into perceptually independent streams constituting accurate representations of individual sound sources. This decomposition, termed auditory scene analysis, is critical for both survival and communication, and is thought to underlie both speech and music perception. The neural underpinnings of auditory scene analysis have been studied utilizing invasive experiments with animal models as well as non-invasive (MEG, EEG, and fMRI) and invasive (intracranial EEG) studies conducted with human listeners. The present article reviews human neurophysiological research investigating the neural basis of auditory scene analysis, with emphasis on two classical paradigms termed streaming and informational masking. Other paradigms - such as the continuity illusion, mistuned harmonics, and multi-speaker environments - are briefly addressed thereafter. We conclude by discussing the emerging evidence for the role of auditory cortex in remapping incoming acoustic signals into a perceptual representation of auditory streams, which are then available for selective attention and further conscious processing. This article is part of a Special Issue entitled Human Auditory Neuroimaging.

Behavioral evidence for auditory induction in a species of rodent: Mongolian gerbil (Meriones unguiculatus)

When a segment of sound of interest is interrupted by a loud extraneous noise, humans perceive that the missing sound continues during the intrusive noise. This restoration of auditory information occurs in perceptions of both speech and non-speech sounds (e.g., tone bursts), a phenomenon referred to as auditory induction. In this study, Mongolian gerbils were trained with standard Go/No-Go operant conditioning to discriminate continuous tone bursts (the Go stimulus) from tone bursts with a silent gap in the middle (the No-Go stimulus). Noise was added to Go and No-Go stimuli to determine the condition under which induction would occur. The Mongolian gerbils engaged in Go responses to No-Go stimuli only when the noise spectrally surrounding the tone was of the same duration as the silent portion of the No-Go stimulus; these results match those previously reported in primates (humans and macaque monkeys). The result presents not only the evidence of the auditory induction in a rodent species but also suggests that similar mechanisms for restoring missing sounds are shared among mammals. Additionally, our findings demonstrated that the rodent can serve as a valuable animal model for future studies of perceptual restoration.

Spatiotemporal dynamics of neural activity related to auditory induction in the core and belt fields of guinea-pig auditory cortex

Auditory induction is a continuity illusion in which missing sounds are perceived under appropriate conditions, for example, when noise is inserted during silent gaps in the sound. To elucidate the neural mechanisms underlying auditory induction, neural responses to tones interrupted by a silent gap or noise were examined in the core and belt fields of the auditory cortex using real-time optical imaging with a voltage-sensitive dye. Tone stimuli interrupted by a silent gap elicited responses to the second tone following the gap as well as early phasic responses to the first tone. Tone stimuli interrupted by broad-band noise (BN), considered to cause auditory induction, considerably reduced or eliminated responses to the tone following the noise. This reduction was stronger in the dorsocaudal field (field DC) and belt fields compared with the anterior field (the primary auditory cortex of guinea pig). Tone stimuli interrupted by notched (band-stopped) noise centered at the tone frequency, considered to decrease the strength of auditory induction, partially restored the second responses from the suppression caused by BN. These results suggest that substantial changes between responses to silent gap-inserted tones and those to BN-inserted tones emerged in field DC and belt fields. Moreover, the findings indicate that field DC is the first area in which these changes emerge, suggesting that it may be an important region for auditory induction of simple sounds.

Hearing an illusory vowel in noise: suppression of auditory cortical activity

Human hearing is constructive. For example, when a voice is partially replaced by an extraneous sound (e.g., on the telephone due to a transmission problem), the auditory system may restore the missing portion so that the voice can be perceived as continuous (Miller and Licklider, 1950; for review, see Bregman, 1990; Warren, 1999). The neural mechanisms underlying this continuity illusion have been studied mostly with schematic stimuli (e.g., simple tones) and are still a matter of debate (for review, see Petkov and Sutter, 2011). The goal of the present study was to elucidate how these mechanisms operate under more natural conditions. Using psychophysics and electroencephalography (EEG), we assessed simultaneously the perceived continuity of a human vowel sound through interrupting noise and the concurrent neural activity. We found that vowel continuity illusions were accompanied by a suppression of the 4 Hz EEG power in auditory cortex (AC) that was evoked by the vowel interruption. This suppression was stronger than the suppression accompanying continuity illusions of a simple tone. Finally, continuity perception and 4 Hz power depended on the intactness of the sound that preceded the vowel (i.e., the auditory context). These findings show that a natural sound may be restored during noise due to the suppression of 4 Hz AC activity evoked early during the noise. This mechanism may attenuate sudden pitch changes, adapt the resistance of the auditory system to extraneous sounds across auditory scenes, and provide a useful model for assisted hearing devices.

Global not local masker features govern the auditory continuity illusion

When an acoustic signal is temporarily interrupted by another sound, it is sometimes heard as continuing through, even when the signal is actually turned off during the interruption-an effect known as the "auditory continuity illusion." A widespread view is that the illusion can only occur when peripheral neural responses contain no evidence that the signal was interrupted. Here we challenge this view using a combination of psychophysical measures from human listeners and computational simulations with a model of the auditory periphery. The results reveal that the illusion seems to depend more on the overall specific loudness than on the peripheral masking properties of the interrupting sound. This finding indicates that the continuity illusion is determined by the global features, rather than the fine-grained temporal structure, of the interrupting sound, and argues against the view that the illusion arises in the auditory periphery.

Effects of inducer continuity on auditory stream segregation: comparison of physical and perceived continuity in different contexts

The factors influencing the stream segregation of discrete tones and the perceived continuity of discrete tones as continuing through an interrupting masker are well understood as separate phenomena. Two experiments tested whether perceived continuity can influence the build-up of stream segregation by manipulating the perception of continuity during an induction sequence and measuring streaming in a subsequent test sequence comprising three triplets of low and high frequency tones (LHL-[ellipsis (horizontal)]). For experiment 1, a 1.2-s standard induction sequence comprising six 100-ms L-tones strongly promoted segregation, whereas a single extended L-inducer (1.1 s plus 100-ms silence) did not. Segregation was similar to that following the single extended inducer when perceived continuity was evoked by inserting noise bursts between the individual tones. Reported segregation increased when the noise level was reduced such that perceived continuity no longer occurred. Experiment 2 presented a 1.3-s continuous inducer created by bridging the 100-ms silence between an extended L-inducer and the first test-sequence tone. This configuration strongly promoted segregation. Segregation was also increased by filling the silence after the extended inducer with noise, such that it was perceived like a bridging inducer. Like physical continuity, perceived continuity can promote or reduce test-sequence streaming, depending on stimulus context.

Perceived tonal continuity through two noise bursts separated by silence

Three experiments measured the perceived continuity of two pure tones "flankers" through a masker containing a silence. Experiment 1 used a 2I-2AFC procedure; one interval contained two noise bursts separated by a silent gap, and the other contained two noise bursts separated by a tone of the same duration as the silence. Discrimination between masker conditions was very accurate when the flankers were absent but was impaired substantially when the flankers were present. This was taken as evidence that illusory flanker continuity during the silent gap was heard as similar to the physical presence of a tone in the gap. In experiment 2, performance remained poor when the flankers were frequency glides aligned along a common trajectory. Performance improved significantly when the flankers were misaligned in trajectory. In experiment 3, listeners rated directly perceived flanker continuity. Strong continuity was reported in the silent gap conditions for which poor performance had been observed in experiments 1 and 2. These findings show that continuity may be heard through a masker that cannot mask a physically continuous tone but can mask the flankers' offset and onset. The results are explained in terms of the perceptual grouping of onsets and offsets of the flankers.

Recalibration of the auditory continuity illusion: sensory and decisional effects

An interrupted sound can be perceived as continuous when noise masks the interruption, creating an illusion of continuity. Recent findings have shown that adaptor sounds preceding an ambiguous target sound can influence listeners' rating of target continuity. However, it remains unclear whether these aftereffects on perceived continuity influence sensory processes, decisional processes (i.e., criterion shifts), or both. The present study addressed this question. Results show that the target sound was more likely to be rated as 'continuous' when preceded by adaptors that were perceived as clearly discontinuous than when it was preceded by adaptors that were heard (illusorily or veridically) as continuous. Detection-theory analyses indicated that these contrastive aftereffects reflect a combination of sensory and decisional processes. The contrastive sensory aftereffect persisted even when adaptors and targets were presented to opposite ears, suggesting a neural origin in structures that receive binaural inputs. Finally, physically identical but perceptually ambiguous adaptors that were rated as 'continuous' induced more reports of target continuity than adaptors that were rated as 'discontinuous'. This assimilative aftereffect was purely decisional. These findings confirm that judgments of auditory continuity can be influenced by preceding events, and reveal that these aftereffects have both sensory and decisional components.